Problem with s3 (ceph) and sync jobs

[fabry-68]

Hello.
I am trying 2 different installations; pbs is the last update, s3 is the same (used for a lot of different jobs, ceph based, and perfectly working).
pmoxs3backuproxy is running on ubuntu 24.04LTS, and after it starts to sync, I have errors on console and the job reamains locked.

2025/02/26 10:51:51 [ ERROR ] Writing object 6e125961648ab7d90cfbc515c3c08f7b9757b1db4db1003b2403844f0e2e0601 failed: Put "https://s3storage.panservice.it:8000/pesce-pbsproxy/chunks/6e/12/5961648ab7d90cfbc515c3c08f7b9757b1db4db1003b2403844f0e2e0601": client disconnected

---

2025-02-26T11:46:04+01:00: Starting datastore sync job 'PBS-Proxy:pesce-pbsproxy:VolumeBackup::s-4c052331-d0be'
2025-02-26T11:46:04+01:00: sync datastore 'VolumeBackup' to 'PBS-Proxy/pesce-pbsproxy'
2025-02-26T11:46:07+01:00: ----
2025-02-26T11:46:07+01:00: Syncing datastore 'VolumeBackup', root namespace into datastore 'pesce-pbsproxy', root namespace
2025-02-26T11:46:07+01:00: Found 3 groups to sync (out of 3 total)
2025-02-26T11:46:07+01:00: Group 'vm/101' not owned by remote user 'A501Z23AX1EM1F1WB8IT@pbs' on target, skipping upload
2025-02-26T11:46:07+01:00: Could not download previous manifest -
2025-02-26T11:51:23+01:00: Percentage done: 37.04% (1/3 groups, 1/9 snapshots in group #2)
2025-02-26T11:51:23+01:00: Encountered errors: Put "https://s3storage.panservice.it:8000/pesce-pbsproxy/chunks/8f/d9/52fe9285dac54ccf5fa135b51326fcc699327399ef5c3ecfd6cb41c15216": net/http: HTTP/1.x transport connection broken: http: ContentLength=4194316 with Body length 0{"data":{"digest":"8fd952fe9285dac54ccf5fa135b51326fcc699327399ef5c3ecfd6cb41c15216","offset":0,"size":4194304,"chunk_is_known":false}}
2025-02-26T11:51:23+01:00: Failed to push group vm/102 to remote!
2025-02-26T11:51:26+01:00: Could not download previous manifest - stream closed because of a broken pipe
2025-02-26T11:51:26+01:00: Percentage done: 70.37% (2/3 groups, 1/9 snapshots in group #3)
2025-02-26T11:51:26+01:00: Encountered errors: connection closed because of a broken pipe
2025-02-26T11:51:26+01:00: Failed to push group vm/120 to remote!
2025-02-26T11:51:26+01:00: Finished syncing root namespace, current progress: 2 groups, 1 snapshots
2025-02-26T11:51:26+01:00: TASK ERROR: Sync failed with some errors!

[abbbi]

net/http: HTTP/1.x transport connection broken: http: ContentLength=4194316 with Body length

and:

Put "https://s3storage.panservice.it:8000/pesce-pbsproxy/chunks/6e/12/5961648ab7d90cfbc515c3c08f7b9757b1db4db1003b2403844f0e2e0601": client disconnected

look like an connectivity issue ..
Probably down/uplink too saturated.

[fabry-68]

Hello abbbi. Is a FTTH 2500/500, and I set the rsync job to 10Mbit/sec max, so the link is really empty. The storage is in my datacenter, and the FTTH line is directly connected to my site. I can't notice latency issues.
But... even if the link was totally used, sync jobs need to continue.
What kind of info can i give you ?
PS: same problem with another link/installation, but always done to the same object storage, that I use for a lot of jobs without any issue.

[abbbi]

i dont know if there is anything we can do here. The minio S3 library we use to put objects uses the underyling golang net/http library which receives an connection reset as it seems. Like a broken http connection or the backend closes the connection (or some third party component). Any logs on the receiving side that might show why connection was dropped?

@tizbac ideas?

Weve had multiple reports in this regard, but in one case it seemed like a backblaze issue and in the other case the solutionw as to use ssl for communication as the plain http transfers were limited.

[fabry-68]

On the remote side I have a "petasan" ceph cluster, with 12 hosts (2 are for s3 gateway and 10 for monitor+osd). I also have haproxy for balancing and HA and of course ssl with a commercial certificate.
No any issues using a lot of services (veeam + s3, rclone, kopia, several storage servers).
Now i try to configure haproxy to use always the same internal ip and I will let you know.

[abbbi]

it seems the minio implementation and the put method has some kind of retry timer:

https://github.com/minio/minio-go/blob/master/retry.go

That actually has some retries implemented for certain http error codes:

https://github.com/minio/minio-go/blob/master/retry.go#L114

I only had a quick look. It doesnt seem the API allow to set specific retry operations and retries are only possible
for certain conditions.

[fabry-68]

This is what I have seen in debug mode:

2025/02/27 11:22:21.528760 [ DEBUG ] Request:/fixed_chunk?digest=0b6174462ec1725dae5e753053eaa022300be22b32daa091ae2d57550eb8ef90&encoded-size=1296591&size=4194304&wid=1 Method: POST
2025/02/27 11:22:21.728844 [ ERROR ] Writing object 986f6f5fb6287502e0ccdf6c29743794ddb7c26e532aaff592686b9a49734167 failed: Put "https://s3storage.panservice.it:8000/pesce-pbsproxy/chunks/98/6f/6f5fb6287502e0ccdf6c29743794ddb7c26e532aaff592686b9a49734167": net/http: HTTP/1.x transport connection broken: http: ContentLength=2635664 with Body length 0
2025/02/27 11:22:21.978877 [WARNING] Removed incomplete backup backups/120|1735254019|vm
2025/02/27 11:22:21.978907 [ INFO ] Backup session by [192.168.20.168:48220] finished

And on my client-ceph.log:

185.109.191.226 - - [27/Feb/2025:12:22:21 +0100] "HEAD /pesce-pbsproxy/chunks/0b/61/74462ec1725dae5e753053eaa022300be22b32daa091ae2d57550eb8ef90 HTTP/1.1" 404 0 "-" "MinIO (linux; amd64) minio-go/v7.0.74"
185.109.191.226 - - [27/Feb/2025:12:22:21 +0100] "PUT /pesce-pbsproxy/chunks/9d/b6/ce9f8baeaea4f47fd09a9725059e7965f611e6342b92635edc359c3c2787 HTTP/1.1" 200 0 "-" "MinIO (linux; amd64) minio-go/v7.0.74"
185.109.191.226 - - [27/Feb/2025:12:22:21 +0100] "PUT /pesce-pbsproxy/chunks/8e/71/f318779cbbcaeb6ef1799c0836be38f5ffbf980fa2bb188da5c201955378 HTTP/1.1" 200 0 "-" "MinIO (linux; amd64) minio-go/v7.0.74"
185.109.191.226 - - [27/Feb/2025:12:22:21 +0100] "PUT /pesce-pbsproxy/chunks/5a/d7/c271a9cd45642f907c8b733739bf6be9dc9482a860c56eb9384aad1b83e3 HTTP/1.1" 200 0 "-" "MinIO (linux; amd64) minio-go/v7.0.74"
185.109.191.226 - - [27/Feb/2025:12:22:21 +0100] "PUT /pesce-pbsproxy/chunks/7e/1c/92616d92b740eb285317c4c82edd19078967569421cc45e1c4e527c8e3e4 HTTP/1.1" 200 0 "-" "MinIO (linux; amd64) minio-go/v7.0.74"
185.109.191.226 - - [27/Feb/2025:12:22:21 +0100] "PUT /pesce-pbsproxy/chunks/c0/d8/d77246227031c72a74df5e808bbaf08ffe618e22ce36d937b6077c442a13 HTTP/1.1" 200 0 "-" "MinIO (linux; amd64) minio-go/v7.0.74"
185.109.191.226 - - [27/Feb/2025:12:22:21 +0100] "PUT /pesce-pbsproxy/chunks/54/d1/4abd03b8ce1b33029f7539864a029ae3bf7e46bd8cbbd229effe057f147b HTTP/1.1" 200 0 "-" "MinIO (linux; amd64) minio-go/v7.0.74"
185.109.191.226 - - [27/Feb/2025:12:22:21 +0100] "PUT /pesce-pbsproxy/chunks/82/19/87620d9ae1428c415cd17833355576e162d1622f0666c06179e174431b62 HTTP/1.1" 200 0 "-" "MinIO (linux; amd64) minio-go/v7.0.74"
185.109.191.226 - - [27/Feb/2025:12:22:21 +0100] "PUT /pesce-pbsproxy/chunks/08/75/2c8e8e9272914318c5c1ba3b0d44726789dcd5d93d440c4310447c236b60 HTTP/1.1" 200 0 "-" "MinIO (linux; amd64) minio-go/v7.0.74"
185.109.191.226 - - [27/Feb/2025:12:22:21 +0100] "PUT /pesce-pbsproxy/chunks/2b/28/9777d456acb8229cd8abd5e2c5659e47062aace29f12f723710f6c0d5435 HTTP/1.1" 200 0 "-" "MinIO (linux; amd64) minio-go/v7.0.74"
185.109.191.226 - - [27/Feb/2025:12:22:21 +0100] "PUT /pesce-pbsproxy/chunks/d8/45/ea1cf2687e85fa60944465d3f051b76718b0ef7bb556bd4eed01314e4e30 HTTP/1.1" 200 0 "-" "MinIO (linux; amd64) minio-go/v7.0.74"
185.109.191.226 - - [27/Feb/2025:12:22:21 +0100] "PUT /pesce-pbsproxy/chunks/91/24/f39bb324e21dd4ce35defada11402df710320595afa742ef3aa8269819e4 HTTP/1.1" 200 0 "-" "MinIO (linux; amd64) minio-go/v7.0.74"
185.109.191.226 - - [27/Feb/2025:12:22:21 +0100] "PUT /pesce-pbsproxy/chunks/3a/3e/1cb05b71762bdf7ca57d73521c368f331bd8627f5cd70134c88f804eb790 HTTP/1.1" 200 0 "-" "MinIO (linux; amd64) minio-go/v7.0.74"
185.109.191.226 - - [27/Feb/2025:12:22:21 +0100] "PUT /pesce-pbsproxy/chunks/8b/ef/d7690f4b78e1035f0106e7e2c0bcd655af4ed83704b06e285240ae8715cc HTTP/1.1" 200 0 "-" "MinIO (linux; amd64) minio-go/v7.0.74"
185.109.191.226 - - [27/Feb/2025:12:22:21 +0100] "PUT /pesce-pbsproxy/chunks/88/44/e7fe28cd7c4c88fbe07ad00133727f70080651dce8011e7827845c46cc4d HTTP/1.1" 200 0 "-" "MinIO (linux; amd64) minio-go/v7.0.74"
185.109.191.226 - - [27/Feb/2025:12:22:21 +0100] "PUT /pesce-pbsproxy/chunks/be/f7/8df30c07c4e5d77ed024d6396e6ccc803a7f06e4592fdfc5ce8b93d99e25 HTTP/1.1" 200 0 "-" "MinIO (linux; amd64) minio-go/v7.0.74"
185.109.191.226 - - [27/Feb/2025:12:22:21 +0100] "PUT /pesce-pbsproxy/chunks/48/39/0b02f7d931d68dfe7a739916972ba401aff0613561976c008bcd32a4892c HTTP/1.1" 200 0 "-" "MinIO (linux; amd64) minio-go/v7.0.74"
185.109.191.226 - - [27/Feb/2025:12:22:21 +0100] "PUT /pesce-pbsproxy/chunks/95/4f/8695baf1c684b472294d11b9bb1285f20e428da0eaec26f08f9b507d231f HTTP/1.1" 200 0 "-" "MinIO (linux; amd64) minio-go/v7.0.74"
185.109.191.226 - - [27/Feb/2025:12:22:21 +0100] "PUT /pesce-pbsproxy/chunks/0b/61/74462ec1725dae5e753053eaa022300be22b32daa091ae2d57550eb8ef90 HTTP/1.1" 200 0 "-" "MinIO (linux; amd64) minio-go/v7.0.74"
185.109.191.226 - - [27/Feb/2025:12:22:21 +0100] "PUT /pesce-pbsproxy/chunks/8b/be/e8fc7e5bd163eeeab08f0689d7a7208ad9f73a4cfc1205a89c62bc1cb9d0 HTTP/1.1" 200 0 "-" "MinIO (linux; amd64) minio-go/v7.0.74"
185.109.191.226 - - [27/Feb/2025:12:22:21 +0100] "GET /pesce-pbsproxy/?delimiter=&encoding-type=url&fetch-owner=true&list-type=2&prefix=backups%2F120%7C1735254019%7Cvm HTTP/1.1" 200 646 "-" "MinIO (linux; amd64) minio-go/v7.0.74"
185.109.191.226 - - [27/Feb/2025:12:22:21 +0100] "POST /pesce-pbsproxy/?delete= HTTP/1.1" 200 220 "-" "MinIO (linux; amd64) minio-go/v7.0.74"

[abbbi]

during object write:

986f6f5fb6287502e0ccdf6c29743794ddb7c26e532aaff592686b9a49734167

it hits:

HTTP/1.x transport connection broken:

and in the ceph log, there is actually no entry for an PUT request of this object.
For me, that means, this http request didnt make it through to your ceph endpoint.

[fabry-68]

Can I give you (in private message) credential to use a bucket on my ceph cluster ? Because a lot of my customers use it with any software without problems, also with nextcloud and primary storage (i don't know if nextcloud use minio libraries).
And, can you manage the Error writing object to retry and don't stop the sync ? This is because object storage servers are tipically in cloud, and some network issue from the customer can always happen.

[abbbi]

it fails here:

https://github.com/tizbac/pmoxs3backuproxy/blob/master/cmd/pmoxs3backuproxy/main.go#L971

i cant see a reason what would be problematic with that code.
I could imagine that the request sent by the proxmox backup server to the proxy is already wrong and/or the body as returned by the request:

https://github.com/tizbac/pmoxs3backuproxy/blob/master/cmd/pmoxs3backuproxy/main.go#L966

is empty. You could add debug output to see if the Body is somehow mangled.

I dont intend to implement any retries at that stage. Pull requests welcome, if they actually help to solve a real issue and no workaround issues caused by lacky uplinks and or IDS Systems or firewalls dropping connections.

[fabry-68]

Hello abbbi; I have started with -debug option, but the only message I have is what I wrote in my previous message.
I have also tried to compile the proxy with minio-go/v7.0.87 (i don't know golang, but i was been able to do it), but I have the same problem.
Can be related to the last pbs release ? Or you can confirm that works with it ?
In the meanwhile i will try to find another s3 account to test with a different storage.
Thanks, Fabrizio

[fabry-68]

2025/02/28 09:11:54.673296 [ ERROR ] Writing object d68304ae587343fd85ee38a53eec51c27ecf43599f83b34d4e0e503ed72fabc9 failed: Put "https://bucketfabry.s3.dualstack.eu-north-1.amazonaws.com/chunks/d6/83/04ae587343fd85ee38a53eec51c27ecf43599f83b34d4e0e503ed72fabc9": net/http: HTTP/1.x transport connection broken: http: ContentLength=1665765 with Body length 0

Same problem with Amazon; no network issues at all... I am using a 2500/500Mbit FTTH line, latency is really low, no jitter. PBS is updated, backup is verified.

[abbbi]

Can you capture https traffic via httptap or similar tool like mitmproxy

[fabry-68]

Of course; I have never used httptap, but If you can help me, i have already installed on the proxy server. (i will back at office in 3 or 4 hours.

[tizbac]

Beware non-enterprise fiber connections may have limits, or stuff killing connections / behavior with too much traffic Il ven 28 feb 2025, 10:14 fabry-68 ***@***.***> ha scritto:

…

2025/02/28 09:11:54.673296 [ ERROR ] Writing object d68304ae587343fd85ee38a53eec51c27ecf43599f83b34d4e0e503ed72fabc9 failed: Put " https://bucketfabry.s3.dualstack.eu-north-1.amazonaws.com/chunks/d6/83/04ae587343fd85ee38a53eec51c27ecf43599f83b34d4e0e503ed72fabc9": net/http: HTTP/1.x transport connection broken: http: ContentLength=1665765 with Body length 0 Same problem with Amazon; no network issues at all... I am using a 2500/500Mbit FTTH line, latency is really low, no jitter. PBS is updated, backup is verified. — Reply to this email directly, view it on GitHub <#69 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABBV6QZSYT2CJYO7NOIB532SASF5AVCNFSM6AAAAABX45XO52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJQGEZDENBWGE> . You are receiving this because you were mentioned.Message ID: ***@***.***> [image: fabry-68]*fabry-68* left a comment (tizbac/pmoxs3backuproxy#69) <#69 (comment)> 2025/02/28 09:11:54.673296 [ ERROR ] Writing object d68304ae587343fd85ee38a53eec51c27ecf43599f83b34d4e0e503ed72fabc9 failed: Put " https://bucketfabry.s3.dualstack.eu-north-1.amazonaws.com/chunks/d6/83/04ae587343fd85ee38a53eec51c27ecf43599f83b34d4e0e503ed72fabc9": net/http: HTTP/1.x transport connection broken: http: ContentLength=1665765 with Body length 0 Same problem with Amazon; no network issues at all... I am using a 2500/500Mbit FTTH line, latency is really low, no jitter. PBS is updated, backup is verified. — Reply to this email directly, view it on GitHub <#69 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABBV6QZSYT2CJYO7NOIB532SASF5AVCNFSM6AAAAABX45XO52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJQGEZDENBWGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[fabry-68]

I know of the limits, but I am the Internet operator and is my own network, so I am sure that is not a network issue. Thank you Il 28 Feb 2025 12:57, Tiziano Bacocco ***@***.***> ha scritto: tizbac left a comment (tizbac/pmoxs3backuproxy#69) Beware non-enterprise fiber connections may have limits, or stuff killing connections / behavior with too much traffic Il ven 28 feb 2025, 10:14 fabry-68 ***@***.***> ha scritto:

…

2025/02/28 09:11:54.673296 [ ERROR ] Writing object d68304ae587343fd85ee38a53eec51c27ecf43599f83b34d4e0e503ed72fabc9 failed: Put " https://bucketfabry.s3.dualstack.eu-north-1.amazonaws.com/chunks/d6/83/04ae587343fd85ee38a53eec51c27ecf43599f83b34d4e0e503ed72fabc9": net/http: HTTP/1.x transport connection broken: http: ContentLength=1665765 with Body length 0 Same problem with Amazon; no network issues at all... I am using a 2500/500Mbit FTTH line, latency is really low, no jitter. PBS is updated, backup is verified. — Reply to this email directly, view it on GitHub <#69 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABBV6QZSYT2CJYO7NOIB532SASF5AVCNFSM6AAAAABX45XO52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJQGEZDENBWGE> . You are receiving this because you were mentioned.Message ID: ***@***.***> [image: fabry-68]*fabry-68* left a comment (tizbac/pmoxs3backuproxy#69) <#69 (comment)> 2025/02/28 09:11:54.673296 [ ERROR ] Writing object d68304ae587343fd85ee38a53eec51c27ecf43599f83b34d4e0e503ed72fabc9 failed: Put " https://bucketfabry.s3.dualstack.eu-north-1.amazonaws.com/chunks/d6/83/04ae587343fd85ee38a53eec51c27ecf43599f83b34d4e0e503ed72fabc9": net/http: HTTP/1.x transport connection broken: http: ContentLength=1665765 with Body length 0 Same problem with Amazon; no network issues at all... I am using a 2500/500Mbit FTTH line, latency is really low, no jitter. PBS is updated, backup is verified. — Reply to this email directly, view it on GitHub <#69 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABBV6QZSYT2CJYO7NOIB532SASF5AVCNFSM6AAAAABX45XO52VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOJQGEZDENBWGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Reply to this email directly or view it on GitHub: #69 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>