Can't list remote backup

[ArnaudIO]

Hi,

I'm actually working on the configuration of clickhouse-backup in a kubernetes environment and a S3 object storage located at Scaleway Cloud.
The sidecar pod is correctly configured and working. Command "clickhouse-backup create_remote TestBackup" is also working as expected.
But even if the backup is present in the S3 bucket, the "clickhouse-backup list remote" is returning an error

2024-09-03 14:55:26.496 INF pkg/storage/s3.go:49 > [s3:DEBUG] Request
GET /clickhouse-staging?versioning= HTTP/1.1
Host: clickhouse-staging.xxxxxx.scw.cloud
User-Agent: m/F aws-sdk-go-v2/1.30.3 os/linux lang/go#1.22.6 md/GOOS#linux md/GOARCH#amd64 api/s3#1.58.3
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: e05fbf4a-958c-45c4-b2d3-8eb327f2bdf7
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=xxxxxxxxxxxxxxxxxx/20240903/xxxxxxx/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=aa02ac44bd4294a9e8a0392ef7785fa83bd00744739eba6d91bcf936cd2517b2
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240903T145526Z

2024-09-03 14:55:26.579 INF pkg/storage/s3.go:49 > [s3:DEBUG] request failed with unretryable error https response error StatusCode: 404, RequestID: xxxxxxxxxxxxx-0066d7235e, HostID: xxxxxxxxxxxxx-0066d7235e, api error NoSuchKey: The specified key does not exist.
2024-09-03 14:55:26.580 INF pkg/storage/s3.go:49 > [s3:DEBUG] Request
GET /clickhouse-staging?delimiter=%2F&list-type=2&max-keys=1000&prefix=backup%2Fshard-0%2F HTTP/1.1
Host: clickhouse-staging.xxxxxx.scw.cloud
User-Agent: m/C,F aws-sdk-go-v2/1.30.3 os/linux lang/go#1.22.6 md/GOOS#linux md/GOARCH#amd64 api/s3#1.58.3
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: b87d7707-9e8c-49c9-ae8d-22d1874e53e7
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=xxxxxxxxxxxxxxxxxx/20240903/xxxxxxx/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=a45ae574c1961b1bd0df4d19023fd9e2481095462543af698b56e01e3068672c
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240903T145526Z


2024-09-03 14:55:26.598 INF pkg/storage/s3.go:49 > [s3:DEBUG] Response
HTTP/2.0 404 Not Found
Content-Length: 256
Content-Type: application/xml
Date: Tue, 03 Sep 2024 14:55:26 GMT
X-Amz-Id-2: txg10f3a1b3f5764118a5cb-0066d7235e
X-Amz-Request-Id: txg10f3a1b3f5764118a5cb-0066d7235e


2024-09-03 14:55:26.599 INF pkg/storage/s3.go:49 > [s3:DEBUG] request failed with unretryable error https response error StatusCode: 404, RequestID: xxxxxxxxxxxxx-0066d7235e, HostID: xxxxxxxxxxxxx-0066d7235e, api error NoSuchKey: The specified key does not exist.
2024-09-03 14:55:26.599 WRN pkg/storage/general.go:248 > BackupList bd.Walk return error: operation error S3: ListObjectsV2, https response error StatusCode: 404, RequestID: xxxxxxxxxxxxx-0066d7235e, HostID: xxxxxxxxxxxxx-0066d7235e, api error NoSuchKey: The specified key does not exist.


2024-09-03 14:55:26.579 INF pkg/storage/s3.go:49 > [s3:DEBUG] Response
HTTP/2.0 404 Not Found

In the meantime I notice that the table system.backup_list is not populated with any "remote" backup during the create_remote command.

Any idea? I wish to fix this behavior because I want to use a cronjob in kube which browse remote backup to define full or incremental backup.

Thanks

[Slach]

It looks similar issue
#505
and
#877

Did you try?

s3:
 force_path_style: false

[ArnaudIO]

Hi, thanks for the answer, indeed I didn't find these issues.
I try the parameter, it changed the error, now I have a certificate error... To fix this I remove S3_bucket in the configuration. Now I am able to list remote.
Good news, but the bad news is the create_remote command doesn't work anymore.

error    b.getTablesDiffFromRemote return error: operation error S3: GetObject, https response error StatusCode: 404, RequestID: txg5bf10cce29764b759560-0066d803af, H │
│ ostID: txg5bf10cce29764b759560-0066d803af, NoSuchKey:

Any advice is appreciate, I continue to play with the parameters.

[ArnaudIO]

It works, I delete the backups in the S3 bucket. I think there is a mismatch with my backup name and for name format clickhouse-backup is looking for during the diff.

Thanks a lot for your help.

[Slach]

could you provide
clickhouse-backup print-config

s3 section without sensitive credentials?

for future searching

[ArnaudIO]

Here it is

s3:
    access_key: XXXXXXXXXXXXXX
    secret_key: XXXXXXX-XXXXXX-XXXXXX-XXXXXXX
    bucket: ""
    endpoint: https://clickhouse-staging.XXXXXX.scw.cloud
    region: XXXXXX
    acl: private
    assume_role_arn: ""
    force_path_style: false
    path: backup/shard-{shard}
    object_disk_path: ""
    disable_ssl: false
    compression_level: 1
    compression_format: tar
    sse: ""
    sse_kms_key_id: ""
    sse_customer_algorithm: ""
    sse_customer_key: ""
    sse_customer_key_md5: ""
    sse_kms_encryption_context: ""
    disable_cert_verification: false
    use_custom_storage_class: false
    storage_class: STANDARD
    custom_storage_class_map: {}
    concurrency: 5
    part_size: 0
    max_parts_count: 4000
    allow_multipart_download: false
    object_labels: {}
    request_payer: ""
    check_sum_algorithm: ""
    debug: true