Timeout when calling FinalizeCacheEntryUpload (OOMKilled)

[nponsard]

Hello,

After getting the cache server to work correctly with Postgres and Minio, I got into another issue where the cache server pods get OOMKilled, this seems to happen when FinalizeCacheEntryUpload is called.

I get this in the action logs:

Sent 6873822700 of 6940931564 (99.0%), 26.5 MBs/sec
Sent 6940931564 of 6940931564 (100.0%), 26.7 MBs/sec
Attempt 1 of 5 failed with error: Request timeout: /twirp/github.actions.results.api.v1.CacheService/FinalizeCacheEntryUpload. Retrying request in 3000 ms...
Warning: Failed to save: Failed to FinalizeCacheEntryUpload: Unable to make request: ECONNRESET

The current memory limit set in the chart in 3Gi, this seems a bit excessive. I've seen in #135 you commented about using changing the NITRO_CLUSTER_WORKERS env variable, does this apply to a kubernetes environment too ?

[nponsard]

Okay I tried increasing the memory limit even more and bumping the replica count to 4, now the pods don't get OOMKilled and the cache action shows a proper timeout message:

 Sent 6375342080 of 6551508059 (97.3%), 55.2 MBs/sec
Sent 6551508059 of 6551508059 (100.0%), 56.5 MBs/sec
Attempt 1 of 5 failed with error: Request timeout: /twirp/github.actions.results.api.v1.CacheService/FinalizeCacheEntryUpload. Retrying request in 3000 ms...
Attempt 2 of 5 failed with error: Request timeout: /twirp/github.actions.results.api.v1.CacheService/FinalizeCacheEntryUpload. Retrying request in 5318 ms...
Attempt 3 of 5 failed with error: Request timeout: /twirp/github.actions.results.api.v1.CacheService/FinalizeCacheEntryUpload. Retrying request in 7943 ms...
Attempt 4 of 5 failed with error: Request timeout: /twirp/github.actions.results.api.v1.CacheService/FinalizeCacheEntryUpload. Retrying request in 14247 ms...
Warning: Failed to save: Failed to FinalizeCacheEntryUpload: Failed to make request after 5 attempts: Request timeout: /twirp/github.actions.results.api.v1.CacheService/FinalizeCacheEntryUpload

[kuhnroyal]

I had OOM killed problems in the past but I the pods more resources so they don't crash anymore.
However the FinalizeCacheEntryUpload often times out with large cache uploads, like a JDK or something.
This took 3 minutes and the builds don't finish until this is done.

Looking at the storage code, the chunks are uploaded to MinIO and then everything is downloaded again, put together and uploaded in full. This seems kind of inefficent.

[LouisHaftmann]

We cannot save the parts as files because it doesn't scale horizontally. And we cannot use S3's multipart uploads because the minimum chunk size of gha cache is too small.

[LouisHaftmann]

I'm open to suggestions

[kuhnroyal]

Do you mean the 1MB buildx chunks?

// 1 MB for docker buildx
// 64 MB for everything else
const chunkSize = userAgent && userAgent.startsWith('azsdk-go-azblob') ? MB : 64 * MB

[LouisHaftmann]

Yes, docker build has 1MB chunk size

[kuhnroyal]

Could we split the implementation for buildx vs the "real" cache?

[LouisHaftmann]

We could but GCS doesn't support out-of-order multipart uploads, so we'd need to keep the current system for that too.

[LouisHaftmann]

We will probably drop direct gcs support in the next major release and focus on improving s3 integration. This will make maintenance much simpler and reduce memory/bandwidth usage.

Using GCS might still be possible because they have an s3 compatible api.