FinalizeCacheEntryUpload 404 Upload Not found

[awoisoak]

I'm trying to stress test the cache server and got into the next error:

Github

Sent 10351934759 of 10486152487 (98.7%), 151.8 MBs/sec
Sent 10486152487 of 10486152487 (100.0%), 153.2 MBs/sec
Attempt 1 of 5 failed with error: Request timeout: /twirp/github.actions.results.api.v1.CacheService/FinalizeCacheEntryUpload. Retrying request in 3000 ms...
Attempt 2 of 5 failed with error: Request timeout: /twirp/github.actions.results.api.v1.CacheService/FinalizeCacheEntryUpload. Retrying request in 5374 ms...
Attempt 3 of 5 failed with error: Request timeout: /twirp/github.actions.results.api.v1.CacheService/FinalizeCacheEntryUpload. Retrying request in 8804 ms...
Warning: Failed to save: Failed to FinalizeCacheEntryUpload: Received non-retryable error: Failed request: (404) Upload not found

Server

[cache-server-node-1] ⚙ Request: POST /twirp/github.actions.results.api.v1.CacheService/FinalizeCacheEntryUpload

[cache-server-node-1]  ERROR  Response: POST /twirp/github.actions.results.api.v1.CacheService/FinalizeCacheEntryUpload > 404
 Upload not found

    at createError$1 (server/index.mjs:647:15)
    at Object.handler (server/chunks/routes/twirp/github.actions.results.api.v1.CacheService/FinalizeCacheEntryUpload.mjs:46:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async Object.handler (server/index.mjs:1633:19)
    at async Server.toNodeHandle (server/index.mjs:1904:7)

  [cause]: { statusCode: 404, statusMessage: 'Upload not found' }

The weird thing is that when I trigger a new GH action, it is actually able to find and download the cached files (with some timeout though)

Run actions/cache@v4
Attempt 1 of 5 failed with error: Request timeout: /twirp/github.actions.results.api.v1.CacheService/GetCacheEntryDownloadURL. Retrying request in 3000 ms...
Cache hit for restore-key: test-foo-Linux-17143879533
Cache Size: ~10000 MB (10486152487 B)
/usr/bin/tar -xf /home/runner/_work/_temp/84dc1a91-1c67-4187-9b4e-6e01ada531a3/cache.tzst -P -C /home/runner/_work/poc-arc/poc-arc --use-compress-program unzstd
Cache restored successfully
Cache restored from key: test-foo-Linux-17143879533

I suspect this might be an issue specific to my environment, given that it involves a large cache, but I wanted to report it just in case.
I’ve set up the server with Postgres and S3, and I’m able to reproduce the issue with the following setup:

  ci:
    name: Run CI
    runs-on: arc-stg-amd64
    steps:
      # At this step it will attempt to restore foo from the cache server (if the corresponding key is found)
      - name: Cache foo directory
        uses: actions/cache@v4
        with:
          path: foo
          key: test-foo-${{ runner.os }}-${{ github.run_id }}
          restore-keys: |
            test-foo-${{ runner.os }}-

      - name: Display contents after restore
        run: |
          echo "Contents of foo:"
          ls -l foo || echo "No cache found"

      # Generate a total of 10GB files under 'foo' to create a large cache for the stress test
      - name: Generate large cache files
        run: |
          mkdir -p foo
          for i in {1..100}; do
            head -c 100M </dev/urandom > foo/artifact_$i.bin
          done

[LouisHaftmann]

Probably a memory issue. Looks like the cache archive is loaded into memory at some point. I already know how to fix it and will do so as soon as I find the time.

[LouisHaftmann]

related to #160

[emmanuelnk]

I'm also facing this issue. If you need help with implementation point me in the right direction and I can maybe take a stab at it.