k8s Go modules cause many requests

[jplitza]

How are you running Renovate?

Self-hosted Renovate CLI

Which platform you running Renovate on?

GitLab (.com or self-hosted)

Which version of Renovate are you using?

43.11.0

Please tell us more about your question or problem

Renovate Bot seems to make a request to the configured $GOPROXY (or proxy.golang.org if unset) for every version of every dependency that is to be inspected.

There are Go modules with hundreds of versions. Particularly the k8s.io/* and sigs.k8s.io/* modules:

$ curl -s https://proxy.golang.org/k8s.io/client-go/@v/list | wc -l
627

And since indirect dependencies seem to be checked as well, our repository with 56 dependencies on k8s modules causes almost 20k requests to the $GOPROXY on each scan. This seems excessive, especially if they hit the community-provided (default) proxy.golang.org.

AFAICS, the requests for the individual versions are needed to fetch the digest and the release timestamp, both of which shouldn't matter for versions not selected to be updated to.

Logs (if relevant)

Logs

Replace this text with your logs, between the starting and ending triple backticks

[jamietanna]

Do you have any caching set up? (related)

If you're running the CLI from fresh each time, with no caching, you'll hit this yes - on the Mend-hosted apps, we have a Redis instance set up to cache these lookups, so it's reduced impact

[jplitza]

AFAICT, the Renovate cache is (and has to be) invalidated regularly (every 30 minutes by default). Since we only executed Renovate every few hours, this does not improve the situation.

[jamietanna]

the Renovate cache is (and has to be) invalidated regularly (every 30 minutes by default).

To confirm, who/what is noting that it has to be regularly invalidated?

[jplitza]

My observation: For 30 minutes after running Renovate Bot with persistent RENOVATE_CACHE_DIR, no requests are made to proxy.golang.org. After 30 minutes, the same amount of requests appears again:

         "proxy.golang.org": {
           "count": 18386,
           "reqAvgMs": 179,
           "reqMedianMs": 36,
           "reqMaxMs": 15537,
           "queueAvgMs": 75,
           "queueMedianMs": 24,
           "queueMaxMs": 14599
         },

My understanding (which might be wrong): If the package version cache weren't invalidated, Renovate Bot couldn't inform me about new versions. Because AFAICT from the files in the cache directory, the cache does not contain the result of those 20k HTTP requests, but instead only "the newest version of package X at time T is Y". And if T is more than 30 minutes ago, the cache is "soft expired" and the requests are repeated.