Very slow performance

[nikiepaydk]

Hi.

We just installed a standard Athens server using default configuration with storage set to disk.

When downloading our dependencies with go mod download we see a very large performance degradation, and are wondering if there are any configurations that are generally recommended to improve the performance?

The server seems to use a lot of CPU time. When we increase the CPU resources so it is no longer hitting 100% we do see an increase in performance, but it is still 6 times slower (40sec) than without athens (7sec).

Package updates such as go get -u is also considerably slower than without athens.

Without Athens

$ time go mod download

real	0m7.504s
user	0m2.132s
sys	0m1.175s

With Athens

$ time go mod download

real	1m32.169s
user	0m2.115s
sys	0m1.202s

All subsequent cached downloads are fine:

$ time go mod download

real	0m4.368s
user	0m2.100s
sys	0m1.181s

[garvit14]

@nikiepaydk were you able to find a solution to this? I am also facing similar performance regression when using Athens.

[r-ashish]

You may wanna add an env override for this config:ATHENS_GO_BINARY_ENV_VARS='GOPROXY=proxy.golang.org,direct, or if you prefer using the config file then it can be set via GoBinaryEnvVars.

By default this config is set to direct which means Athens will try to download it directly from the source repository instead of checking the public Go proxy first, which is much slower.

This of course will only work for public modules so if you’d like more control over the download behaviour then I’d suggest checking The Download Mode File.

[garvit14]

@r-ashish I already have all the modules in Athens cache (s3) so changing ATHENS_GO_BINARY_ENV_VARS will not have any impact.

[r-ashish]

Ok my original response was based on what @nikiepaydk had mentioned above i.e. cached downloads are working fine and only the first go get for a package version seems slow, so your problem is probably different from Niki.

Anyway, do you mind sharing more about your setup? Redacted logs with timestamps might also be helpful.

Normally serving from cache shouldn't have any such performance bottlenecks unless the instances aren't scaled properly to handle the QPS. I'd also suggest checking CPU and memory usage patterns and tweak your scaling params if you do find them overloaded during slow response times.

[nikiepaydk]

Hi @r-ashish thanks for the heads up!

We just updated our config to include the mentioned proxy. When I run a go get -u it is still quite slow.

time go get -u

real    1m0.469s
user    0m0.932s
sys     0m0.405s

We can send our config or perhabs some logs if this helps detect the issue? If logs would help, what kind of logs and do we need to enable any sort of verbose mode beforehand?

[nikiepaydk]

@r-ashish Our proxy server with 8 CPU threads is getting overloaded and runs constantly at 100% CPU load during a go get -u.

On the client side things fails after a while:

$ time go get -u
go: downloading github.com/philhofer/fwd v1.2.0
go: downloading golang.org/x/image v0.29.0
go: downloading github.com/aws/aws-sdk-go-v2 v1.37.0
go: downloading github.com/aws/smithy-go v1.22.5
go: downloading github.com/aws/aws-sdk-go-v2/service/s3 v1.85.0
go: downloading github.com/aws/aws-sdk-go-v2/config v1.30.0
go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.18.0
go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.35.0
go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.31.0
go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.26.0
go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.17.0
go: downloading github.com/thomaspoignant/go-feature-flag v1.45.5
go: downloading github.com/diegoholiveira/jsonlogic/v3 v3.8.4
go: downloading github.com/antlr4-go/antlr/v4 v4.13.1
go: downloading github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.0
go: downloading github.com/diegoholiveira/jsonlogic v1.0.0
go: downloading golang.org/x/exp v0.0.0-20250718183923-645b1fa84792
go: downloading github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.0
go: downloading github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.0
go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.0
go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.0
go: downloading github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.8.0
go: module github.com/aws/aws-sdk-go-v2/aws/ratelimit: reading https://athens.our-server.com/github.com/aws/aws-sdk-go-v2/aws/ratelimit/@v/list: 502 Bad Gateway
go: module github.com/aws/aws-sdk-go-v2/internal/endpoints: reading https://athens.our-server.com/github.com/aws/aws-sdk-go-v2/internal/endpoints/@v/list: 502 Bad Gateway
go: module github.com/aws/aws-sdk-go-v2/internal/endpoints: reading https://athens.our-server.com/github.com/aws/aws-sdk-go-v2/internal/endpoints/@v/list: 502 Bad Gateway
go: module github.com/aws/aws-sdk-go-v2/internal/timeconv: reading https://athens.our-server.com/github.com/aws/aws-sdk-go-v2/internal/timeconv/@v/list: 502 Bad Gateway
go: module github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding: reading https://athens.our-server.com/github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding/@v/list: 502 Bad Gateway
go: module github.com/aws/aws-sdk-go-v2/service/internal/presigned-url: reading https://athens.our-server.com/github.com/aws/aws-sdk-go-v2/service/internal/presigned-url/@v/list: 502 Bad Gateway
go: module github.com/aws/smithy-go/container/private/cache: reading https://athens.our-server.com/github.com/aws/smithy-go/container/private/cache/@v/list: 502 Bad Gateway
go: module github.com/aws/smithy-go/container/private/cache/lru: reading https://athens.our-server.com/github.com/aws/smithy-go/container/private/cache/lru/@v/list: 502 Bad Gateway
go: module github.com/aws/smithy-go/waiter: reading https://athens.our-server.com/github.com/aws/smithy-go/waiter/@v/list: 502 Bad Gateway
go: github.com/thomaspoignant/go-feature-flag@v1.45.5: reading https://athens.our-server.com/github.com/thomaspoignant/go-feature-flag/@v/v1.45.5.zip: 502 Bad Gateway
go: module github.com/thomaspoignant/go-feature-flag/exporter: reading https://athens.our-server.com/github.com/thomaspoignant/go-feature-flag/exporter/@v/list: 502 Bad Gateway
go: github.com/thomaspoignant/go-feature-flag@v1.45.5: reading https://athens.our-server.com/github.com/thomaspoignant/go-feature-flag/@v/v1.45.5.zip: 502 Bad Gateway
go: github.com/thomaspoignant/go-feature-flag@v1.45.5: reading https://athens.our-server.com/github.com/thomaspoignant/go-feature-flag/@v/v1.45.5.zip: 502 Bad Gateway
go: github.com/thomaspoignant/go-feature-flag@v1.45.5: reading https://athens.our-server.com/github.com/thomaspoignant/go-feature-flag/@v/v1.45.5.zip: 502 Bad Gateway
go: module github.com/thomaspoignant/go-feature-flag/internal/flag: reading https://athens.our-server.com/github.com/thomaspoignant/go-feature-flag/internal/flag/@v/list: 502 Bad Gateway
go: module github.com/thomaspoignant/go-feature-flag/internal/flagstate: reading https://athens.our-server.com/github.com/thomaspoignant/go-feature-flag/internal/flagstate/@v/list: 502 Bad Gateway
go: github.com/thomaspoignant/go-feature-flag@v1.45.5: reading https://athens.our-server.com/github.com/thomaspoignant/go-feature-flag/@v/v1.45.5.zip: 502 Bad Gateway
go: github.com/thomaspoignant/go-feature-flag@v1.45.5: reading https://athens.our-server.com/github.com/thomaspoignant/go-feature-flag/@v/v1.45.5.zip: 502 Bad Gateway
go: github.com/thomaspoignant/go-feature-flag@v1.45.5: reading https://athens.our-server.com/github.com/thomaspoignant/go-feature-flag/@v/v1.45.5.zip: 502 Bad Gateway
go: github.com/thomaspoignant/go-feature-flag@v1.45.5: reading https://athens.our-server.com/github.com/thomaspoignant/go-feature-flag/@v/v1.45.5.zip: 502 Bad Gateway
go: module github.com/thomaspoignant/go-feature-flag/notifier: reading https://athens.our-server.com/github.com/thomaspoignant/go-feature-flag/notifier/@v/list: 502 Bad Gateway
go: module github.com/thomaspoignant/go-feature-flag/notifier/logsnotifier: reading https://athens.our-server.com/github.com/thomaspoignant/go-feature-flag/notifier/logsnotifier/@v/list: 502 Bad Gateway
go: github.com/thomaspoignant/go-feature-flag@v1.45.5: reading https://athens.our-server.com/github.com/thomaspoignant/go-feature-flag/@v/v1.45.5.zip: 502 Bad Gateway
go: module github.com/thomaspoignant/go-feature-flag/retriever/fileretriever: reading https://athens.our-server.com/github.com/thomaspoignant/go-feature-flag/retriever/fileretriever/@v/list: 502 Bad Gateway
go: github.com/thomaspoignant/go-feature-flag@v1.45.5: reading https://athens.our-server.com/github.com/thomaspoignant/go-feature-flag/@v/v1.45.5.zip: 502 Bad Gateway

real    4m29.019s
user    0m2.318s
sys     0m0.943s

We end up seeing a lot of 502 Bad Gateway.

[kounne]

@nikiepaydk try to use this tag instead - v0.15.4
I've suffered the same performance problems and after rollback everything is fine. Not sure what's the reason behind this.

go env -w GOPROXY="http://localhost:3000,direct"
go clean -modcache
time go mod download
go: downloading go1.24.1 (linux/amd64)

real 0m15.890s
user 0m11.319s
sys 0m3.199s

go env -w GOPROXY="https://proxy.golang.org,direct"
go clean -modcache
time go mod download
go: downloading go1.24.1 (linux/amd64)

real 0m47.353s
user 0m11.491s
sys 0m3.808s

[DrPsychick]

I tried to reproduce this with athens 16.0, but apart from the initial delay (which is due to athens needing to fetch mods in order to cache them, so it is kind of unavoidable), I do get fast go mod download after that.

Can you possibly share your go.mod, so we can see if we can reproduce it with that?

[kounne]

Hello!
Here is some information I gathered while trying to reproduce this issue on my local machine. And i think i've found the issue on my side.

The tests were conducted on a machine with 20 vCPUs, 32 GB of RAM, and a 1 TB M.2 SSD. During the tests with version 16, the CPU usage peaked at around 70%. After downgrading, it dropped significantly to 1–5%.

Below are the detailed results:

v16 – First run:
TotalMinutes: 2.45

v16 – Second run:
TotalMinutes: 0.8

v15.4 – First run:
TotalMinutes: 0.23

v15.4 – Second run:
TotalMinutes: 0.18

After that i set GONOSUMDB for our Gitlab repo.
v16 - First run:
TotalMinutes : 0,08(4,8 seconds)
v16 - Second run:
TotalMinutes : 0,06(4,7 seconds)

Good to know i guess. My bad :D

[nikiepaydk]

@kounne & @DrPsychick & @r-ashish

We have made a reproducible docker composer setup that illustrate our performance issues here: https://github.com/nikiepaydk/athens-performance

You can see how to run it in the readme. But the time goes from about 2 seconds to 4,5 minutes when using Athens.

[kounne]

@nikiepaydk, my guess is that Athens doesn't work well with go get -u.
As you can see in the logs, there are hundreds of 404 errors. I'm not entirely sure why you're using the -u flag (I'm not a developer myself), but from what I can tell, removing it resolves the issue and makes everything work normally.
The -u flag forces Go to look for the latest versions of all dependencies, including transitive ones, which generates a large number of invalid or non-existent module queries (like /github.com/redis/@v/list). Athens tries to process each of these, often timing out or failing with EOF/404, which drastically slows down the process.

command: ["sh", "-c", "time go get -d -x -v"]
go-app-athens | real 0m 6.11s
go-app-athens | user 0m 0.32s
go-app-athens | sys 0m 0.40s

[nikiepaydk]

@kounne The purpose is to update all dependencies to their latest versions. So the -u is important for this behaviour. But the default module proxy is able to handle this with a breeze. So another module proxy should be able to do the same, or atleast proxy the requests to the default proxy so it can handle it.

[DrPsychick]

I did a "short" testrun with your docker-compose, awesome that you can reproduce it with that 👍 Also allows to play with some settings.

Athens downloads over 12GB of data, so it's doing a lot more than what a direct download is doing (a few MB) and obviously this takes time.

This is just FYI, I haven't looked deeper into what athens does there. What I did notice was periods of athens apparently idling, maybe due to slow downloads.

But I'm intrigued to learn more about your use case. Athens is a proxy and it's purpose is to cache upstream packages and provide access to private packages. As such, it's purpose is to live long and have storage to cache packages. So why exactly do you need athens to be fast on the very first run with an empty storage? While the performance of athens for this case could possibly be improved (I don't know, just a guess), it seems you're using athens in a way it's not meant to be used (or at least not fast).
Or is it rather that athens is always slow on a go get -u? If so, how exactly is that an issue when it's slower than direct?