fix: adjust file healthcheck durations

[mzglinski]

Legal Boilerplate

Look, I get it. The entity doing business as "Sentry" was incorporated in the State of Delaware in 2015 as Functional Software, Inc. and is gonna need some rights from me in order to utilize my contributions in this here PR. So here's the deal: I retain all rights, title and interest in and to my contributions, and by keeping this boilerplate intact I confirm that Sentry can use, modify, copy, and redistribute my contributions, under Sentry's choice of terms.

[aminvakil]

What is the necessity for adding swap? AFAICS it already has 16GB memory which is above self-hosted limit (14GB)
https://github.com/getsentry/self-hosted/actions/runs/16963036969/job/48080155328?pr=3874#step:4:4255

               total        used        free      shared  buff/cache   available
Mem:            15Gi       1.6Gi       4.3Gi        48Mi        10Gi        13Gi

[aminvakil]

Sorry, I've just seen #3859 (comment)

I think there is another problem here if it wants to use more than 16GB.

[mzglinski]

There are 72 containers, when they start (almost all) at once, the RAM requirements are much higher than 16GB, it settles later, but during startup, swap is required.

[mzglinski]

Adding healthcheck just exposed that problem, before --wait finished without error, even if containers without healthcheck were killed or not started fully.

[aminvakil]

There are 72 containers, when they start (almost all) at once, the RAM requirements are much higher than 16GB, it settles later, but during startup, swap is required.

We should increase memory requirement if that's the case, there are many scenarios which swap cannot be added, my environments are one of them:)

[mzglinski]

We should increase memory requirement

That is not a bad idea; however, we cannot do much about GitHub-hosted runners, having the amount of resources they have. There are large runners, but those are not free to use.

[aminvakil]

Adding healthcheck just exposed that problem, before --wait finished without error, even if containers without healthcheck were killed or not started fully.

Oops! Has there been any other complaint about this in issues or discord? @aldy505

We should increase memory requirement

That is not a bad idea; however, we cannot do much about GitHub-hosted runners, having the amount of resources they have. There are large runners, but those are not free to use.

Yes, you're right. We have no other option aside from adding swap in GHA.

[aldy505]

There are 72 containers, when they start (almost all) at once, the RAM requirements are much higher than 16GB, it settles later, but during startup, swap is required.

We should increase memory requirement if that's the case, there are many scenarios which swap cannot be added, my environments are one of them:)

No, please don't do that. I already recommend people to add some swapfile to battle with the insane amount of RAM usage. I'd prefer adding more info regarding swap.

[aldy505]

Adding healthcheck just exposed that problem, before --wait finished without error, even if containers without healthcheck were killed or not started fully.

Oops! Has there been any other complaint about this in issues or discord? @aldy505

I was the one who complained 😆. SH e2e tests are failing everywhere. And I don't want to merge the revert PR.

[aldy505]

Your knowledge is insane!!

[aldy505]

@aminvakil Burak is on vacation and I don't wanna wait until Hubert is awake. If you're ok with this, I'll go merge this.

[aminvakil]

I'm fine with adding swap on GHA.

[aminvakil]

Adding healthcheck just exposed that problem, before --wait finished without error, even if containers without healthcheck were killed or not started fully.

Oops! Has there been any other complaint about this in issues or discord? @aldy505

I was the one who complained 😆. SH e2e tests are failing everywhere. And I don't want to merge the revert PR.

I still cannot understand how integration tests passed before, it only needs more than 16GB upon starting up?

[mzglinski]

docker compose up --wait stops observing a service without a defined healthcheck once it reaches the running state. After that, it may restart or be killed, and up --wait will succeed. Once up --wait finishes, nothing is really happening on the machine, so failed or killed services restart quickly.

However, with healthchecks defined, those services not only have to report a healthy state, but also stay in that state for the entire duration of the wait. It was still possible for the test to fail before, but I suspect the odds were quite small for that to happen.

I also did not check what exactly the integration tests are testing, it might be that they would pass even with some snuba or sentry consumers not running ;)

[aminvakil]

I also did not check what exactly the integration tests are testing, it might be that they would pass even with some snuba or sentry consumers not running ;)

I think most probably this was the case.

[aldy505]

I also did not check what exactly the integration tests are testing, it might be that they would pass even with some snuba or sentry consumers not running ;)

Some of it test user creation, login, and some basic interaction with the API. Other than that, it sends events through the Python SDK: error, transaction, and profile. Then, query to the Sentry API to see if there are any submitted event from the SDK.