API falls over

[peterdudfield]

After about 45 days of running the api it falls over.
Could it be

• storage on server?
• memory runs out?
• Other memory leaks?

Could it be fastapi/fastapi#11079?

[peterdudfield]

The way we run the api is defined here

["uvicorn", "nowcasting_api.main:app", "--host", "0.0.0.0", "--port", "80"]

[peterdudfield]

I connected to a machine that has been running for 13 days

[peterdudfield]

Where as if I connect to a machine thats been running for 3 days

The memory is only 17% (3 days), compared to 33% (13 days) to instance above

[peterdudfield]

On development, weve had 2 apis runnig for 13 days, they are on 41% and 52% memory
If I start a new API up, it is on about 9%

[peterdudfield]

Lots of discussion here, but a bit old - fastapi/fastapi#1624
Some people hinted it could be a pydantic memoery leak - https://github.com/pydantic/pydantic/issues?q=memory+leak, we currently use pydantic-2.11.7, which is the latest and we are using the latest sqlalchemy-2.0.41

Could be pydantic/pydantic#9429

[peterdudfield]

If I reboot the machine, the memory goes down to ~9%, similar to a new machine

[peterdudfield]

One idea could be when

1. scale the elb cluster to have +1 machines, (this happens already)
2. we then reboot any old instances,
3. and then scale down again (this happens already)

This could be inserted here

Of course ideally we would solve memoery leak, but sometimes easier said than done

[peterdudfield]

Other things it could be https://talsabag.hashnode.dev/needle-in-a-haystack-how-we-found-and-solved-a-memory-leak-in-our-integration-framework

[peterdudfield]

Could there be something funny about the redirect of pvlive/gsp_id

[peterdudfield]

Could try something like this https://www.uvicorn.org/settings/#resource-limits

[peterdudfield]

On dev, verison 1.6.5 has been running for 16 hours, and its on ~16% memoery on top