RSS slowly grows over time

[matthew-hibachi-xyz]

RSS grows as the service stays up and does not appear to be cleared over time. This is running the containerized version ghcr.io/celestiaorg/pda-proxy:v0.2.2

[nuke-web3]

Sorry I missed this before @matthew-hibachi-xyz - please do DM so I know when you put up an issue so I can prioritize it 🙏

My suspicion is this is somewhat expected behavior based on use of sled (see this issue). If that is the case, this might motivate us to move away from sled, that would be a non-trivial refactor, but would enable more control of the DB for things like TTL or other sliding retention window much easier, and multi-proxy setups w/ a single job DB backing them.

In the meantime I think the unfortunate work around is stop the proxy, clear it's DB, and restart it. (The DB is only caching intermediate states for jobs in case of crash, and finished/erroneous ones long term) This could be scripted to run after some time and a job completes successfully so as not to redo any work on a job in-progress.

[matthew-hibachi-xyz]

Sorry I missed this before @matthew-hibachi-xyz - please do DM so I know when you put up an issue so I can prioritize it 🙏

My suspicion is this is somewhat expected behavior based on use of sled (see this issue). If that is the case, this might motivate us to move away from sled, that would be a non-trivial refactor, but would enable more control of the DB for things like TTL or other sliding retention window much easier, and multi-proxy setups w/ a single job DB backing them.

In the meantime I think the unfortunate work around is stop the proxy, clear it's DB, and restart it. (The DB is only caching intermediate states for jobs in case of crash, and finished/erroneous ones long term) This could be scripted to run after some time and a job completes successfully so as not to redo any work on a job in-progress.

Thanks @nuke-web3, this is low priority from our side. We restart the service weekly with our deployment cycle so that's enough to clear to RSS usage, so it's not currently impacting us. I just wanted to note that this was a behavior of the service.