Memory and database storage grow indefinitely

[ben221199]

As many people may be aware, this hub implementation has bugs that causes memory or database storage to grow. I will describe both bugs here with all information that is known about it:

Memory:

This bug is also mentioned in lbryio#100. Sometimes, the hub needs a restart in order to fix this. I don't have any indication on what could be the cause.

Database Storage:

The only part of the hub that has the task to write to the database is the scribe part. Other parts (TCP Server and Elasticsearch Updater) only read from it, so the bug is very likely in the scribe part too. It is guessed that scribe writes bullshit to the RocksDB database storage when a blockchain reorganization has happened.

[Freedom-BJ]

Comment from u/MarcSetGo2...

"Just glancing at the repo. It doesn’t look like mempool is thread safe. I assume there’s one mempool. Is it referenced from multiple threads as requests come in? That would definitely be a problem."

How should I reply to Marc?

[Freedom-BJ]

Comment from u/redfacedquark

"I don't know how critical this bug is to your project, if it's a show-stopper I can understand that you would prioritise it.

Is it possible to create the smallest reproducible reliable demo of the bug? If it is intermittent or different on different machines/nodes/users/systems/blockchain data, it would be difficult to say for sure if the bug is properly fixed by any particular change.

This sort of bug suggests that your tests are insufficient. Certainly, your development team is not keeping up with updating dependencies. The lack of commits in the last three years gives me the impression that there actually is no development team.

When I had a quick look at the tests for one part of the project it was just one huge file, suggesting that your architecture may benefit from more modularisation. By doing this it's easier to see if individual components are fully tested.

Personally I don't have the head-space to take on learning a project of this complexity at the moment, especially given there are lots of niche dependencies that I am not familiar with. Perhaps you could use one of the bug bounty platforms, although I think you would need to provide more information about the bug as the link is very terse and vague.

If you're looking for suggestions I would try creating a feature branch where you simply update all the dependencies to the latest version and see if the bug persists. Since it sounds like it was working until some bad data appeared, try to narrow down what data causes the bug by bisecting the blockchain to localise the bad data.

Since you're talking about OOM issues my first thought would be that circular data relations might be at play.

Good luck with your project!"