High Performance Remote Database Read/Write/Sync Library

[pkjmesra]

We need a python library (either already available some open source library or build a new one) that can connect to https://app.turso.tech/ database.

We should be able to at the very least do the following:

1. Allow running it as a standalone service or a module in another service/application.
2. Connect to a remote database like app.turso.tech or other similar remote database service that can be free for large number of read/writes and have sufficiently large free storage allowance.
3. Be able to write at least 4000 writes per minute (60-70 writes per second) remotely without causing dirty writes.
4. Should allow spanning the read/write capabilities using multiprocessing AND multi-threading so that we should be able to distribute the workload with simultaneous read/write needs.
5. Should be able to read from the remote database @ at least 100-150 per second.
6. Should be able to reconcile database syncs from other instances (slaves) of running services of itself and then write or read remotely when running at a master node.
7. Should be able to capture all data writes to a local database file (SQLite?) and then be able to write (sync) later without any loss of data and along with reconciliation.
8. Should be able to use database level caching to avoid repeated reads.
9. Everything should be configuration driven and should be importable/exportable. So, tomorrow we should be able to move to another database provider.
10. Should fallback on local database if remote service is unavailable. It should follow exponential backoff in such cases and later sync the data, as described above.
11. Be able to monitor the number of reads/writes during a session to specific table(s) and at database level.
12. Be able to monitor stale data to identify which stock data/ticks are not getting updated during market hours.

This library will be used for the following purposes:

1. Register and create new user accounts using their Telegram usage via PKScreener bot
2. Read historical data (last 1 year+) of 2000+ stocks/instruments on-demand for running various multiple scans simultaneously. Do the same for NASDAQ's 8000+ instruments.
3. Write historical data for 2000+ stocks from NSE and 8000+ stocks for NASDAQ.
4. Write instrument ticks during market hours. This will be extremely write-heavy (4000+ writes per minute).
5. Read daily ticks for the saved stocks to run various scans triggered by multiple users.
6. Save each unique scan results every minute during market hours, but save throughout during off market hours. The data should be marked stale at the end of the minute (during market hours), but should be cached for the minute during market hours (or throughout off market hours).
7. Export/import data on-demand.