Force Reindex Data from Postgres to Typesense In Case of Crashes

[stkegoul]

Description

Currently, the system relies heavily on Typesense for searching by balance ID or reference code. However, when Typesense crashes, data remains stored in Postgres but becomes unavailable for querying in Typesense. This creates a data inconsistency issue. We need a mechanism to force reindex data from Postgres to Typesense to ensure data availability after a crash.

Expected behaviour

1. After a Typesense crash, there should be a way to force reindex all data from Postgres to Typesense.
2. Queried data should be consistently available in Typesense after reindexing.

Proposed solution

1. Implement a script or process to force reindex data from Postgres to Typesense.
2. Override the meta_data field (change from string to object as part of the migration).
3. Document the migration steps in the process.

[jerry-enebeli]

I'd appreciate any help with this 🙏

[stkegoul]

Worked on a script that reindexes existing data in the DB into typesense — https://github.com/stkegoul/blnk-reindex. Can be used multiple times as well.