Feature request: Mergable persistance in Postgres tabular #237
Description
Hi, me again, another issue to track and get feedback on what improvements we're doing to our fork of TinyBase and what we'll raise PR to contribute back.
Problem / use-case
Is your feature request related to a problem? Please describe.
As docs mention using mergable data structure with tabular store isn't supported in any SQL backed database
The following database-oriented Persister types can be used to persist a MergeableStore, but only in the 'JSON-serialization' mode:
- PostgresPersister
Our use-case for having this (related to #236) is that especially during migration towards TinyBase we have an existing application that is writing directly to postgres and we want to sync the changes to a store which is then later synced to DurableObjects and to clients.
Feature Request
Describe the solution you'd like
tl;dr: Allow to use updated_at column from the table to create approximate HLC during load from DB. But to be able to do that we'd probably need to change HLC logic.
I understand that full support is not possible because it would mean storing HLC and the hash for every cell which in tabular mode isn't really possible/practical. However I believe that we can achieve partial support that would be good in 99% of cases and better than having none.
disclaimer: following is how I understand how it works. And I'm probably wrong
If I understand how HLC generally works it has combination of physical time and logical component. And if we could use updated_at timestamp from the database as source of physical time we could create an approximate HLC which could be used for merging data together. If we fix the logical counter for approx HLC to 0 we would have a predictable behaviour that in case of 2 writes within same millisecond TinyBase wins. That would work great for us.
However, looking through Tinybase hlc implementation it seems that it's not using physical time at all. I'm assuming that it's because we can't trust browsers to have correct time set and wrong time could cause completely wrong resolutions. But often HLC implementations have built in protection by checking for back-in-time jumps and adjusting physical time (from wall clock time) to handle that gracefully. As guessed at in the original write down, I was wrong. The Tinybase HLC implementation uses timestamp
Alternatives
Describe alternatives you've considered
unfortunately we don't see an alternative that would work for us within current TinyBase capabilities. Our main alternative is to completely fork the persister and implement this.
Additional context
This partially relates to my existing PR for whereCondition and Feature Request #236 for partial load of postgres changes but it's not coupled with either of them.