Fix timestamp representation

[simolus3]

The mongo and postgres replicators used to represent timestamp values with a space between the date and time parts, whereas ISO-6801 would require the letter T. While we would like to align the format with that standard, doing so is a backwards-incompatible change since existing rows would sync differently.

This PR introduces two new concepts to fix this issue:

1. SqliteInputValue: This expands the SqliteValue type used by sync rules to represent SQLite rows to also support a CustomSqliteValue. Values of that type can decay dynamically into SqliteValue by giving them additional context. We can thus represent date values as CustomSqliteValues, that, depending on context, use the legacy or the ISO-6801 format.
2. As part of the context as pass to CustomSqliteValues, this adds the Quirk class representing a historical issue of the sync service we can't fix without a backwards-incompatible change. Users can opt in to fixes with the fixed_quirks option in the sync-rules config, and there's also a CompatibilityLevel to fix some quirks by default. The idea is that streams would implicitly raise that level internally.

When evaluating parameter and data rows, CustomSqliteValues are mapped into the actual values using the context specified in sync rules.
The Mongo and Postgres adapters have been fixed to return these custom values instead of applying a fixed format for date values.

Users can opt into this fix by adding this section to their sync rules:

fixed_quirks:
  - non_iso8601_timestamps

There are a number of other issues also worth fixing (in particular, #299, handling postgis types properly, fixing the ->> operator, encoding sync rule id in bucket identifiers). Since most of this PR is just laying the groundwork for those changes, I'll open additional PRs for those.

Closes #286.

[changeset-bot]

🦋 Changeset detected

Latest commit: b8338d3

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 17 packages

Name	Type
@powersync/service-sync-rules	Minor
@powersync/service-image	Minor
@powersync/service-jpgwire	Patch
@powersync/service-core-tests	Patch
@powersync/service-core	Minor
@powersync/lib-services-framework	Patch
@powersync/service-module-mongodb-storage	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres-storage	Patch
@powersync/service-module-postgres	Patch
@powersync/lib-service-postgres	Patch
@powersync/service-module-core	Patch
test-client	Patch
@powersync/service-rsocket-router	Patch
@powersync/lib-service-mongodb	Patch
@powersync/service-schema	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[rkistner]

I'm sure I posted this earlier, but can't find it now...

I'm happy with the implementation, but I'm wondering if we could phrase the settings as "config" rather than "quirks", and provide some form of version number to easily get the latest recommended defaults. I'm thinking something like this (other ideas welcome):

config:
  edition: 2 # single version number to select a set of defaults - we can include this by default in new sync rules. Another option here is `preset`
  timestamps_iso8601: true # or set this one directly
  json_extract_quirks: false # not sure if there's a better name here
  versioned_bucket_ids: true

For reference, some examples I know of from other projects:

Rust:

edition = "2024"

Rails:

config.load_defaults "6.1"
config.yjit = true

[simolus3]

I agree with giving the configuration keys a more positive name 👍

How would we deal with a combination of edition: 1 and sync streams being used, should that be an error? Or do we want the option of sync steams using the old behavior as an opt-out strategy?

[rkistner]

How would we deal with a combination of edition: 1 and sync streams being used, should that be an error? Or do we want the option of sync steams using the old behavior as an opt-out strategy?

Migration could perhaps be easier if we do also allow explicitly opting in to the old behavior. In that case we'd default to say edition 2 for sync streams, but still allow specifying edition 1. Does that complicate anything in the implementation?

[simolus3]

Does that complicate anything in the implementation?

Not substantially - just need more tests.

Did we want want to allow sync streams and sync rules in the same file to ease the transition? I think there was a discussion on this at some point, but I don't remember the outcome (edit: we do, but secretly). But if so, I think defaulting to edition: 2 if sync streams are used may even be harmful, because if you don't have a config block, adding a new sync stream now alters the behavior of existing sync rules.

I think my preferred approach would be to:

1. Make edition: 1 the default.
2. Forbid the use of sync streams with edition < 2.
3. Enable the new config items by default with edition: 2, but allow disabling them (e.g. {edition: 2, versioned_bucket_ids: false}).
4. Add edition: 3 that is like edition: 2 but disallows sync rules.
5. In the near-ish future, add edition: 3 by default when new instances are created (still allowing users to downgrade editions).
6. Much further into the future, make edition: 3 the default and disallow older editions.

This approach is a bit similar to some features requiring new editions in Rust, and step 6 is similar to how Dart 3.x still supports older language versions but at least 2.12 for null-safety. I think that approach has generally worked quite well.

[rkistner]

I'm happy with that approach.

I'm starting to think it makes sense to just encourage specifying an explicit edition (i.e. have it in all our templates), rather than setting the default with sync streams.

Much further into the future, make edition: 3 the default and disallow older editions.

Before we drop support for older editions we'll need to have a proper versioning and support policy. For example, we could potentially drop it in version 2.x or 3.x of the service, while still maintaining some level of support (bugfixes) for older major versions.

[simolus3]

I've implemented that approach now 👍