What would a 'strict mode' look like for PyAirbyte?

[aaronsteers]

PyAirbyte started with a somewhat idealized framing of how connectors would work together - as if you could pipe data directly from a source to a destination.

In reality, the Airbyte Platform softens rough edges, and in some cases hides what would otherwise be a failure condition. As PyAirbyte has to do some of these same issues, we should document those cases and see if we can detect/fix them (in the long run) in sources themselves.

Category A: Mitigating errors in the connnectors

For expediency and practicality, we iron our issues that can't be fixed in the source within a time-sensitive manner. But in the longer run, the connectors should eventually be fixed/improved.

Case A1: Undeclared fields

Case A2: Schema validation errors

Examples:

• Data is string instead of timestamp.
• Sub-properties not declared.

Category B: TBD

Category C: Unsupported / Divergent Behaviors

• Support config write-back #8 (via CONTROL message type)
• PyAirbyte doesn't do anything for State Migration.
  • Ideally, sources would do this themselves. If the platform is taking a role, PyAirbyte would not be able to (yet) do the same.
• Config Migration
  • Similarly, PyAirbyte doesn't attempt to migrate source config. If users have a pinned connector version, they should be stable.