How does Sling deal with schema evolution?

[alberttwong]

What happens when it sees a target schema that is different? Is there an internal way to deal with schema changes? eg. comparison to https://flywaydb.org/

[flarco]

Sling will attempt to match whatever columns are present in both the source stream and target table.
If an extra column is present in the source stream, it will add it in the target table.
If no columns match from source stream at all, it will error. At least primary_key or update_key must be present in target table.

See below.

# Initial data

$ echo 'a,b,c
1,2,3
4,5,6' > test1.csv

$ sling run \
  --src-stream file://./test1.csv \
  --tgt-conn postgres \
  --tgt-object public.test1

...

$ sling run \
  --src-conn postgres \
  --src-stream public.test1 \
  --stdout

a,b,c,_sling_loaded_at
1,2,3,1707869559
4,5,6,1707869559

# test2.csv is missing column b

echo 'a,c
7,8' > test2.csv

$ sling run \
  --src-stream file://./test2.csv \
  --tgt-conn postgres \
  --tgt-object public.test1 \
  --mode incremental \
  --primary-key a

...

$ sling run \
  --src-conn postgres \
  --src-stream public.test1 \
  --stdout

a,b,c,_sling_loaded_at
1,2,3,1707869559
4,5,6,1707869559
7,,8,1707869689

# test3.csv is missing column b, c and has extra column d

$ echo 'a,d
9,10' > test3.csv

$ sling run \
  --src-stream file://./test3.csv \
  --tgt-conn postgres \
  --tgt-object public.test1 \
  --mode incremental \
  --primary-key a

...

$ sling run \
  --src-conn postgres \
  --src-stream public.test1 \
  --stdout

a,b,c,_sling_loaded_at,d
1,2,3,1707869559,
4,5,6,1707869559,
7,,8,1707869689,
9,,,1707870320,10

[alberttwong]

this should also be published.

[flarco]

https://blog.slingdata.io/how-does-sling-deal-with-schema-evolution

[alberttwong]

i'd also put in the documentation next to transformation since it's related.