How to make the stream schema dynamic?

[somiandras]

I'm building replication from MySQL transactional db to Postgres, and it turned out that the db I can use for dev/testing purposes is named differently from the final (eg. mydb_staging and mydb_prod), ie. if I write stream keys like mydb_staging.my_table in replication.yaml, it won't work on prod (or the other way around, if I use mydb_prod, I can't run it in dev). I don't have control over the db names and how I can access them.

Is there a way to define the schema somewhere and use interpolation in the stream keys? Something like

streams:
  {schema_from_envvar}.my_table:
    ...

[flarco]

Just checked over the logic, the stream name doesn't get formatted, so it won't work. I'll enable that next release.

As a workaround, try this (the custom SQL does get formatted):

defaults:
  sql: "select * from {schema}.{stream_name}"

streams:
  my_table:

env:
  schema: '${SCHEMA}'   # From Environment

[somiandras]

Wait, I'm in a weird place actually... I'm installing Sling via dagster-embedded-elt in a Docker container.

[flarco]

Could you share your config? I ran this and it worked fine:

source: postgres
target: sqlite

defaults:
  mode: full-refresh
  object: 'main.pg_{stream_table}'
  sql: "select *, 'hi' there from {stream_name}"

streams:
  public.my_table*:

It rendered to select *, 'hi' there from "public"."my_table2"

[somiandras]

So I was behind... sorry for the confusion.

[flarco]

Ah yes, the python deps were changed. Can you uninstall all and reinstall?

[somiandras]

Just want to confirm: after cleaning up my env the workaround with the default sql works. Thanks again!