[Bug]: SDK 2.59.0 Java SpannerIO ChangeStream Can't Use the same metadata database for multiple pipelines

[bangau1]

What happened?

With the latest apache beam sdk 2.59.0, the SpannerIO changestream's metadata table is adding the indexes: https://github.com/apache/beam/blame/160dffd88e5a60077a48e3e2f8fff331aecced08/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/changestreams/dao/PartitionMetadataAdminDao.java#L82-L86

Since the indexes' name are hardcoded to WatermarkIndex and CreatedAtStartTimestampIndex, this poses problem when there are multiple pipelines that are using the same metadata database but using different metadata tablename, since the index name must be unique within the same database.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[bangau1]

After upgraded to dataflow SDK 2.62.0, the behavior is now changed. It assigns UUID to the index. However it's still using the database name as base name. #32689

Question is: why don't we use the metadata table name as the base name instead? So we can let user choose their own metadata table name as the unit of job. @thiagotnunes

[thiagotnunes]

The original intent was to keep the index names with a meaningful prefix while not overflowing to the max name length. Since the generated metadata table could be long (Metadata__), we would waste part of the prefix with that.

Can I ask what problem are you trying to solve here? Are you having issues with the generated table name?

[bangau1]

The original intent was to keep the index names with a meaningful prefix while not overflowing to the max name length. Since the generated metadata table could be long (Metadata__), we would waste part of the prefix with that.

Can I ask what problem are you trying to solve here? Are you having issues with the generated table name?

My use case right now is to have a long running streaming pipeline, that ingest Spanner Changstream to GCS. Sometime there are cases when I need to perform adhoc pipeline, targeting the same source (changestream), but with different (job) checkpoint metadata table. Currently with v2.59.0, it's not possible to do that since the index name is hardcoded (see my original issue description). There are also cases when I need to setup smoke test in real infrastructure.

Now with v2.62.0, i think it's probably possible, but the index name is derived from the database, making it harder to do some cleanup. It would be great if naming the index the same with metadata table name, or let it customizeable from the end user. This way, user has some control on end to end the resources created by the pipeline and do some cleanup later on

[thiagotnunes]

Got it, I think we could parameterize the metadata table prefix and use that in the index definitions.

Meanwhile, could you get the UUID of the generated metadata table name (from the Dataflow job parameters) and use that to clean up the schema? The same UUID is used for the metadata table and the indexes.

[bangau1]

Got it, I think we could parameterize the metadata table prefix and use that in the index definitions.

Meanwhile, could you get the UUID of the generated metadata table name (from the Dataflow job parameters) and use that to clean up the schema? The same UUID is used for the metadata table and the indexes.

Thank you @thiagotnunes for considering my use case, really appreciate it 🙏

In the meantime, I can manually cleanup the resources for now

[github-actions]

This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.

[github-actions]

This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.