sql: add well-known protobuf types for CSR schema compilation #34630
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When creating a Protobuf source using CONFLUENT SCHEMA REGISTRY CONNECTION, Materialize previously failed to compile schemas that import well-known types like google/protobuf/timestamp.proto. This happened because the compile_proto function only added files fetched from the schema registry to the VirtualSourceTree, but well-known types are typically not registered in the schema registry (they are implicitly available to protoc). This commit adds a new protobuf module that embeds the standard Google protobuf well-known types: - any.proto - api.proto - duration.proto - empty.proto - field_mask.proto - source_context.proto - struct.proto - timestamp.proto - type.proto - wrappers.proto These types are now added to the VirtualSourceTree before compiling schemas fetched from the schema registry, allowing schemas that import well-known types to compile successfully. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Add a test that verifies Materialize can compile Protobuf schemas that import well-known types (google/protobuf/timestamp.proto, google/protobuf/duration.proto) even when those types are NOT registered in the Confluent Schema Registry. This tests the fix in commit ca7584c where we embed well-known types in the VirtualSourceTree before compiling CSR schemas. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
bosconi
requested review from
def-,
ggevay,
martykulma,
ohbadiah and
petrosagg
and removed request for
ggevay
- Reorder mod declarations alphabetically in pure.rs - Split long method chain in protobuf.rs 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Wrap URL in angle brackets to satisfy rustdoc::bare-urls lint. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Replaced by #34635
Context for my making this PR is here.
When creating a Protobuf source using CONFLUENT SCHEMA REGISTRY
CONNECTION, Materialize fails to compile schemas that import
well-known types like google/protobuf/timestamp.proto. This happens
because the compile_proto function only adds files fetched from the
schema registry to the VirtualSourceTree, but well-known types are
typically not registered in the schema registry (they are implicitly
available to protoc).
This change adds a new protobuf module that embeds the standard
Google protobuf well-known types:
These types are now added to the VirtualSourceTree before compiling
schemas fetched from the schema registry, allowing schemas that import
well-known types to compile successfully.
Motivation
https://github.com/MaterializeInc/database-issues/issues/10003
Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.