Conversation
…unctions (MaterializeInc#31097) This PR changes the MySQL source to support ingesting the `bit` type as `uint8`. It also adds two new Postgres functions, `bit_count(bytea)` and `get_bit(bytea, int32)` to making working with byte strings easier. ### Motivation Progress towards: MaterializeInc/database-issues#8891 ### Checklist - [x] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [x] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [x] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [x] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](MaterializeInc/cloud#5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [x] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post. --------- Co-authored-by: Dennis Felsing <dennis@felsing.org>
Enabled widely by default with zero consequences so far.
Don't need an explicit %N anymore
The materialized binary generates massive debuginfo: 8GB with full debuginfo and 3.5GB with limited debuginfo. (This is not the compressed size of the debuginfo on disk, but the amount of memory required to load the debuginfo into memory so that backtraces can be symbolized.) We've historically shipped the materialized binary with full debuginfo, so that we get rich backtraces for any crashes users run into when running the emulator. Unfortunately, the full debuginfo is so large that it is itself the *cause* of OOMs [0], which is unacceptable. (When a console query encounters certain routine errors, like a connection failing to validate, the adapter attempts to log a backtrace, which requires loading the debuginfo.) Even the limited debuginfo size (3.5GB) is unacceptable for a Docker image that's meant to be run on developer laptops. So, this commit adjusts the materialized image to strip all debuginfo from the binary. If a user reports a crash with an unsymbolized backtrace, it's still *possible* (just painful) to manually symbolize the backtrace as long as they give us the exact version of Materialize they were running. We'll just need to manually run `addr2line` on each address reported in the backtrace. (We do irrevocably lose access to frames for inlined functions, but that seems tolerable.) One silver lining here is that the Docker image will get much smaller. It's currently about 1GB. I expect this change to shave off a huge chunk of that. [0]: https://materializeinc.slack.com/archives/C07FX1W1Y03/p1737414021061139
…alizeInc#31127) This refactors the impl of `Blob` for Azure in a way that should be faster. The `BlobClient` we use from the `azure_storage_blob` crate returns a `Stream` that when `await`-ed sends a ranged GET request for a chunk of a blob. This PR refactors our impl so we await each ranged request in a `tokio::task` which increases the concurrency at which we fetch chunks of a `Part`. It also refactors how we handle the case when the `content-length` header is missing, and adds metrics so we can track how often this occurs. ### Motivation Maybe progress against MaterializeInc/database-issues#8892 ### Checklist - [ ] This PR has adequate test coverage / QA involvement has been duly considered. ([trigger-ci for additional test/nightly runs](https://trigger-ci.dev.materialize.com/)) - [ ] This PR has an associated up-to-date [design doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md), is a design doc ([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)), or is sufficiently small to not require a design. <!-- Reference the design in the description. --> - [ ] If this PR evolves [an existing `$T ⇔ Proto$T` mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md) (possibly in a backwards-incompatible way), then it is tagged with a `T-proto` label. - [ ] If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label ([example](MaterializeInc/cloud#5021)). <!-- Ask in #team-cloud on Slack if you need help preparing the cloud PR. --> - [ ] If this PR includes major [user-facing behavior changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note), I have pinged the relevant PM to schedule a changelog post.
Previously, INSPECT SHARD would panic if it was given an ID that didn't belong to any item. This commit fixes the issue by returning an error instead. Fixes #MaterializeInc/database-issues/issues/8910
We built this for the streaming iterator, but it works well here too.
In particular, don't decode K/Vs when we have an override set, and make sure every K/V only gets decoded once.
To prevent stuff like MaterializeInc#31482
Our license currently prohibits the use of multiple clusters, which
prevents folks from using Materialize in our recommended configurations,
and to experience some of the cool features. Change the license to
instead constraint the total resources consumed by Materialize across
any number of clusters.
### Motivation
<!--
Which of the following best describes the motivation behind this PR?
* This PR fixes a recognized bug.
[Ensure issue is linked somewhere.]
* This PR adds a known-desirable feature.
[Ensure issue is linked somewhere.]
* This PR fixes a previously unreported bug.
[Describe the bug in detail, as if you were filing a bug report.]
* This PR adds a feature that has not yet been specified.
[Write a brief specification for the feature, including justification
for its inclusion in Materialize, as if you were writing the original
feature specification.]
* This PR refactors existing code.
[Describe what was wrong with the existing code, if it is not obvious.]
-->
### Tips for reviewer
<!--
Leave some tips for your reviewer, like:
* The diff is much smaller if viewed with whitespace hidden.
* [Some function/module/file] deserves extra attention.
* [Some function/module/file] is pure code movement and only needs a
skim.
Delete this section if no tips.
-->
### Checklist
- [ ] This PR has adequate test coverage / QA involvement has been duly
considered. ([trigger-ci for additional test/nightly
runs](https://trigger-ci.dev.materialize.com/))
- [ ] This PR has an associated up-to-date [design
doc](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/README.md),
is a design doc
([template](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/design/00000000_template.md)),
or is sufficiently small to not require a design.
<!-- Reference the design in the description. -->
- [ ] If this PR evolves [an existing `$T ⇔ Proto$T`
mapping](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/command-and-response-binary-encoding.md)
(possibly in a backwards-incompatible way), then it is tagged with a
`T-proto` label.
- [ ] If this PR will require changes to cloud orchestration or tests,
there is a companion cloud PR to account for those changes that is
tagged with the release-blocker label
([example](MaterializeInc/cloud#5021)).
<!-- Ask in #team-cloud on Slack if you need help preparing the cloud
PR. -->
- [ ] If this PR includes major [user-facing behavior
changes](https://github.com/MaterializeInc/materialize/blob/main/doc/developer/guide-changes.md#what-changes-require-a-release-note),
I have pinged the relevant PM to schedule a changelog post.
- Moves the code inside `datadriven_testcase()`, where there is already other similar code. - Places the code to after the normal (non-redacted) reparse check. It's better like this, because when the redacted check was before the normal check, it used to just complain about the redacted reparse even when the normal reparse would also fail, which was misleading me to think that there is something wrong specifically with the redaction.
Newlines are represented as a ⏎ character in the result specification.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
13 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Tips for reviewer
Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.