Skip to content

snapshot: allow DBI dumps >2GB #113

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wojas merged 2 commits into from
Jan 7, 2026
Merged

snapshot: allow DBI dumps >2GB #113

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
db4ce19
snapshot: allow DBI dumps >2GB
wojas Jan 7, 2026
5bcd945
snapshot: use datasize.GB
wojas Jan 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,11 @@ require (
golang.org/x/term v0.37.0 // indirect
golang.org/x/text v0.31.0 // indirect
google.golang.org/genproto v0.0.0-20211208223120-3a66f561d7aa // indirect
google.golang.org/protobuf v1.36.10 // indirect
google.golang.org/protobuf v1.36.11 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
pluginrpc.com/pluginrpc v0.5.0 // indirect
)

// See https://github.com/PowerDNS/lightningstream/issues/112
// Can be removed once https://github.com/CrowdStrike/csproto/pull/208 is merged and released
replace github.com/CrowdStrike/csproto => github.com/wojas/csproto v0.0.0-20260107092112-0e013c7984a2
8 changes: 4 additions & 4 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,6 @@ dario.cat/mergo v1.0.1/go.mod h1:uNxQE+84aUszobStD9th8a29P2fMDhsBdgRYvZOxGmk=
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c h1:udKWzYgxTojEKWjV8V+WSxDXJ4NFATAsZjh8iIbsQIg=
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/CrowdStrike/csproto v0.35.0 h1:9vEV5PwDHXqinbobsI1Doy6kF5uHJyo4znYqXaUKQNs=
github.com/CrowdStrike/csproto v0.35.0/go.mod h1:KVvLr7dlTgOmX5oT4cw9mZ0rhMIBG2LHb8yt9DMg7ic=
github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY=
github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU=
github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
Expand Down Expand Up @@ -322,6 +320,8 @@ github.com/tklauser/numcpus v0.6.1 h1:ng9scYS7az0Bk4OZLvrNXNSAO2Pxr1XXRAPyjhIx+F
github.com/tklauser/numcpus v0.6.1/go.mod h1:1XfjsgE2zo8GVw7POkMbHENHzVg3GzmoZ9fESEdAacY=
github.com/vbatts/tar-split v0.12.1 h1:CqKoORW7BUWBe7UL/iqTVvkTBOF8UvOMKOIZykxnnbo=
github.com/vbatts/tar-split v0.12.1/go.mod h1:eF6B6i6ftWQcDqEn3/iGFRFRo8cBIMSJVOpnNdfTMFA=
github.com/wojas/csproto v0.0.0-20260107092112-0e013c7984a2 h1:T9S4tRljv07jn5KNFyKhecgU6rbsq0OTxxar8ATxP/U=
github.com/wojas/csproto v0.0.0-20260107092112-0e013c7984a2/go.mod h1:s4KQGP1RDkm7AyniRi6BUAWySd8RtXZmk1ZBm/CF1F8=
github.com/wojas/go-healthz v0.2.0 h1:Pm+V2mCkEMvLoppiOW6zV/VpnbOfnwIA7F8Cd+ucmyg=
github.com/wojas/go-healthz v0.2.0/go.mod h1:uAGwtiPYhG63AgJoWM+EFzSfwgW3rj58ZalNogNFRRk=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
Expand Down Expand Up @@ -473,8 +473,8 @@ google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlba
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
google.golang.org/protobuf v1.36.10 h1:AYd7cD/uASjIL6Q9LiTjz8JLcrh/88q5UObnmY3aOOE=
google.golang.org/protobuf v1.36.10/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE=
google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
Expand Down
8 changes: 8 additions & 0 deletions snapshot/snapshot.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,13 @@ const (
FieldSnapshotCompatVersion = 4
)

// MaxFieldLength overrides the default 2GB limit on variable length protobuf
// fields to allow larger dumps with our current schema.
const (
GB = 1024 * 1024 * 1024
MaxFieldLength = 100 * GB
)

// Snapshot is the root object in a snapshot protobuf
type Snapshot struct {
FormatVersion uint32 // version of this snapshot format
Expand All @@ -25,6 +32,7 @@ type Snapshot struct {
func (s *Snapshot) Unmarshal(data []byte) error {
d := csproto.NewDecoder(data)
d.SetMode(csproto.DecoderModeFast)
d.SetMaxFieldLength(MaxFieldLength) // allow DBI dumps larger than 2GB
for d.More() {
tag, wireType, err := d.DecodeTag()
if err != nil {
Expand Down