Skip to content

Commit 4522ad0

Browse files
NathanFlurryNathanFlurry
committed
init commit
0 parents  commit 4522ad0

37 files changed

+3789
-0
lines changed

‎.github/workflows/pkg-pr-new.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
on:
2+
pull_request:
3+
4+
jobs:
5+
publish:
6+
runs-on: ubuntu-latest
7+
steps:
8+
- uses: actions/checkout@v4
9+
- run: corepack enable
10+
- uses: actions/setup-node@v4
11+
- name: Install dependencies
12+
run: pnpm install
13+
working-directory: ./typescript
14+
- name: Build packages
15+
run: pnpm build
16+
working-directory: ./typescript
17+
- name: Publish preview packages
18+
run: pnpm dlx pkg-pr-new publish 'compiler' 'vbare' --packageManager pnpm
19+
working-directory: ./typescript

‎README.md‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
# Versioned Binary Application Record Encoding (VBARE)
2+
3+
Fearless schema migrations at theoretical maximum performance
4+
5+
VBARE is a tiny extension to BARE
6+
7+
## Preface: What is BARE?
8+
9+
- https://baremessages.org/
10+
- https://www.ietf.org/archive/id/draft-devault-bare-11.html
11+
12+
## Project goals
13+
14+
- fast -- self-contained binary encoding, akin to a tuple ->
15+
- simple -- can rewrite in under an hour
16+
- portable -- cross-language & well standardized
17+
18+
non-goals:
19+
20+
- data compactness -> that's what gzip is for
21+
22+
## Use cases
23+
24+
- Defining network protocols
25+
- Storing data at rest that needs to be able to be upgraded
26+
- Binary data in the database
27+
- File formats
28+
29+
## At a glance
30+
31+
- Every message has a version associated with it
32+
- either pre-negotiated (via something like an http request query parameter/handshake) or embedded int he message itself
33+
- Applications provide functions to upgrade between protocol versions
34+
- There is no migration in the schema itself, just copy and paste the schema to write the new one
35+
36+
## Migration philosophy
37+
38+
- declare discrete versions with predefined version indexes
39+
- manual migrations simplify the application logic by putting complex defaults in your app code
40+
- stop making big breaking v1 -> v2 changes, make much smaller changes with more flexibility
41+
- reshaping structures is important -- not just changing types and names
42+
43+
## Code examples
44+
45+
## Current users
46+
47+
- Rivet
48+
- Network protocol
49+
- All internal communication
50+
- All data stored at rest
51+
- RivetKit
52+
- Protocol for communicating with clients
53+
54+
## FAQ
55+
56+
### Why not include RPC?
57+
58+
- why the fuck does your protocol need to define an rpc schema
59+
- keep it simple, use a union
60+
61+
### Why is copying the entire schema for every version better than using decorators for gradual migrations
62+
63+
### Isn't copying the schema going to result in a lot of duplicate code?
64+
65+
- yes. after enough pain and suffering of running production APIS, this is what you will end up doing manually, but in a much more painful way.
66+
- having schema versions also makes it much easier to reason about how clients are connecting to your system/the state of an application. incremental migrations dno't let you consider other properties/structures.
67+
- this also lets you reshape your structures.
68+
69+
### Why copying instead of decorators for migrations?
70+
71+
- decorators are limited and get very complicated
72+
- it's unclear what version of the protocol a decorator takes effect -- this is helpful
73+
- generated sdks become more and more bloated with every change
74+
- you need a validation build step for your validators
75+
- things you can do with manual migrations
76+
77+
### Don't migration steps get repetitive?
78+
79+
- most of the time, structures will match exactly. most languages can provide a 1:1 migration.
80+
- the most eggrarious offendors will be deeply nested structures, but even that isn't terrible
81+
82+
## Comparison
83+
84+
- Protobuf (versioned: yes)
85+
- unbelievably poorly designed protocol
86+
- makes it your problem by making everything optional
87+
- even worse, makes properties have a default value (ie integers) which leads to subtle bugs with serious concequenses
88+
- tracking indexes in the file is ass
89+
- Cap'n'proto (versioned: yes)
90+
- simplicity
91+
- quality of output languages
92+
- cbor/messagepack/that mongodb one (versioned: self-describing)
93+
- requires encoding the entire key
94+
- Flatbuffers (versioned: yes)
95+
- still uses indexes like protobuf, unless you use structs
96+
- structs are ass
97+
- cdoegen is ass
98+
- https://crates.io/crates/bebop & https://crates.io/crates/borsh (versioned: TODO)
99+
- provides cross platform
100+
- TODO: more complicated than i'd like
101+
- bebop includes an extra ??? step
102+
- rust options like postcard/etc (versioned: no)
103+
- not cross platform
104+
105+
## Implementations
106+
107+
| Language | BARE | VBARE |
108+
| --- | --- | --- |
109+
| TypeScript | X | X |
110+
| Rust | X | X |
111+
112+
[Full list of BARE implementations](https://baremessages.org/)
113+

‎STORY.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
2+
## How we got here
3+
4+
- updating apis is incredibly error prone
5+
- we would frequently find ourselves catching breaking changes by:
6+
- adding request properties that were not optional
7+
- adding enum variants in responses that clients were not adept to handle
8+
- to solve this, we:
9+
- started frequently copying and pasting our entire api router
10+
- the latest version keeps the business logic
11+
- the previous version would include logic to mutate the schema to match the newest
12+
- this frequently involved making database queries to mutate data to match the new API
13+
- additional pain points:
14+
- we kept using protobuf because it was the only option with descent code generation that would let us mutate the schema
15+
- but we knew it was terrible in how it handled version migration
16+
- kept introducing edge cases with default values
17+
- how vbare fixes this:
18+
- builds infrastructure around the pattern that emerged: making micromigrations for many small changes
19+
- provides custom hooks to handle migrations between versions
20+

‎rust/.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
target/
2+
Cargo.lock
3+
**/*.rs.bk
4+

‎rust/Cargo.toml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
[workspace]
2+
members = ["bare-gen", "compiler", "vbare"]
3+
resolver = "2"
4+
5+
[workspace.package]
6+
version = "0.1.0"
7+
edition = "2021"
8+
authors = ["Your Name"]
9+
license = "MIT OR Apache-2.0"
10+
11+
[workspace.dependencies]
12+
anyhow = "1.0"
13+
heck = "0.5"
14+
indoc = "2.0"
15+
pest = "2.7"
16+
pest_derive = "2.7"
17+
prettyplease = "0.2"
18+
proc-macro2 = "1.0"
19+
quote = "1.0"
20+
serde = { version = "1.0", features = ["derive"] }
21+
serde_bare = "0.5"
22+
syn = "2.0"

‎rust/README.md‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# Rust Workspace
2+
3+
This workspace contains two Rust libraries:
4+
5+
## Libraries
6+
7+
### compiler
8+
A build script compiler for processing schema files. This library provides utilities to:
9+
- Process schema files in a directory
10+
- Generate Rust code from schemas
11+
- Create module declarations for generated code
12+
- Handle build script integration with proper cargo rerun-if-changed directives
13+
14+
**Usage in build.rs:**
15+
```rust
16+
use compiler::SchemaCompiler;
17+
use std::path::Path;
18+
19+
fn main() -> Result<(), Box<dyn std::error::Error>> {
20+
let schema_dir = Path::new("schemas");
21+
22+
// Use with a custom processor function
23+
SchemaCompiler::process_schemas_for_build_script(
24+
schema_dir,
25+
|path| {
26+
// Your schema processing logic here
27+
Ok(String::from("generated code"))
28+
}
29+
)?;
30+
31+
Ok(())
32+
}
33+
```
34+
35+
### vbare
36+
A versioned data serialization library that provides traits for:
37+
- Versioned data serialization/deserialization
38+
- Automatic version conversion between different schema versions
39+
- Embedded version encoding in payloads
40+
- Support for both borrowed and owned data types
41+
42+
**Features:**
43+
- `VersionedData<'a>` trait for borrowed data
44+
- `OwnedVersionedData` trait for owned data
45+
- Automatic version migration via converter functions
46+
- Embedded version support for self-describing payloads
47+
48+
## Building
49+
50+
```bash
51+
cargo build
52+
```
53+
54+
## Testing
55+
56+
```bash
57+
cargo test
58+
```
59+
60+
## License
61+
62+
MIT OR Apache-2.0

‎rust/bare-gen/Cargo.toml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
[package]
2+
name = "bare_gen"
3+
version.workspace = true
4+
authors.workspace = true
5+
license.workspace = true
6+
edition.workspace = true
7+
8+
[dependencies]
9+
heck.workspace = true
10+
pest_derive.workspace = true
11+
pest.workspace = true
12+
proc-macro2.workspace = true
13+
quote.workspace = true
14+
serde.workspace = true
15+
syn.workspace = true

‎rust/bare-gen/src/grammar.pest‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
//schema = { SOI ~ user_type+ ~ EOI }
2+
schema = { SOI ~ user_type+ ~ EOI }
3+
type_l = _{ "type" }
4+
user_type = { "type" ~ user_type_name ~ any_type }
5+
user_type_name = @{ ASCII_ALPHA_UPPER ~ (ASCII_ALPHANUMERIC | "_" | "-")* }
6+
unsigned_t = { "uint" | "u64" | "u32" | "u16" | "u8" }
7+
signed_t = { "int" | "i64" | "i32" | "i16" | "i8" }
8+
void_t = { "void" }
9+
str_t = { "str" }
10+
bool_t = { "bool" }
11+
float_t = { "f32" | "f64" }
12+
data_t = { "data" ~ (length)? }
13+
enum_t = { "enum" ~ "{" ~ enum_value+ ~ "}" }
14+
enum_value = { enum_value_name ~ ("=" ~ integer)? }
15+
enum_value_name = @{ ASCII_ALPHA_UPPER ~ (ASCII_ALPHANUMERIC | "_" | "-")* }
16+
list_t = { "list" ~ type_t ~ length? }
17+
type_t = _{ "<" ~ any_type ~ ">" }
18+
struct_t = { "struct" ~ "{" ~ struct_field+ ~ "}" }
19+
map_t = { "map" ~ type_t ~ type_t }
20+
union_t = { "union" ~ "{" ~ any_type ~ ("|" ~ any_type)* ~ "}" }
21+
optional_t = { "optional" ~ type_t }
22+
struct_field = { struct_field_name ~ ":" ~ any_type }
23+
struct_field_name = { (ASCII_ALPHANUMERIC | "_" | "-")+ }
24+
length = _{ "[" ~ integer ~ "]" }
25+
integer = !{ ASCII_NONZERO_DIGIT ~ ASCII_DIGIT* }
26+
primative_type = _{
27+
unsigned_t
28+
| signed_t
29+
| bool_t
30+
| float_t
31+
| data_t
32+
| str_t
33+
| void_t
34+
}
35+
any_type = _{
36+
user_type_name
37+
| list_t
38+
| struct_t
39+
| enum_t
40+
| map_t
41+
| union_t
42+
| optional_t
43+
| primative_type
44+
}
45+
WHITESPACE = _{ " " | "\n" | "\t" | NEWLINE }
46+
COMMENT = _{ "#" ~ (!NEWLINE ~ ANY)* ~ NEWLINE }

0 commit comments

Comments
 (0)