Commit 55d0a17
authored
Improve dictionary struct encoder in Go (#156)
Related to #153
### Summary of changes
- Structs can be now marked frozen, i.e. unchangeable. Once frozen they can be safely referenced by pointer from multiple places (including dictionaries) without cloning.
- We now have Set methods on fields that are dictionary structs. The setter works fast if the value being set is frozen - instead of full copy it just copies the pointer.
- CopyFrom does a similar optimization. Any encounter of a frozen struct just copies the pointer.
- Structs which are encoded remember (cache) their refNum. Next attempt to encode the same struct no longer requires slow dictionary lookup by value.
- OTLP to STEF converters use Freeze() when creating data, which improves encoding performance.
### Performance impact
Performance improvement of encoding of dictionary structs drammatic - about *10x faster* and *5 times less memory*.
Here is Profile serialization benchmarks:
```
pkg: github.com/splunk/stef/examples/profile
cpu: Apple M2 Pro
│ bench_base.txt │ bench_current.txt │
│ sec/op │ sec/op vs base │
Serialization/file=deser_stef.prof/format=stef-10 2901.2µ ± 4% 471.7µ ± 1% -83.74% (p=0.002 n=6)
Serialization/file=otelcol_otlp.prof/format=stef-10 660.41µ ± 2% 62.30µ ± 1% -90.57% (p=0.002 n=6)
Serialization/file=wsstreamasync.prof/format=stef-10 7253.3µ ± 3% 672.3µ ± 1% -90.73% (p=0.002 n=6)
geomean 2.404m 270.3µ -88.76%
│ bench_base.txt │ bench_current.txt │
│ sec/sample │ sec/sample vs base │
Serialization/file=deser_stef.prof/format=stef-10 2637.5n ± 4% 428.9n ± 1% -83.74% (p=0.002 n=6)
Serialization/file=otelcol_otlp.prof/format=stef-10 7591.0n ± 2% 716.0n ± 1% -90.57% (p=0.002 n=6)
Serialization/file=wsstreamasync.prof/format=stef-10 6523.0n ± 3% 604.6n ± 1% -90.73% (p=0.002 n=6)
geomean 5.074µ 570.5n -88.76%
│ bench_base.txt │ bench_current.txt │
│ B/op │ B/op vs base │
Serialization/file=deser_stef.prof/format=stef-10 1961.0Ki ± 0% 448.9Ki ± 0% -77.11% (p=0.002 n=6)
Serialization/file=otelcol_otlp.prof/format=stef-10 624.63Ki ± 0% 73.44Ki ± 0% -88.24% (p=0.002 n=6)
Serialization/file=wsstreamasync.prof/format=stef-10 3288.5Ki ± 0% 716.1Ki ± 0% -78.22% (p=0.002 n=6)
geomean 1.554Mi 286.9Ki -81.97%
│ bench_base.txt │ bench_current.txt │
│ allocs/op │ allocs/op vs base │
Serialization/file=deser_stef.prof/format=stef-10 12007.0 ± 0% 989.0 ± 0% -91.76% (p=0.002 n=6)
Serialization/file=otelcol_otlp.prof/format=stef-10 2353.0 ± 0% 330.0 ± 0% -85.98% (p=0.002 n=6)
Serialization/file=wsstreamasync.prof/format=stef-10 27628.0 ± 0% 931.0 ± 0% -96.63% (p=0.002 n=6)
geomean 9.207k 672.3 -92.70%
```
It comes at a cost of a percent slower deserialization due to additional bookkeeping, which is an acceptable tradeoff since deserialization is already very fast:
```
pkg: github.com/splunk/stef/examples/profile
cpu: Apple M2 Pro
│ bench_base.txt │ bench_current.txt │
│ sec/op │ sec/op vs base │
Deserialization/file=deser_stef.prof/format=stef-10 486.8µ ± 1% 519.4µ ± 1% +6.70% (p=0.000 n=9)
Deserialization/file=otelcol_otlp.prof/format=stef-10 113.5µ ± 2% 115.2µ ± 1% ~ (p=0.063 n=9)
Deserialization/file=wsstreamasync.prof/format=stef-10 933.2µ ± 3% 958.4µ ± 2% +2.70% (p=0.000 n=9)
geomean 372.2µ 385.6µ +3.60%
│ bench_base.txt │ bench_current.txt │
│ sec/sample │ sec/sample vs base │
Deserialization/file=deser_stef.prof/format=stef-10 442.6n ± 1% 472.2n ± 1% +6.69% (p=0.000 n=9)
Deserialization/file=otelcol_otlp.prof/format=stef-10 1.305µ ± 2% 1.324µ ± 1% ~ (p=0.054 n=9)
Deserialization/file=wsstreamasync.prof/format=stef-10 839.2n ± 3% 861.8n ± 2% +2.69% (p=0.000 n=9)
geomean 785.5n 813.7n +3.59%
│ bench_base.txt │ bench_current.txt │
│ B/op │ B/op vs base │
Deserialization/file=deser_stef.prof/format=stef-10 1.163Mi ± 0% 1.041Mi ± 0% -10.51% (p=0.000 n=9)
Deserialization/file=otelcol_otlp.prof/format=stef-10 390.6Ki ± 0% 360.6Ki ± 0% -7.67% (p=0.000 n=9)
Deserialization/file=wsstreamasync.prof/format=stef-10 1.590Mi ± 0% 1.507Mi ± 0% -5.23% (p=0.000 n=9)
geomean 911.6Ki 840.3Ki -7.83%
│ bench_base.txt │ bench_current.txt │
│ allocs/op │ allocs/op vs base │
Deserialization/file=deser_stef.prof/format=stef-10 1.673k ± 0% 1.638k ± 0% -2.09% (p=0.000 n=9)
Deserialization/file=otelcol_otlp.prof/format=stef-10 627.0 ± 0% 618.0 ± 0% -1.44% (p=0.000 n=9)
Deserialization/file=wsstreamasync.prof/format=stef-10 1.523k ± 0% 1.492k ± 0% -2.04% (p=0.000 n=9)
geomean 1.169k 1.147k -1.85%
```
The slowdown is less pronounced for Otel schema:
```
pkg: github.com/splunk/stef/benchmarks
cpu: Apple M2 Pro
│ bench_base.txt │ bench_current.txt │
│ sec/op │ sec/op vs base │
DeserializeNative/STEF/deser-10 1.552m ± 0% 1.612m ± 0% +3.89% (p=0.000 n=30)
│ bench_base.txt │ bench_current.txt │
│ sec/point │ sec/point vs base │
DeserializeNative/STEF/deser-10 23.21n ± 0% 24.11n ± 0% +3.90% (p=0.000 n=30)
│ bench_base.txt │ bench_current.txt │
│ B/op │ B/op vs base │
DeserializeNative/STEF/deser-10 911.5Ki ± 0% 925.4Ki ± 0% +1.53% (p=0.000 n=30)
│ bench_base.txt │ bench_current.txt │
│ allocs/op │ allocs/op vs base │
DeserializeNative/STEF/deser-10 669.0 ± 0% 465.0 ± 0% -30.49% (p=0.000 n=30)
```
Converting from pdata to STEF also benefits significantly from the faster encoding:
```
pkg: github.com/splunk/stef/benchmarks
cpu: Apple M2 Pro
│ bench_base.txt │ bench_current.txt │
│ sec/op │ sec/op vs base │
SerializeFromPdata/STEF/serialize-10 92.83m ± 0% 64.01m ± 0% -31.05% (p=0.000 n=15)
│ bench_base.txt │ bench_current.txt │
│ sec/point │ sec/point vs base │
SerializeFromPdata/STEF/serialize-10 1388.0n ± 0% 957.5n ± 0% -31.02% (p=0.000 n=15)
│ bench_base.txt │ bench_current.txt │
│ B/op │ B/op vs base │
SerializeFromPdata/STEF/serialize-10 169.20Mi ± 0% 78.28Mi ± 0% -53.74% (p=0.000 n=15)
│ bench_base.txt │ bench_current.txt │
│ allocs/op │ allocs/op vs base │
SerializeFromPdata/STEF/serialize-10 256.4k ± 0% 134.8k ± 0% -47.43% (p=0.000 n=15)
```1 parent fb37d45 commit 55d0a17
File tree
72 files changed
+5074
-3587
lines changed- benchmarks
- examples
- jsonl/internal/jsonstef
- profile
- internal/profile
- go
- otel
- oteltef
- pdata/metrics
- sortedbymetric
- sortedbyresource
- stefgen
- generator
- templates/go
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
72 files changed
+5074
-3587
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
61 | 61 | | |
62 | 62 | | |
63 | 63 | | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
64 | 68 | | |
65 | 69 | | |
66 | 70 | | |
| |||
87 | 91 | | |
88 | 92 | | |
89 | 93 | | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
90 | 98 | | |
91 | 99 | | |
92 | 100 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
24 | 28 | | |
25 | 29 | | |
26 | 30 | | |
| |||
124 | 128 | | |
125 | 129 | | |
126 | 130 | | |
127 | | - | |
| 131 | + | |
128 | 132 | | |
129 | 133 | | |
130 | 134 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
207 | 207 | | |
208 | 208 | | |
209 | 209 | | |
210 | | - | |
| 210 | + | |
211 | 211 | | |
212 | 212 | | |
213 | 213 | | |
214 | | - | |
| 214 | + | |
215 | 215 | | |
216 | 216 | | |
217 | 217 | | |
218 | | - | |
| 218 | + | |
219 | 219 | | |
220 | 220 | | |
221 | 221 | | |
| |||
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
0 commit comments