Commit fcbed38
LogRecord and SpanByte changes (#1186)
* Refactors prior to storage-v2:
- Move CountdownWrapper to its own file
- Rename Lockable* to Transactional*
- Rename Transient* to Ephemeral*
* WIP; first cut at LogRecord etc. implementations
* WIP: more LogRecord
* More *LogRecord, and DiskRecord
* WIP: Convert allocators to new LogRecord and DiskRecord
* WIP: incorporate LogRecord into ISessionFunctions
* More logRecord WIP
* More WIP on MainSessionFunctions RMWMethods (and PrivateMethods) with LogRecord
* WIP LogRecord (MainStore RMWMethods)
* WIP: Consolidate to a single LogRecord; convert InternalRMW
* TKey removal
* A few more TKey removals
* WIP more logRecord
* Remove pendingConterxt.recordInfo as it is no longer in RecordMetadata
Other work to get CreateNewRecordRMW clean
* WIP; Internal(RUMD) converted.
* added PendingContext implementation of ISourceLogRecord
* Finish ContinuePending
* Iterator changes - consolidate to RecordScanIterator
* Mostly IO work
* WIP: Reintroduce TValue through ObjectAllocator, LogRecord, etc.
* WIP Convert Garnet to LogRecord and ObjectAllocator (MainStoreFunctions first pass done)
* WIP converting Garnet to non-ref SpanByte-only keys
* More WIP on Garnet migration to LogRecord
* WIP Tsavorite LogRecord odds and ends
* Rename TsavoriteLog to TsavoriteAof
* More processing-layer LogRecord etc. fixes
* fix iterator as ISourceLogRecord
* More Expiration etc. cleanup in processing layer
* WIP BlockAllocate work
* FInalize removal of SpanByte.Metadata
* More param fixes in processing layer
* SpanField and OptionalFieldsShift structs
* Propagate some SpanField etc. changes
* First pass at LogRecord-based Revivification
* Finish OverflowAllocator
* OK now I finished this iteration of OverflowAllocator
* improve callers' use of OverflowAllocator
* More LogRecord-related cleanup
* // TODOMigrate: temporarily clear errors related to Expiration in Key Migration
* Fixes for first test runs
* More fixes for CopyRecord and setting of Expiration etc
* Cleanup OversizeAllocator.BlockHeader and usage
* SpanField and LogRecord documentation and cleanup; BlockHeader GetUserSize
* More improvements to overflow allocator and usage
* More fixes for tests
* Remove unneeded IAllocator(Callbacks) methods (superseded by LogRecord)
* more overflow allocator fixes
* sizeInfo on ISessionFunctions and TrySetValueSpanLength improvements
* Remove SpanByte serialized usage (but leave an assert on .Serialized in case I missed some). Also fix CopyUpdater copies from old record and some more OverflowAllocator tweaks
* Improved FreeList handling and ObjectIdMap growth/FreeList via SimpleConcurrentStack and TailAndLatch
* Make sure DiskLogRecord Value and Record lengths are long
* Remove ISerializer (not used and not planned to be)
* TIghten up SpanByte !IsSerialized assert and fix SETEX "expiration set mismatch"
* Some TODO cleanup
* Begin converting Tsavorite tests and benchmarks to LogRecord
* Tsavorite RevivificationTests and ObjectTests
* More ObjectIdMap and OverflowAllocator work and tests
* More overflowAllocator work: FixedSizePages and other fixes for YCSB, add tests and fixes for OversizePages
* Clean up Locking.md (had some referenced to obsolete locking modes); start of LogRecord-related docs
* support for "inline struct" as SpanByte for Object value optimization
* finish "inline struct" testing; update processing layer to Tsavorite changes; update LogRecord docs
* Convert RecordFieldInfo Key/Value TotalSize to DataSize - caller should not need to know about the length prefix (especially bc it changes between Inline, Overflow, ObjectId)
* Fixes from merging main
* More testing and fixes
* More merge fixes
* fix more tests
* minor comment cleanup
* some doc-related updates, mostly comments
* Add DisposeReason.Deleted, .Elided
Add YCSB --di (Delete and reInsert) option
* WIP Span<byte>
* WIP Span<byte>
* WIP Span<byte>
* WIP Span<byte>
* WIP Span<byte>
* WIP Span<byte>
* WIP Span<byte>
* WIP Span<byte> (Tsavorite builds, with a couple hacks)
* WIP Span<byte>: YCSB fixes
* WIP Span<byte>: more fixes found during YCSB
* Switching CTT, CTRC, TempKv insertions to TSourceLogRecord direct copies (eliminating need for ISessionFunctions calls for those operations)
* WIP Span<byte>: more cleanup in Compaction (no TSessionFunctions needed), other fixes related to prior changes
* WIP Span<byte>: LogRecord revisions in functions and prep Migrate/Repl for change to LogRecord based
* ISessionFunctions revisions
- Rename SingleWriter() => InitialWriter(), ConcurrentWriter() => InPlaceWriter()
- Rename SinglecDeleter() => InitialDeleter(), ConcurrentDeleter() => InPlaceDeleter()
- Consolidate SingleReader() and ConcurrentReader() to just Reader()
* Remove TInput, TOutput, TContext from Compact and PrepareIOForConditionalOperation chain
* Remove WriteReason as it is no longer needed with Compaction, CopyToTail, and CopyToReadCache now being fully internal operations.
* Missed a few WriteReason
* WIP Span<byte>: Cleanup DiskLogRecord deserialization and remove some no-longer-needed allocator functions that are now done by DLR
* Varebyte in DiskLogRecord
* More LogRecord testing
* FInish DiskLogRecord serialization tests
* Rename SpanField to LogField
* Various cleanup, including PrepareForRevivification
* WIP on converting Replication and Migration to LogRecord
* WIP on Repl (done-ish) and Migrate (mostly done except ISF) conversion to DiskLogRecord
* Split IHO.Size into .MemorySize, DiskSize
Almost done with Migrate changeover to LogRecord
* Finish LogRecordization of Migrate
* First fixes for Migration testing
* More Migrate testing; use SBAM.MemorySpan
* SImple Migrate tests now run
* Simple diskless replica runs
* Fixes for in-memory tests
* Add UpsertValueSelector
* Remove PendingContext.IsAsync (a remnant of earlier, since-removed async APIs) from Tsavorite
* Cleanup Read initialization of PendingContext
* Fix record alignment in YCSB test
* tune HashBytes
* Fix initial inserting in YCSB
* Update LogRecord.md and some cleanup
* Restore TsavoriteAof naming back to TsavoriteLog
* Add changes for AddressType in logicalAddresses and enable minimal IO for SpanByteAllocator. The AddressType change is a breaking on-disk format change: it shuffles bits around in RecordInfo to add an additional bit adjacent to the old ReadCache bit to mark an address as:
- 00: Reserved
- 11: ReadCache
- 10: InMemory portion of the main log
- 01: On-Disk
* Fix a few more cursor tests (int => long)
* Fixes for review comments (except for "ref RecordSizeInfo")
* TSourceLogRecord and RecordSizeInfo parameters are readonly so make them "in" rather than "ref"
* Fix some missing ref => in for TSourceLogRecord
* Merge fixes; MigrateKeysWithObjects test still fails
* Fix Object store migration typo
* format
* ObjectAllocator serialization - not yet complete
* Garnet Storage v2 (#1384)
* wip
* wip
* wip
* Added unified store session
* Correcting generic typing
* Added MEMORY USAGE + TYPE to unified ops
* Added TTL, EXPIRETIME and EXISTS to unified ops
* implemented DEL in unified ops
* wip - expire & persist (broken)
* wip - adding expire to unified ops
* wip - expire
* add cref to server-side replication inter-node commands
* fix server-side BeginRecoverReplica
* wip
* Fix transaction key locking
* format
* Some test fixes
* Fixing tests
* reverting a couple of unnecessary changes
* Eliminating more multi-context methods from API
* Removed some unnecessary stuff
* Some more cleanup to TransactionManager
* merge tedhar/storage-v2 (ObjectAllocator serialization updates)
* Updating memory usage values
* format
* Handling wrong type ops
* Revert "Updating memory usage values"
This reverts commit 88ba307.
* fix no-object-log case
* Fixes for Tsavorite UTs
* Fixes for Tsavorite UTs (mostly ReadCache, TsavoriteLog, Compaction)
* Tsavorite Iterator work and UT re-enabling
* Fixes to Object iteration, LogRecord.ToString()
* Add RecordMetadata.ETag
* Readding --no-obj config
* fix
* test fix
* Prep for Recovery
* wip
* wip
* fix
* More fixes for UT (mostly Recovery, Migration)
* Moving DELIFEXPIM to unified store
* ObjectLogTail in Recovery, and more UT fixes (Migration record serialization, ReadCache size and tailAddress verification calculations, etc.)
* More Tsavorite recovery tests
* Removing unnecessary isObject flag from record serialization
---------
Co-authored-by: Vasileios Zois <[email protected]>
Co-authored-by: TedHartMS <[email protected]>
* Removing unnecessary output flag
* Fixes for StateMachineTests (Recovery)
* Fixes, mostly for Replication to include object log for UnifiedStore; correctly initialized SegmentSize for ObjectLogDevice; other fixes to UTs to work with UnifiedStore
* fixes
* Fixes for RESP UTs for Unified; clear logRecord on deserialization error
* Remove SerializedSize(IsExact) from IHeapObject; various UT fixes
* All Cluster.test now passes
* small fixes
* Fix TailAddress from Rescovery not being PageHeader-aligned
* Move RENAME to Unified Store and modify for LogRecord
* Fixing rename with etag + removing remaining unnecessary GarnetObjectTypes
* Add ITsavoriteRecordObserver and fix Readcache size tracking
* Convert LogRecord to use RecordLength
* More work on LogRecord.RecordLength
* Finish LogRecord.RecordLength
* Remove ObjectDb* from Sample*Txn
* MultiLevelPageArray stress fixes; improved LogRecord and RecordDataHeader ToString()
* Fix GetObjectLogSegmentSize()
* update docs
* Fix a couple comments
* Trim ObjectIdMap arrays on ClearPage
* wip
* wip
* wip
* wip
* wip
* wip
* wip
* wip
* formatting tsavo
* Fixing website build error
* some renames
* small rename
* some small changes
* Version update
* Address review comments; implement OA Truncate; add --value-overflow-threshold configuration option
* ObjectAllocator testing
* Add large (multi-GB) object tests (and fix associated issues, mostly object-log segments); PendingContext is no longer itself a TSourceLogRecord;
* readcache fixes
* Move KVSettings from GarnetServer to the individual Database, so we can dispose the log devices.
Remove KVSettings from CachedSizeTracker as it was unused
Fix some Tsavorite LogRecord-lifetime issues on scan and conditional insert/push
* WIP to fix replica for object log segments
* Fixes for ObjectLog recovery
* Some additional ObjectRecovery test conversion, not ready yet
* Convert and re-enable more Tsavorite ObjectRecovery tests.
Fix CacheSerializedData to not overwrite if CopyUpdater already set the object
Fix foldover checkpoint storage of object log tail
* format
* A couple fixes for DiskLogRecord serialization
* Fix SimpleIncrementByFloatWithOutOfRangeFloat exception verification to use Throws.TypeOf<>().Or.TypeOf<>())
Make SpanByteRcMultiThreadTest [Explicit] until I figure out the request_key issue
* Address review comments
Fix merge error that resulted in Cluster timeouts
Minor cleanup
* format
* Comment out [Repeat]
---------
Co-authored-by: Tal Zaccai <[email protected]>
Co-authored-by: Vasileios Zois <[email protected]>1 parent 935cf3f commit fcbed38
File tree
588 files changed
+31149
-26764
lines changed- benchmark
- BDN.benchmark/Custom
- Resp.benchmark
- libs
- client
- ClientSession
- cluster
- Server
- Migration
- Replication
- PrimaryOps
- DisklessReplication
- ReplicaOps
- Session
- SlotVerification
- common
- Metrics
- Networking
- host
- Configuration
- server
- AOF
- ReplayCoordinator
- API
- ArgSlice
- Cluster
- Custom
- Databases
- Lua
- Metrics
- Info
- Slowlog
- Objects
- Hash
- ItemBroker
- List
- Set
- SortedSetGeo
- SortedSet
- Types
- Providers
- PubSub
- Resp
- Bitmap
- HyperLogLog
- Objects
- Parser
- Servers
- Sessions
- Storage
- Functions
- MainStore
- ObjectStore
- UnifiedStore
- Session
- Common
- MainStore
- ObjectStore
- UnifiedStore
- SizeTracker
- Transaction
- storage/Tsavorite/cs
- benchmark
- BDN-Tsavorite.Benchmark
- YCSB.benchmark
- src
- core
- Allocator
- ObjectSerialization
- Async
- ClientSession
- Compaction
- Device
- Epochs
- Index
- CheckpointManagement
- Checkpointing
- Common
- Interfaces
- Recovery
- StoreFunctions
- Tsavorite
- Implementation
- Locking
- Revivification
- TsavoriteLog
- Utilities
- VarLen
- devices/AzureStorageDevice
- test
- main/GarnetServer/Extensions
- metrics/HdrHistogram/Utilities
- modules
- GarnetJSON
- NoOpModule
- playground/ClusterStress
- samples/MetricsMonitor
- test
- Garnet.fuzz/Targets
- Garnet.test.cluster
- CustomProcs
- RedirectTests
- ReplicationTests
- Garnet.test
- Extensions
- website
- docs/dev/tsavorite
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
588 files changed
+31149
-26764
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
4 | | - | |
| 4 | + | |
5 | 5 | | |
6 | 6 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
25 | | - | |
26 | | - | |
27 | | - | |
28 | | - | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
29 | 29 | | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| |||
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
55 | | - | |
56 | | - | |
57 | | - | |
58 | | - | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
23 | | - | |
| 23 | + | |
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
38 | | - | |
| 38 | + | |
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
43 | | - | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | 43 | | |
52 | 44 | | |
53 | 45 | | |
| |||
113 | 105 | | |
114 | 106 | | |
115 | 107 | | |
116 | | - | |
| 108 | + | |
| 109 | + | |
117 | 110 | | |
118 | 111 | | |
119 | 112 | | |
| |||
188 | 181 | | |
189 | 182 | | |
190 | 183 | | |
191 | | - | |
192 | | - | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
193 | 187 | | |
194 | 188 | | |
195 | 189 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
102 | 102 | | |
103 | 103 | | |
104 | 104 | | |
105 | | - | |
106 | | - | |
107 | | - | |
108 | 105 | | |
109 | 106 | | |
110 | 107 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | | - | |
| 29 | + | |
| 30 | + | |
30 | 31 | | |
31 | 32 | | |
32 | 33 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
306 | 306 | | |
307 | 307 | | |
308 | 308 | | |
| 309 | + | |
309 | 310 | | |
310 | 311 | | |
311 | 312 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
209 | 209 | | |
210 | 210 | | |
211 | 211 | | |
212 | | - | |
| 212 | + | |
213 | 213 | | |
214 | 214 | | |
215 | 215 | | |
| |||
563 | 563 | | |
564 | 564 | | |
565 | 565 | | |
566 | | - | |
| 566 | + | |
567 | 567 | | |
568 | 568 | | |
569 | 569 | | |
| |||
607 | 607 | | |
608 | 608 | | |
609 | 609 | | |
610 | | - | |
611 | | - | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
| 614 | + | |
| 615 | + | |
612 | 616 | | |
613 | 617 | | |
614 | 618 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
209 | 209 | | |
210 | 210 | | |
211 | 211 | | |
212 | | - | |
| 212 | + | |
213 | 213 | | |
214 | 214 | | |
215 | 215 | | |
| |||
323 | 323 | | |
324 | 324 | | |
325 | 325 | | |
326 | | - | |
| 326 | + | |
327 | 327 | | |
328 | 328 | | |
329 | 329 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
132 | 132 | | |
133 | 133 | | |
134 | 134 | | |
135 | | - | |
| 135 | + | |
136 | 136 | | |
137 | 137 | | |
138 | 138 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
32 | | - | |
| 32 | + | |
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| |||
0 commit comments