Pre-hash rows and add SmallHashSets #318

kazimuth · 2025-05-16T17:05:31Z

Description of Changes

Annotate rows with their hashes. This is halfway to the old solution of annotating code with their serialized forms; it speeds up insertion into BTreeIndexes while saving a byte array allocation per-row.
(Wait, why do we hash when inserting into BTreeIndexes, you ask? It's because we use a HashSet to store the multiple rows corresponding to a non-unique key.)

This saves work on the main thread repeatedly hashing values. Once we parallelize message processing, this work will be spread out over multiple threads too.

Also adds a data type SmallHashSet<T> for use in BTreeIndexes. This is a struct that can store at most one element without allocating. This gives dramatic allocation reduction on initial connection -- it seems that most rows in BTreeIndexes don't have any other rows with the same key.
Then I manually specialized this, since it's only used on one type -- PreHashedRow. using SmallHashSetOfPreHashedRow instead of SmallHashSet<PreHashedRow> allows us to inline some methods and pass around fewer pointers while hashing things, which is probably good.

API

This is an API breaking change to the SDK

Requires SpacetimeDB PRs

Testsuite

SpacetimeDB branch name: master

Testing

I will test:

blackholio
bitcraft

Some API updates

More API updates

…w updates and reducer events

API callback improvements

Add sslEnabled option to Connect function

* Fix compilation issue with master * Initial push for SATS (still WIP) * Small build fix * Small update * Updates from meeting with Tyler * Fixed issues with LayoutExplicit and added some missing functions * AlgebraicValue cleanup * Copied changes from BitCraftMini * Removed TypeDef --------- Co-authored-by: Clockwork Labs <[email protected]> Co-authored-by: Tyler Cloutier <[email protected]> Co-authored-by: John Detter <[email protected]>

* Upload script to help deploying to our partners * You have to specify hosts individually * Small fix * Another small fix - tested + working --------- Co-authored-by: John Detter <[email protected]>

* Serialize byte arrays as hex * Fix namespace * Small fix --------- Co-authored-by: John Detter <[email protected]>

* First pass * Committing meta file * Removed option type - unneeded * Implementing new some converter * Some converter updates * Tons of fixes here * Cleaned up the some/enum serialization implementation --------- Co-authored-by: John Detter <[email protected]>

Co-authored-by: John <[email protected]>

Co-authored-by: John Detter <[email protected]>

* Several updates to the way ClientCache + processing updates works * Client cache changes working * Removed log statements --------- Co-authored-by: John Detter <[email protected]>

* Copied changes from BitCraft * Applied state diff changes * State diffs complete * Compilation fix * Fix build issues --------- Co-authored-by: John Detter <[email protected]>

* Pulled SpacetimeUnitySDK from the BitCraft project * Removed unused logs * Removed primary key functionality * Fix compilation issues with older version of C# * Primary key client implementation * Another row update fix --------- Co-authored-by: John <[email protected]>

* Pulled SpacetimeUnitySDK from the BitCraft project * Removed unused logs * Removed primary key functionality * Fix compilation issues with older version of C# --------- Co-authored-by: John <[email protected]>

* Adding event parameter to table callbacks * Reverted a namespace change --------- Co-authored-by: Steve <[email protected]> Co-authored-by: John Detter <[email protected]>

Co-authored-by: Derek Brinkmann <[email protected]>

* Add onSubscriptionUpdate callback * Remove unnecesary onTransactionComplete --------- Co-authored-by: Derek Brinkmann <[email protected]>

…ed from callbacks (#20) Co-authored-by: Steve <[email protected]>

* Deserializing reducer args before table updates so they can be accessed from callbacks * Reducer arguments in table events * Removed redundant onSubscriptionUpdate --------- Co-authored-by: Steve <[email protected]> Co-authored-by: John Detter <[email protected]> Co-authored-by: John Detter <[email protected]>

Co-authored-by: Steve <[email protected]>

Co-authored-by: John Detter <[email protected]>

* Deserializing reducer args before table updates so they can be accessed from callbacks * Reducer arguments in table events * Adding OnBeforeDelete event to tables * Renaming ReducerCallInfo to ReducerEvent * Removed extra onSubscriptionUpdate --------- Co-authored-by: Steve <[email protected]> Co-authored-by: John Detter <[email protected]>

* Deserializing reducer args before table updates so they can be accessed from callbacks * Reducer arguments in table events * Adding OnBeforeDelete event to tables --------- Co-authored-by: Steve <[email protected]> Co-authored-by: John Detter <[email protected]>

kazimuth · 2025-05-16T19:17:19Z

This does mildly pessimize calls to Filter -- it adds a second allocation to them.
SmallHashSet is so good I'm tempted to just use it and get rid of PreHashedRow, not sure.

## Description of Changes Just bumping version numbers in preparation for a release. The Unity SDK and C# SDK had different version numbers. That seems confusing, so I bumped them both to 1.1.5. ## API No breaking changes ## Requires SpacetimeDB PRs None, afaik ## Testsuite SpacetimeDB branch name: master ## Testing None Co-authored-by: Zeke Foppa <[email protected]>

## Description of Changes Update DLLs to the 1.1.2 versions that have been pushed to NuGet. The SpacetimeDB tag that they were built from is `v1.1.2-csharp-release`. ## API - [ ] This is an API breaking change to the SDK *If the API is breaking, please state below what will break* ## Requires SpacetimeDB PRs None, because we're updating the DLLs to essentially include SpacetimeDB PRs. ## Testsuite SpacetimeDB branch name: master ## Testing CI I guess? Co-authored-by: Zeke Foppa <[email protected]>

## Description of Changes Bump versions to 1.2.0, as well as update the DLLs from SpacetimeDB (commit `fc8f7dcc8`). They don't have any changes, but the versions needed bumping. ## API This itself is not a breaking changes. ## Requires SpacetimeDB PRs clockworklabs/SpacetimeDB#2837 ## Testsuite SpacetimeDB branch name: release/v1.2.0 ## Testing CI only --------- Co-authored-by: Zeke Foppa <[email protected]>

## Description of Changes *Describe what has been changed, any new features or bug fixes* ## API - [ ] This is an API breaking change to the SDK *If the API is breaking, please state below what will break* ## Requires SpacetimeDB PRs *List any PRs here that are required for this SDK change to work* ## Testsuite *If you would like to run the your SDK changes in this PR against a specific SpacetimeDB branch, specify that here. This can be a branch name or a link to a PR.* SpacetimeDB branch name: master ## Testing *Write instructions for a test that you performed for this PR* - [ ] Describe a test for this PR that you have completed --------- Co-authored-by: Alessandro Asoni <[email protected]> Co-authored-by: James Gilles <[email protected]>

## Description of Changes `NetworkRequestTracker` previously was keeping all historical reducer request data, and searching through this every frame to get statistics. I've modified it to throw out much more data -- it's much faster now, but only updates every few seconds. ## API Not an API break, but deprecates an argument of one of NetworkRequestTracker's methods to no longer do anything. Adds new APIs. ## Requires SpacetimeDB PRs N/A ## Testsuite SpacetimeDB branch name: master ## Testing - [x] Tested Bitcraft. **Their F9 debug menu will require an update, since we now only keep one time window of request data, rather than being able to give information about multiple windows.** But it works. - [x] Blackholio CI

kazimuth · 2025-06-13T16:24:08Z

I've added pooling of HashSets within the SmallHashSet class.

…ed with ObjectEqualityComparer.

## Description of Changes Addresses clockworklabs/SpacetimeDBPrivate#1786 and #329. Marks messages as parsed from the background parsing thread, and then marks them as applied using a separate tracker once they are applied. ## API - [ ] This is an API breaking change to the SDK *If the API is breaking, please state below what will break* ## Requires SpacetimeDB PRs ## Testsuite SpacetimeDB branch name: master ## Testing - [x] Testing against Bitcraft

rekhoff

Assuming all testing goes well, and there are not any significant performance impact, the changes look good and are in-line with the stated goal of the pull request. I've added some minor notes to some comment lines, but that is all. I'm marking this approved.

rekhoff · 2025-06-18T17:51:03Z

src/SpacetimeDBClient.cs

@@ -226,7 +226,7 @@ internal struct ParsedDatabaseUpdate
            // Map: table handles -> (primary key -> IStructuralReadWrite).


This comment could use an update, as it still references IStructuralReadWrite.

rekhoff · 2025-06-18T18:27:51Z

src/SmallHashSet.cs

+}
+
+/// <summary>
+/// This is a silly object.


Though the struct is indeed silly, we may want to provide more context or remove this summary.

rekhoff · 2025-06-18T18:28:58Z

src/SmallHashSet.cs

+}
+
+/// <summary>
+/// This is a very silly object.


Removal of summary or adding additional context is also needed for this very silly struct.

kazimuth · 2025-06-19T15:30:48Z

I'm not happy with this yet. I'm not entirely convinced it improves performance; in my testing, allocating HashSets from the pool is taking 5% of time to apply an update. But that's under deep profiling, so I have no idea if it's true or not. We could really use better benchmarks.

## Description of Changes Updates the DLLs for clockworklabs/SpacetimeDB#2762, and bumps our version numbers as well. ## API - [ ] This is an API breaking change to the SDK Not a breaking change ## Requires SpacetimeDB PRs None afaik ## Testsuite SpacetimeDB branch name: master ## Testing *Write instructions for a test that you performed for this PR* - [x] Ran through Bitcraft tutorial - [x] Blackholio on CI --------- Co-authored-by: James Gilles <[email protected]> Co-authored-by: Zeke Foppa <[email protected]>

## Description of Changes These version bumps were supposed to happen in #332, but apparently I messed something up while I was switching branches. ## API - [ ] This is an API breaking change to the SDK No breaking changes ## Requires SpacetimeDB PRs None. ## Testsuite SpacetimeDB branch name: master ## Testing CI only Co-authored-by: Zeke Foppa <[email protected]>

## Description of Changes Missed in #333. We should update the 1.2.1 tag and the `release/latest` branch after this. ## API - [ ] This is an API breaking change to the SDK Nope ## Requires SpacetimeDB PRs None ## Testsuite SpacetimeDB branch name: master ## Testing CI only Co-authored-by: Zeke Foppa <[email protected]>

bfops · 2025-06-25T17:18:34Z

@kazimuth is doing some more testing on this and then we may merge + release it

bfops · 2025-08-08T17:52:33Z

Migrated to clockworklabs/SpacetimeDB#3146, as we are merging this repo into that one.

jdetter and others added 30 commits November 2, 2022 12:01

Merge pull request #1 from clockworklabs/jdetter/some-api-updates

583b612

Some API updates

Merge pull request #2 from clockworklabs/jdetter/some-api-updates2

a9f8bef

More API updates

Improved names

7832181

.gitignore change

b37b468

Several fixes to sending when the websocket is closed.

fe3a8ad

Give SDK users access to the token key

355b9e9

Changes to the Unity SDK to support callbacks for individual table ro…

842cb00

…w updates and reducer events

Merge pull request #3 from clockworklabs/Derek-Callback-Improvements

40b19b7

API callback improvements

Add sslEnabled option to Connect function

ab12cef

Merge pull request #4 from clockworklabs/derek-add-SSL-option

3364cd7

Add sslEnabled option to Connect function

SDK Upload Script (#7)

7fc5463

* Upload script to help deploying to our partners * You have to specify hosts individually * Small fix * Another small fix - tested + working --------- Co-authored-by: John Detter <[email protected]>

Json Byte Arrays Serialized as Hex (#8)

28897a3

* Serialize byte arrays as hex * Fix namespace * Small fix --------- Co-authored-by: John Detter <[email protected]>

C# Enum Generation (#9)

9286779

* First pass * Committing meta file * Removed option type - unneeded * Implementing new some converter * Some converter updates * Tons of fixes here * Cleaned up the some/enum serialization implementation --------- Co-authored-by: John Detter <[email protected]>

Fix compilation issue (#11)

ca5eefe

Co-authored-by: John <[email protected]>

Automatically find Unity on macOS (#12)

6151939

Co-authored-by: John Detter <[email protected]>

GetEntries now returns both AlgebraicValue and object as a tuple (#15)

cddda19

Co-authored-by: John Detter <[email protected]>

Use Network Thread for Processing DB Rows (#14)

9f9e549

* Several updates to the way ClientCache + processing updates works * Client cache changes working * Removed log statements --------- Co-authored-by: John Detter <[email protected]>

State Diffs (#16)

51086a2

* Copied changes from BitCraft * Applied state diff changes * State diffs complete * Compilation fix * Fix build issues --------- Co-authored-by: John Detter <[email protected]>

SpacetimeDB SDK Version 0.4.1 (#17)

aad6c4b

* Pulled SpacetimeUnitySDK from the BitCraft project * Removed unused logs * Removed primary key functionality * Fix compilation issues with older version of C# --------- Co-authored-by: John <[email protected]>

Adding event parameter to table callbacks (#19)

11801d4

* Adding event parameter to table callbacks * Reverted a namespace change --------- Co-authored-by: Steve <[email protected]> Co-authored-by: John Detter <[email protected]>

Fix compile error in sdk (#22)

bc6ed3e

Co-authored-by: Derek Brinkmann <[email protected]>

Add onSubscriptionUpdate callback (#21)

c15c54e

* Add onSubscriptionUpdate callback * Remove unnecesary onTransactionComplete --------- Co-authored-by: Derek Brinkmann <[email protected]>

Deserializing reducer args before table updates so they can be access…

fb72396

…ed from callbacks (#20) Co-authored-by: Steve <[email protected]>

Changed guid (#25)

30f4347

Co-authored-by: Steve <[email protected]>

Fixed API (#26)

cf771d3

Co-authored-by: John Detter <[email protected]>

kazimuth added 2 commits May 16, 2025 14:29

Allocate fewer hash sets

39f53af

Add test for SmallHashSet

b32c9ea

kazimuth changed the title ~~Pre-hash rows on the message pre-processing thread~~ Pre-hash rows and add SmallHashSets May 16, 2025

Save an allocation when calling Filter

1bad0b6

kazimuth and others added 9 commits May 16, 2025 15:28

Fix regression tests & make them more aggressive

2410e12

Merge branch 'master' into jgilles/pre_hashed_row

d5d2296

Specialize MultiDictionary to TValue : struct

c409695

Pool HashSets used by SmallHashSet

866011d

kazimuth added 4 commits June 13, 2025 12:33

Reduce pool size

b5dec1c

Hand-monomorphize SmallHashSet. This also fixes their being initializ…

ccf74e2

…ed with ObjectEqualityComparer.

Merge branch 'master' into jgilles/pre_hashed_row

a497966

rekhoff approved these changes Jun 18, 2025

View reviewed changes

bfops and others added 4 commits June 23, 2025 18:39

Merge branch 'master' into jgilles/pre_hashed_row

5b14731

bfops force-pushed the master branch from aecb141 to a916977 Compare August 5, 2025 21:13

bfops mentioned this pull request Aug 8, 2025

C#/Unity SDK - Pre-hash rows and add SmallHashSets clockworklabs/SpacetimeDB#3146

Draft

bfops closed this Aug 8, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Pre-hash rows and add SmallHashSets #318

Pre-hash rows and add SmallHashSets #318

Uh oh!

kazimuth commented May 16, 2025 •

edited

Loading

Uh oh!

kazimuth commented May 16, 2025

Uh oh!

kazimuth commented Jun 13, 2025

Uh oh!

rekhoff left a comment

Uh oh!

rekhoff Jun 18, 2025

Uh oh!

rekhoff Jun 18, 2025

Uh oh!

rekhoff Jun 18, 2025

Uh oh!

kazimuth commented Jun 19, 2025

Uh oh!

bfops commented Jun 25, 2025

Uh oh!

bfops commented Aug 8, 2025

Uh oh!

Uh oh!

		@@ -226,7 +226,7 @@ internal struct ParsedDatabaseUpdate
		// Map: table handles -> (primary key -> IStructuralReadWrite).

Pre-hash rows and add SmallHashSets #318

Pre-hash rows and add SmallHashSets #318

Uh oh!

Conversation

kazimuth commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of Changes

API

Requires SpacetimeDB PRs

Testsuite

Testing

Uh oh!

kazimuth commented May 16, 2025

Uh oh!

kazimuth commented Jun 13, 2025

Uh oh!

rekhoff left a comment

Choose a reason for hiding this comment

Uh oh!

rekhoff Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

rekhoff Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

rekhoff Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

kazimuth commented Jun 19, 2025

Uh oh!

bfops commented Jun 25, 2025

Uh oh!

bfops commented Aug 8, 2025

Uh oh!

Uh oh!

kazimuth commented May 16, 2025 •

edited

Loading