Are there performance implications to having many keyspaces? #236
-
|
For example, is having 1,000,000 keyspaces OK, or is it better to think on how to structure the data differently so that I use prefixes instead? For the type of workload I have in mind, I think either could work, but keyspaces feel like a bit of a better match. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 1 reply
-
|
Prefer common prefixes as modelling mechanism. Every keyspace has some files on disk, and its own write buffer. It becomes pretty impossible at that number of keyspaces to maintain the journal (unless you allowed it to be huge) and keeping write buffer size sane. So for high cardinality, and especially arbitrarily many "prefixes", using structured keys is better. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the response (and for fjall)! |
Beta Was this translation helpful? Give feedback.
Prefer common prefixes as modelling mechanism.
It's not uncommon to virtualize many table/collections in a single, flat keyspace(e.g. MyRocks or CockroachDB).
Every keyspace has some files on disk, and its own write buffer. It becomes pretty impossible at that number of keyspaces to maintain the journal (unless you allowed it to be huge) and keeping write buffer size sane.
So for high cardinality, and especially arbitrarily many "prefixes", using structured keys is better.