feat(tiering): Serialize hashes #6015

dranikpg · 2025-11-05T18:17:19Z

Ok, so this changed a lot. Add active path for serializing hashes:

Generic EstimateSerializedSize
Generic Serialize
Propagating ExternalRep in CompactObj, including cooling
Handle upload

Signed-off-by: Vladislav Oleshko <[email protected]>

romange · 2025-11-18T15:13:55Z

src/core/compact_object.h


+  // Prequisite: IsCool() is true.
+  // Keeps cool record only as external value and discard in-memory part.
+  void KeepExternal(size_t offset, size_t sz);


KeepExternal is not clear imho.

Freeze - a more creative option, and maybe DropCool is more direct explaining what it does?

Ok.. The names just become more and more cryptic 😃

dranikpg · 2025-11-20T08:25:43Z

src/server/tiered_storage.cc

+  auto estimated = EstimateSerializedSize(*value);
+  if (OccupiesWholePages(estimated->first)) {


This was a bug, we have to compute it not by value->Size(), but by the estimated serialization size. This might become an issue with large maps

Signed-off-by: Vladislav Oleshko <[email protected]>

dranikpg · 2025-11-20T11:06:46Z

src/server/tiered_storage.cc

+      if (pv.IsInline())
+        return {};
+      return std::make_pair(pv.GetRawString().view().size(), CompactObj::ExternalRep::STRING);


Because we can't call GetRawString unconditionally. I'll try to make it nicer in the future, for now it just becomes too much for a single PR

romange

It is a big PR, I wish it could be split into multiple parts.

romange · 2025-11-20T15:50:57Z

src/server/tiering/serialized_map_test.cc

-                                                             {"third key", "third value"},
-                                                             {"fourth key", "fourth value"},
-                                                             {"fifth key", "fifth value"}};
+  const vector<std::pair<string, string>> kBase = {{"first key", "first value"},


curious, what's the reason for string?

Turing listpack into vector<p<string_view, string_view>> is even more code, so I just changed the type to string. In the future, the serialized map constructor needs to be updated eitehr way to be copy-less, but its too much for this PR. Its all temporary

romange · 2025-11-20T15:52:59Z

src/server/tiered_storage.cc

+      if (pv.Encoding() == kEncodingListPack) {
+        auto* lp = static_cast<uint8_t*>(pv.RObjPtr());
+        size_t bytes = lpBytes(lp);
+        bytes += lpLength(lp) * 2 * 4;


can you add a comment explaining the formula?

src/server/tiered_storage.cc

romange · 2025-11-20T16:11:34Z

src/server/tiered_storage.cc

+// TODO(vlad): Maybe split into different accessors?
+// Do NOT enforce rules depending on dynamic runtime values as this is called
+// when scheduling stash and just before succeeeding and is expected to return the same results
+optional<pair<size_t /*size*/, CompactObj::ExternalRep>> EstimateSerializedSize(


It does more than EstimateSize, and with having optional and precise ExternalRep the behavior is confusing. Maybe call it GetSerializationDescriptor which will return

struct SerializationDescriptor { size estimated_size; CompactObj::ExternalRep repr;

and add NONE to ExternalRep enum? or alternatively add is_valid() { return size> 0; } and return size=0 for unfit objects.

Personally, I am not a big fan of chaining optionals—or other types like std::expected—one within another, especially when we control the wrapped class and it can describe the "undef" state itself. For example, in our codebase, we have using Result = std::optional; where ResultType is another optional, or std::optional<facade::ErrorReply> where ErrorReply can hold an empty state. These levels of indirection decrease readability, imho.

I agree here, but in a properly structured code base type composition is always almost preferable

romange · 2025-11-20T16:23:23Z

src/server/tiered_storage.cc

      stats->tiered_used_bytes += segment.length;
      stats_.total_stashes++;

+      CompactObj::ExternalRep rep = EstimateSerializedSize(*pv)->second;


ADD a DCHECK before dereferencing.

the nullopt access with throw an assert either way

dranikpg · 2025-11-21T11:11:43Z

Removed the optional and just keeped a pair

feat(tiering): Serialize hashes

2f77302

Signed-off-by: Vladislav Oleshko <[email protected]>

dranikpg force-pushed the encode-hash branch from 1839c60 to a6f53a6 Compare November 17, 2025 12:17

fixes

4205592

Signed-off-by: Vladislav Oleshko <[email protected]>

dranikpg force-pushed the encode-hash branch from a6f53a6 to 4205592 Compare November 17, 2025 12:18

fixes and small tests

c178a09

Signed-off-by: Vladislav Oleshko <[email protected]>

dranikpg force-pushed the encode-hash branch from 9dce7a9 to c178a09 Compare November 18, 2025 14:32

dranikpg added 2 commits November 18, 2025 17:43

small fix

b819f16

extend test

bd9fccd

dranikpg requested a review from romange November 18, 2025 14:46

dranikpg marked this pull request as ready for review November 18, 2025 14:46

more fixes and uploading

dc970d2

romange previously approved these changes Nov 19, 2025

View reviewed changes

dranikpg mentioned this pull request Nov 19, 2025

fix(server): Move LpGetView to core target #6076

Merged

dranikpg added 2 commits November 19, 2025 20:50

Merge branch 'main' into encode-hash

52630ac

fix: use size estimation for page deduction

fcf65e9

dranikpg dismissed romange’s stale review via fcf65e9 November 20, 2025 08:18

rename

6cc2419

dranikpg commented Nov 20, 2025

View reviewed changes

dranikpg mentioned this pull request Nov 20, 2025

feat(tiering): Mutable hash commands #6085

Draft

more fixes

40d054b

Signed-off-by: Vladislav Oleshko <[email protected]>

dranikpg force-pushed the encode-hash branch from ecdfad3 to 40d054b Compare November 20, 2025 10:36

dranikpg commented Nov 20, 2025

View reviewed changes

dranikpg requested a review from romange November 20, 2025 11:47

romange reviewed Nov 20, 2025

View reviewed changes

Unwrap optional

731dbba

dranikpg requested a review from romange November 21, 2025 09:41

		auto estimated = EstimateSerializedSize(*value);
		if (OccupiesWholePages(estimated->first)) {

feat(tiering): Serialize hashes #6015

Are you sure you want to change the base?

feat(tiering): Serialize hashes #6015

Conversation

dranikpg commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

romange left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dranikpg commented Nov 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dranikpg commented Nov 5, 2025 •

edited

Loading