Parallel snapshot decompress experiment #7406
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ripatel-fd
commented
Nov 24, 2025
ripatel-fd
commented
Nov 24, 2025
ripatel-fd
force-pushed
the
ripatel/snapdc-para
branch
2 times, most recently
from
c23da83 to
30da3ff
Compare
ripatel-fd
force-pushed
the
ripatel/snapdc-para
branch
from
30da3ff to
287dc29
Compare
Performance Measurements ⏳
|
ripatel-fd
force-pushed
the
ripatel/snapdc-para
branch
from
287dc29 to
8f25332
Compare
Performance Measurements ⏳
|
Ensures that our streaming Zstandard frame boundary detector computes the same result as libzstd.
ripatel-fd
force-pushed
the
ripatel/snapdc-para
branch
from
8f25332 to
e9fc516
Compare
Performance Measurements ⏳
|
ripatel-fd
commented
Nov 25, 2025
| return; | ||
| FD_LOG_WARNING(( "corrupt snapshot: unexpected end of TAR file" )); | ||
| transition_malformed( ctx, stem ); | ||
| return 0; |
ripatel-fd
commented
Nov 25, 2025
| atexit( fd_halt ); | ||
| //fd_log_level_stderr_set(4); | ||
| //fd_log_level_logfile_set(4); | ||
| //fd_log_level_core_set(4); |
ripatel-fd
commented
Nov 25, 2025
|
|
||
| if( FD_LIKELY( out_produced ) ) { | ||
| fd_stem_publish( stem, 0UL, FD_SNAPSHOT_MSG_DATA, ctx->out.chunk, out_produced, 0UL, 0UL, 0UL ); | ||
| ulong ctl = fd_frag_meta_ctl( 0UL, 0, frame_res==0UL, 0 ); |
Contributor
Author
There was a problem hiding this comment.
separate out into eom var
ripatel-fd
commented
Nov 25, 2025
| if( snapwr_tile ) fputs( "--------------", stdout ); | ||
| if( snapls_tile ) fputs( "[ct],[ld],[dc],[in],[lts]--------[ct],[ld],[dc],[in],[lts]", stdout ); | ||
| else fputs( "[ct],[ld],[dc],[in]--------[ct],[ld],[dc],[in]", stdout ); | ||
| if( snapls_tile ) fputs( "[ld],[dc],[in],[lts]--------[ld],[dc],[in],[lts]", stdout ); |
Contributor
Author
There was a problem hiding this comment.
lthash and multi dc tiles are incompatible
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Updates Firedancer's snapshot load topology to run multiple Zstandard decompressor tiles in parallel, taking turns compressing frames produced by the writer in #7401.
This entirely eliminates the Zstandard bottleneck, decompressing a mainnet snapshot in 22 seconds with 16 threads on Zen 4.