feat(chunkserver): add HDD_CHUNK_BULK_SIZE option#824
Conversation
81ac1a7 to
92e2d09
Compare
There was a problem hiding this comment.
Code Review
This pull request introduces a new configuration option, HDD_CHUNK_BULK_SIZE, which allows users to define the number of chunks processed in a single bulk operation when reporting to the master server. This change replaces several hardcoded instances of the value 1000 with a reloadable atomic variable. The review feedback suggests enforcing a reasonable upper bound on this configuration value using cfg_get_minmaxvalue to prevent potential memory exhaustion or network communication issues caused by excessively large bulk sizes.
There was a problem hiding this comment.
Pull request overview
Adds a new reloadable chunkserver configuration option (HDD_CHUNK_BULK_SIZE) to control how many chunks are grouped per bulk when communicating chunk lists to the master, and propagates this option through code/config/docs.
Changes:
- Introduce a reloadable
HDD_CHUNK_BULK_SIZEsetting (default 1000) backed by an atomic (gChunkBulkSize). - Use the configured bulk size for chunk registration bulks and damaged/lost/new chunk report bulks.
- Update config template, admin default dump, and chunkserver config manpage with the new option.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/data/sfschunkserver.cfg.in | Documents the new HDD_CHUNK_BULK_SIZE option in the sample config. |
| src/chunkserver/masterconn.cc | Uses the configured bulk size for damaged/lost/new chunk report bulks. |
| src/chunkserver/master_connection.cc | Uses the configured bulk size for registration chunk bulks. |
| src/chunkserver/hddspacemgr.h | Adds kDefaultChunkBulkSize and atomic gChunkBulkSize; updates bulk API signature. |
| src/chunkserver/hddspacemgr.cc | Loads/stores HDD_CHUNK_BULK_SIZE on init and reload. |
| src/admin/dump_config_command.cc | Adds the default value for HDD_CHUNK_BULK_SIZE to chunkserver defaults. |
| doc/sfschunkserver.cfg.5.adoc | Documents the new HDD_CHUNK_BULK_SIZE option in the manpage. |
92e2d09 to
cafe7b7
Compare
cafe7b7 to
9671305
Compare
GigaCronos
left a comment
There was a problem hiding this comment.
Looks good. Please check my suggestion. Also consider creating and linking the corresponding Linear Issue.
9671305 to
82eba92
Compare
antuan96314
left a comment
There was a problem hiding this comment.
LGTM, However, consider complementing PR with a test to make sure it works as expected.
82eba92 to
f2d7c22
Compare
I added a simple one. |
Introduce a reloadable HDD_CHUNK_BULK_SIZE option for chunkserver. Use it for chunk registration and damaged/lost/new report bulks. Add default (1000) to config template, admin defaults, and docs. Signed-off-by: guillex <guillex@leil.io>
f2d7c22 to
3a78802
Compare
Introduce a reloadable HDD_CHUNK_BULK_SIZE option for chunkserver. Use it for chunk registration and damaged/lost/new report bulks. Add default (1000) to config template, admin defaults, and docs.
LS-564