v1.33.1

This is a patch release.

It is strongly advised to update to this version if you use 1.33 due to regression below.

What's Changed

• fix : stack corruption in MultiCommandSquasher #5697
  It also fixed the io_uring compatibility issue with older kernels (#5693)

v1.33.0

Dragonfly v1.33.0

Important stability fixes and changes:

• Fix stability bugs in georadius #5629 #5634
• Cluster related improvements: #5614 #5621 and #5618

What's Changed

• server: Log warning if server stops accepting connections by @abhijat in #5596
• feat(server): support command help command by @guozhihao-224 in #5601
• chore: refactor JournalItem by @kostasrim in #5585
• feat(linear_search_map): Add keys method by @BagritsevichStepan in #5603
• core: Minor cleanups in bloom filters by @abhijat in #5595
• fix(tiering): Block on concurrent backing file grow by @dranikpg in #5609
• feat: use SET instead of RESTORE for strings during slot migrations by @BorysTheDev in #5614
• chore: introduce shard_thread_busy_polling_usec to control busy polling by @romange in #5624
• fix: UB in georadius by @BorysTheDev in #5629
• fix: test_throttle_on_commands_squashing_replies_bytes by @kostasrim in #5628
• chore: more pipeline latency coverage by @romange in #5632
• fix: ub in GEORADIUSBYMEMBER by @kostasrim in #5634
• chore: more pipeline latency coverage (#5632) by @romange in #5635
• feat: use PXAT flag instead of separate command for strings during migration by @BorysTheDev in #5631
• refactor: CmdArgParser::Error() by @BorysTheDev in #5636
• test(acl): Fix flaky ACL tests with long-running Lua scripts by @vyavdoshenko in #5627
• chore: string_view overload for BaseFamilyTest::Run by @kostasrim in #5637
• fix: FlushSlots test by @BorysTheDev in #5639
• fix: macos build geo_family.cc by @vyavdoshenko in #5640
• fix: preserve null characters in hash values after HEXPIRE by @vyavdoshenko in #5642
• test(acl): make ACL tests with concurrent scripts deterministic by @vyavdoshenko in #5641
• test: skip test_acl_del_user_while_running_lua_script becuase it constantly fails on CI by @vyavdoshenko in #5646
• chore: do not shutdown after takeover only in full cluster mode by @kostasrim in #5621
• chore: add more I/O run-time settings by @romange in #5644
• fix: Prevent race condition during concurrent SAVE operations by @vyavdoshenko in #5633
• core: Track objects skipped during defragmentation by @abhijat in #5647
• chore(tiered_storage): Rename flags, make them updatable by @dranikpg in #5608
• chore: allow replicaof in cluster mode when state is TAKEN_OVER by @kostasrim in #5618
• fix: timeout overflow during migration finalization by @BorysTheDev in #5652
• fix(json): support json keys containing dots in bracket notation by @vyavdoshenko in #5650
• fix: accounting of metrics under squash_stats_latency_lower_limit filter by @romange in #5659
• chore: sort per shard sorted sets for SINTER by @kostasrim in #5617
• fix: macos build tiered_storage.h by @vyavdoshenko in #5663
• fix: RoundRobinSharder for all proactors by @kostasrim in #5654
• workflows: Add a dispatch action which will run pytests on repeat by @abhijat in #5668
• fix(pytest): Don't delete and modify huge value keys in test_replication_all by @mkaruza in #5657
• chore: move tiering flags by @kostasrim in #5665
• fix(json_family): Remove expiry of existing keys in JSON.SET by @mkaruza in #5670
• fix: test_migration_timeout_on_sync by @BorysTheDev in #5667
• fix: refactor save synchronization to prevent deadlocks and crashes by @vyavdoshenko in #5655
• fix(pytest): Increase timeout in test_seeder_key_target by @mkaruza in #5677
• chore: calibrate regression tests running time by @kostasrim in #5658
• chore(lua): Add lua redis.log function by @mkaruza in #5672
• chore: add more detailed timings for slow squash hops by @romange in #5679

Full Changelog: v1.32.0...v1.33.0

v1.32.0

Dragonfly v1.32.0

Important stability fixes and changes:

• Lots of performance improvements around Search #5341#5327 #5350 #5545 #5533
• Add optional latency percentile stats #5372 and #5428
• Update mimalloc version to 2.24
• Added BZMPOP and BLMPOP commands #5436 and #5370
• Added CONFIG REWRITE command #5471

What's Changed

• test(search): add unified benchmarks for prefix/suffix/infix search by @vyavdoshenko in #5319
• chore: get rid of quicklist encoding by @romange in #5282
• feat: add flag pause_wait_timeout by @BorysTheDev in #5328
• test: enable test_multidim_knn python test by @vyavdoshenko in #5329
• test: make random determenistic in test_multidim_knn python test by @vyavdoshenko in #5335
• fix: data race when doing a snapshot in cache mode by @romange in #5336
• chore(search_family): Minor performance improvements to GetAllResults methods. FIRST PR by @BagritsevichStepan in #5341
• fix: JSON.DEL to return 0 for non-existing keys instead of error by @vyavdoshenko in #5343
• chore: improve test_full_sync test by @romange in #5340
• chore: Refactor CompactObject encoding by @dranikpg in #5342
• feat: Added Sort_RO Command. by @H4R5H1T-007 in #5332
• chore: avoid deduplicating docs ids when not needed by @romange in #5344
• chore: add hdr_histogram to the project by @romange in #5330
• chore(tiering_test): tune memory settings to be below 6GB by @romange in #5345
• chore: suffix search by @dranikpg in #5327
• test: Vector operations benchmark was added by @vyavdoshenko in #5353
• feat(replication): Aggregate data until threshold before replication by @mkaruza in #5308
• SimSIMD integration for Vector operations by @vyavdoshenko in #5348
• feat (server) : introduce no point in time replication by @adiholden in #5103
• chore: move 3rd party libraries into a separate cmake file by @romange in #5362
• fix(pytest): Increase allowed diff in test_partial_replication_on_same_source_master by @mkaruza in #5363
• fix(replication): reduce ERROR logging during normal shutdown by @vyavdoshenko in #5366
• chore: remove unneeded mimalloc-specific code by @romange in #5367
• chore: add mimalloc2 third_party library by @romange in #5364
• fix(replication): reduce ERROR to WARNING for connection issues during sync by @vyavdoshenko in #5369
• feat(search): Suffix/Infix search for Tags by @dranikpg in #5350
• fix(zset): expand parsing of rank arguments beyond int32 by @romange in #5377
• test: replication_test.py::test_bug_in_json_memory_tracking enabled by @vyavdoshenko in #5375
• chore: Disable test_replication_all cache mode by @mkaruza in #5385
• chore: Make fakeredis part of the core CI checks by @romange in #5380
• fix(zset_family): Return early if range is expected to be zero by @mkaruza in #5386
• fix: JSON.STRLEN behavior for non-existent keys by @vyavdoshenko in #5389
• chore: add psync metrics for replica by @kostasrim in #5365
• fix(server): fix serialization on move with big values by @adiholden in #5390
• feat(pytest): Set and log random seed used in pytest by @mkaruza in #5391
• chore: remove zset/t_zset files by @romange in #5387
• feat: add support for BLMPOP command by @EricHayter in #5370
• feat(pytest): Initialize random seed for each test by @mkaruza in #5395
• fix(values): fix typos in Helm README and values file by @adimoragbloch in #5400
• fix: no set/hmap expiration during serialization by @BorysTheDev in #5349
• feat(search_family): Add SortedVector<pair<DocId, double>> by @BagritsevichStepan in #5405
• feat: add snapshot-id field into dfs files by @BorysTheDev in #5399
• chore: Enable point_in_time_snapshot by default by @mkaruza in #5403
• chore: Increase BLMPopBlocking sleep duration by @mkaruza in #5408
• chore: skip flaky test_pipeline_cache_size by @kostasrim in #5410
• chore: introduce class scope fixtures by @romange in #5409
• fix(pytest): Increase REPLTAKOVER time in test_take_over_seeder by @mkaruza in #5415
• server_family: Expose command latency stats as part of info by @abhijat in #5372
• fix(search): Cut off results before serializing by @dranikpg in #5412
• test: Add Dragonfly search benchmark tests by @vyavdoshenko in #5406
• feat(search_family): Introduce RangeTree by @BagritsevichStepan in #5414
• fix: search_benchmark_utils on python 3.12 by @romange in #5422
• feat(search_family): Add RangeTree to NumericIndex by @BagritsevichStepan in #5416
• chore: switch to mimalloc2 by @romange in #5368
• feat(search_family): Add optimized merging logic for one and two blocks in RangeResult by @BagritsevichStepan in #5417
• chore(range_tree): Simplify RangeTree code by @BagritsevichStepan in #5418
• Feat config management by @boomballa in #5388
• Lua library Integer overflow can cause the DragonFly crash. (CVE-2020-14147) by @KIMDONGYEON00 in #5421
• chore(search): LIMIT=0 Fast path and remove LOAD parameter from FT.SEARCH by @dranikpg in #5419
• server_family: Support resetting latency histogram by @abhijat in #5428
• test: search benchmark updated to use class fixture for multiple tests by @vyavdoshenko in #5425
• chore: harden checks in _mi_page_malloc_zero by @romange in #5427
• fix(pytest): Wait replication to be in sync in test_stream_approximate_trimming by @mkaruza in #5430
• chore: avoid squashing when squashing_current_reply_size crosses limit by @kostasrim in #4924
• fix: acl log missing return after reply by @kostasrim in #5454
• chore: add acl help by @kostasrim in #5456
• chore: allow initializing a separate huffman table for string values by @romange in #5470
• feat(search_family): Introduce SingleBlockRangeResult and TwoBlocksRangeResult by @BagritsevichStepan in #5424
• chore: add tls_caching flags by @kostasrim in #5485
• chore: add more logs during replication by @BorysTheDev in #5489
• chore(set_family): Fast path SPOP with small count by @dranikpg in #5491
• fix: add a sleep to relax slot migration process by @romange in #5478
• chore: add tls_socket accept error metric by @romange in #5492
• chore: add io_read_cnt metric by @romange in #5494
• chore: add tls handshake statistics by @kostasrim in #5493
• feat: add more auxillary strings to the snapshot by @romange in #5411
• chore: log frequency for IsReplySizeOverLimit by @kostasrim in #5501
• fix: adjust BLMPOP command options and implementation for proper journaling by @EricHayter in #5503
• chore: add more metrics by @romange in https://github.com/dragonflydb/dragonfly/p...

v1.31.2

This is a patch release.

What's Changed

• fix: prevent idle disconnects for replica flow connections (#5506)
• feat: add debug segments command (#5505)
• server: Make memory defrag blocking, accept optional threshold (#5498)
• fix: allow eviction of zero-allocation entries in cache mode (#5500)
• chore: add more metrics (#5504)

Full Changelog: v1.31.1...v1.31.2

v1.31.1

This is a patch release.

What's Changed

• fix: two tls issues
  1. Excessive logging during tls handshake in some cases.
  2. TLS Short read that hinders read performance.
• chore: add tls handshake statistics (#5493)
• chore: add io_read_cnt metric (#5494)
• chore: add tls_socket accept error metric (#5492)
• fix: add sleep to relax slot migration process (#5478)
• fix(set_family): Fast path SPOP with small count (#5491)
• chore: add more logs during replication (#5489)
• chore: add tls_caching flags (#5485)

Full Changelog: v1.31.0...v1.31.1

v1.31.0

Dragonfly v1.31.0

This release is mostly focused on stability features and bug fixes.

Important stability fixes and changes:

1. Introduce partial sync feature that avoids full replica reload during master/replica short disconnects #5182
2. Add support for GAT/GATS commands #5257
3. Introduce experimental huffman encoding for keys #5179

What's Changed

• chore: support fetching a huffman encoded string from CompactObj by @romange in #5086
• fix(hset_family): Fix val being overwritten by TTL by @abhijat in #5094
• fix: memcached authentication handling by @vyavdoshenko in #5098
• fix: memcached port has been already used crash by @vyavdoshenko in #5097
• chore: Add suffix and infix search syntax support in query parser by @vyavdoshenko in #5091
• chore(server): cleanup unused noop opcode in journal by @adiholden in #5101
• chore: add arch type to dfly bench release binary by @kostasrim in #5090
• feat: add HELP SCAN and MINMSZ filter by @romange in #5099
• fix test: rdb test keep loading expired keys by @adiholden in #5105
• refactor: provide CommandContext into InvokeCmd by @BorysTheDev in #5100
• dfly_bench: Send auth if required before cluster setup by @abhijat in #5107
• chore: tls improvements by @romange in #5116
• fix: XTRIM command crash by @vyavdoshenko in #5121
• chore: update module versions by @romange in #5126
• chore: provide metrics for huffman encoding by @romange in #5117
• fix: make search keywords case insensitive by @romange in #5129
• chore: replica clean ups by @romange in #5127
• fix: collapsing logic in CapturingReplyBuilder by @romange in #5133
• fix: improved socket error logging for connection diagnostics by @vyavdoshenko in #5062
• fix: out of bounds SIMD access by @romange in #5140
• chore: improve logs around the parser error by @romange in #5144
• feat: introduce resp validator by @romange in #5147
• chore: minor fixes in geo_family by @romange in #5149
• refactor: geosearch cmd with CmdArgParser by @BorysTheDev in #5153
• feat: add Georadius command by @BorysTheDev in #5158
• pytest: Add sleep to let master stop propagate to replica by @abhijat in #5157
• chore: enable simd optimizations for aarch64 by @romange in #5150
• fix: prevent crash in GEORADIUS argument parsing by @vyavdoshenko in #5166
• chore: more tests around dash table by @romange in #5168
• chore: Add option to ignore parsing errors in replay tool by @vyavdoshenko in #5167
• fix: Prevent crash in PFCOUNT when using keys of different types by @vyavdoshenko in #5161
• chore: lower log level for harmless "Invalid Json path" warnings by @vyavdoshenko in #5163
• chore: remove global current_reply_size by @romange in #5057
• replication: Ensure snapshot fiber joined in all code paths by @abhijat in #5171
• chore: refactorings in dash code. by @romange in #5178
• chore(dash_table): pass memory resource to segment by @romange in #5176
• fix: show proactor kind on test failures by @kostasrim in #5170
• fix(parser): Parse correctly split simple response or error msg in client mode by @mkaruza in #5177
• chore: introduce azure snapshot storage skeleton by @romange in #5104
• fix: improve parsing of client facing responses by @romange in #5181
• server(fix) : SCAN command reduce latency by @vyavdoshenko in #5151
• chore(json_family): Clean up the memory tracking code for JSON mutate operations. FOURTH PR by @BagritsevichStepan in #5070
• feat(search_family): Ignore unsupported options in the FT.CREATE command by @BagritsevichStepan in #5188
• fix (server): fix replication partial sync flow by @adiholden in #5189
• feat: add flags support for huffman encoding by @romange in #5179
• fix server: expire if need check valid condition fix by @adiholden in #5193
• chore: minor dash fixes by @romange in #5191
• fix(json_family): Fix memory tracking for the JSON.DEL command. FIFTH PR by @BagritsevichStepan in #5075
• fix: prevent crash on KEYLOCK_ACQUIRED check for NO_KEY_TRANSACTIONAL commands by @vyavdoshenko in #5185
• fix(cluster_family): Cancel slot migration from incoming node on OOM by @mkaruza in #5000
• fix: timeout for test_seeder_key_target by @BorysTheDev in #5198
• fix: crash in HLL PFMERGE with corrupted data by @vyavdoshenko in #5201
• fix: ZCOUNT crash when min > max by @vyavdoshenko in #5205
• fix: SearchStats race condition crash during concurrent FT operations by @vyavdoshenko in #5207
• fix(pytest): Sleep in sporadically failing test_slot_migration_oom pytest by @mkaruza in #5214
• fix(transaction): DCHECK fail in non-atomic transactions by @romange in #5217
• fix: seeder hset commands generation by @BorysTheDev in #5220
• fix(http_api): Return valid JSON string by @mkaruza in #5210
• feat: add Lua GC flags by @BorysTheDev in #5194
• fix: Unsupported status for Copy/Move/Rename operations by @vyavdoshenko in #5223
• (string_family): Add priliminary support for GAT by @abhijat in #5229
• chore: reduce leaky interface of DashCursor by @romange in #5230
• chore: minor changes in dash code by @romange in #5224
• test: add migration_restart_test with another slots set by @BorysTheDev in #5231
• chore: bring back partial sync by @kostasrim in #5182
• fix: macos build by @vyavdoshenko in #5234
• chore: maintain meta information around continous transactions by @romange in #5232
• fix: dcheck trigger in JournalFlushGuard constructor by @kostasrim in #5241
• fix: typo in AclFamily by @vyavdoshenko in #5244
• fix(pubsub): Skip stale pubsub messages by @dranikpg in #5246
• chore: improve partial sync and test by @kostasrim in #5195
• chore(search): Add self time and resp3 format to ft.profile by @dranikpg in #5250
• fix: test_pubsub_unsubscribe test timeout increased by @vyavdoshenko in #5253
• chore: add corruption check for lpSetTotalBytes by @romange in #5252
• chore: Pull latest helio by @mkaruza in #5254
• fix: BITFIELD: overflow handling, type validation, and return values by @vyavdoshenko in #5242
• fix: prevent writing zero size fields into log file by @BorysTheDev in #5256
• fix(transaction): Fix non-transactional multi/exec transaction access by @dranikpg in #5260
• fix(search_family): Fix empty key bug by @BagritsevichStepan in #5264
• chore: remove old ZIPLIST encoding during snapshotting by @romange in #5267
• chore: stats for lsn buffer by @kostasrim in https://github.com/dragonflyd...

v1.30.3

This is a patch release.

What's Changed

• fix: reduce scan effect on latency of other operations - #5193 #5151
• fix replication partial sync flow (#5189)
• fix(search_family): Ignore unsupported options in the FT.CREATE command (#5188)

Full Changelog: v1.30.2...v1.30.3

v1.30.2

This is a patch release.

What's Changed

• replication: Ensure snapshot fiber joined in all code paths (#5171)
• chore: lower log level for harmless "Invalid Json path" warnings (#5163)
• chore: improve logs around the parser error (#5144)
• fix: out of bounds SIMD access (#5140)
• chore: enable simd optimizations for aarch64 (#5150)
• fix: Prevent crash in PFCOUNT when using keys of different types (#5161)

Full Changelog: v1.30.1...v1.30.2

v1.30.1

This is a patch release.

This release fixes a regression from v1.29.0 when Dragonfly crashes during pipelining (#5012).
It is strongly advised to upgrade for those who use v1.30.0.

What's Changed

• fix: "chore: get rid of possible recursion when unwinding structured reply (#5012)"
• fix(hset_family): Fix val being overwritten by TTL (#5094)
• fix: update module versions (#5126)
• fix: make search keywords case insensitive (#5129)

Full Changelog: v1.30.0...v1.30.1

v1.30.0

Dragonfly v1.30.0

Important stability fixes and changes:

1. More JSON/SEARCH stability fixes: #4973 #5026 #5036 #5040
2. Performance improvements around pipleining #4994
3. Memcached suppport for GETS #5087 as well as supporting tls without authentication #5085

What's Changed

• feat(dfly_bench): allow regulated throughput in 3 modes by @romange in #4962
• feat(server): Add support for command aliasing by @abhijat in #4932
• fix: FT.SEARCH KNN crash fixed by @vyavdoshenko in #4973
• Add workflow to run mastodon ruby test suite by @abhijat in #4946
• chore(acl_family): Move some variables outside loop by @abhijat in #4975
• feat: availability_zone flag support was added by @vyavdoshenko in #4976
• chore(hset_family): Support resp3 format for hrandfield by @abhijat in #4978
• Fix test hypothesis by @cunla in #4927
• chore: time sensitive throttle for scan command by @kostasrim in #4954
• feat(tools/replay): Add pipeline latency distribution data by @Pothulapati in #4990
• refactor(debug-cmd): migrate ParsePopulateArgs to CmdArgParser by @XueSongTap in #4838
• fix: potential vulnerable cloned functions by @npt-1707 in #4996
• chore: split unit test steps in ci workflow by @kostasrim in #5002
• feat: allow sharding by cluster slot id by @romange in #5006
• fix: potential crash with multi-sharded pfmerge by @romange in #5008
• chore: Pipelining fixes by @romange in #4994
• fix: bitops_family crash fixed by @vyavdoshenko in #4989
• chore: fix some clang-tidy warnings by @romange in #5009
• chore: clean ups around command squasher by @romange in #5011
• chore: StoredCmd to support both owned and external arguments by @romange in #5010
• fix: RENAME in cluster mode crash by @vyavdoshenko in #4995
• fix: regtest failures by @romange in #5013
• fix(glob): Correctly match longer strings with '*' or '**' pattern by @mkaruza in #5001
• refactor: update contributing doc by @BorysTheDev in #5016
• fix: HNSW schema deleting document crash by @vyavdoshenko in #4987
• chore: add DCHECK to easier tests debugging by @BorysTheDev in #5018
• chore: polishing. put basic usage info to stdout using one call to a void messing lines by @vyavdoshenko in #5019
• feat(server): support partial sync from last master by @adiholden in #5015
• feat(pytest): Generate huge val string in seeder by @mkaruza in #4957
• fix: JSON.ARRPOP command crash fixed by @vyavdoshenko in #5026
• chore: more performance improvements around MGET and pipelining by @romange in #5022
• feat: add copy cmd by @BorysTheDev in #5032
• fix: json depth limit has been changed to 64 by @vyavdoshenko in #5036
• chore: show moving P99 latency in dfly_bench by @romange in #5027
• chore: get rid of possible recursion when unwinding structured reply by @romange in #5012
• fix: Support for additional flags in expireat, pexpire and pexpireat commands by @H4R5H1T-007 in #5007
• chore: bump up max_busy_read_usec in tests by @romange in #5039
• fix: make FreeMemWithEvictionStep atomic by @kostasrim in #4885
• fix(stream_family): Fix stream memory tracking issues by @BagritsevichStepan in #5024
• chore: skip failing test_migration_rebalance_node by @kostasrim in #5042
• chore: add dfly_bench to release workflow by @kostasrim in #5045
• refactor: add WARNING regarding max_client_iobuf_len flag usage by @BorysTheDev in #5029
• refactor: reduce number of cluster logs by @BorysTheDev in #5044
• chore(json_family): Clean up code. FIRST PR by @BagritsevichStepan in #5049
• fix: send response error for long sequence instead of command crash by @vyavdoshenko in #5048
• feat: introduce 'debug iostats' command by @romange in #5051
• fix: json.set recursive processing crash by @vyavdoshenko in #5040
• bug(server): fix lns mismatch in replication by @adiholden in #4967
• feat(dfly_bench): introduce connect only flow by @romange in #5052
• chore: skip test_bug_in_json_memory_tracking by @kostasrim in #5066
• feat(memory_cmd): Add WITHOUTKEY option for the MEMORY USAGE command. SECOND PR by @BagritsevichStepan in #5068
• chore: dash table clean ups by @romange in #5064
• chore: further extend the compression analysis by @romange in #5065
• fix(json_family): Fix memory tracking for the JSON.SET command. THIRD PR by @BagritsevichStepan in #5069
• chore: introduce huffman encoder class by @romange in #5076
• chore(metrics): add rdb_bgsave_in_progress and rdb_last_bgsave_status by @kostasrim in #5061
• chore: futher dash table clean ups by @romange in #5072
• chore: add password flag to dfly_bench by @kostasrim in #5074
• chore: reorganize compact object mask bits by @romange in #5077
• feat: add migrated keys statistic by @BorysTheDev in #5043
• chore: Add HuffmanDecoder class by @romange in #5078
• fix: remove code duplicate in GetSlice by @romange in #5079
• chore: allow skipping cluster probing by @romange in #5082
• feat: add huffman coding to compact object by @romange in #5080
• chore: allow setting huffman tables via DEBUG COMPRESSION SET by @romange in #5083
• fix: Support TLS for Memcached without password authentication by @vyavdoshenko in #5085
• test: tune test_cluster_migration_while_seeding by @BorysTheDev in #5088
• feat(memcache): Add support for GETS by @abhijat in #5087
• feat(github-workflow): upload dev binaries to artifact buckets by @Abhra303 in #5031

Huge thanks to all the contributors! ❤️

New Contributors

• @XueSongTap made their first contribution in #4838
• @npt-1707 made their first contribution in #4996

Full Changelog: v1.29.0...v1.30.0