Skip to content

Commit e9337bb

Browse files
committed
docs(changelog): add go-ds-flatfs batch memory improvement
documents reduced memory usage during data import with the new flatfs batch implementation that writes to temp files instead of buffering in RAM.
1 parent 153c870 commit e9337bb

File tree

1 file changed

+12
-1
lines changed

1 file changed

+12
-1
lines changed

docs/changelogs/v0.40.md

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,15 +10,26 @@ This release was brought to you by the [Shipyard](https://ipshipyard.com/) team.
1010

1111
- [Overview](#overview)
1212
- [🔦 Highlights](#-highlights)
13+
- [Reduced memory usage during data import](#reduced-memory-usage-during-data-import)
1314
- [Routing V1 HTTP API now exposed by default](#routing-v1-http-api-now-exposed-by-default)
14-
- [Track total size when adding pins](#track-total-size-when-adding-pins]
15+
- [Track total size when adding pins](#track-total-size-when-adding-pins)
1516
- [📝 Changelog](#-changelog)
1617
- [👨‍👩‍👧‍👦 Contributors](#-contributors)
1718

1819
### Overview
1920

2021
### 🔦 Highlights
2122

23+
#### Reduced memory usage during data import
24+
25+
The flatfs datastore batch implementation has been rewritten to reduce memory usage during large imports (`ipfs add`, `ipfs dag import`). Instead of buffering all blocks in RAM until commit, blocks are now written to temporary files asynchronously and renamed atomically on commit.
26+
27+
For example, importing a 10 GiB file with 1 MiB chunks previously required buffering up to 10 GiB of block data in memory. Now, only a small number of concurrent writes (up to 16 MiB) are buffered at any time, regardless of total import size. This makes it possible to onboard large files and directories on machines with limited RAM.
28+
29+
Additionally, if kubo crashes during import, uncommitted data is automatically cleaned up on restart, preventing orphan blocks in the datastore.
30+
31+
See [go-ds-flatfs#142](https://github.com/ipfs/go-ds-flatfs/pull/142) for implementation details.
32+
2233
#### Routing V1 HTTP API now exposed by default
2334

2435
The [Routing V1 HTTP API](https://specs.ipfs.tech/routing/http-routing-v1/) is now exposed by default at `http://127.0.0.1:8080/routing/v1`. This allows light clients in browsers to use Kubo Gateway as a delegated routing backend instead of running a full DHT client. Support for [IPIP-476: Delegated Routing DHT Closest Peers API](https://github.com/ipfs/specs/pull/476) is included. Can be disabled via [`Gateway.ExposeRoutingAPI`](https://github.com/ipfs/kubo/blob/master/docs/config.md#gatewayexposeroutingapi).

0 commit comments

Comments
 (0)