Failure to add more than ~1200 files

[fabricedesre]

While debugging a failure to import largish directories, I found that adding more than ~1200 files would always fail with a variation of "Maximum block size exceeded". I can reproduce with the test case added in main...capyloon:rs-wnfs:perf-test

You can reproduce by running cargo test --release test_perf_add_files and will get something similar to:

fabrice@framework:~/dev/capyloon/rs-wnfs/wnfs perf-test$ cargo test --release test_perf_add_files -- --nocapture
warning: some crates are on edition 2021 which defaults to `resolver = "2"`, but virtual workspaces default to `resolver = "1"`
note: to keep the current resolver, specify `workspace.resolver = "1"` in the workspace root's manifest
note: to use the edition 2021 resolver, specify `workspace.resolver = "2"` in the workspace root's manifest
   Compiling wnfs v0.1.26 (/home/fabrice/dev/capyloon/rs-wnfs/wnfs)
    Finished release [optimized] target(s) in 15.49s
     Running unittests src/lib.rs (/home/fabrice/dev/capyloon/rs-wnfs/target/release/deps/wnfs-85ceb7b84c448bfe)

running 1 test
#0 : dir stored in 11ms
#100 : dir stored in 793ms
#200 : dir stored in 783ms
#300 : dir stored in 798ms
#400 : dir stored in 776ms
#500 : dir stored in 757ms
#600 : dir stored in 757ms
#700 : dir stored in 792ms
#800 : dir stored in 817ms
#900 : dir stored in 820ms
#1000 : dir stored in 866ms
#1100 : dir stored in 833ms
#1200 : dir stored in 886ms
thread 'private::directory::tests::test_perf_add_files' panicked at wnfs/src/private/directory.rs:1429:63:
called `Result::unwrap()` on an `Err` value: Maximum block size exceeded: Encountered block with 263232 bytes
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test private::directory::tests::test_perf_add_files ... FAILED

failures:

failures:
    private::directory::tests::test_perf_add_files

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 72 filtered out; finished in 19.68s

There is also a degradation of the dir.as_node().store(...) performance as we add more files. Is that a known issue?

[ehsan6sha]

Is this resolved?

[matheus23]

No, PrivateDirectoryContent::entires is still a BTreeMap, so if that (in its serialized form) grows beyond the IPFS block size limit, then it'll fail to be stored in the blockstore.