Exploring fastsafetensors for Network Storage: Seeking Community Input on Custom Reader Integration

[ABNER-1]

Hi maintainers,

The two-phase abstraction in fastsafetensors is excellent:

1. Parallel model loading with multiple workers
2. Efficient tensor distribution via NCCL broadcast, leveraging high-speed GPU memory and NVLink bandwidth

While this design excels in GDS scenarios, we believe it's equally well-suited for network storage workloads that benefit from:

1. High-concurrency file loading (different files in parallel)
2. Sequential bulk data reads

Building on fastsafetensors' architecture (referenced in issue #29), we've implemented a zero-copy reader optimized for network storage and achieved promising results. Since zero-copy implementations vary across storage systems, we extended this approach to 3FS (an open-source distributed filesystem), creating a usrbio-based reader.

Performance highlights:

• Peak throughput: 35 GB/s (saturating single RDMA 400 Gbps link)
• Setup: 8 processes via usrbio SDK
• Real-world result: Loading 640GB DeepSeek-R1 in ~27 seconds

We're excited about these results and would love to open-source our 3FS reader. However, we'd like to check whether fastsafetensors has plans to support such custom readers, or if you'd be open to this type of contribution?

Our initial implementation consists of three main components, though we're happy to refine the architecture based on your feedback if there's interest in moving forward.

Looking forward to your thoughts!

[takeshi-yoshimura]

@ABNER-1
Thank you for your excellent proposal again!

My recent concern is unit/integration tests. Do we need to set up 3FS and GPUs for testing the new code? What is the minimal requirements?

[takeshi-yoshimura]

Also, cpp directories should not be separated. Is it possible to merge them into one?
Adding a new loader is fine because we already have Pipeline Loader as you introduced, but what is the purpose of theefs_reader?

[ABNER-1]

Also, cpp directories should not be separated. Is it possible to merge them into one? Adding a new loader is fine because we already have Pipeline Loader as you introduced, but what is the purpose of theefs_reader?

Thank you for your response and encouragement!

The directory structure is flexible and can certainly be adjusted. The current layout was primarily intended to clearly illustrate the CPP code that interfaces with the 3FS usrbio SDK.

Regarding threefs_reader.py: it serves as a Python wrapper for the CPP-level interfaces. The core purpose is to optimize Stage 1 (the download phase) by enabling more efficient file reads from network storage.

To elaborate: in traditional FUSE-based read modes, direct file reads incur multiple kernel-space copy operations. Solutions like the 3FS usrbio SDK provide a much shorter I/O path, allowing the read pipeline to saturate the full network bandwidth.

[ABNER-1]

@ABNER-1 Thank you for your excellent proposal again!

My recent concern is unit/integration tests. Do we need to set up 3FS and GPUs for testing the new code? What is the minimal requirements?

Thank you for raising this excellent question!
This is indeed a critical aspect we need to address.
I'll systematically organize the specific testing environment setup and minimal requirements shortly and get back to you.

[ABNER-1]

@ABNER-1 Thank you for your excellent proposal again!
My recent concern is unit/integration tests. Do we need to set up 3FS and GPUs for testing the new code? What is the minimal requirements?

Thank you for raising this excellent question! This is indeed a critical aspect we need to address. I'll systematically organize the specific testing environment setup and minimal requirements shortly and get back to you.

To test the full 3FS reader implementation, a 3FS cluster and GPUs are indeed required.

However, considering CI complexity, we can take inspiration from how SGLang handles 3FS testing—they use a mock client in their CI pipeline to simulate 3FS client responses without requiring the actual infrastructure. I believe this approach is worth adopting for our case.

[takeshi-yoshimura]

Regarding threefs_reader.py: it serves as a Python wrapper for the CPP-level interfaces. The core purpose is to optimize Stage 1 (the download phase) by enabling more efficient file reads from network storage.

We already have cpp.pyi for cpp wrapper for type hints and checks. If it's just a wrapper, then please also merge it to cpp.pyi.

To test the full 3FS reader implementation, a 3FS cluster and GPUs are indeed required.
However, considering CI complexity, we can take inspiration from how SGLang handles 3FS testing—they use a mock client in their CI pipeline to simulate 3FS client responses without requiring the actual infrastructure. I believe this approach is worth adopting for our case.

Mock client sounds better, but having its implementation here is a bit too much. I am wondering if 3FS provides such utilities or python packages.

Also, 3fs_client should be put outside of this repository ideally. Do we have 3fs client library? Is 3fs_client here specific to your optimization?

[ABNER-1]

Sounds like we've reached a preliminary consensus, which is great! To summarize my understanding:

• fastsafetensors is open to accepting network-storage-specific loaders that demonstrate significant performance improvements
• The remaining work involves refining code organization, implementation details, and CI/testing infrastructure

I'm happy to proceed with these topics. Here's my plan:

1. Code Integration

• Merge CPP bindings into the existing cpp.pyi structure as suggested
• Design a clean integration point for the 3FS reader that aligns with fastsafetensors' architecture

2. CI/Testing Strategy (inspired by SGLang)

• SGLang recommends building from source to install their pip wheel, and they've implemented a complete set of classes including mock implementations for CI
• We'll follow a similar pattern: implement mock classes for the 3FS backend interface to enable regression testing without requiring full 3FS infrastructure

3. 3FS Client Repository Structure

• Our current 3fs_client is implemented on top of the 3FS usrbio SDK, primarily interfacing with the copier layer (two key interfaces)
• I will explore the possibility of extracting this into a standalone repository to keep fastsafetensors core lean, while maintaining it as an optional dependency

[takeshi-yoshimura]

Sounds good! Please open PR if you get ready. I am pleased to review your change.