Skip to content

Commit 902ca40

Browse files
committed
Updated compression RFC
1 parent 3de77ba commit 902ca40

File tree

1 file changed

+4
-2
lines changed

1 file changed

+4
-2
lines changed

beyond-bitswap/rfc/rfcBBL203A.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -32,12 +32,14 @@ Some of the compression approaches to be explored in this RFC are:
3232
## Implementation plan
3333
- [x] Perform a simple test to evaluate the benefits of "on-the-fly" compression on blocks (to determine if IPFS could benefit from directly exchanging compressed messages and blocks). Evaluate different compression algorithms used in the web.
3434

35-
- [x] Evaluate the compression of full Bitswap messages (`bs.compressionStrategy = "full"`): To achieve this we add a compression flag in [Bitswap messages](https://github.com/adlrocha/go-bitswap/blob/master/message/message.go) to be able to identify when messages are compressed. Afterwards, if compression is enabled we need to [compress the stream](https://github.com/adlrocha/go-bitswap/blob/d151875a94048c3db59de52b9cb99d0246d74613/network/ipfs_impl.go#L240) before sending it. Compressed messages are identified in the [newMessageFromProto](https://github.com/adlrocha/go-bitswap/blob/d151875a94048c3db59de52b9cb99d0246d74613/message/message.go#L199) of receiving peers, they are uncompressed and processed seamlessly by Bitswap. In order to open the door to the use different compression algorithms and different full-message compression strategies a compressionType has been added to [message.proto](https://github.com/adlrocha/go-bitswap/blob/master/message/pb/message.proto).
35+
- [x] Evaluate the compression of full Bitswap messages (`bs.compressionStrategy = "full"`): To achieve this we add a compression flag in [Bitswap messages](https://github.com/adlrocha/go-bitswap/blob/master/message/message.go) to be able to identify when messages are compressed. Afterwards, if compression is enabled we need to [compress the message](https://github.com/adlrocha/go-bitswap/blob/d151875a94048c3db59de52b9cb99d0246d74613/network/ipfs_impl.go#L240) before sending it. Compressed messages are identified in the [newMessageFromProto](https://github.com/adlrocha/go-bitswap/blob/d151875a94048c3db59de52b9cb99d0246d74613/message/message.go#L199) of receiving peers, they are uncompressed and processed seamlessly by Bitswap. In order to open the door to the use different compression algorithms and different full-message compression strategies a compressionType has been added to [message.proto](https://github.com/adlrocha/go-bitswap/blob/master/message/pb/message.proto).
3636

37-
- [x] Evaluate the compression of blocks only (`bs.compressionStrategy = "blocks"`): We compress each block before adding it to the message in [nextEnvelope](https://github.com/adlrocha/go-bitswap/blob/d151875a94048c3db59de52b9cb99d0246d74613/internal/decision/engine.go#L469) function of engine.go, and then uncompress them in [ReceiveMessage](https://github.com/adlrocha/go-bitswap/blob/d151875a94048c3db59de52b9cb99d0246d74613/bitswap.go#L458) from bitswap.go. For the compression of blocks, the only thing that is changed for the transmission of the block is the RawData, the CID is kept without change so the block is conveniently identified. It required several changes over unit tests.
37+
- [x] Evaluate the compression of blocks only (`bs.compressionStrategy = "blocks"`): We compress each block before adding it to the protobuf message in [ToProtoV1](https://github.com/adlrocha/go-bitswap/blob/d151875a94048c3db59de52b9cb99d0246d74613/message/message.go#L583) function of message.go, and then uncompress them in [newMessageFromProto](https://github.com/adlrocha/go-bitswap/blob/d151875a94048c3db59de52b9cb99d0246d74613/message/message.go#L199). For the compression of blocks, the only thing that is changed for the transmission of the block is the RawData, the CID is kept without change so the block is conveniently identified.
3838

3939
- [x] Use [GZip](https://golang.org/pkg/compress/gzip/) as the default compression algorithm (`engine.compressor = "Gzip"`).
4040

41+
- [ ] Instead of compressing fields of the protobuf message, evaluate the compression of the full stream in the [bitswap network](https://github.com/adlrocha/go-bitswap/blob/d151875a94048c3db59de52b9cb99d0246d74613/network/ipfs_impl.go).
42+
4143
- [ ] Evaluate other compression algorithms (Brotli and gzip are the best alternative, but in case we want to test with other algorithms):
4244

4345
- [ ] [ZDAG](https://github.com/mikeal/zdag) Blocks

0 commit comments

Comments
 (0)