fix(hermes): New implementation of IPFS get and add file #750

bkioshn · 2026-01-06T07:28:04Z

Description

Apply new implementation of add_ipfs_file and get_ipfs_file

Related Issue(s)

#736

Description of Changes

Properly convert CID bytes to readable CID
Store data in the document when successfully POST
compute_cid expected CBOR document encoded
Change doc_sync db schema CID as TEXT not BLOB
Properly handle module when sending event
Remove all CID conversion (remove HACK)
Apply change made to add_ipfs_file and get_ipfs_file, add GetFileWithProviders

Related Pull Requests

input-output-hk/catalyst-libs#763

Please confirm the following checks

My code follows the style guidelines of this project
I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
Any dependent changes have been merged and published in downstream module

Signed-off-by: bkioshn <[email protected]>

hermes/bin/src/ipfs/api.rs

hermes/bin/src/runtime_extensions/hermes/doc_sync/host.rs

Signed-off-by: bkioshn <[email protected]>

cong-or

LGTM

…-block Signed-off-by: bkioshn <[email protected]>

* fix(hermes): doc-sync P2P message format and subscription conflicts WHAT THIS FIXES: - CID format: Use dag-cbor codec (0x51) instead of dag-pb (0x70) for protocol compliance - Message encoding: Publish CBOR-encoded payload::New with CID lists instead of raw document bytes - Subscription conflict: Remove auto-subscription that prevented doc-sync module from subscribing CHANGES: - hermes/bin/src/runtime_extensions/hermes/doc_sync/host.rs: * Import minicbor crate directly (not via cardano_chain_follower) * Compute both dag-pb CID (storage) and dag-cbor CID (protocol) in add_file() * Construct proper payload::New structure in publish() * Encode to CBOR before publishing to PubSub - hermes/bin/src/ipfs/mod.rs: * Remove auto-subscription to documents.new during bootstrap * Prevents subscription kind conflicts with doc-sync module - hermes/bin/src/ipfs/task.rs: * Add extensive documentation of blocking operations issue * Document root cause and recommended async solution REMAINING ISSUE: P2P propagation still fails due to blocking operations in async context. The doc_sync_topic_message_handler (added in 944360f) calls file_pin() and file_get() which use blocking_send()/blocking_recv(). This causes deadlock when called from async PubSub handler. NEXT STEPS: Convert file_pin() and file_get() to async versions. See detailed implementation guide in task.rs comments (lines 354-374). TEST: cd p2p-testing && just test-pubsub-propagation Currently FAILS with No Propagation due to blocking operations issue. Will PASS once async conversion is complete. Related: #691 (introduced blocking operations bug) * fix(hermes): make doc-sync PubSub handler non-blocking with async file operations The doc_sync_topic_message_handler was blocking the PubSub event loop by using blocking_send/blocking_recv for IPFS file operations, preventing new messages from being processed while fetching content. Changes: - Add file_get_async() and file_pin_async() methods in ipfs/mod.rs that use send().await instead of blocking_send() - Update doc_sync_topic_message_handler() in ipfs/task.rs to spawn async task with tokio::spawn() for file operations - Add RECEIVED PubSub message with CID logging for test detection - Fix test-pubsub-propagation in p2p-testing/justfile to properly detect message reception using wc -l instead of grep -q Result: - PubSub handler no longer blocks - can process new messages while previous content is still being fetched - test-pubsub-propagation now passes with 100% success rate (5/5 nodes) - Resolves doc-sync P2P message propagation issue * fix(hermes): make doc-sync PubSub handler non-blocking with async file operations The doc_sync_topic_message_handler was blocking the PubSub event loop by using blocking_send/blocking_recv for IPFS file operations, preventing new messages from being processed while fetching content. Changes: - Add file_get_async() and file_pin_async() methods in ipfs/mod.rs that use send().await instead of blocking_send() - Update doc_sync_topic_message_handler() in ipfs/task.rs to spawn async task with tokio::spawn() for file operations - Add RECEIVED PubSub message with CID logging for test detection - Fix test-pubsub-propagation in p2p-testing/justfile to properly detect message reception using wc -l instead of grep -q Result: - PubSub handler no longer blocks - can process new messages while previous content is still being fetched - test-pubsub-propagation now passes with 100% success rate (5/5 nodes) - Resolves doc-sync P2P message propagation issue * fix(hermes): make doc-sync PubSub handler non-blocking with async file operations The doc_sync_topic_message_handler was blocking the PubSub event loop by using blocking_send/blocking_recv for IPFS file operations, preventing new messages from being processed while fetching content. Changes: - Add file_get_async() and file_pin_async() methods in ipfs/mod.rs that use send().await instead of blocking_send() - Update doc_sync_topic_message_handler() in ipfs/task.rs to spawn async task with tokio::spawn() for file operations - Add RECEIVED PubSub message with CID logging for test detection - Fix test-pubsub-propagation in p2p-testing/justfile to properly detect message reception using wc -l instead of grep -q Result: - PubSub handler no longer blocks - can process new messages while previous content is still being fetched - test-pubsub-propagation now passes with 100% success rate (5/5 nodes) - Resolves doc-sync P2P message propagation issue * fix(hermes): make doc-sync PubSub handler non-blocking with async file operations The doc_sync_topic_message_handler was blocking the PubSub event loop by using blocking_send/blocking_recv for IPFS file operations, preventing new messages from being processed while fetching content. Changes: - Add file_get_async() and file_pin_async() methods in ipfs/mod.rs that use send().await instead of blocking_send() - Update doc_sync_topic_message_handler() in ipfs/task.rs to spawn async task with tokio::spawn() for file operations - Add RECEIVED PubSub message with CID logging for test detection - Fix test-pubsub-propagation in p2p-testing/justfile to properly detect message reception using wc -l instead of grep -q Result: - PubSub handler no longer blocks - can process new messages while previous content is still being fetched - test-pubsub-propagation now passes with 100% success rate (5/5 nodes) - Resolves doc-sync P2P message propagation issue * fix(hermes): make doc-sync PubSub handler non-blocking with async file operations The doc_sync_topic_message_handler was blocking the PubSub event loop by using blocking_send/blocking_recv for IPFS file operations, preventing new messages from being processed while fetching content. Changes: - Add file_get_async() and file_pin_async() methods in ipfs/mod.rs that use send().await instead of blocking_send() - Update doc_sync_topic_message_handler() in ipfs/task.rs to spawn async task with tokio::spawn() for file operations - Add RECEIVED PubSub message with CID logging for test detection - Fix test-pubsub-propagation in p2p-testing/justfile to properly detect message reception using wc -l instead of grep -q Result: - PubSub handler no longer blocks - can process new messages while previous content is still being fetched - test-pubsub-propagation now passes with 100% success rate (5/5 nodes) - Resolves doc-sync P2P message propagation issue * use type CID instead of string Signed-off-by: bkioshn <[email protected]> * small cleanup Signed-off-by: bkioshn <[email protected]> * reverts commit 7043008 Signed-off-by: bkioshn <[email protected]> * change cid string to CID Signed-off-by: bkioshn <[email protected]> * change ipfs-cid type string to bstr Signed-off-by: bkioshn <[email protected]> * revert change Signed-off-by: bkioshn <[email protected]> * fix linter Signed-off-by: bkioshn <[email protected]> * cleanup task.rs and add comment about unixfs Signed-off-by: bkioshn <[email protected]> * cleanup Signed-off-by: bkioshn <[email protected]> * hack, add conversion Signed-off-by: bkioshn <[email protected]> * fix node 3 connection + log capture Signed-off-by: bkioshn <[email protected]> * add log necessary for test Signed-off-by: bkioshn <[email protected]> * revert peer docker config Signed-off-by: bkioshn <[email protected]> * fix linter Signed-off-by: bkioshn <[email protected]> * filter non-empty cid array Signed-off-by: bkioshn <[email protected]> * fix(hermes): New implementation of IPFS get and add file (#750) * use new implementation of add and get ipfs file Signed-off-by: bkioshn <[email protected]> * hermes-ipfs points to branch Signed-off-by: bkioshn <[email protected]> * add error to hermes-ipfs wit Signed-off-by: bkioshn <[email protected]> * move cbor encode logic Signed-off-by: bkioshn <[email protected]> * fix get_ipfs_file Signed-off-by: bkioshn <[email protected]> * fix syntax and format Signed-off-by: bkioshn <[email protected]> * update hermes-ipfs to 0.0.11 Signed-off-by: bkioshn <[email protected]> * update function Signed-off-by: bkioshn <[email protected]> * handle store documenet to db after success post Signed-off-by: bkioshn <[email protected]> * handle ipfs/doc-sync module id Signed-off-by: bkioshn <[email protected]> * change cid to string Signed-off-by: bkioshn <[email protected]> * minor fix Signed-off-by: bkioshn <[email protected]> * fix compute cid Signed-off-by: bkioshn <[email protected]> * fix get file and add file Signed-off-by: bkioshn <[email protected]> * event module all Signed-off-by: bkioshn <[email protected]> * handle event module Signed-off-by: bkioshn <[email protected]> * fix doc data encode Signed-off-by: bkioshn <[email protected]> * fix linter and format Signed-off-by: bkioshn <[email protected]> * minor fix Signed-off-by: bkioshn <[email protected]> --------- Signed-off-by: bkioshn <[email protected]> * fix linter and format Signed-off-by: bkioshn <[email protected]> * fix channel name Signed-off-by: bkioshn <[email protected]> * remove unused err Signed-off-by: bkioshn <[email protected]> * fix spelling Signed-off-by: bkioshn <[email protected]> * Bump CI * Update `deny.toml` * fix event-on-new-doc Signed-off-by: bkioshn <[email protected]> * bump hermes-ipfs to v0.0.11 Signed-off-by: bkioshn <[email protected]> * fix linter and format Signed-off-by: bkioshn <[email protected]> --------- Signed-off-by: bkioshn <[email protected]> Co-authored-by: cong-or <[email protected]> Co-authored-by: rafal-ch <[email protected]>

bkioshn added 3 commits January 6, 2026 14:18

use new implementation of add and get ipfs file

1d950e9

Signed-off-by: bkioshn <[email protected]>

hermes-ipfs points to branch

a67f4fc

Signed-off-by: bkioshn <[email protected]>

add error to hermes-ipfs wit

4f5f23a

Signed-off-by: bkioshn <[email protected]>

bkioshn self-assigned this Jan 6, 2026

bkioshn added this to Catalyst Jan 6, 2026

bkioshn added the squad: hermetics Hermes Backend, System Development & Integration Team label Jan 6, 2026

github-project-automation bot moved this to New in Catalyst Jan 6, 2026

bkioshn added 3 commits January 6, 2026 14:37

move cbor encode logic

6137f36

Signed-off-by: bkioshn <[email protected]>

fix get_ipfs_file

cca69ed

Signed-off-by: bkioshn <[email protected]>

fix syntax and format

184f34b

Signed-off-by: bkioshn <[email protected]>

bkioshn mentioned this pull request Jan 7, 2026

fix(rust/hermes-ipfs): Fix add_ipfs_file and get_ipfs_file input-output-hk/catalyst-libs#763

Closed

8 tasks

bkioshn marked this pull request as ready for review January 7, 2026 08:51

bkioshn requested review from aido-mth, cong-or, no30bit, rafal-ch and stevenj as code owners January 7, 2026 08:51

bkioshn moved this from New to 👀 In review in Catalyst Jan 7, 2026

update hermes-ipfs to 0.0.11

d065c49

Signed-off-by: bkioshn <[email protected]>

rafal-ch reviewed Jan 7, 2026

View reviewed changes

hermes/bin/src/ipfs/api.rs Outdated Show resolved Hide resolved

hermes/bin/src/runtime_extensions/hermes/doc_sync/host.rs Outdated Show resolved Hide resolved

bkioshn added 10 commits January 8, 2026 15:14

update function

94f56d7

Signed-off-by: bkioshn <[email protected]>

handle store documenet to db after success post

31ca41d

Signed-off-by: bkioshn <[email protected]>

handle ipfs/doc-sync module id

75f6ae6

Signed-off-by: bkioshn <[email protected]>

change cid to string

17eaf48

Signed-off-by: bkioshn <[email protected]>

minor fix

d8a6da1

Signed-off-by: bkioshn <[email protected]>

fix compute cid

d53dcff

Signed-off-by: bkioshn <[email protected]>

fix get file and add file

ff10658

Signed-off-by: bkioshn <[email protected]>

event module all

ba5f781

Signed-off-by: bkioshn <[email protected]>

handle event module

ef0d92e

Signed-off-by: bkioshn <[email protected]>

fix doc data encode

36abf43

Signed-off-by: bkioshn <[email protected]>

bkioshn added 2 commits January 14, 2026 13:45

fix linter and format

8b436c0

Signed-off-by: bkioshn <[email protected]>

minor fix

59f38a9

Signed-off-by: bkioshn <[email protected]>

cong-or approved these changes Jan 14, 2026

View reviewed changes

Merge branch 'fix/doc-sync-p2p-propagation' into fix/736/ipfs-get-add…

56e44ff

…-block Signed-off-by: bkioshn <[email protected]>

bkioshn merged commit 77e5a88 into fix/doc-sync-p2p-propagation Jan 15, 2026
16 of 19 checks passed

bkioshn deleted the fix/736/ipfs-get-add-block branch January 15, 2026 06:37

github-project-automation bot moved this from 👀 In review to ✅ Done in Catalyst Jan 15, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(hermes): New implementation of IPFS get and add file #750

fix(hermes): New implementation of IPFS get and add file #750

Uh oh!

bkioshn commented Jan 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

cong-or left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fix(hermes): New implementation of IPFS get and add file #750

fix(hermes): New implementation of IPFS get and add file #750

Uh oh!

Conversation

bkioshn commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

Description of Changes

Related Pull Requests

Please confirm the following checks

Uh oh!

Uh oh!

Uh oh!

cong-or left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

bkioshn commented Jan 6, 2026 •

edited

Loading