doc: add shrex-sub spec #4541

vgonkivs · 2025-09-10T11:54:49Z

Added spec for the shree-sub sub-component

Co-authored-by: Wondertan <[email protected]>

codecov-commenter · 2025-09-10T12:01:04Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (shrex_spec@c6f56e4). Learn more about missing BASE report.

Additional details and impacted files

@@              Coverage Diff              @@
##             shrex_spec    #4541   +/-   ##
=============================================
  Coverage              ?   36.00%           
=============================================
  Files                 ?      304           
  Lines                 ?    20170           
  Branches              ?        0           
=============================================
  Hits                  ?     7263           
  Misses                ?    11957           
  Partials              ?      950

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Wondertan

Rendered link would help even with no diagram
Misses some form of motivation section where we explain the purpose behind this protocol. Yes, it delivers notifications, but what's the point?

Wondertan · 2025-09-10T13:01:01Z

specs/src/shrex/shrex-sub.md

+
+## Abstract
+
+**ShrEx/Sub** (Share Exchange/Subscribe) is a push-based notification protocol implementing the publish-subscribe pattern for Extended Data Square (EDS) hash distribution in the Celestia data availability network. The protocol enables efficient dissemination of new EDS availability notifications across different node types.


I wish we had a place to point to what EDS is

Maybe core specs?

I did not find anything specific. But probably we can use https://github.com/celestiaorg/celestia-app/blob/6ec2d1cab99a130cbfd9a2ddecf5d59e20084fa3/specs/src/data_structures.md the Erasure Coding Section..

I think this is a bit misleading to say that it distributes eds notifications. It is just new blocks being available. Notification mainly carries the height and eds root hash only to verify notification

Wondertan · 2025-09-10T13:06:13Z

specs/src/shrex/shrex-sub.md

+// Publish publishes data hash and height to the topic
+Publish(context, dataHash []byte, height uint64) -> error


Lets aim to be consistent with notations. Discovery detached from golang, even in implementation section, but here we start explaining the protocol directly with interfaces.

I think its fine to use golang in the spec generally, but in the some for of implementation detail section in the end.

I reworked discovery. Please double check. No golang notations anymore.

That's what I mean here by it's not consistent with the discovery spec
I just don't think stepping away from golang is necessary in the implementation section. Up to you tho, as long as it consistent in our specs

I'd leave language agnostic if you are ok with it.

if we are going language agnostic then context is not needed here or in other signature definitions.

Wondertan · 2025-09-10T13:07:36Z

specs/src/shrex/shrex-sub.md

+**Lifecycle Management:**
+
+- All components must be started before message processing
+- Proper shutdown ensures cleanup of resources and connections
+


Is this AI generated?

It was editing small parts and helping me to fix misspelling.

Will remove this section.

walldiss · 2025-09-16T10:29:52Z

specs/src/shrex/shrex-sub.md

+
+## Abstract
+
+**ShrEx/Sub** (Share Exchange/Subscribe) is a push-based notification protocol implementing the publish-subscribe pattern for Extended Data Square (EDS) hash distribution in the Celestia data availability network. The protocol enables efficient dissemination of new EDS availability notifications across different node types.


I think this is a bit misleading to say that it distributes eds notifications. It is just new blocks being available. Notification mainly carries the height and eds root hash only to verify notification

walldiss · 2025-09-16T10:31:23Z

specs/src/shrex/shrex-sub.md

+```text
+Notification {
+    data_hash: bytes[32]  // EDS root hash
+    height: uint64        // Block height
+}
+```


need to specify encoding schema (json / proto )

walldiss · 2025-09-16T10:32:21Z

specs/src/shrex/shrex-sub.md

+- **Message Distribution**: Flood-based (sends to all connected peers)
+- **Overlay**: No mesh topology (unlike GossipSub)


Those two points looks like one. floodsub is no mesh

walldiss · 2025-09-16T10:33:27Z

specs/src/shrex/shrex-sub.md

+**Properties:**
+
+- Messages MUST have a fixed 40-byte payload (32 bytes hash + 8 bytes height)
+- Serialization overhead SHOULD be minimal


Serialisation must be defined in spec

Otherwise its like spec saying to itself that it should have a minimal overhead 😁

walldiss · 2025-09-16T10:34:55Z

specs/src/shrex/shrex-sub.md

+
+- Messages MUST have a fixed 40-byte payload (32 bytes hash + 8 bytes height)
+- Serialization overhead SHOULD be minimal
+- Each EDS at a specific height MUST generate only a single notification


This needs is not a MUST right now. Currently it is ok if some message is not send or duplicates are also acceptable. So for current state of implementation lets replace MUST with SHOULD.

I would also double check grammar here. "generate" might need to be to replaced with "produce" or "send"

produce sgtm

walldiss · 2025-09-16T10:37:54Z

specs/src/shrex/shrex-sub.md

+### Validation Interface
+
+**MessageValidator Interface:**
+
+```text
+// Validate validates a message from a peer
+Validate(context, peerID PeerID, message []byte) -> ValidationResult
+```


Need to add explanation why this is exposed and how it will be used by consumer of Validate (peer-manager)

Wondertan · 2025-09-16T11:50:03Z

Misses some form of motivation section where we explain the purpose behind this protocol. Yes, it delivers notifications, but what's the point?

Motivation section is still missing

specs/src/shrex/shrex-sub.md

renaynay · 2025-09-22T12:10:43Z

specs/src/shrex/shrex-sub.md

+
+- Messages MUST have a fixed 40-byte payload (32 bytes hash + 8 bytes height)
+- Serialization overhead SHOULD be minimal
+- Each EDS at a specific height SHOULD produce only a single notification


why SHOULD here?

Because we don't have any restrictions about that. The message will be ignored in this case and the peer won't be added to the pool.

renaynay · 2025-09-22T12:11:23Z

specs/src/shrex/shrex-sub.md

+// Publish publishes data hash and height to the topic
+Publish(context, dataHash []byte, height uint64) -> error


if we are going language agnostic then context is not needed here or in other signature definitions.

renaynay · 2025-09-22T12:16:01Z

specs/src/shrex/shrex-sub.md

+
+ShrEx/Sub implementations MUST implement the following validation process:
+
+**Format Validation Requirements:**


we need to add here that the header corresponding to the given height + datahash must be fully verified + included as part of implementation's subjective chain in order for notification to be considered valid.

Maybe this was implied by

Height MUST be a valid block height

but we should be more concrete about what that means.

You could optionally also split this up into sanity checks vs verification check to make it more clear.

specs/src/shrex/shrex-sub.md

Co-authored-by: rene <[email protected]>

renaynay · 2025-10-06T11:24:11Z

specs/src/shrex/shrex-sub.md

+
+- `ACCEPT`: Message is valid and MUST be processed
+- `REJECT`: Message is invalid and MUST be discarded
+- `IGNORE`: Message is valid but duplicate/stale and SHOULD be ignored


we should only ignore a message if sender has already been seen for the exact same message.

Peer manager detects duplication. Shrex-sub does not use IGNORE at all. Mistakenly added deduplication here.

renaynay · 2025-10-06T11:26:08Z

specs/src/shrex/shrex-sub.md

+
+### Publisher Behavior (BN)
+
+- **Trigger**: Publishers MUST publish when a new EDS becomes available locally


please distinguish "new EDS becomes available locally" from a syncing BN

publishers should only publish message if subjectively it understands that it is synced up and that this EDS is from a height at the tip of the network

renaynay · 2025-10-06T11:26:50Z

specs/src/shrex/shrex-sub.md

+- **Trigger**: Publishers MUST publish when a new EDS becomes available locally
+- **Action**: Publishers MUST publish EDS hash and height to `/eds-sub/0.0.1` topic
+- **Frequency**: Each EDS MUST be published exactly once per height
+- **Validation**: Publishers MUST only publish hashes for locally validated and available EDS


should we define "available" at top of file?

Removed available as it is already mentioned in Trigger

renaynay · 2025-10-06T11:28:32Z

specs/src/shrex/shrex-sub.md

+- **Subscription**: Subscribers MUST maintain an active subscription to `/eds-sub/0.0.1`
+- **Processing**: Subscribers MUST validate received hash format and height
+- **Action**: Subscribers MUST process notifications through registered listeners
+- **Deduplication**: Subscribers MUST ignore duplicate notifications


need to specify what a duplicate is

also this feels like a SHOULD

removed deduplication section

specs/src/shrex/shrex-sub.md

vgonkivs self-assigned this Sep 10, 2025

vgonkivs requested review from Wondertan and renaynay as code owners September 10, 2025 11:54

vgonkivs added the kind:docs For solely documentation PRs label Sep 10, 2025

vgonkivs requested review from gupadhyaya and walldiss as code owners September 10, 2025 11:54

vgonkivs added the specs label Sep 10, 2025

vgonkivs changed the base branch from main to specs/shwap September 10, 2025 11:58

vgonkivs requested review from adlerjohn and liamsi as code owners September 10, 2025 11:58

vgonkivs changed the base branch from specs/shwap to main September 10, 2025 11:58

vgonkivs changed the base branch from main to shrex_spec September 10, 2025 11:58

doc: add shrex-sub spec

cba3aa9

Co-authored-by: Wondertan <[email protected]>

vgonkivs force-pushed the add_shrex-sub_spec branch from 0af89d1 to cba3aa9 Compare September 10, 2025 12:02

Wondertan reviewed Sep 10, 2025

View reviewed changes

add fixes

292f1de

walldiss reviewed Sep 16, 2025

View reviewed changes

add fixes

8534985

vgonkivs requested review from Wondertan and walldiss September 22, 2025 11:19

renaynay reviewed Sep 22, 2025

View reviewed changes

vgonkivs and others added 3 commits September 24, 2025 14:19

apply suggestion

6ded979

Co-authored-by: rene <[email protected]>

remove context

1f71718

split the verification part

591eb64

renaynay reviewed Oct 6, 2025

View reviewed changes

walldiss approved these changes Oct 23, 2025

View reviewed changes

fixes after review

a5a825a

renaynay approved these changes Oct 28, 2025

View reviewed changes

specs/src/shrex/shrex-sub.md Outdated Show resolved Hide resolved

specs/src/shrex/shrex-sub.md Outdated Show resolved Hide resolved

Merge branch 'shrex_spec' into add_shrex-sub_spec

cd6b5c4

vgonkivs merged commit 76db37c into celestiaorg:shrex_spec Oct 28, 2025
11 of 12 checks passed


		## Abstract

		ShrEx/Sub (Share Exchange/Subscribe) is a push-based notification protocol implementing the publish-subscribe pattern for Extended Data Square (EDS) hash distribution in the Celestia data availability network. The protocol enables efficient dissemination of new EDS availability notifications across different node types.

		// Publish publishes data hash and height to the topic
		Publish(context, dataHash []byte, height uint64) -> error

		- Message Distribution: Flood-based (sends to all connected peers)
		- Overlay: No mesh topology (unlike GossipSub)


		ShrEx/Sub implementations MUST implement the following validation process:

		Format Validation Requirements:


		### Publisher Behavior (BN)

		- Trigger: Publishers MUST publish when a new EDS becomes available locally

doc: add shrex-sub spec #4541

doc: add shrex-sub spec #4541

Uh oh!

Conversation

vgonkivs commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Wondertan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vgonkivs Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Wondertan commented Sep 16, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vgonkivs commented Sep 10, 2025 •

edited

Loading

codecov-commenter commented Sep 10, 2025 •

edited

Loading

vgonkivs Sep 10, 2025 •

edited

Loading