GIP-0083: Substreams On The Network. (#63)

* First draft of Substreams On The Network.

* Do NOT suggest enabling rewards.

* Remove artifact

* Move to GIP 0082. Address Pablo's comments.

* Massive improvements to the GIP, clarified what is part of the specs, what is
informational. Linked to documents with payments flow details.

* Add final commitment.

* another update

* 0083 instead

* Added a nudge to GIP-0056 Permissionless Payers.

* More details about pricing, and ECON details.

* Fix yaml issue

* Some updates after ffeedback.

* Rework the attestation, no more DataEdge, a Deployment ID would be best, and
can be allocated and disputed on.

More clarity on the attestation signature payload, including the module_hash
being covered, which would facilitate pinning down culprits.

* Unescessarry.

* Update 0083.md

Added details about network subgraph, fixed a few typos.

* Update 0083.md

Take out arbitration charter modifications.

* Update 0009-arbitration-charter.md

Revert arbitration charter changes.

Author: abourget

gips/0083.md

@@ -0,0 +1,232 @@
+---
+GIP: "0083"
+Title: Make Substreams an official The Graph Network product
+Authors: Alexandre Bourget <[email protected]>
+Created: 2024-10-27
+Updated: 2024-10-27
+Stage: Draft
+Category: Protocol Logic, Protocol Interfaces
+Implementations: https://thegraph.market, https://susbtreams.dev, https://github.com/streamingfast/substreams
+---
+
+# Abstract
+
+This GIP proposes the formal recognition of Substreams as an official product within The Graph Network. Upon approval, this GIP represents the community's consensus to fully integrate Substreams into the network's suite of products, documentation, and marketing materials. While this integration paves the way for future indexing rewards, this GIP does not authorize such rewards.
+
+# Motivation
+
+The Graph ecosystem has successfully developed and deployed Substreams over several years, demonstrating significant performance improvements for indexing, particularly with large datasets and complex computations. However, the lack of formal network approval has limited its full integration and adoption.
+
+Official recognition would:
+1. Increase developer and user trust in the product
+2. Enable confident marketing and go-to-market efforts
+3. Allow the community to fully leverage Substreams' performance benefits
+4. Maintain The Graph's decentralization and security principles
+5. Strengthen The Graph's position as a premier blockchain infrastructure provider
+
+# Non-normative sections
+
+IMPORTANT: Non-normative (or Informative) sections were added for the sake of clarity, providing clearer examples, but are not part of the specification proposed in this GIP. They also do not hold back the value accrued to the Network if this proposal is approved.
+
+# Specification
+
+## Permissionless Discovery
+
+This section discusses how Indexers may be listed on the Payments Gateway UI.
+
+Substreams providers MUST register their service on-chain by publishing a _Substreams Service Deployment_ manifest to IPFS, and [`allocate`ing](https://github.com/graphprotocol/contracts/blob/main/packages/contracts/contracts/staking/Staking.sol#L296) to it through the Staking contract on the Arbitrum chain.
+
+Here is a sample _Substreams Service Deployment_ manifest:
+```yaml
+specVersion: 0.0.5
+description: "Substreams Data Service for MyNetwork"
+service:
+  type: substreams-v1
+  endpoint: https://mynetwork.substreams.example.com
+  network: mynetwork
+  provider:
+    id: my-company
+    name: My Company
+    logo: https://company.example.com/logo.png
+```
+
+The service `substreams-v1` means that the endpoint makes available a gRPC endpoint responding to these methods:
+
+- sf.substreams.rpc.v2.Stream/Block
+- sf.substreams.rpc.v2.EndpointInfo/Info
+
+as specified by https://github.com/streamingfast/substreams/blob/develop/proto/sf/substreams/rpc/v2/service.proto updated from time-to-time in backwards compatible ways.
+
+The Payment Gateway MUST listen to the Arbitrum Staking contract for such registrations (through a Substreams module streamed, a Subgraph like the canonical Network Subgraph or other means), and update its local view of the network.
+
+The Payment Gateway CAN have systems to perform health checks (e.g. on `/health` or `/info` endpoints), or other checks to ensure active, up-to-date and properly configured providers are offered to end users. A Payment Gateway CAN check block height to ensure the backing provider is close to chain head before offering it to users.
+
+
+## Indexer Selection Algorithm (ISA)
+
+To provide consumers with fair access to Substreams providers, an Indexer Selection Algorithm is needed.
+
+Given that there's no gateway involved with Substreams (it's direct point-to-point between consumer and provider), the existing Indexer Selection Algorithm (ISA), which happens at the time of query, cannot be applied for Substreams.  Therefore, a round-robin selection of providers is offered on the front-end of the Payment Gateway (https://thegraph.market), ensuring fair distribution of request loads, while still allowing consumers to choose their provider.
+
+> NON-NORMATIVE: A reason for selecting a provider could be that it has pre-cached data for a given dataset, which would speed up and lower the costs for a consumer.
+
+> NON-NORMATIVE: Future GIPs will propose improved ISA, based on better data and better metrics, which might be desirable to consumers.
+
+## Payments
+
+Payments are handled by the Payment Gateway, which is responsible for collecting payments from consumers and distributing them to indexers.
+
+> INFORMATIVE: The Graph Market acts as the first Payments Gateway. Such a gateway DOES NOT route traffic, but does route payments and receive consumption signals from Indexers.
+
+Such a Payment Gateway is responsible for invoking the `collect` function in the staking contract, thereby transfering funds from Consumers to Indexers.
+
+This builds on work done in [GIP-0056 Permissionless Payers](https://github.com/graphprotocol/graph-improvement-proposals/blob/main/gips/0056-permissionless-payers.md), which opened up an opportunity for such new data services to use the network for payment.
+
+> INFORMATIVE: [TAP](https://docs.rs/tap_core/latest/tap_core/index.html) is not required for Substreams to honor the promises of the protocol. However, as TAP brings augmented trust minimization properties, it would be incorporated in future work.
+
+After collecting consumption signals, the Payment Gateway MUST sum up all what is due to the Indexer, and transfer a payment to the Indexer's wallet, using the `collect` function in the staking contract.
+
+### Current implementation and future opportunities
+
+THIS SECTION IS NON NORMATIVE
+
+An example of the payment flow today, which ends up abiding by this specification (calling `collect`) can be seen here:
+
+https://github.com/streamingfast/network-payments-cli
+
+It involves a small dance, of publishing a manifest similar to:
+
+```yaml
+specVersion: 0.0.5
+description: "thegraph.market Payment Gateway Usage"
+usage:
+  serviceType: substreams-v1
+  network: mainnet
+  nonce: some-uuid
+```
+
+then allocating to it, sharing the allocation ID with the Payment Gateway, in order for the Indexer to receive payment. Mind you, this isn't the same thing as the _Substreams Service Deployment_ manifest, as this one is short-lived as to not be exposed to the curation fee market.
+
+This flow is expected to be simplified in the future, with the Payment Gateway being able to automatically detect new Substreams services, and allocate to them, without manual intervention.
+
+Future opportunities would include self-serve APIs for Indexers to claim payment and do the dance automatically. Thus an Indexer would be able to lower the risks of not receiving payments from the Gateway.
+
+Eventually, TAP could be used to send payments alongside blockchain data payloads (BlockScopedData), or some other trust-minimized payment mechanism.
+
+
+## Pricing
+
+Pricing is currently outside of the scope of this proposal. The protocol (contracts) itself does not provide pricing mechanisms, so this responsibility is passed to the Payment Gateway.
+
+The Payment Gateway MUST transfer funds to the Indexer according to a Pricing Period. Ideally this period is the shortest possible, to reduce counterparty risks.
+
+
+### Current implementation and future opportunities
+
+THIS SECTION IS NON-NORMATIVE
+
+Current pricing is set at a fixed price, defined on The Graph Market (https://thegraph.market). It has been measured by the Core Devs to have unit economics that makes sense for operators, and will be revised as the system matures.
+
+Consumption is measured in terms of Terabytes read from the backing stores and/or Egress bytes. Fees are cumulated during a period of time, until payment is settled. This time period will start at 1 month and be adjusted to become lower and lower as the system matures, to address counterparty risks.
+
+Although initial integration starts with a simple pricing strategy, the integration of Substreams on The Graph Network by no means limits the future opportunities to adjust and augment the business models and pricing strategies.
+
+Future GIPs will be proposed to enable more complex pricing strategies, such as pay-per-cpu-time, pay-per-storage, pay-per-query, pay-per-block, pay-per-attestation, etc.
+
+### Curation
+
+THIS SECTION IS NON-NORMATIVE
+
+Curation can happen on the published Subgraph manifest above, although a nonce may be added to avoid paying curation fees. Because there are no indexer rewards, curation is not adding value, therefore it is reasonable for participants to want to avoid it.
+
+
+
+## Economic Security and Disputes
+
+Economic security is achieved through slashing and disputes, similar to subgraphs as of Jan 24th 2025. Indexers providing incorrect data CAN have their stake slashed. Therefore, to be an Indexer recognized by the Payment Gateway, one must staked the Network token, just like for subgraphs.
+
+The means are similar to subgraphs, by providing signed attestations, attached to allocations, and having an Arbiter validate those attestations, and slash the Indexer's stake if necessary.
+
+### Attestations
+
+Having no queries like subgraphs have, Proofs of Indexing for Substreams are always null.
+
+> [!NOTE]
+> This is why enabling indexing rewards for Substreams is not part of this GIP. Future GIPs, potentially with Horizon, will propose a mechanism to reward Indexers for providing Substreams services.
+
+Substreams endpoints provide signed attestations with each block's response, as per [the `attestation` field](https://github.com/streamingfast/substreams/blob/develop/proto/sf/substreams/rpc/v2/service.proto#L96) in the `BlockScopedData` response, matching the operator key specified in [the `attestation_public_key` field](https://github.com/streamingfast/substreams/blob/develop/proto/sf/substreams/rpc/v2/service.proto#L106) of the `SessionInit` response.
+
+The payload signed over is composed of:
+- the Substreams output module's payload, hashed with SHA256 (32 bytes), followed by:
+- the 'b' character, followed by:
+- the block ID as UTF-8 encoded string, followed by:
+- the 'm' character, followed by:
+- the module hash for the top-level Substreams module being attested (32 bytes)
+
+The attestation payload MUST be signed by a special _attestation key_, derived from the operator's key, so that disputes can be opened on the _Substreams Service Deployment ID_.
+
+A valid _attestation key_, attached to such an allocation MUST be verified by the Payment Gateway, before giving the assurance to Consumers that the Indexer can be slashed for misbehaving.
+
+An Arbiter MUST be able to validate payloads, through multiple providers and investigation, and after judgement, be able to slash the Indexer's stake, using the information provided in those payloads via an on-chain allocations.
+
+> INFORMATIVE: The module hash covers everything needed for deterministic execution, and nothing more. It is easily computed by all known compliant Substreams library.
+
+
+#### Methods of analysis
+
+THIS SECTION IS NON-NORMATIVE
+
+This section specifies potential methods an Arbiter can use to analyze disputes. Other more optimal tools could be written, and are out of scope of this specification.
+
+**Detecting Faults:** Discrepancies in data returned to a Fisherman or a Consumer, by different providers for the same stream and block, is material to open an investigation.
+
+The Arbiter can use the `substreams gui` tool or the `substreams run --show-attestations` tool to inspect attestations and contents, and compare stream data at specific block ranges, verifying data integrity and determinism.
+
+`substreams tools` will contain a way to validate those attestations, and help with investigation.  This tool will take the content being disputed (BlockScopedData) from the replay.log, which contain the attestations over the content within, and validate their output, in order to compare against providers, and end up slashing an Indexer.
+
+```
+substreams tools validate-attestation --indexer-address 0x123123123 --attestation eth:120398102398102938109283012983 --replay-log-file ./path/to/replay.log
+```
+
+This command will process all messages in the provided replay log and validate their attestations. Replay logs can be generated using  `substreams gui --with-replay`.
+
+The Payment Gateway can keep an in-memory mapping will link the operator key to the staking key.
+
+- The `setOperator` function ([Staking.sol#L226](https://github.com/graphprotocol/contracts/blob/ce3ec16484dacc89a1b9cf08455256be830790e3/packages/contracts/contracts/staking/Staking.sol#L226)) links the operator key to the staking key and emits the `SetOperator` event ([IStakingBase.sol#L99](https://github.com/graphprotocol/contracts/blob/main/packages/contracts/contracts/staking/IStakingBase.sol#L99)).
+
+Such a system can hydrate from the Network Subgraph.
+
+
+### Tools
+
+THIS SECTION IS NON-NORMATIVE. Different tools can exist to do those operations. We are listing some for informational purposes here.
+
+For Indexers, in order to register a new Substreams provider on-chain, and show up on the Payment Gateway, you can use commands such as:
+
+```
+substreams network register --operator-wallet 0x123123123 --operator-priv-key-file my.key --service substreams-v1 --endpoint mainnet.eth.example.com --provider-logo https://www.example.com/static/mylogo.png --provider-id example-company --provider-name "Example Company" --network ethereum
+```
+
+This command allows providers to register their service, specifying their operator wallet, private key file, the service type (`substreams-v1`), the endpoint URL, logo, name, and the supported network. The registration data is signed by the operator's key.  An update to an existing registration can be achieved by re-running the command with updated parameters, based on the `service`, `providerId` and `network` tripled.  A corresponding `unregister` or `revoke` command would be added to allow providers to take down their service from the front-end.
+
+
+# Dependencies & Backwards Compatibility
+
+The only dependency for this GIP is the ratification of the changes to GIP-0009 as proposed in this Pull Request.
+
+Most of the elements described above are already rolled out and currently work.
+
+To account for signed attestation, some backwards compatible fields were be added to the Substreams requests and responses.  These fields should not affect current Substreams rollout, but will be necessary for those wanting to join the Network.
+
+
+# Risks and Security Considerations
+
+Risks associated with data integrity and malicious actors will be mitigated through signed attestations, the arbitration process, and economic security enforced via slashing.
+
+There are some centralization risks in the Payment Gateway until TAP is integrated. For instance, Indexers need to trust the Gateway for payment. In the same vein, there is some counterparty risk for Indexers, as they have to trust that the Payment Gateway _will pay_ - this can be mitigated by requesting payment early and often.
+
+No other risks have been identified during the consultation period.
+
+# Copyright Waiver
+
+Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).