Skip to content

Commit 63204cb

Browse files
pcarranzavpcarranzav
committed
feat: add DIPs (Distributed Indexing Payments) support
This squashed commit adds support for DIPs (Distributed Indexing Payments), which allows indexers to receive indexing fees for indexing subgraphs requested via the DIPs system. Key changes: - Add 'dips' as a new IndexingDecisionBasis enum value - Add indexing agreements model and database support - Add DIPs client for interacting with the gateway DIPs service - Support for matching DIPs agreements with allocations - Allow actions on deployments that are not published yet (for DIPs) - Update allocation management to handle DIPs-based allocations - Add proper handling for cancelled agreements Co-authored-by: Multiple contributors from the DIPs development team
1 parent 30e3682 commit 63204cb

33 files changed

+2353
-89
lines changed

‎REBASE_STRATEGY.md

Lines changed: 232 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,232 @@
1+
# DIPs-Horizon Rebase Strategy
2+
3+
## Overview
4+
This document tracks the merge conflict resolution strategy for rebasing DIPs (Distributed Indexing Payments) onto the Horizon branch.
5+
6+
- **DIPs Branch**: Adds distributed indexing payments functionality
7+
- **Horizon Branch**: Adds Graph Horizon protocol upgrade with GraphTally/RAV v2
8+
9+
## Conflict Files (8 total)
10+
11+
### 1. packages/indexer-common/src/network.ts
12+
**Status**: ❌ Unresolved
13+
14+
**Horizon Changes**:
15+
- Imports `GraphTallyCollector` and `encodeRegistrationData`
16+
- Adds `graphTallyCollector: GraphTallyCollector | undefined` property
17+
- Adds `isHorizon: Eventual<boolean>` property
18+
- Creates GraphTallyCollector instance for RAV v2
19+
20+
**DIPs Changes**:
21+
- Imports `DipsCollector`
22+
- Adds `dipsCollector: DipsCollector | undefined` property
23+
- Adds `queryFeeModels: QueryFeeModels` property
24+
- Adds `managementModels: IndexerManagementModels` property
25+
26+
**Key Conflicts**:
27+
1. Import section (lines 41-46)
28+
2. Class properties (lines 58-72)
29+
3. Constructor parameters (lines 86-92)
30+
4. Constructor body (lines 106-112)
31+
5. Network instantiation (lines 396-399)
32+
33+
**Resolution Strategy**:
34+
- [ ] Merge both collectors (GraphTallyCollector AND DipsCollector)
35+
- [ ] Keep all properties from both branches
36+
- [ ] Update constructor to accept all parameters
37+
- [ ] Ensure both collectors can be initialized
38+
39+
---
40+
41+
### 2. packages/indexer-common/src/operator.ts
42+
**Status**: ❌ Unresolved
43+
44+
**Horizon Changes**:
45+
- `createAllocation` method takes `isHorizon: boolean` parameter
46+
- Uses `isHorizon` to determine `isLegacy` flag on actions
47+
- Passes `isLegacy: !isHorizon` to queueAction
48+
- Also sets `isLegacy: allocation.isLegacy` when closing allocations
49+
50+
**DIPs Changes**:
51+
- `createAllocation` method takes `forceAction: boolean = false` parameter
52+
- `queueAction` method signature changed to `async queueAction(action: ActionItem, forceAction: boolean = false)`
53+
- Passes forceAction as second parameter to queueAction
54+
55+
**Key Conflicts**:
56+
1. createAllocation method signature (line 366-370)
57+
2. queueAction calls - Horizon passes object with isLegacy, DIPs passes forceAction as 2nd param
58+
3. closeEligibleAllocations also has forceAction parameter in DIPs
59+
4. refreshExpiredAllocations has similar conflicts
60+
61+
**Resolution Strategy**:
62+
- [ ] Need both isHorizon AND forceAction parameters in allocation methods
63+
- [ ] Update method signatures: `createAllocation(logger, decision, lastClosed, isHorizon, forceAction = false)`
64+
- [ ] Merge queueAction calls to include both isLegacy (from Horizon) and forceAction (from DIPs)
65+
66+
---
67+
68+
### 3. packages/indexer-common/src/query-fees/models.ts
69+
**Status**: ❌ Unresolved
70+
71+
**Horizon Changes**:
72+
- Uses simpler Model type: `extends Model<ScalarTapReceiptsAttributes>`
73+
- id property is `public id!: number`
74+
75+
**DIPs Changes**:
76+
- Uses Model with creation attributes: `extends Model<ScalarTapReceiptsAttributes, ScalarTapReceiptsCreationAttributes>`
77+
- id property is `public id!: CreationOptional<number>`
78+
79+
**Key Conflicts**:
80+
- Single conflict at line 28-37 in ScalarTapReceipts class definition
81+
82+
**Resolution Strategy**:
83+
- [ ] Use DIPs version (more complete typing with CreationOptional)
84+
85+
---
86+
87+
### 4. packages/indexer-common/package.json
88+
**Status**: ❌ Unresolved
89+
90+
**Horizon Changes**:
91+
- `@graphprotocol/common-ts`: "3.0.1" (newer)
92+
- `@graphprotocol/toolshed`: "0.6.5"
93+
- `@semiotic-labs/tap-contracts-bindings`: "2.0.0" (newer)
94+
95+
**DIPs Changes**:
96+
- `@graphprotocol/common-ts`: "2.0.11" (older)
97+
- `@semiotic-labs/tap-contracts-bindings`: "^1.2.1" (older)
98+
- Adds DIPs-specific dependencies:
99+
- `@bufbuild/protobuf`: "2.2.3"
100+
- `@graphprotocol/dips-proto`: "0.2.2"
101+
- `@grpc/grpc-js`: "^1.12.6"
102+
103+
**Key Conflicts**:
104+
- Dependency version mismatches
105+
106+
**Resolution Strategy**:
107+
- [ ] Use Horizon's newer versions
108+
- [ ] Add DIPs-specific dependencies
109+
110+
---
111+
112+
### 5. packages/indexer-common/src/indexer-management/allocations.ts
113+
**Status**: ❌ Unresolved
114+
115+
**Horizon Changes**:
116+
- Empty constructor body
117+
118+
**DIPs Changes**:
119+
- Constructor initializes DipsManager if dipperEndpoint is configured
120+
- Adds `dipsManager: DipsManager | null` property
121+
122+
**Key Conflicts**:
123+
- Constructor body (lines 131-139)
124+
125+
**Resolution Strategy**:
126+
- [ ] Keep DIPs initialization logic
127+
128+
---
129+
130+
### 6. packages/indexer-common/src/indexer-management/resolvers/allocations.ts
131+
**Status**: ❌ Unresolved
132+
133+
**Horizon Changes**:
134+
- Destructures `graphNode` from resolver context
135+
136+
**DIPs Changes**:
137+
- Destructures `actionManager` from resolver context
138+
139+
**Key Conflicts**:
140+
- reallocateAllocation resolver context destructuring (lines 1720-1724)
141+
142+
**Resolution Strategy**:
143+
- [ ] Include BOTH in destructuring: `{ logger, models, multiNetworks, graphNode, actionManager }`
144+
- [ ] The IndexerManagementResolverContext interface already has both properties
145+
146+
---
147+
148+
### 7. packages/indexer-agent/src/agent.ts
149+
**Status**: ❌ Unresolved
150+
151+
**Horizon Changes**:
152+
- Passes `isHorizon` to createAllocation
153+
154+
**DIPs Changes**:
155+
- Passes `forceAction` to createAllocation
156+
157+
**Key Conflicts**:
158+
- createAllocation call (lines 1243-1247)
159+
160+
**Resolution Strategy**:
161+
- [ ] Pass both parameters: `createAllocation(logger, decision, lastClosed, isHorizon, forceAction)`
162+
163+
---
164+
165+
### 8. yarn.lock
166+
**Status**: ❌ Unresolved
167+
168+
**Resolution Strategy**:
169+
- [ ] Will regenerate after resolving package.json conflicts
170+
171+
---
172+
173+
## General Notes
174+
- Both branches introduce different payment/collection systems that need to coexist
175+
- Horizon introduces protocol upgrade detection and legacy/horizon mode switching
176+
- DIPs introduces indexing agreements and gateway payment integration
177+
178+
## Important Context
179+
- **Current branch**: dips-horizon-rebase
180+
- **Base commit**: Squashed DIPs changes into single commit (35ceac2a) on top of 32d8f174
181+
- **Rebase status**: `git rebase origin/horizon` in progress with conflicts
182+
- **To continue rebase**: After resolving conflicts, use `git add <files>` then `git rebase --continue`
183+
- **To abort**: `git rebase --abort` if needed
184+
185+
## Key Files/Imports Added by Each Branch
186+
**Horizon**:
187+
- `GraphTallyCollector` from './allocations/graph-tally-collector'
188+
- `encodeRegistrationData` from '@graphprotocol/toolshed'
189+
- `isHorizon` property for protocol upgrade detection
190+
- `isLegacy` flag on actions
191+
192+
**DIPs**:
193+
- `DipsCollector` from './indexing-fees/dips'
194+
- `DipsManager` class
195+
- DIPs-specific dependencies in package.json
196+
- `forceAction` parameter for manual allocation management
197+
- New directory: `indexing-fees/` with DIPs implementation
198+
199+
## Important Note: Method Call Analysis
200+
201+
**Call sites found for modified methods:**
202+
- `createAllocation`: Only 1 call in agent.ts (already in conflict)
203+
- `closeEligibleAllocations`: 2 calls in agent.ts (already have forceAction parameter)
204+
- `refreshExpiredAllocations`: 1 call in agent.ts (already has forceAction parameter)
205+
- `queueAction`: 5 calls in operator.ts (all in conflicts)
206+
207+
**Good news**: All method calls appear to be either:
208+
1. Already in merge conflicts (so we'll handle them)
209+
2. Already updated with the DIPs parameters (forceAction)
210+
211+
**Action needed**: When resolving conflicts, ensure we add BOTH parameters where needed.
212+
213+
## Resolution Summary
214+
215+
### High Priority Decisions Needed:
216+
1. **Method Signatures**: Most conflicts are about method parameters. We need both `isHorizon` (from Horizon) AND `forceAction` (from DIPs)
217+
2. **Collectors**: We need both GraphTallyCollector (Horizon) and DipsCollector (DIPs) to coexist
218+
3. **Dependencies**: Use Horizon's newer versions but add DIPs-specific dependencies
219+
220+
### Recommended Approach:
221+
1. Start with package.json - merge dependencies
222+
2. Fix network.ts - ensure both collectors can exist
223+
3. Fix operator.ts - update method signatures to accept both parameters
224+
4. Fix agent.ts - pass both parameters
225+
5. Fix remaining files with minor conflicts
226+
6. Regenerate yarn.lock
227+
228+
### Key Principle:
229+
Both payment systems (Horizon's GraphTally and DIPs) should coexist. The system should support:
230+
- Legacy allocations (pre-Horizon)
231+
- Horizon allocations (with GraphTally/RAV v2)
232+
- DIPs agreements (with distributed indexing payments)

‎docs/action-queue.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,8 +8,8 @@ The action execution worker will only grab items from the action queue to execut
88

99
## Allocation management modes:
1010
- `auto`: The indexer-agent will act similarly to the legacy paradigm. When it identifies allocation actions it will add them to the queue with ActionStatus = `approved`; the execution worker process will pick up the approved actions within 30 seconds and execute them.
11-
- `manual`: The indexer-agent will not add any items to the action queue in this mode. It will spin up an indexer-management server which can be interacted with manually or integrated with 3rd party tools to add actions to the action queue and execute them.
12-
- `oversight`: The indexer-agent will add run its reconciliation loop to make allocation decisions and when actions are identified it will queue them. These actions will then require approval before they can be executed.
11+
- `manual`: The indexer-agent will not add any items to the action queue in this mode. It will spin up an indexer-management server which can be interacted with manually or integrated with 3rd party tools to add actions to the action queue and execute them. An exception to this is indexing agreements (DIPs), for which actions will be queued and executed even in this mode.
12+
- `oversight`: The indexer-agent will add run its reconciliation loop to make allocation decisions and when actions are identified it will queue them. These actions will then require approval before they can be executed. An exception to this is indexing agreements (DIPs), for which actions will be queued as approved and executed even in this mode.
1313

1414
## Actions CLI
1515
The indexer-cli provides an `actions` module for manually working with the action queue. It uses the #Graphql API hosted by the indexer management server to interact with the actions queue.

‎packages/indexer-agent/src/__tests__/indexer.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,7 @@ const setup = async () => {
147147
const network = await Network.create(
148148
logger,
149149
networkSpecification,
150+
models,
150151
queryFeeModels,
151152
graphNode,
152153
metrics,

0 commit comments

Comments
 (0)