Skip to content

Add SubQuery indexer for pallet-robonomics-cps#411

Closed
Copilot wants to merge 9 commits intorelease/4.1from
copilot/create-subquery-project-pallet-robonomics-cps
Closed

Add SubQuery indexer for pallet-robonomics-cps#411
Copilot wants to merge 9 commits intorelease/4.1from
copilot/create-subquery-project-pallet-robonomics-cps

Conversation

Copy link
Contributor

Copilot AI commented Nov 29, 2025

Implements a production-ready SubQuery indexer for the CPS pallet, exposing hierarchical node structures, ownership data, and audit trails via GraphQL.

Implementation

Event Handlers (5 total)

  • NodeCreated, MetaSet, PayloadSet, NodeMoved, NodeDeleted
  • Statistics tracking (global + daily aggregates)
  • Owner indexing for fast lookups
  • Complete audit trail with old/new value tracking

GraphQL Schema

  • Node: Hierarchical tree with parent-child relationships via auto-generated parentId
  • NodeHistory: Audit log (action, block, timestamp, actor, values)
  • OwnerIndex: O(1) owner → nodes mapping
  • Statistics, DailyStats: Aggregated metrics

Type Safety

  • unwrapOption() helper for Substrate Option handling
  • Global type declarations for SubQuery runtime (logger, store)
  • No unsafe type assertions in production code

Example Queries

Get tree structure with history:

query {
  node(id: "0") {
    id
    owner
    children { id owner }
    history(orderBy: BLOCK_NUMBER_DESC) {
      action
      timestamp
      oldValue
      newValue
    }
  }
}

Get all nodes by owner:

query {
  ownerIndices(filter: { owner: { equalTo: "5Grw..." } }) {
    nodes {
      node { id parentId createdAt isDeleted }
    }
  }
}

Project Structure

subquery-cps-indexer/
├── src/mappings/cpsHandlers.ts    # Event processing
├── src/mappings/utils.ts          # Statistics, timestamps, Option unwrapping
├── schema.graphql                 # 5 entities with relationships
├── project.yaml                   # SubQuery manifest
└── docker-compose.yml             # Local dev stack

Notes

  • MetaSet/PayloadSet events mark updates but don't contain new values (SubQuery limitation)
  • Clients query chain state separately: api.query.cps.nodes(nodeId)
  • TypeScript strict mode disabled (SubQuery codegen incompatibility)
  • 42 unit/integration tests included

Deployment

Local: npm run start:docker → GraphQL at http://localhost:3000
Production: subql publish && subql deployment:deploy

Original prompt

This section details on the original issue you should resolve

<issue_title>SubQuery indexer for pallet-robonomics-cps</issue_title>
<issue_description># SubQuery Integration

Architecture:

Robonomics Node → SubQuery Indexer → PostgreSQL → GraphQL API → UI/Apps

Implementation:

  • Create a SubQuery project specifically for pallet-robonomics-cps
  • Subscribe to all CPS pallet events: NodeCreated, NodeUpdated, NodeMoved, NodeDeleted, MetaSet, PayloadSet, CryptoProfileCreated
  • Build relational schema in PostgreSQL:
    nodes (id, parent_id, owner, meta_type, meta_data, payload_type, payload_data, created_at, updated_at)
    node_history (node_id, action, block_number, timestamp, old_value, new_value)
    crypto_profiles (id, algorithm, public_params)
    node_ownership_index (owner, node_id)
  • Expose GraphQL queries:
    • getTreeByRoot(rootId) - materialized tree structure
    • getNodesByOwner(account) - all nodes for owner
    • getNodeHistory(nodeId) - audit trail
    • searchNodes(filters) - advanced filtering

Benefits:

  • Battle-tested in Polkadot ecosystem
  • Built-in GraphQL API
  • Historical data & audit trails
  • Automatic schema generation
  • Multi-chain support

Example SubQuery Manifest:

specVersion: 1. 0.0
name: robonomics-cps-indexer
version: 1. 0.0
runner:
  node:
    name: '@subql/node-substrate'
    version: '*'
description: 'Indexer for Robonomics CPS Pallet'
repository: 'https://github.com/airalab/robonomics-cps-indexer'
schema:
  file: ./schema.graphql
network:
  chainId: '0x.. .'
  endpoint: 'wss://kusama.rpc.robonomics.network'
dataSources:
  - kind: substrate/Runtime
    startBlock: 1
    mapping:
      file: ./dist/index.js
      handlers:
        - handler: handleNodeCreated
          kind: substrate/EventHandler
          filter:
            module: cps
            method: NodeCreated
        - handler: handleNodeUpdated
          kind: substrate/EventHandler
          filter:
            module: cps
            method: NodeUpdated
```</issue_description>

<agent_instructions>Focus on simple indexer with well documentation and test coverage, including integration tests.</agent_instructions>

## Comments on the Issue (you are @copilot in this section)

<comments>
</comments>

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI and others added 2 commits November 29, 2025 21:12
* Initial plan

* Implement pallet-robonomics-cps with core functionality

Co-authored-by: akru <786394+akru@users.noreply.github.com>

* Fix code review issues: assert_ok usage and weight calculations

Co-authored-by: akru <786394+akru@users.noreply.github.com>

* Change MaxSize to module-level constant and remove id fields from structs

- Changed MaxSize from generic parameter to module-level constant ConstU32<2048>
- Removed id field from Node struct (ID is DB-level only)
- Removed id field from CryptoProfile struct (ID is DB-level only)
- Removed MaxDataSize from Config trait (now using module constant)
- Updated all type signatures to remove MaxSize generic parameter
- All tests pass

Co-authored-by: akru <786394+akru@users.noreply.github.com>

* Cleanup

* Refactor crypto profiles to use enum and add encrypted data tests

- Changed crypto profile from storage-based ID system to CryptoAlgorithm enum
- Removed CryptoProfiles and NextProfileId storage items
- Removed create_crypto_profile extrinsic
- Removed CryptoProfileNotFound and DataTooLarge errors
- Updated NodeData to use CryptoAlgorithm::XChaCha20Poly1305
- Added comprehensive tests for encrypted node data (meta, payload, both)
- Updated README with encryption examples and privacy model
- Fixed README MaxDataSize documentation (2048 bytes)
- Updated benchmarking and weights to remove crypto profile extrinsic
- All 19 tests pass

Co-authored-by: akru <786394+akru@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add delete_node extrinsic, storage version, and security improvements

High Priority:
- Added delete_node extrinsic with proper index cleanup
- Added storage version v1 for future migrations
- Added NodeHasChildren error for delete validation
- Removed NodesByOwner storage (off-chain indexing)
- Changed to Blake2_128Concat hasher for better security

Changes:
- New delete_node extrinsic validates ownership and prevents deletion of nodes with children
- Storage version tracking enables safe future migrations
- Blake2_128Concat provides cryptographic security vs Twox64Concat
- Removed MaxNodesPerOwner config and TooManyNodesPerOwner error
- Added 4 comprehensive delete_node tests (leaf, root, with children, non-owner)
- Updated benchmarking and weights for delete_node
- Updated README with security notes and delete_node documentation

All 23 tests pass

Co-authored-by: akru <786394+akru@users.noreply.github.com>

* Store complete ancestor path in nodes to eliminate loops

Performance improvement:
- Added path field to Node struct storing complete ancestor path from root
- Replaced loop-based is_ancestor() with O(1) path.contains() check
- Replaced loop-based check_tree_depth() with O(1) path.len() check
- Added update_descendant_paths() to recursively update paths after moves

Benefits:
- O(1) cycle detection during node moves
- O(1) depth validation during node creation
- No loop instructions in pallet logic
- Predictable gas costs regardless of tree depth
- Path automatically maintained during create and move operations

Changes:
- Node struct now generic over Config to access MaxTreeDepth
- Path built incrementally during node creation
- Path updated for entire subtree when node is moved
- Replaced is_ancestor test with path_tracking test
- Added move_node_updates_descendant_paths test

All 24 tests pass

Co-authored-by: akru <786394+akru@users.noreply.github.com>

* Add SCALE compact encoding for node paths to reduce storage costs

Storage optimization:
- Implemented CompactNodePath wrapper type with custom SCALE encoding
- Each NodeId in path encoded as Compact<u64> instead of plain u64
- Path length also encoded as compact
- Manual Clone, Debug, Index implementations for CompactNodePath
- Updated all path usage throughout the codebase

Benefits:
- 50-87% storage reduction for paths with small node IDs (< 2^14)
- Typical path with 3 small IDs: ~6 bytes vs 24 bytes
- No performance overhead - O(1) operations remain O(1)
- Maintains all existing functionality

Technical details:
- Added Get trait import for generic bounds
- CompactNodePath<S: Get<u32>> wraps BoundedVec<NodeId, S>
- Custom Encode/Decode implementations use Compact wrapper
- DecodeWithMemTracking derived automatically
- All 24 tests pass

All 24 tests pass

Co-authored-by: akru <786394+akru@users.noreply.github.com>

* Replace CompactNodePath with NodeId newtype for simpler compact encoding

Simplified compact encoding approach:
- Removed custom CompactNodePath wrapper type
- Changed NodeId from type alias to newtype struct with #[codec(compact)] attribute
- BoundedVec<NodeId> automatically uses compact encoding for each element
- Added saturating_add method to NodeId for node ID increments
- Added From<u64> and Into<u64> implementations for NodeId
- Updated all tests to use NodeId(value) constructor

Benefits:
- Cleaner, more idiomatic Rust code
- Leverages built-in SCALE codec compact support
- No manual Encode/Decode implementations needed
- Same storage savings (50-87% for small IDs)
- Better type safety with newtype pattern
- Simpler API - BoundedVec operations work directly

Technical changes:
- Added DecodeWithMemTracking derive to NodeId, CryptoAlgorithm, NodeData, and Node
- Node.path changed from CompactNodePath<T::MaxTreeDepth> to BoundedVec<NodeId, T::MaxTreeDepth>
- All test cases updated to use NodeId constructor
- README updated to reflect newtype approach
- All 24 tests pass

Co-authored-by: akru <786394+akru@users.noreply.github.com>

* Restructure documentation: conceptual README + technical lib.rs docs

Improved documentation structure per @akru feedback:

README.md (Conceptual):
- Added real-world use cases (supply chain, smart city, medical devices)
- Visual tree diagrams with ASCII art
- Conceptual explanations of hierarchy benefits
- Security guarantees and threat model
- Storage efficiency tables with examples
- Integration guide for runtime and dApp developers
- Comparison with alternative approaches
- Roadmap for future features

lib.rs (Technical):
- Comprehensive module-level documentation (120+ lines)
- Architecture overview with storage layout
- Performance characteristics analysis
- Compact encoding details with tables
- Usage examples with code snippets
- Security invariants documentation
- Query patterns and best practices
- Detailed inline docs for all types with examples

Benefits:
- README now targets end-users and integrators
- lib.rs targets developers working on/with the pallet
- Clear separation of concerns
- Better discoverability with examples
- All 24 tests pass

Co-authored-by: akru <786394+akru@users.noreply.github.com>

* Fix benchmarking to use NodeId newtype wrapper consistently

Updated all benchmarking code to use NodeId wrapper:
- Changed raw integer literals (0, 1, 2) to NodeId(0), NodeId(1), NodeId(2)
- Updated extrinsic call parameters to use NodeId wrapper
- Updated storage access calls to use NodeId wrapper
- Updated assertions to use NodeId wrapper for comparisons

This ensures type safety and consistency with the NodeId newtype pattern used throughout the codebase.

All 24 tests pass

Co-authored-by: akru <786394+akru@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: akru <786394+akru@users.noreply.github.com>
Co-authored-by: Alexander Krupenkin <mail@akru.me>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI and others added 7 commits November 29, 2025 18:35
Co-authored-by: akru <786394+akru@users.noreply.github.com>
Co-authored-by: akru <786394+akru@users.noreply.github.com>
Co-authored-by: akru <786394+akru@users.noreply.github.com>
Co-authored-by: akru <786394+akru@users.noreply.github.com>
Co-authored-by: akru <786394+akru@users.noreply.github.com>
Co-authored-by: akru <786394+akru@users.noreply.github.com>
Co-authored-by: akru <786394+akru@users.noreply.github.com>
Copilot AI changed the title [WIP] Add SubQuery indexer for pallet-robonomics-cps Add SubQuery indexer for pallet-robonomics-cps Nov 29, 2025
Copilot AI requested a review from akru November 29, 2025 18:57
Base automatically changed from feat/cps1_0 to release/4.1 January 27, 2026 15:04
@akru
Copy link
Member

akru commented Feb 2, 2026

Used #488

@akru akru closed this Feb 2, 2026
@akru akru deleted the copilot/create-subquery-project-pallet-robonomics-cps branch February 2, 2026 09:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants