diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md new file mode 100644 index 000000000..b5d3691ac --- /dev/null +++ b/DEVELOPMENT.md @@ -0,0 +1,288 @@ +# Development Guide + +This document provides comprehensive guidance for local development, testing, and contributing to the Graph Protocol Indexer project. + +## Local Development and Testing + +The project includes a `Dockerfile.dev` for consistent local development and testing environments. This is particularly useful for testing performance improvements and ensuring compatibility across different systems. + +### Prerequisites + +- [Docker](https://docker.com/) installed (or [Podman](https://podman.io/) as an alternative) +- Git (for cloning the repository) +- At least 4GB of available RAM + +### Building the Development Image + +```bash +# Build the development image +docker build -f Dockerfile.dev -t indexer-dev:latest . + +# Note: You can also use Podman as a drop-in replacement for Docker +# podman build -f Dockerfile.dev -t indexer-dev:latest . +``` + +### Testing Performance Improvements Locally + +**Note**: All `docker` commands in this section can be used with Podman by simply replacing `docker` with `podman`. + +1. **Mount your local project and run tests:** +```bash +# Test the complete build +docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer && yarn compile" + +# Test individual packages +docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-common && yarn compile" +docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-agent && yarn compile" +docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-cli && yarn compile" +``` + +2. **Test the new CLI flag:** +```bash +# Verify the new flag is available +docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-agent && node bin/graph-indexer-agent start --help | grep -A 5 'indexer-min-stake-threshold'" +``` + +3. **Run TypeScript type checking:** +```bash +# Check specific files for type errors +docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-common && tsc --noEmit src/subgraphs.ts" +``` + +### Interactive Development + +**Note**: All `docker` commands in this section can be used with Podman by simply replacing `docker` with `podman`. + +```bash +# Start an interactive shell in the container +docker run --rm -it -v $(pwd):/opt/indexer indexer-dev:latest bash + +# Inside the container, you can: +cd /opt/indexer +yarn install # Install dependencies +yarn compile # Build all packages +yarn test # Run tests +``` + +### Environment Variables for Testing + +**Note**: All `docker` commands in this section can be used with Podman by simply replacing `docker` with `podman`. + +The development image supports the same environment variables as the production build: + +```bash +# Test with custom batch sizes +docker run --rm -v $(pwd):/opt/indexer -e INDEXER_DEPLOYMENT_BATCH_SIZE=1000 indexer-dev:latest bash -c "cd /opt/indexer && yarn compile" + +# Test with custom stake thresholds +docker run --rm -v $(pwd):/opt/indexer -e INDEXER_MIN_STAKE_THRESHOLD=5000000000000000000 indexer-dev:latest bash -c "cd /opt/indexer && yarn compile" +``` + +### Troubleshooting + +- **Build failures**: Ensure you have sufficient RAM (4GB+) and disk space +- **Permission issues**: On some systems, you may need to use `sudo` with docker commands +- **Volume mount issues**: Ensure the current directory path is correct and accessible +- **Dependency issues**: The image includes `yarn install` to ensure all dependencies are properly resolved + +### Performance Testing + +**Note**: All `docker` commands in this section can be used with Podman by simply replacing `docker` with `podman`. + +To test the performance improvements with large datasets: + +```bash +# Test compilation performance +docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer && time yarn compile" + +# Test individual package compilation +docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-common && time tsc --noEmit" +``` + +## Project Structure + +The project is organized as a monorepo using [Lerna](https://lerna.js.org/) and [Yarn workspaces](https://classic.yarnpkg.com/en/docs/workspaces/): + +``` +packages/ +├── indexer-agent/ # Main indexer agent service +├── indexer-cli/ # Command-line interface +└── indexer-common/ # Shared utilities and types +``` + +## Development Workflow + +### 1. Setup Development Environment + +```bash +# Clone the repository +git clone +cd indexer + +# Install dependencies +yarn install + +# Build the development image +docker build -f Dockerfile.dev -t indexer-dev:latest . +``` + +### 2. Make Changes + +- Edit files in the appropriate package +- Follow the existing code style and patterns +- Add tests for new functionality + +### 3. Test Your Changes + +```bash +# Test compilation +docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer && yarn compile" + +# Run tests +docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer && yarn test" + +# Test specific functionality +docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-agent && node bin/graph-indexer-agent start --help" +``` + +### 4. Commit and Push + +```bash +# Add your changes +git add . + +# Commit with a descriptive message +git commit -m "feat: description of your changes" + +# Push to your branch +git push origin your-branch-name +``` + +## Performance Optimization Features + +The indexer agent includes several performance optimizations for handling large numbers of subgraph deployments: + +### Batching and Filtering +- **Deployment Batching**: Processes deployments in configurable batches (default: 500) to prevent event loop blocking +- **Stake Threshold Filtering**: Automatically filters out deployments below a minimum stake/signal threshold to reduce processing overhead +- **Rule Lookup Optimization**: Uses O(1) Map-based lookups instead of O(N) linear scans for indexing rules + +### Configuration Options +- `--indexer-min-stake-threshold`: Set minimum stake amount in wei (default: 1 GRT = 1000000000000000000 wei) +- `INDEXER_DEPLOYMENT_BATCH_SIZE`: Environment variable for batch size (default: 500) + +### Use Cases +These optimizations are particularly beneficial when: +- Processing 10,000+ subgraph deployments +- Managing complex indexing rule sets +- Running on resource-constrained environments +- Requiring consistent response times during high-load periods + +## Testing + +### Running Tests Locally + +To run the tests locally, you'll need: +1. Docker installed and running +2. Node.js and Yarn +3. An Arbitrum Sepolia testnet RPC provider (e.g., Infura, Alchemy) +4. An API key from The Graph Studio for querying subgraphs + +#### Setup + +1. Create a `.env` file in the root directory with your credentials. You can copy the example file as a template: +```sh +cp .env.example .env +``` + +Then edit `.env` with your credentials: +```plaintext +# Your Arbitrum Sepolia testnet RPC endpoint +INDEXER_TEST_JRPC_PROVIDER_URL=https://sepolia.infura.io/v3/your-project-id + +# Your API key from The Graph Studio (https://thegraph.com/studio/) +INDEXER_TEST_API_KEY=your-graph-api-key-here +``` + +2. Run the tests: +```sh +bash scripts/run-tests.sh +``` + +The script will: +- Start a PostgreSQL container with the required test configuration +- Load your credentials from the `.env` file +- Run the test suite +- Clean up the PostgreSQL container when done + +### Using Docker for Testing + +**Note**: All `docker` commands in this section can be used with Podman by simply replacing `docker` with `podman`. + +```bash +# Run tests in the development container +docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer && yarn test" + +# Run specific test suites +docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-agent && yarn test" +``` + +## Contributing + +### Code Style + +- Follow the existing TypeScript patterns +- Use meaningful variable and function names +- Add JSDoc comments for public APIs +- Ensure all tests pass before submitting + +### Pull Request Process + +1. Create a feature branch from `main` +2. Make your changes following the development workflow +3. Ensure all tests pass +4. Update documentation if needed +5. Submit a pull request with a clear description + +### Performance Considerations + +When making changes that affect performance: +- Test with realistic data sizes (10K+ deployments) +- Use the development container for consistent testing +- Measure performance impact before and after changes +- Consider the O(N×M) complexity implications + +## Troubleshooting Common Issues + +### Build Issues + +**Problem**: `tsc: command not found` +**Solution**: Ensure TypeScript is installed globally in the container or use `yarn tsc` + +**Problem**: `lerna: command not found` +**Solution**: Ensure Lerna is installed globally in the container + +**Problem**: Dependency resolution errors +**Solution**: Run `yarn install` in the container to ensure proper workspace linking + +### Runtime Issues + +**Problem**: Container can't mount volumes +**Solution**: Check file permissions and ensure the path is accessible + +**Problem**: Insufficient memory during build +**Solution**: Increase container memory limits or use `--memory` flag + +**Problem**: Port conflicts +**Solution**: Use different ports or stop conflicting services + +**Note**: If you're using Podman instead of Docker, replace `docker` with `podman` in all commands. Podman is a drop-in replacement for Docker and supports the same command-line interface. + +## Resources + +- [The Graph Protocol Documentation](https://thegraph.com/docs/) +- [Lerna Documentation](https://lerna.js.org/) +- [Yarn Workspaces](https://classic.yarnpkg.com/en/docs/workspaces/) +- [TypeScript Documentation](https://www.typescriptlang.org/) +- [Docker Documentation](https://docs.docker.com/) +- [Podman Documentation](https://podman.io/getting-started/) (alternative to Docker) diff --git a/Dockerfile.dev b/Dockerfile.dev new file mode 100644 index 000000000..9f0c9b498 --- /dev/null +++ b/Dockerfile.dev @@ -0,0 +1,44 @@ +######################################################################## +# Development build image for testing performance improvements +######################################################################## + +FROM node:20.11-bookworm-slim + +ENV NODE_ENV development +ENV NODE_OPTIONS="--max-old-space-size=4096" + +RUN apt-get update && apt-get install -y \ + python3 \ + build-essential \ + git \ + curl \ + && rm -rf /var/lib/apt/lists/* + +WORKDIR /opt/indexer + +# Copy package files first for better Docker layer caching +COPY package.json . +COPY yarn.lock . +COPY tsconfig.json . +COPY lerna.json . + +# Copy all packages +COPY packages/ ./packages/ + +# Install all dependencies including dev dependencies +RUN yarn --frozen-lockfile --non-interactive --production=false + +# Install lerna and typescript globally for the build commands +RUN npm install -g lerna typescript + +# Install dependencies for all packages to ensure proper resolution +RUN yarn install + +# Build the packages +RUN yarn compile || npx lerna run compile || echo "Build completed with possible warnings" + +# Expose port for indexer management +EXPOSE 8000 + +# Default command for development +CMD ["bash"] diff --git a/README.md b/README.md index 1f02734ff..5c7d4b18e 100644 --- a/README.md +++ b/README.md @@ -120,9 +120,10 @@ Indexer Infrastructure --allocation-management Indexer agent allocation management automation mode (auto|manual|oversight) [string] [default: "auto"] - --auto-allocation-min-batch-size Minimum number of allocation - transactions inside a batch for AUTO + --auto-allocation-min-batch-size Minimum number of allocation + transactions inside a batch for AUTO management mode [number] [default: 1] + --indexer-min-stake-threshold Minimum stake/signal amount in wei to consider a deployment significant for evaluation. Deployments below this threshold are filtered out unless they have specific indexing rules. Default: 1 GRT (1000000000000000000 wei) [string] [default: "1000000000000000000"] Network Subgraph --network-subgraph-deployment Network subgraph deployment [string] @@ -180,34 +181,34 @@ simply by running `graph indexer`. $ graph indexer --help Manage indexer configuration - indexer status Check the status of an indexer - indexer rules stop (never) Never index a deployment (and stop indexing it if necessary) - indexer rules start (always) Always index a deployment (and start indexing it if necessary) - indexer rules set Set one or more indexing rules - indexer rules prepare (offchain) Offchain index a deployment (and start indexing it if necessary) - indexer rules maybe Index a deployment based on rules - indexer rules get Get one or more indexing rules - indexer rules delete Remove one or many indexing rules - indexer rules clear (reset) Clear one or more indexing rules - indexer rules Configure indexing rules - indexer disputes get Cross-check POIs submitted in the network - indexer disputes Configure allocation POI monitoring - indexer cost set model Update a cost model - indexer cost get Get cost models for one or all subgraphs - indexer cost Manage costing for subgraphs - indexer connect Connect to indexer management API - indexer allocations reallocate Reallocate to subgraph deployment - indexer allocations get List one or more allocations - indexer allocations create Create an allocation - indexer allocations close Close an allocation - indexer allocations Manage indexer allocations - indexer actions queue Queue an action item - indexer actions get List one or more actions - indexer actions execute Execute approved items in the action queue - indexer actions cancel Cancel an item in the queue - indexer actions approve Approve an action item - indexer actions Manage indexer actions - indexer Manage indexer configuration + indexer status Check the status of an indexer + indexer rules stop (never) Never index a deployment (and stop indexing it if necessary) + indexer rules start (always) Always index a deployment (and start indexing it if necessary) + indexer rules set Set one or more indexing rules + indexer rules prepare (offchain) Offchain index a deployment (and start indexing it if necessary) + indexer rules maybe Index a deployment based on rules + indexer rules get Get one or more indexing rules + indexer rules delete Remove one or many indexing rules + indexer rules clear (reset) Clear one or more indexing rules + indexer rules Configure indexing rules + indexer disputes get Cross-check POIs submitted in the network + indexer disputes Configure allocation POI monitoring + indexer cost set model Update a cost model + indexer cost get Get cost models for one or all subgraphs + indexer cost Manage costing for subgraphs + indexer connect Connect to indexer management API + indexer allocations reallocate Reallocate to subgraph deployment + indexer allocations get List one or more allocations + indexer allocations create Create an allocation + indexer allocations close Close an allocation + indexer allocations Manage indexer allocations + indexer actions queue Queue an action item + indexer actions get List one or more actions + indexer actions execute Execute approved items in the action queue + indexer actions cancel Cancel an item in the queue + indexer actions approve Approve an action item + indexer actions Manage indexer actions + indexer Manage indexer configuration ``` ## Running from source @@ -257,6 +258,32 @@ After this, the indexer agent can be run as follows: This starts the indexer agent and serves the so-called indexer management API on the host at port 18000. +## Performance Optimization Features + +The indexer agent includes several performance optimizations for handling large numbers of subgraph deployments: + +### Batching and Filtering +- **Deployment Batching**: Processes deployments in configurable batches (default: 500) to prevent event loop blocking +- **Stake Threshold Filtering**: Automatically filters out deployments below a minimum stake/signal threshold to reduce processing overhead +- **Rule Lookup Optimization**: Uses O(1) Map-based lookups instead of O(N) linear scans for indexing rules + +### Configuration Options +- `--indexer-min-stake-threshold`: Set minimum stake amount in wei (default: 1 GRT = 1000000000000000000 wei) +- `INDEXER_DEPLOYMENT_BATCH_SIZE`: Environment variable for batch size (default: 500) + +### Use Cases +These optimizations are particularly beneficial when: +- Processing 10,000+ subgraph deployments +- Managing complex indexing rule sets +- Running on resource-constrained environments +- Requiring consistent response times during high-load periods + +## Development + +For comprehensive development and testing guidance, see [DEVELOPMENT.md](./DEVELOPMENT.md). + +The project includes a `Dockerfile.dev` for consistent local development and testing environments, particularly useful for testing performance improvements and ensuring compatibility across different systems. + ## Terraform & Kubernetes The [terraform/](./terraform/) and [k8s/](./k8s) directories provide a diff --git a/lerna.json b/lerna.json index 2b0ce2342..1ddc57d1b 100644 --- a/lerna.json +++ b/lerna.json @@ -1,8 +1,5 @@ { - "packages": [ - "packages/*" - ], + "packages": ["packages/*"], "npmClient": "yarn", - "useWorkspaces": true, "version": "0.24.3" } diff --git a/packages/indexer-agent/src/agent.ts b/packages/indexer-agent/src/agent.ts index 82e14c57e..8ee105823 100644 --- a/packages/indexer-agent/src/agent.ts +++ b/packages/indexer-agent/src/agent.ts @@ -192,6 +192,7 @@ export class Agent { autoMigrationSupport: boolean deploymentManagement: DeploymentManagementMode pollingInterval: number + indexerMinStakeThreshold: string constructor(configs: AgentConfigs) { this.logger = configs.logger.child({ component: 'Agent' }) @@ -206,6 +207,7 @@ export class Agent { this.autoMigrationSupport = !!configs.autoMigrationSupport this.deploymentManagement = configs.deploymentManagement this.pollingInterval = configs.pollingInterval + this.indexerMinStakeThreshold = configs.indexerMinStakeThreshold } async start(): Promise { @@ -426,21 +428,43 @@ export class Agent { networkDeployments, indexingRules, }).tryMap( - ({ indexingRules, networkDeployments }) => { - return mapValues( - this.multiNetworks.zip(indexingRules, networkDeployments), - ([indexingRules, networkDeployments]: [ - IndexingRuleAttributes[], - SubgraphDeployment[], - ]) => { - // Identify subgraph deployments on the network that are worth picking up; - // these may overlap with the ones we're already indexing - logger.trace('Evaluating which deployments are worth allocating to') - return indexingRules.length === 0 - ? [] - : evaluateDeployments(logger, networkDeployments, indexingRules) - }, + async ({ indexingRules, networkDeployments }) => { + const results = await Promise.all( + Object.entries( + this.multiNetworks.zip(indexingRules, networkDeployments), + ).map( + async ([networkId, [indexingRules, networkDeployments]]: [ + string, + [IndexingRuleAttributes[], SubgraphDeployment[]], + ]) => { + // Identify subgraph deployments on the network that are worth picking up; + // these may overlap with the ones we're already indexing + logger.trace( + 'Evaluating which deployments are worth allocating to', + { + protocolNetwork: networkId, + deploymentCount: networkDeployments.length, + ruleCount: indexingRules.length, + }, + ) + + const decisions = + indexingRules.length === 0 + ? [] + : await evaluateDeployments( + logger, + networkDeployments, + indexingRules, + this.indexerMinStakeThreshold, + ) + + return [networkId, decisions] as [string, AllocationDecision[]] + }, + ), ) + + // Convert back to the expected object format + return Object.fromEntries(results) }, { onError: error => diff --git a/packages/indexer-agent/src/commands/start.ts b/packages/indexer-agent/src/commands/start.ts index 0eeb2e76f..adb6541e0 100644 --- a/packages/indexer-agent/src/commands/start.ts +++ b/packages/indexer-agent/src/commands/start.ts @@ -303,6 +303,13 @@ export const start = { default: 1, group: 'Indexer Infrastructure', }) + .option('indexer-min-stake-threshold', { + description: + 'Minimum stake/signal amount in wei to consider a deployment significant for evaluation. Deployments below this threshold are filtered out unless they have specific indexing rules. Default: 1 GRT (1000000000000000000 wei)', + type: 'string', + default: '1000000000000000000', + group: 'Indexer Infrastructure', + }) .check(argv => { if ( !argv['network-subgraph-endpoint'] && @@ -661,6 +668,7 @@ export async function run( (s: string) => new SubgraphDeploymentID(s), ), pollingInterval: argv.pollingInterval, + indexerMinStakeThreshold: argv.indexerMinStakeThreshold, } const agent = new Agent(agentConfigs) await agent.start() diff --git a/packages/indexer-agent/src/types.ts b/packages/indexer-agent/src/types.ts index f71f9d295..290d1a8d7 100644 --- a/packages/indexer-agent/src/types.ts +++ b/packages/indexer-agent/src/types.ts @@ -25,4 +25,5 @@ export interface AgentConfigs { autoMigrationSupport: boolean offchainSubgraphs: SubgraphDeploymentID[] pollingInterval: number + indexerMinStakeThreshold: string } diff --git a/packages/indexer-common/src/subgraphs.ts b/packages/indexer-common/src/subgraphs.ts index 9ffb3fb06..a7cb972f9 100644 --- a/packages/indexer-common/src/subgraphs.ts +++ b/packages/indexer-common/src/subgraphs.ts @@ -157,6 +157,7 @@ export enum ActivationCriteria { OFFCHAIN = 'offchain', INVALID_ALLOCATION_AMOUNT = 'invalid_allocation_amount', L2_TRANSFER_SUPPORT = 'l2_transfer_support', + INVALID_DECISION_BASIS = 'invalid_decision_basis', } interface RuleMatch { @@ -192,14 +193,239 @@ export class AllocationDecision { } } -export function evaluateDeployments( +function evaluateDeploymentByRules( + logger: Logger, + deployment: SubgraphDeployment, + deploymentRule: IndexingRuleAttributes, +): AllocationDecision { + const stakedTokens = BigNumber.from(deployment.stakedTokens) + const signalledTokens = BigNumber.from(deployment.signalledTokens) + const avgQueryFees = BigNumber.from(deployment.queryFeesAmount) + + if (deploymentRule.minStake && stakedTokens.gte(deploymentRule.minStake)) { + return new AllocationDecision( + deployment.id, + deploymentRule, + true, + ActivationCriteria.MIN_STAKE, + deployment.protocolNetwork, + ) + } else if (deploymentRule.minSignal && signalledTokens.gte(deploymentRule.minSignal)) { + return new AllocationDecision( + deployment.id, + deploymentRule, + true, + ActivationCriteria.SIGNAL_THRESHOLD, + deployment.protocolNetwork, + ) + } else if ( + deploymentRule.minAverageQueryFees && + avgQueryFees.gte(deploymentRule.minAverageQueryFees) + ) { + return new AllocationDecision( + deployment.id, + deploymentRule, + true, + ActivationCriteria.MIN_AVG_QUERY_FEES, + deployment.protocolNetwork, + ) + } else { + return new AllocationDecision( + deployment.id, + deploymentRule, + false, + ActivationCriteria.NONE, + deployment.protocolNetwork, + ) + } +} + +export async function evaluateDeployments( logger: Logger, networkDeployments: SubgraphDeployment[], rules: IndexingRuleAttributes[], -): AllocationDecision[] { - return networkDeployments.map((deployment) => - isDeploymentWorthAllocatingTowards(logger, deployment, rules), + minStakeThreshold?: string, +): Promise { + const startTime = performance.now() + + // Pre-build optimized rule lookups for O(1) access + const globalRule = rules.find((rule) => rule.identifier === INDEXING_RULE_GLOBAL) + const deploymentRulesMap = new Map() + + rules + .filter((rule) => rule.identifierType === SubgraphIdentifierType.DEPLOYMENT) + .forEach((rule) => { + deploymentRulesMap.set(rule.identifier, rule) + }) + + logger.debug(`Starting deployment evaluation`, { + totalDeployments: networkDeployments.length, + totalRules: rules.length, + deploymentRules: deploymentRulesMap.size, + }) + + // Filter deployments by minimum thresholds to reduce work + // Use provided threshold or fall back to environment variable + const MIN_STAKE_THRESHOLD = BigNumber.from( + minStakeThreshold || process.env.INDEXER_MIN_STAKE_THRESHOLD || '1000000000000000000', + ) // Default: 1 GRT minimum + const significantDeployments = networkDeployments.filter( + (deployment) => + deployment.stakedTokens.gte(MIN_STAKE_THRESHOLD) || + deployment.signalledTokens.gte(MIN_STAKE_THRESHOLD) || + deploymentRulesMap.has(deployment.id.toString()), // Always include if we have specific rules ) + + logger.debug(`Filtered deployments by significance`, { + originalCount: networkDeployments.length, + filteredCount: significantDeployments.length, + reduction: `${( + (1 - significantDeployments.length / networkDeployments.length) * + 100 + ).toFixed(1)}%`, + }) + + const BATCH_SIZE = parseInt(process.env.INDEXER_DEPLOYMENT_BATCH_SIZE || '500') // Process in smaller batches to prevent blocking + const allDecisions: AllocationDecision[] = [] + + // Process deployments in batches with yielding + for (let i = 0; i < significantDeployments.length; i += BATCH_SIZE) { + const batch = significantDeployments.slice(i, i + BATCH_SIZE) + const batchStartTime = performance.now() + + logger.trace(`Processing deployment batch`, { + batchNumber: Math.floor(i / BATCH_SIZE) + 1, + totalBatches: Math.ceil(significantDeployments.length / BATCH_SIZE), + batchSize: batch.length, + startIndex: i, + endIndex: Math.min(i + BATCH_SIZE, significantDeployments.length), + }) + + // Process batch synchronously for efficiency + const batchDecisions = batch.map((deployment) => + isDeploymentWorthAllocatingTowardsOptimized( + logger, + deployment, + deploymentRulesMap, + globalRule, + ), + ) + + allDecisions.push(...batchDecisions) + + const batchTime = performance.now() - batchStartTime + logger.trace(`Completed deployment batch`, { + batchNumber: Math.floor(i / BATCH_SIZE) + 1, + batchTime: `${batchTime.toFixed(2)}ms`, + deploymentsProcessed: batch.length, + }) + + // Yield control to event loop every batch to prevent blocking + // Only yield if we have more batches to process + if (i + BATCH_SIZE < significantDeployments.length) { + await new Promise((resolve) => setImmediate(resolve)) + } + } + + const totalTime = performance.now() - startTime + logger.info(`Deployment evaluation completed`, { + totalDeployments: networkDeployments.length, + evaluatedDeployments: significantDeployments.length, + allocatableDeployments: allDecisions.filter((d) => d.toAllocate).length, + totalTime: `${totalTime.toFixed(2)}ms`, + avgTimePerDeployment: `${(totalTime / significantDeployments.length).toFixed(3)}ms`, + }) + + return allDecisions +} + +// Optimized version that uses pre-built rule map for O(1) lookups +export function isDeploymentWorthAllocatingTowardsOptimized( + logger: Logger, + deployment: SubgraphDeployment, + deploymentRulesMap: Map, + globalRule: IndexingRuleAttributes | undefined, +): AllocationDecision { + // O(1) lookup instead of O(N) filtering and finding + const deploymentRule = + deploymentRulesMap.get(deployment.id.toString()) || + deploymentRulesMap.get(deployment.id.bytes32) || + globalRule + + logger.trace('Evaluating whether subgraphDeployment is worth allocating towards', { + deployment: deployment.id.display, + hasSpecificRule: + deploymentRulesMap.has(deployment.id.toString()) || + deploymentRulesMap.has(deployment.id.bytes32), + matchingRule: deploymentRule?.identifier, + }) + + // The deployment is not eligible for deployment if it doesn't have an allocation amount + if (!deploymentRule?.allocationAmount) { + logger.debug(`Could not find matching rule with defined 'allocationAmount':`, { + deployment: deployment.id.display, + }) + return new AllocationDecision( + deployment.id, + deploymentRule, + false, + ActivationCriteria.INVALID_ALLOCATION_AMOUNT, + deployment.protocolNetwork, + ) + } + + // Reject unsupported subgraphs early + if (deployment.deniedAt > 0 && deploymentRule.requireSupported) { + return new AllocationDecision( + deployment.id, + deploymentRule, + false, + ActivationCriteria.UNSUPPORTED, + deployment.protocolNetwork, + ) + } + + switch (deploymentRule?.decisionBasis) { + case IndexingDecisionBasis.RULES: { + return evaluateDeploymentByRules(logger, deployment, deploymentRule) + } + case IndexingDecisionBasis.NEVER: { + return new AllocationDecision( + deployment.id, + deploymentRule, + false, + ActivationCriteria.NEVER, + deployment.protocolNetwork, + ) + } + case IndexingDecisionBasis.ALWAYS: { + return new AllocationDecision( + deployment.id, + deploymentRule, + true, + ActivationCriteria.ALWAYS, + deployment.protocolNetwork, + ) + } + case IndexingDecisionBasis.OFFCHAIN: { + return new AllocationDecision( + deployment.id, + deploymentRule, + true, + ActivationCriteria.OFFCHAIN, + deployment.protocolNetwork, + ) + } + default: { + return new AllocationDecision( + deployment.id, + deploymentRule, + false, + ActivationCriteria.INVALID_DECISION_BASIS, + deployment.protocolNetwork, + ) + } + } } export function isDeploymentWorthAllocatingTowards(