Setup local testnet for integration testing #679

neithanmo · 2025-04-03T18:53:58Z

This PR introduces a local-testnet and an integration test to validate that the receipt aggregation and RAV generation functionality works correctly in a real-world environment.

Key Changes

Added a TAP RAV generation test that verifies receipts are properly aggregated into RAVs(this is a custom rust binary that is part of the workspace)
Implemented metrics monitoring to detect successful RAV generation
Added proper indexer-service and tap-agent configurations to make testing times shorter
Implement testing checks that compares pre-test and post-test metrics

Testing Details

The tests deploy and interact with a complete local testnet environment using Docker containers, including:

Indexer service
TAP agent
TAP aggregator
Blockchain node with deployed contracts
tap-contracts deployments
Database
block oracles
Gateway
indexer-agent(not use directly)

The integration test:

Funds the escrow contract with sufficient GRT(using a bash script)
Sends multiple receipts in batches
Waits for the timestamp buffer period to ensure receipts age appropriately
Verifies that RAVs are successfully created via metrics examination

Missing Checks

The current test does not check for RAV claims/redemption from the indexer-agent
Test does not check for the RAV amounts to be equal to a pre-computed value

How to setup and run:

In order to try this locally:

Check if your local setup is already running a posgress database, if that is the case stop that service
run just setup to build and run containers
If previous step was completed successfully, then, run the test: just test-local

Set up a comprehensive Docker-based testing environment that simulates a production-like setup for indexer services. This includes: - Integration with local-network components (chain, IPFS, graph-node, contracts) - Custom services configuration with proper database migrations - Test scripts to validate service functionality(WIP) - Fix for database schema creation to support cost models and deny lists - Automated service startup with correct dependency ordering(WIP) This environment allows testing our services against real subgraphs and contracts, providing better validation than isolated unit tests. The setup creates a three-tier architecture that mimics production deployment. Tests can now validate payment receipt processing and query routing in a realistic scenario.

…ntrib dir

… local network testing

…ocal network

…le and remove logging from indexer start script

coveralls · 2025-04-03T19:01:26Z

Pull Request Test Coverage Report for Build 14310527874

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

0 of 294 (0.0%) changed or added relevant lines in 4 files are covered.
9 unchanged lines in 1 file lost coverage.
Overall coverage decreased (-1.9%) to 74.244%

Changes Missing Coverage	Changed/Added Lines	%
integration-tests/src/main.rs	4	0.0%
integration-tests/src/receipt.rs	26	0.0%
integration-tests/src/metrics.rs	87	0.0%
integration-tests/src/rav_tests.rs	177	0.0%

Files with Coverage Reduction	New Missed Lines	%
crates/tap-agent/src/agent/sender_account.rs	9	77.53%

Totals
Change from base Build 14193652489:	-1.9%
Covered Lines:	8858
Relevant Lines:	11931

💛 - Coveralls

…ployed already

…gent for testing on local network

…ation thresholds

…re tests

…ting considerations

…ation files

contrib/README.md

rav_e2e/testing_plan.md

…running tests

.github/workflows/tap_integration_test.yml.disable

neithanmo · 2025-04-04T20:01:06Z

I just tried running this with just setup in one terminal then just test-local in another. I got this far:
Error: contract 0x0355b7b8cb128fa5692729ab3aaa199c1753f726 does not have any code
error: Recipe `test-local` failed on line 73 with exit code 1

What's happening, I think, is that there's a subtle timing gap between when the contract deployment container finishes submitting transactions and when those contracts are actually mined and confirmed on the local network.
that did not happen on my target, so it could be a matter of timing, performance.
This could be fixed by mining some blocks between graph-contract/tap-contracts deployment and its verification in the setup script. I will add that

setup-test-network.sh

suchapalaver · 2025-04-04T20:08:51Z

I just tried running this with just setup in one terminal then just test-local in another. I got this far:
Error: contract 0x0355b7b8cb128fa5692729ab3aaa199c1753f726 does not have any code
error: Recipe `test-local` failed on line 73 with exit code 1
What's happening, I think, is that there's a subtle timing gap between when the contract deployment container finishes submitting transactions and when those contracts are actually mined and confirmed on the local network. that did not happen on my target, so it could be a matter of timing, performance. This could be fixed by mining some blocks between graph-contract/tap-contracts deployment and its verification in the setup script. I will add that

I deleted my comment because I jumped the gun running the local-test directive, not everything had built yet. Apologies for any confusion!

suchapalaver · 2025-04-04T20:11:41Z

I just tried running this with just setup in one terminal then just test-local in another. I got this far:
Error: contract 0x0355b7b8cb128fa5692729ab3aaa199c1753f726 does not have any code
error: Recipe `test-local` failed on line 73 with exit code 1
What's happening, I think, is that there's a subtle timing gap between when the contract deployment container finishes submitting transactions and when those contracts are actually mined and confirmed on the local network. that did not happen on my target, so it could be a matter of timing, performance. This could be fixed by mining some blocks between graph-contract/tap-contracts deployment and its verification in the setup script. I will add that
I deleted my comment because I jumped the gun running the local-test directive, not everything had built yet. Apologies for any confusion!

Ah yeah I ran into Waiting for gateway service... forever.

neithanmo · 2025-04-04T20:19:57Z

Ah yeah I ran into Waiting for gateway service... forever.

That happened to me as well, but it got better after mining some blocks before the gateway check,
the timeout there is 5 minutes.

suchapalaver · 2025-04-04T20:22:33Z

Ah yeah I ran into Waiting for gateway service... forever.

That happened to me as well, but it got better after mining some blocks before the gateway check, the timeout there is 5 minutes.

Your fix worked

suchapalaver · 2025-04-04T20:23:20Z

These values aren't changing in my output: RAVs created: 0, Unaggregated fees: 0 even at 41/100. Is that to be expected?

This is running just local-test.

Update:

=== Summary ===
Total receipts sent successfully: 140
Total value sent: 0.14 GRT
❌ TEST FAILED: No RAV generation detected
Error: Failed to detect RAV generation

…carted

suchapalaver · 2025-04-07T11:26:35Z

These values aren't changing in my output: RAVs created: 0, Unaggregated fees: 0 even at 41/100. Is that to be expected?

This is running just local-test.

Update:
=== Summary ===

Total receipts sent successfully: 140

Total value sent: 0.14 GRT

❌ TEST FAILED: No RAV generation detected

Error: Failed to detect RAV generation

Confirming this is fixed for me now 👍

suchapalaver

Amazing to have this infrastructure in place, what a great start!

I was having an issue with a very old version of protoc in apt on my new laptop and now that's fixed I can confirm this is working as described and as expected! Nice work!

TypeLevelConsoli

That's so valuable, thanks!

neithanmo · 2025-04-07T13:05:15Z

Glad to hear
I am changing the name to integration-test/ and making it part of the workspace as well
This ensures that its compilation is also verified by CI.

integration-tests/src/rav_tests.rs

neithanmo added 8 commits April 3, 2025 11:42

fix(testing): Organize containers and configurations into its owne co…

f53c3d1

…ntrib dir

fix(testing): add reload script and container builder script to setup…

968ecdf

… local network testing

fix(testing): Remove unnecessary tests and files

adf2908

fix(testing): Update justfile and add a Makefile(for future CI)

06cfef5

feat(testing): Add a minimal workflow for integration testing using l…

519e2e1

…ocal network

fix(service): tap-agent start script to use settings from .env variab…

0afaa14

…le and remove logging from indexer start script

fix(testing): Fix docker command in makefile

740a947

neithanmo added 15 commits April 3, 2025 13:01

chore(testnet): Add more logs and double check for contracts to be de…

cad130e

…ployed already

fix(testing_setup): Update local network and enable missing services

92170b5

feat(local_network): Enable the gateway service

5480a80

fix(indexer-testing): Use same auth token as gateway

059b2f6

fix(testing): Use reasonable values to configure RAV request in tap-a…

d6ebb30

…gent for testing on local network

chore(testing): Remove old testing script

3e449a5

feat(testing): Add end to end testing for RAV generation

62bfb87

fix(git): Add new items to gitignore

663c73d

fix(makefile): Avoid triggering errors if no container is active

4790efb

fix(ci): Enable test execution on push

74ce3fe

fix(rav_test): Increase number of receipts to ensure we hit rav gener…

fe2af78

…ation thresholds

fix(ci): Disable local testnet testing for now

51bbc4d

fix(Makefile): Update help and ensure funding script is executed befo…

7def4a3

…re tests

fix(testing_plan): Remove code example and improve description of tes…

6ae72c0

…ting considerations

chore(local_testnet): Update README and Fix down action in justfile

73f00e4

neithanmo marked this pull request as draft April 3, 2025 19:07

neithanmo added 3 commits April 3, 2025 13:23

fix(license): Add license headers

41a22f2

fix(ci): Ignore Dockerfile, docker compose, bash scripts and configur…

7275a17

…ation files

fix(ci): Restore original Dockerfiles for indexer and tap-agent services

0015b69

neithanmo force-pushed the feat/integration_test branch from 53a66d3 to 0015b69 Compare April 3, 2025 19:25

fix(ci): Ignore bash scripts in rav_e2e testing

2c62eca