Skip to content

Commit b609d4d

Browse files
Abdul Rabbanii-norden
Abdul Rabbani
authored and
i-norden
committed
Statediff Geth
* Handle conflicts (#244) * Handle conflicts * Update go mod file versions * Make lint changes * Disassociate block number from the indexer object * Update ipld-eth-db ref * Refactor builder code to make it reusable * Use prefix comparison for account selective statediffing * Update builder unit tests * Add mode to write to CSV files in statediff file writer (#249) * Change file writing mode to csv files * Implement writer interface for file indexer * Implement option for csv or sql in file mode * Close files in CSV writer * Add tests for CSV file mode * Implement CSV file for watched addresses * Separate test configs for CSV and SQL * Refactor common code for file indexer tests * Update indexer to include block hash in receipts and logs (#256) * Update indexer to include block hash in receipts and logs * Upgrade ipld-eth-db image in docker-compose to run tests
1 parent 8f2416a commit b609d4d

File tree

180 files changed

+25883
-86
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

180 files changed

+25883
-86
lines changed

‎.github/workflows/checks.yml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
name: checks
2+
3+
on: [pull_request]
4+
5+
jobs:
6+
linter-check:
7+
runs-on: ubuntu-latest
8+
steps:
9+
- uses: actions/setup-go@v3
10+
with:
11+
go-version: ">=1.18.0"
12+
check-latest: true
13+
- uses: actions/checkout@v2
14+
- name: Run linter
15+
run: go run build/ci.go lint

‎.github/workflows/on-pr.yml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
name: Build and test
2+
3+
on: [pull_request]
4+
5+
jobs:
6+
run-tests:
7+
uses: ./.github/workflows/tests.yml

‎.github/workflows/publish.yaml‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
name: Publish geth to release
2+
on:
3+
release:
4+
types: [published]
5+
jobs:
6+
run-tests:
7+
uses: ./.github/workflows/tests.yml
8+
build:
9+
name: Run docker build and publish
10+
needs: run-tests
11+
runs-on: ubuntu-latest
12+
steps:
13+
- uses: actions/checkout@v2
14+
- name: Run docker build
15+
run: docker build -t vulcanize/go-ethereum -f Dockerfile .
16+
- name: Get the version
17+
id: vars
18+
run: echo ::set-output name=sha::$(echo ${GITHUB_SHA:0:7})
19+
- name: Tag docker image
20+
run: docker tag vulcanize/go-ethereum docker.pkg.github.com/vulcanize/go-ethereum/go-ethereum:${{steps.vars.outputs.sha}}
21+
- name: Docker Login
22+
run: echo ${{ secrets.GITHUB_TOKEN }} | docker login https://docker.pkg.github.com -u vulcanize --password-stdin
23+
- name: Docker Push
24+
run: docker push docker.pkg.github.com/vulcanize/go-ethereum/go-ethereum:${{steps.vars.outputs.sha}}
25+
push_to_registries:
26+
name: Publish assets to Release
27+
runs-on: ubuntu-latest
28+
needs: build
29+
steps:
30+
- name: Get the version
31+
id: vars
32+
run: |
33+
echo ::set-output name=sha::$(echo ${GITHUB_SHA:0:7})
34+
echo ::set-output name=tag::$(echo ${GITHUB_REF#refs/tags/})
35+
- name: Docker Login to Github Registry
36+
run: echo ${{ secrets.GITHUB_TOKEN }} | docker login https://docker.pkg.github.com -u vulcanize --password-stdin
37+
- name: Docker Pull
38+
run: docker pull docker.pkg.github.com/vulcanize/go-ethereum/go-ethereum:${{steps.vars.outputs.sha}}
39+
- name: Copy ethereum binary file
40+
run: docker run --rm --entrypoint cat docker.pkg.github.com/vulcanize/go-ethereum/go-ethereum:${{steps.vars.outputs.sha}} /usr/local/bin/geth > geth-linux-amd64
41+
- name: Docker Login to Docker Registry
42+
run: echo ${{ secrets.VULCANIZEJENKINS_PAT }} | docker login -u vulcanizejenkins --password-stdin
43+
- name: Tag docker image
44+
run: docker tag docker.pkg.github.com/vulcanize/go-ethereum/go-ethereum:${{steps.vars.outputs.sha}} vulcanize/vdb-geth:${{steps.vars.outputs.tag}}
45+
- name: Docker Push to Docker Hub
46+
run: docker push vulcanize/vdb-geth:${{steps.vars.outputs.tag}}
47+
- name: Get release
48+
id: get_release
49+
uses: bruceadams/[email protected]
50+
env:
51+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
52+
- name: Upload Release Asset
53+
id: upload-release-asset
54+
uses: actions/upload-release-asset@v1
55+
env:
56+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
57+
with:
58+
upload_url: ${{ steps.get_release.outputs.upload_url }}
59+
asset_path: geth-linux-amd64
60+
asset_name: geth-linux-amd64
61+
asset_content_type: application/octet-stream

‎.github/workflows/tests.yml‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
name: Tests for Geth that are used in multiple jobs.
2+
3+
on:
4+
workflow_call:
5+
6+
env:
7+
stack-orchestrator-ref: ${{ github.event.inputs.stack-orchestrator-ref || '382aca8e42bc5e33f301f77cdd2e09cc80602fc3'}}
8+
ipld-eth-db-ref: ${{ github.event.inputs.ipld-ethcl-db-ref || '65b7bee7a6757c1fc527c8bfdc4f99ab915fcf36' }}
9+
GOPATH: /tmp/go
10+
11+
jobs:
12+
build:
13+
name: Run docker build
14+
runs-on: ubuntu-latest
15+
steps:
16+
- uses: actions/checkout@v2
17+
- name: Run docker build
18+
run: docker build -t vulcanize/go-ethereum .
19+
20+
geth-unit-test:
21+
name: Run geth unit test
22+
runs-on: ubuntu-latest
23+
env:
24+
GO111MODULE: on
25+
steps:
26+
- name: Create GOPATH
27+
run: mkdir -p /tmp/go
28+
29+
- uses: actions/setup-go@v3
30+
with:
31+
go-version: ">=1.18.0"
32+
check-latest: true
33+
34+
- name: Checkout code
35+
uses: actions/checkout@v2
36+
37+
- name: Run unit tests
38+
run: |
39+
make test
40+
41+
statediff-unit-test:
42+
name: Run state diff unit test
43+
runs-on: ubuntu-latest
44+
steps:
45+
- name: Create GOPATH
46+
run: mkdir -p /tmp/go
47+
48+
- uses: actions/setup-go@v3
49+
with:
50+
go-version: ">=1.18.0"
51+
check-latest: true
52+
53+
- name: Checkout code
54+
uses: actions/checkout@v2
55+
56+
- name: Run docker compose
57+
run: |
58+
docker-compose up -d
59+
60+
- name: Give the migration a few seconds
61+
run: sleep 30;
62+
63+
- name: Run unit tests
64+
run: make statedifftest
65+
66+
private-network-test:
67+
name: Start Geth in a private network.
68+
runs-on: ubuntu-latest
69+
steps:
70+
- name: Create GOPATH
71+
run: mkdir -p /tmp/go
72+
73+
- uses: actions/setup-go@v3
74+
with:
75+
go-version: ">=1.18.0"
76+
check-latest: true
77+
78+
- name: Checkout code
79+
uses: actions/checkout@v3
80+
with:
81+
path: "./go-ethereum"
82+
83+
- uses: actions/checkout@v3
84+
with:
85+
ref: ${{ env.stack-orchestrator-ref }}
86+
path: "./stack-orchestrator/"
87+
repository: vulcanize/stack-orchestrator
88+
fetch-depth: 0
89+
90+
- uses: actions/checkout@v3
91+
with:
92+
ref: ${{ env.ipld-eth-db-ref }}
93+
repository: vulcanize/ipld-eth-db
94+
path: "./ipld-eth-db/"
95+
fetch-depth: 0
96+
97+
- name: Create config file
98+
run: |
99+
echo vulcanize_ipld_eth_db=$GITHUB_WORKSPACE/ipld-eth-db/ > $GITHUB_WORKSPACE/config.sh
100+
echo vulcanize_go_ethereum=$GITHUB_WORKSPACE/go-ethereum/ >> $GITHUB_WORKSPACE/config.sh
101+
echo db_write=true >> $GITHUB_WORKSPACE/config.sh
102+
cat $GITHUB_WORKSPACE/config.sh
103+
104+
- name: Compile Geth
105+
run: |
106+
cd $GITHUB_WORKSPACE/stack-orchestrator/helper-scripts
107+
./compile-geth.sh -e docker -p $GITHUB_WORKSPACE/config.sh
108+
cd -
109+
110+
- name: Run docker compose
111+
run: |
112+
docker-compose \
113+
-f "$GITHUB_WORKSPACE/stack-orchestrator/docker/local/docker-compose-go-ethereum.yml" \
114+
-f "$GITHUB_WORKSPACE/stack-orchestrator/docker/local/docker-compose-db-sharding.yml" \
115+
--env-file $GITHUB_WORKSPACE/config.sh \
116+
up -d --build
117+
118+
- name: Make sure the /root/transaction_info/STATEFUL_TEST_DEPLOYED_ADDRESS exists within a certain time frame.
119+
shell: bash
120+
run: |
121+
COUNT=0
122+
ATTEMPTS=15
123+
until $(docker compose -f "$GITHUB_WORKSPACE/stack-orchestrator/docker/local/docker-compose-go-ethereum.yml" cp go-ethereum:/root/transaction_info/STATEFUL_TEST_DEPLOYED_ADDRESS ./STATEFUL_TEST_DEPLOYED_ADDRESS) || [[ $COUNT -eq $ATTEMPTS ]]; do echo -e "$(( COUNT++ ))... \c"; sleep 10; done
124+
[[ $COUNT -eq $ATTEMPTS ]] && echo "Could not find the successful contract deployment" && (exit 1)
125+
cat ./STATEFUL_TEST_DEPLOYED_ADDRESS
126+
sleep 15;
127+
128+
- name: Create a new transaction.
129+
shell: bash
130+
run: |
131+
docker compose -f "$GITHUB_WORKSPACE/stack-orchestrator/docker/local/docker-compose-go-ethereum.yml" exec go-ethereum /bin/bash /root/transaction_info/NEW_TRANSACTION
132+
echo $?
133+
134+
- name: Make sure we see entries in the header table
135+
shell: bash
136+
run: |
137+
rows=$(docker compose -f "$GITHUB_WORKSPACE/stack-orchestrator/docker/local/docker-compose-go-ethereum.yml" exec ipld-eth-db psql -U vdbm -d vulcanize_testing -AXqtc "SELECT COUNT(*) FROM eth.header_cids")
138+
[[ "$rows" -lt "1" ]] && echo "We could not find any rows in postgres table." && (exit 1)
139+
echo $rows
140+
docker compose -f "$GITHUB_WORKSPACE/stack-orchestrator/docker/local/docker-compose-go-ethereum.yml" exec ipld-eth-db psql -U vdbm -d vulcanize_testing -AXqtc "SELECT * FROM eth.header_cids"

‎.gitignore‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,3 +47,15 @@ profile.cov
4747
/dashboard/assets/package-lock.json
4848

4949
**/yarn-error.log
50+
foundry/deployments/local-private-network/geth-linux-amd64
51+
foundry/projects/local-private-network/geth-linux-amd64
52+
53+
# Helpful repos
54+
related-repositories/foundry-test/**
55+
related-repositories/hive/**
56+
related-repositories/ipld-eth-db/**
57+
statediff/indexer/database/sql/statediffing_test_file.sql
58+
statediff/statediffing_test_file.sql
59+
statediff/known_gaps.sql
60+
related-repositories/foundry-test/
61+
related-repositories/ipld-eth-db/

‎.gitmodules‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,4 +5,4 @@
55
[submodule "evm-benchmarks"]
66
path = tests/evm-benchmarks
77
url = https://github.com/ipsilon/evm-benchmarks
8-
shallow = true
8+
shallow = true

‎Dockerfile.amd64‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# Build Geth in a stock Go builder container
2+
FROM golang:1.15.5 as builder
3+
4+
#RUN apk add --no-cache make gcc musl-dev linux-headers git
5+
6+
ADD . /go-ethereum
7+
RUN cd /go-ethereum && make geth

‎Makefile‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,31 @@
44

55
.PHONY: geth android ios evm all test clean
66

7+
BIN = $(GOPATH)/bin
8+
9+
## Migration tool
10+
GOOSE = $(BIN)/goose
11+
$(BIN)/goose:
12+
go get -u github.com/pressly/goose/cmd/goose
13+
714
GOBIN = ./build/bin
815
GO ?= latest
916
GORUN = env GO111MODULE=on go run
1017

18+
#Database
19+
HOST_NAME = localhost
20+
PORT = 5432
21+
USER = vdbm
22+
PASSWORD = password
23+
24+
# Set env variable
25+
# `PGPASSWORD` is used by `createdb` and `dropdb`
26+
export PGPASSWORD=$(PASSWORD)
27+
28+
#Test
29+
TEST_DB = vulcanize_public
30+
TEST_CONNECT_STRING = postgresql://$(USER):$(PASSWORD)@$(HOST_NAME):$(PORT)/$(TEST_DB)?sslmode=disable
31+
1132
geth:
1233
$(GORUN) build/ci.go install ./cmd/geth
1334
@echo "Done building."
@@ -48,3 +69,13 @@ devtools:
4869
env GOBIN= go install ./cmd/abigen
4970
@type "solc" 2> /dev/null || echo 'Please install solc'
5071
@type "protoc" 2> /dev/null || echo 'Please install protoc'
72+
73+
.PHONY: statedifftest
74+
statedifftest: | $(GOOSE)
75+
GO111MODULE=on go get github.com/stretchr/testify/[email protected]
76+
GO111MODULE=on MODE=statediff go test -p 1 ./statediff/... -v
77+
78+
.PHONY: statediff_filewriting_test
79+
statediff_filetest: | $(GOOSE)
80+
GO111MODULE=on go get github.com/stretchr/testify/[email protected]
81+
GO111MODULE=on MODE=statediff STATEDIFF_DB=file go test -p 1 ./statediff/... -v

0 commit comments

Comments
 (0)