Skip to content

Commit 41f997a

Browse files
LynxLynxxLynxLynxx
authored
Merge branch 'main' into feat/face-performance-optimization-3352
2 parents f6fb0e4 + a82ccf0 commit 41f997a

File tree

61 files changed

+640
-412
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

61 files changed

+640
-412
lines changed

‎.github/workflows/gateway-tests.yml‎

Lines changed: 0 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -106,18 +106,6 @@ jobs:
106106
run: |
107107
docker compose -f catalyst-gateway/tests/docker-compose.yml up api-tests-runner --exit-code-from api-tests-runner
108108
109-
# Skip for now as it adds ~10mins to CI execution time, will try to incorporate this in nightly instead
110-
- name: Integration health thru proxy tests
111-
if: false
112-
env:
113-
API_TEST_MARKS: "health_with_proxy_endpoint"
114-
EVENT_DB_URL: "haproxy:18080"
115-
INDEX_DB_URL: "haproxy:18090"
116-
run: |
117-
docker stop cat-gateway || true
118-
docker compose -f catalyst-gateway/tests/docker-compose.yml up haproxy --detach
119-
docker compose -f catalyst-gateway/tests/docker-compose.yml up api-tests-runner --exit-code-from api-tests-runner
120-
121109
- name: Collect and upload test reports
122110
uses: actions/upload-artifact@v4
123111
if: success() || failure()

‎.github/workflows/nightly-ci.yml‎

Lines changed: 51 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -69,28 +69,65 @@ jobs:
6969
id: gateway-healthy
7070
if: ${{ steps.gateway-started.conclusion == 'success' }}
7171
run: |
72-
echo "Waiting for container to be healthy..."
73-
for i in {1..500}; do
74-
STATUS=$(docker inspect --format='{{.State.Health.Status}}' cat-gateway)
75-
echo "Health status: $STATUS"
76-
if [ "$STATUS" == "healthy" ]; then
77-
echo "Container is healthy!"
78-
break
79-
fi
80-
if [ "$STATUS" == "unhealthy" ]; then
81-
echo "Container became unhealthy."
82-
exit 1
83-
fi
84-
sleep 5
85-
done
72+
echo "Waiting for container to be healthy..."
73+
for i in {1..500}; do
74+
STATUS=$(docker inspect --format='{{.State.Health.Status}}' cat-gateway)
75+
echo "Health status: $STATUS"
76+
if [ "$STATUS" == "healthy" ]; then
77+
echo "Container is healthy!"
78+
break
79+
fi
80+
if [ "$STATUS" == "unhealthy" ]; then
81+
echo "Container became unhealthy."
82+
exit 1
83+
fi
84+
sleep 5
85+
done
8686
8787
- name: Schemathesis tests
88+
id: schemathesis-tests
8889
if: ${{ steps.gateway-healthy.conclusion == 'success' }}
8990
run: |
9091
export HYPOTHESIS_MAX_EXAMPLES=5000
9192
export MAX_RESPONSE_TIME=25000
9293
docker compose -f catalyst-gateway/tests/docker-compose.yml up schemathesis-runner --exit-code-from schemathesis-runner
9394
95+
- name: Spin up catalyst-gateway with haproxy
96+
id: gateway-with-proxy-started
97+
env:
98+
EVENT_DB_URL: "haproxy:18080"
99+
INDEX_DB_URL: "haproxy:18090"
100+
run: |
101+
docker stop cat-gateway || true
102+
docker compose -f catalyst-gateway/tests/docker-compose.yml up haproxy --detach
103+
docker compose -f catalyst-gateway/tests/docker-compose.yml up cat-gateway --detach
104+
105+
- name: Wait for cat-gateway to become healthy
106+
id: gateway-with-proxy-healthy
107+
if: ${{ steps.gateway-with-proxy-started.conclusion == 'success' }}
108+
run: |
109+
echo "Waiting for container to be healthy..."
110+
for i in {1..500}; do
111+
STATUS=$(docker inspect --format='{{.State.Health.Status}}' cat-gateway)
112+
echo "Health status: $STATUS"
113+
if [ "$STATUS" == "healthy" ]; then
114+
echo "Container is healthy!"
115+
break
116+
fi
117+
if [ "$STATUS" == "unhealthy" ]; then
118+
echo "Container became unhealthy."
119+
exit 1
120+
fi
121+
sleep 5
122+
done
123+
124+
- name: Integration health thru proxy tests
125+
if: ${{ steps.gateway-with-proxy-healthy.conclusion == 'success' }}
126+
env:
127+
API_TEST_MARKS: "health_with_proxy_endpoint"
128+
run: |
129+
docker compose -f catalyst-gateway/tests/docker-compose.yml up api-tests-runner --exit-code-from api-tests-runner
130+
94131
slack:
95132
name: slack notification
96133
needs: tests

‎Earthfile‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
VERSION 0.8
22

3-
IMPORT github.com/input-output-hk/catalyst-ci/earthly/mdlint:v3.6.1 AS mdlint-ci
4-
IMPORT github.com/input-output-hk/catalyst-ci/earthly/cspell:v3.6.1 AS cspell-ci
5-
IMPORT github.com/input-output-hk/catalyst-ci/earthly/postgresql:v3.6.1 AS postgresql-ci
3+
IMPORT github.com/input-output-hk/catalyst-ci/earthly/mdlint:v3.6.2 AS mdlint-ci
4+
IMPORT github.com/input-output-hk/catalyst-ci/earthly/cspell:v3.6.2 AS cspell-ci
5+
IMPORT github.com/input-output-hk/catalyst-ci/earthly/postgresql:v3.6.2 AS postgresql-ci
66

77
ARG --global REGISTRY="harbor.shared-services.projectcatalyst.io/dockerhub/library"
88
FROM ${REGISTRY}/debian:stable-slim

‎catalyst-gateway/Earthfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
VERSION 0.8
22

3-
IMPORT github.com/input-output-hk/catalyst-ci/earthly/rust:v3.6.1 AS rust-ci
3+
IMPORT github.com/input-output-hk/catalyst-ci/earthly/rust:v3.6.2 AS rust-ci
44
IMPORT ../ AS repo-ci
55
IMPORT github.com/input-output-hk/catalyst-voices/catalyst-gateway:main AS cat-gateway-main
66

‎catalyst-gateway/README.md‎

Lines changed: 57 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,14 @@
33
* [Catalyst Data Gateway](#catalyst-data-gateway)
44
* [Code Organization](#code-organization)
55
* [`./bin`](#bin)
6-
* [`./crates`](#crates)
76
* [`./event-db`](#event-db)
87
* [Build and Run](#build-and-run)
98
* [Docker images](#docker-images)
9+
* [Build](#build)
10+
* [Run](#run)
1011
* [Rust binary](#rust-binary)
12+
* [Build](#build-1)
13+
* [Run](#run-1)
1114

1215
The Catalyst Data Gateway is the backend of the Catalyst Voices hosted stack.
1316

@@ -20,11 +23,6 @@ In future it will also act as a gateway from Centralized catalyst infrastructure
2023

2124
This is the main Catalyst Gateway Application.
2225

23-
### `./crates`
24-
25-
These are fully re-usable generalized `rust` crates that the Catalyst Gateway uses and are developed with it.
26-
They are also able to be used stand-alone in other projects and can be published separately.
27-
2826
### `./event-db`
2927

3028
Defines the Postgres Catalyst Event Database that the Catalyst gateway uses for running Catalyst Events.
@@ -40,30 +38,60 @@ or you can build a docker image and run everything with the `docker-compose`.
4038

4139
To build and run docker images follow these steps:
4240

43-
1. Run `earthly +package` to build a cat-gateway docker image.
44-
2. Run `earthly ./event-db+build` to build an event-db docker image.
45-
3. Run `docker-compose up cat-gateway` to spin up cat-gateway with event-db from already built images.
41+
#### Build
42+
43+
* Build `cat-gateway`:
44+
45+
```sh
46+
earthly ./catalyst-gateway+docker
47+
```
48+
49+
* Build `event-db`:
4650

47-
Note that every time when you are building an image it obsoletes an old image but does not remove it,
48-
so don't forget to clean up dangling images of the event-db and cat-gateway in your docker environment.
51+
```sh
52+
earthly ./catalyst-gateway/event-db+docker
53+
```
54+
55+
#### Run
56+
57+
```sh
58+
docker compose -f ./catalyst-gateway/docker-compose.yml up cat-gateway
59+
```
4960

5061
### Rust binary
5162

52-
To build and run a Rust binary follow these steps:
53-
54-
1. Run `cargo build -p cat-gateway --release`
55-
to compile a release version of the cat-gateway
56-
2. Run `earthly ./event-db+build` to build an event-db docker image
57-
3. If you need to have a `preprod-snapshot` unarchive snapshot data to the `/tmp/preprod/` dir.
58-
You can download `preprod-snapshot` from this
59-
[resource](https://mithril.network/explorer/?aggregator=https%3A%2F%2Faggregator.release-preprod.api.mithril.network%2Faggregator).
60-
4. Run
61-
62-
```sh
63-
./target/release/cat-gateway run \
64-
--address "127.0.0.1:3030" \
65-
--database-url=postgres://catalyst-event-dev:CHANGE_ME@localhost/CatalystEventDev \
66-
--log-level=debug \
67-
--log-format=compact \
68-
--metrics-address "127.0.0.1:3032"
69-
```
63+
To build and run `cat-gateway` natively,
64+
as a first step it will be needed to anyway build and run `event-db` and `index-db` as docker containers
65+
66+
#### Build
67+
68+
* Build `cat-gateway`:
69+
70+
```sh
71+
cd catalyst-gateway
72+
cargo b --release
73+
```
74+
75+
* Build `event-db`:
76+
77+
```sh
78+
earthly ./catalyst-gateway/event-db+docker
79+
```
80+
81+
#### Run
82+
83+
* Run `event-db` and `index-db`:
84+
85+
```sh
86+
docker compose -f ./catalyst-gateway/docker-compose.yml up event-db index-db
87+
```
88+
89+
* Run `cat-gateway`:
90+
91+
```sh
92+
export SIGNED_DOC_SK="0x6455585b5dcc565c8975bc136e215d6d4dd96540620f37783c564da3cb3686dd"
93+
export CHAIN_NETWORK="Preprod"
94+
export INTERNAL_API_KEY="123"
95+
export EVENT_DB_URL="postgres://catalyst-event-dev:CHANGE_ME@localhost:5432/CatalystEventDev"
96+
./catalyst-gateway/target/release/cat-gateway run
97+
```

‎catalyst-gateway/docker-compose.yml‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
# cspell: words statusgossip
2+
3+
services:
4+
event-db:
5+
image: event-db:latest
6+
container_name: event-db
7+
environment:
8+
- DB_HOST=localhost
9+
- DB_PORT=5432
10+
- DB_NAME=CatalystEventDev
11+
- DB_DESCRIPTION="Catalyst Event DB"
12+
- DB_SUPERUSER=postgres
13+
- DB_SUPERUSER_PASSWORD=postgres
14+
- DB_USER=catalyst-event-dev
15+
- DB_USER_PASSWORD=CHANGE_ME
16+
17+
- INIT_AND_DROP_DB=true
18+
- WITH_MIGRATIONS=true
19+
ports:
20+
- "5432:5432"
21+
healthcheck:
22+
test:
23+
[
24+
"CMD-SHELL",
25+
"pg_isready -U $${DB_SUPERUSER} -d $${DB_SUPERUSER_PASSWORD}",
26+
]
27+
interval: 10s
28+
timeout: 5s
29+
retries: 10
30+
31+
index-db:
32+
image: scylladb/scylla:latest
33+
container_name: index-db
34+
restart: unless-stopped
35+
command: --smp 2 --memory 1G --overprovisioned 1 --api-address 0.0.0.0
36+
healthcheck:
37+
test: cqlsh -e "SHOW VERSION" || exit 1
38+
interval: 15s
39+
timeout: 30s
40+
retries: 10
41+
ports:
42+
- "9042:9042"
43+
44+
cat-gateway:
45+
image: cat-gateway:latest
46+
container_name: cat-gateway
47+
healthcheck:
48+
test: "curl -s -i localhost:3030/api/v1/health/ready | head -n 1 | grep 204"
49+
start_period: 30s
50+
interval: 20s
51+
timeout: 30s
52+
retries: 45
53+
environment:
54+
- EVENT_DB_URL=postgres://catalyst-event-dev:CHANGE_ME@event-db/CatalystEventDev
55+
- CASSANDRA_PERSISTENT_URL=index-db:9042
56+
- CASSANDRA_VOLATILE_URL=index-db:9042
57+
- CHAIN_NETWORK=Preprod
58+
- SIGNED_DOC_SK=0x6455585b5dcc565c8975bc136e215d6d4dd96540620f37783c564da3cb3686dd
59+
- INTERNAL_API_KEY=123
60+
# - LOG_LEVEL=error
61+
# - RUST_LOG="error,cat_gateway=debug,cardano_chain_follower=debug,mithril-client=debug"
62+
ports:
63+
- "3030:3030"
64+
depends_on:
65+
event-db:
66+
condition: service_healthy
67+
index-db:
68+
condition: service_healthy

‎catalyst-gateway/event-db/Earthfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
# the database and its associated software.
44
VERSION 0.8
55

6-
IMPORT github.com/input-output-hk/catalyst-ci/earthly/postgresql:v3.6.1 AS postgresql-ci
6+
IMPORT github.com/input-output-hk/catalyst-ci/earthly/postgresql:v3.6.2 AS postgresql-ci
77

88
# cspell: words
99

‎catalyst-gateway/tests/Earthfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# cspell: words unittests socat
22

33
VERSION 0.8
4-
IMPORT github.com/input-output-hk/catalyst-ci/earthly/spectral:v3.6.1 AS spectral-ci
4+
IMPORT github.com/input-output-hk/catalyst-ci/earthly/spectral:v3.6.2 AS spectral-ci
55
IMPORT .. AS gateway
66

77
# build all necessary docker images required to run `docker-compose.yml` services

‎catalyst-gateway/tests/api_tests/Earthfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
VERSION 0.8
22

3-
IMPORT github.com/input-output-hk/catalyst-ci/earthly/python:v3.6.1 AS python-ci
3+
IMPORT github.com/input-output-hk/catalyst-ci/earthly/python:v3.6.2 AS python-ci
44
IMPORT github.com/input-output-hk/catalyst-libs/rust:catalyst-signed-doc/v0.0.8 AS cat-libs-rust
55
IMPORT github.com/input-output-hk/catalyst-libs/rust:catalyst-signed-doc/v.0.0.4-fix-earthly-build AS dep-cat-libs-rust
66
IMPORT github.com/input-output-hk/catalyst-storage AS cat-storage

‎catalyst-gateway/tests/api_tests/integration/test_health_thru_proxy.py‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
import pytest
44

55
from api.v1 import rbac
6-
from api.v1 import document
6+
from api.v2 import document
77
from utils import health
88
from utils.rbac_chain import rbac_chain_factory, Chain
99
from utils import ProxyHelper
@@ -39,7 +39,7 @@ def test_ready_endpoint_with_event_db_outage(event_db_proxy, rbac_chain_factory)
3939
event_db_proxy.disable()
4040
health.is_ready() #assertion
4141
# event-db threshold to start returning 503
42-
sleep(35)
42+
sleep(120)
4343
resp = document.post(filter={},limit=10,page=0)
4444
assert(resp.status_code == 503), f"Expected document index to fail: {resp.status_code} - {resp.text}"
4545
health.is_not_ready(5) #assertion
@@ -84,19 +84,20 @@ def test_ready_endpoint_with_index_db_outage(index_db_proxy, rbac_chain_factory)
8484
index_db_proxy.disable()
8585
health.is_ready() #assertion
8686
# index-db threshold to start returning 503
87-
sleep(180)
88-
# Index DB testing
89-
resp = rbac.get(lookup=stake_address_not_registered, token=auth_token)
90-
assert(resp.status_code == 503), f"Expected RBAC lookup to fail: {resp.status_code} - {resp.text}"
87+
health.is_not_ready(380) #assertion
9188
# Event DB testing
9289
resp = document.post(filter={},limit=10,page=0)
9390
assert(resp.status_code == 503), f"Expected document index to fail: {resp.status_code} - {resp.text}"
91+
# Index DB testing
92+
resp = rbac.get(lookup=stake_address_not_registered, token=auth_token)
93+
assert(resp.status_code == 503), f"Expected RBAC lookup to fail: {resp.status_code} - {resp.text}"
9494

9595
# resume index db comms
9696
index_db_proxy.enable()
9797
# wait for cat-gateway API to recover
9898
health.is_ready() #assertion
99-
99+
# sleep needs to stay until bug is fixed https://github.com/input-output-hk/catalyst-voices/issues/3705
100+
sleep(20)
100101
# Index DB testing
101102
resp = rbac.get(lookup=stake_address_not_registered, token=auth_token)
102103
assert(resp.status_code == 404), f"Expected not registered stake address: {resp.status_code} - {resp.text}"

0 commit comments

Comments
 (0)