Skip to content

Commit 979cc39

Browse files
cdeckerclaude
cdecker
and
claude
committed
Add comprehensive coverage infrastructure with clang source-based coverage
This commit introduces a modern coverage infrastructure for Core Lightning: - Migrate from ad-hoc coverage script to integrated Makefile targets - Add LLVM source-based coverage support with per-test profraw organization - Integrate coverage collection into pytest framework via TailableProc - Add GitHub Actions workflow for nightly coverage reports - Add Taskfile.yml for convenient task automation - Add codecov.yml for Codecov integration - Add comprehensive coverage documentation in COVERAGE.md - Update contributor workflow docs with new coverage script path - Add coverage data files to .gitignore (*.profraw, *.profdata) - Remove obsolete contrib/clang-coverage-report.sh - Remove obsolete tests/conftest.py (now using pyln-testing markers) - Update pyproject.toml to include pyln-testing in main dependencies The new infrastructure automatically collects coverage data when CLN_COVERAGE_DIR is set, organizing profraw files by test name for granular analysis. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
1 parent 5c0827e commit 979cc39

File tree

14 files changed

+656
-85
lines changed

14 files changed

+656
-85
lines changed

‎.github/workflows/coverage-nightly.yaml‎

Lines changed: 177 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,177 @@
1+
name: Coverage (Nightly)
2+
3+
on:
4+
schedule:
5+
# Run at 2 AM UTC every day
6+
- cron: '0 2 * * *'
7+
# Allow manual triggers for testing
8+
workflow_dispatch:
9+
10+
concurrency:
11+
group: coverage-${{ github.ref }}
12+
cancel-in-progress: true
13+
14+
jobs:
15+
compile:
16+
name: Build with Coverage
17+
runs-on: ubuntu-22.04
18+
steps:
19+
- name: Checkout
20+
uses: actions/checkout@v4
21+
22+
- name: Set up Python 3.10
23+
uses: actions/setup-python@v5
24+
with:
25+
python-version: "3.10"
26+
27+
- name: Install uv
28+
uses: astral-sh/setup-uv@v5
29+
30+
- name: Install dependencies
31+
run: bash -x .github/scripts/setup.sh
32+
33+
- name: Build with coverage instrumentation
34+
run: |
35+
./configure --enable-debugbuild --enable-coverage CC=clang
36+
uv run make -j $(nproc) testpack.tar.bz2
37+
38+
- name: Upload build artifact
39+
uses: actions/upload-artifact@v4
40+
with:
41+
name: cln-coverage-build
42+
path: testpack.tar.bz2
43+
44+
test:
45+
name: Test (${{ matrix.name }})
46+
runs-on: ubuntu-22.04
47+
needs: compile
48+
strategy:
49+
fail-fast: false
50+
matrix:
51+
include:
52+
- name: sqlite
53+
db: sqlite3
54+
pytest_par: 10
55+
- name: postgres
56+
db: postgres
57+
pytest_par: 10
58+
59+
steps:
60+
- name: Checkout
61+
uses: actions/checkout@v4
62+
63+
- name: Set up Python 3.10
64+
uses: actions/setup-python@v5
65+
with:
66+
python-version: "3.10"
67+
68+
- name: Install uv
69+
uses: astral-sh/setup-uv@v5
70+
71+
- name: Install dependencies
72+
run: bash -x .github/scripts/setup.sh
73+
74+
- name: Install Bitcoin Core
75+
run: bash -x .github/scripts/install-bitcoind.sh
76+
77+
- name: Download build artifact
78+
uses: actions/download-artifact@v4
79+
with:
80+
name: cln-coverage-build
81+
82+
- name: Unpack build
83+
run: tar -xaf testpack.tar.bz2
84+
85+
- name: Run tests with coverage
86+
env:
87+
CLN_COVERAGE_DIR: ${{ github.workspace }}/coverage-raw
88+
TEST_DB_PROVIDER: ${{ matrix.db }}
89+
PYTEST_PAR: ${{ matrix.pytest_par }}
90+
SLOW_MACHINE: 1
91+
TIMEOUT: 900
92+
run: |
93+
mkdir -p "$CLN_COVERAGE_DIR"
94+
uv run eatmydata pytest tests/ -n ${PYTEST_PAR} -vvv
95+
96+
- name: Upload coverage data
97+
uses: actions/upload-artifact@v4
98+
if: always()
99+
with:
100+
name: coverage-raw-${{ matrix.name }}
101+
path: coverage-raw/*.profraw
102+
if-no-files-found: error
103+
104+
report:
105+
name: Generate Coverage Report
106+
runs-on: ubuntu-22.04
107+
needs: test
108+
if: always()
109+
110+
steps:
111+
- name: Checkout
112+
uses: actions/checkout@v4
113+
114+
- name: Install LLVM tools
115+
run: |
116+
wget https://apt.llvm.org/llvm.sh
117+
chmod +x llvm.sh
118+
sudo ./llvm.sh 18
119+
sudo ln -sf /usr/bin/llvm-profdata-18 /usr/bin/llvm-profdata
120+
sudo ln -sf /usr/bin/llvm-cov-18 /usr/bin/llvm-cov
121+
122+
- name: Download build artifact
123+
uses: actions/download-artifact@v4
124+
with:
125+
name: cln-coverage-build
126+
127+
- name: Unpack build
128+
run: tar -xaf testpack.tar.bz2
129+
130+
- name: Download all coverage artifacts
131+
uses: actions/download-artifact@v4
132+
with:
133+
pattern: coverage-raw-*
134+
path: coverage-artifacts
135+
136+
- name: Merge coverage data
137+
run: |
138+
mkdir -p coverage-raw coverage
139+
find coverage-artifacts -name "*.profraw" -exec cp {} coverage-raw/ \;
140+
PROFRAW_COUNT=$(ls -1 coverage-raw/*.profraw 2>/dev/null | wc -l)
141+
echo "Found $PROFRAW_COUNT profile files"
142+
if [ "$PROFRAW_COUNT" -eq 0 ]; then
143+
echo "ERROR: No coverage data found"
144+
exit 1
145+
fi
146+
chmod +x contrib/coverage/collect-coverage.sh
147+
CLN_COVERAGE_DIR=coverage-raw ./contrib/coverage/collect-coverage.sh
148+
149+
- name: Generate HTML report
150+
run: |
151+
chmod +x contrib/coverage/generate-coverage-report.sh
152+
./contrib/coverage/generate-coverage-report.sh
153+
154+
- name: Upload to Codecov
155+
uses: codecov/codecov-action@v4
156+
with:
157+
files: coverage/merged.profdata
158+
flags: integration-tests
159+
name: cln-nightly-coverage
160+
token: ${{ secrets.CODECOV_TOKEN }}
161+
fail_ci_if_error: false
162+
163+
- name: Upload HTML report
164+
uses: actions/upload-artifact@v4
165+
with:
166+
name: coverage-html-report
167+
path: coverage/html
168+
retention-days: 90
169+
170+
- name: Add summary to job
171+
run: |
172+
echo "## Coverage Summary" >> $GITHUB_STEP_SUMMARY
173+
echo '```' >> $GITHUB_STEP_SUMMARY
174+
cat coverage/summary.txt >> $GITHUB_STEP_SUMMARY
175+
echo '```' >> $GITHUB_STEP_SUMMARY
176+
echo "" >> $GITHUB_STEP_SUMMARY
177+
echo "📊 Download detailed HTML report from workflow artifacts" >> $GITHUB_STEP_SUMMARY

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,9 @@ gen_*.h
2424
wire/gen_*_csv
2525
cli/lightning-cli
2626
coverage
27+
# Coverage profiling data files
28+
*.profraw
29+
*.profdata
2730
ccan/config.h
2831
__pycache__
2932
config.vars

‎Makefile‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -674,6 +674,21 @@ coverage/coverage.info: check pytest
674674
coverage: coverage/coverage.info
675675
genhtml coverage/coverage.info --output-directory coverage
676676

677+
# Clang coverage targets (source-based coverage)
678+
coverage-clang-collect:
679+
@./contrib/coverage/collect-coverage.sh "$(CLN_COVERAGE_DIR)" coverage/merged.profdata
680+
681+
coverage-clang-report: coverage/merged.profdata
682+
@./contrib/coverage/generate-coverage-report.sh coverage/merged.profdata coverage/html
683+
684+
coverage-clang: coverage-clang-collect coverage-clang-report
685+
@echo "Coverage report: coverage/html/index.html"
686+
687+
coverage-clang-clean:
688+
rm -rf coverage/ "$(CLN_COVERAGE_DIR)"
689+
690+
.PHONY: coverage-clang-collect coverage-clang-report coverage-clang coverage-clang-clean
691+
677692
# We make libwallycore.la a dependency, so that it gets built normally, without ncc.
678693
# Ncc can't handle the libwally source code (yet).
679694
ncc: ${TARGET_DIR}/libwally-core-build/src/libwallycore.la

‎Taskfile.yml‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
version: '3'
2+
3+
vars:
4+
PYTEST_PAR: 4
5+
6+
tasks:
7+
build:
8+
cmds:
9+
- uv run make cln-grpc/proto/node.proto
10+
- uv run make default -j {{ .PYTEST_PAR }}
11+
test:
12+
dir: '.'
13+
deps:
14+
- build
15+
cmds:
16+
- uv run pytest --force-flaky -vvv -n {{ .PYTEST_PAR }} tests {{ .CLI_ARGS }}
17+
18+
test-liquid:
19+
env:
20+
TEST_NETWORK: "liquid-regtest"
21+
cmds:
22+
- sed -i 's/TEST_NETWORK=regtest/TEST_NETWORK=liquid-regtest/g' config.vars
23+
- uv run make cln-grpc/proto/node.proto
24+
- uv run make default -j {{ .PYTEST_PAR }}
25+
- uv run pytest --color=yes --force-flaky -vvv -n {{ .PYTEST_PAR }} tests {{ .CLI_ARGS }}
26+
27+
clean:
28+
cmds:
29+
- poetry run make distclean
30+
31+
32+
tester-docker-image:
33+
cmds:
34+
- docker build --build-arg DOCKER_USER=$(whoami) --build-arg UID=$(id -u) --build-arg GID=$(id -g) --network=host -t cln-tester - <contrib/docker/Dockerfile.tester
35+
36+
isotest:
37+
dir: '.'
38+
deps:
39+
- tester-docker-image
40+
cmds:
41+
- docker run -ti -v $(pwd):/repo cln-tester bash -c 'task -t /repo/Taskfile.yml in-docker-test'
42+
43+
in-docker-build-deps:
44+
cmds:
45+
- sudo apt-get update -q
46+
- sudo apt-get install -y clang jq libsqlite3-dev libpq-dev systemtap-sdt-dev autoconf libtool zlib1g-dev libsodium-dev gettext git
47+
48+
in-docker-init:
49+
# pre-flight tasks, independent of the source code.
50+
dir: '/'
51+
cmds:
52+
- git config --global --add safe.directory /repo/.git
53+
- mkdir -p /test
54+
- python3 -m venv /test/.venv
55+
- /test/.venv/bin/pip3 install wheel
56+
- /test/.venv/bin/pip3 install 'poetry>=1.8,<2'
57+
58+
in-docker-test:
59+
# Just the counterpart called by `isotest` to actually initialize,
60+
# build and test CLN.
61+
dir: '/test'
62+
deps:
63+
- in-docker-init
64+
- in-docker-build-deps
65+
cmds:
66+
# This way of copying allows us to copy the dirty tree, without
67+
# triggering any of the potentially configured hooks which might
68+
# not be available in the docker image.
69+
- (cd /repo && git archive --format tar $(git stash create)) | tar -xvf -
70+
# Yes, this is not that smart, but the `Makefile` relies on
71+
# `git` being able to tell us about the version.
72+
- cp -R /repo/.git /test
73+
- git submodule update --init --recursive
74+
- python3 -m pip install poetry
75+
- poetry run make distclean
76+
- poetry install --with=dev
77+
- poetry run ./configure --disable-valgrind CC='clang'
78+
- poetry run make -j 4
79+
- poetry run pytest --color=yes -vvv -n {{ .PYTEST_PAR }} tests {{ .CLI_ARGS }}
80+
81+
kill:
82+
cmds:
83+
- killall -v bitcoind || true
84+
- killall -v elementsd || true
85+
- killall -v valgrind.bin || true

‎codecov.yml‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
coverage:
2+
status:
3+
project:
4+
default:
5+
# Coverage can decrease by up to 1% and still pass
6+
target: auto
7+
threshold: 1%
8+
patch:
9+
default:
10+
# New code should maintain coverage
11+
target: auto
12+
13+
comment:
14+
# Post coverage comments on PRs (if we add PR coverage later)
15+
behavior: default
16+
layout: "header, diff, files"
17+
require_changes: false
18+
19+
# Ignore files that shouldn't affect coverage metrics
20+
ignore:
21+
- "external/**"
22+
- "ccan/**"
23+
- "*/test/**"
24+
- "tools/**"
25+
- "contrib/**"
26+
- "doc/**"
27+
- "devtools/**"
28+
29+
# Don't fail if coverage data is incomplete
30+
codecov:
31+
require_ci_to_pass: false

‎contrib/clang-coverage-report.sh‎

Lines changed: 0 additions & 27 deletions
This file was deleted.

‎contrib/pyln-testing/pyln/testing/utils.py‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -197,6 +197,29 @@ class TailableProc(object):
197197
def __init__(self, outputDir, verbose=True):
198198
self.logs = []
199199
self.env = os.environ.copy()
200+
201+
# Add coverage support: inject LLVM_PROFILE_FILE if CLN_COVERAGE_DIR is set
202+
if os.getenv('CLN_COVERAGE_DIR'):
203+
coverage_dir = os.getenv('CLN_COVERAGE_DIR')
204+
205+
# Organize profraw files by test name for per-test coverage analysis
206+
test_name = os.getenv('CLN_TEST_NAME')
207+
if test_name:
208+
test_coverage_dir = os.path.join(coverage_dir, test_name)
209+
os.makedirs(test_coverage_dir, exist_ok=True)
210+
profraw_path = test_coverage_dir
211+
else:
212+
os.makedirs(coverage_dir, exist_ok=True)
213+
profraw_path = coverage_dir
214+
215+
# %p=PID, %m=binary signature prevents collisions across parallel processes
216+
# Note: We don't use %c (continuous mode) as it causes "__llvm_profile_counter_bias"
217+
# errors with our multi-binary setup. Instead, we validate and filter corrupt files
218+
# during collection (see contrib/coverage/collect-coverage.sh)
219+
self.env['LLVM_PROFILE_FILE'] = os.path.join(
220+
profraw_path, '%p-%m.profraw'
221+
)
222+
200223
self.proc = None
201224
self.outputDir = outputDir
202225
if not os.path.exists(outputDir):
@@ -1635,6 +1658,10 @@ def __init__(self, request, testname, bitcoind, executor, directory,
16351658
else:
16361659
self.valgrind = VALGRIND
16371660
self.testname = testname
1661+
1662+
# Set test name in environment for coverage file organization
1663+
os.environ['CLN_TEST_NAME'] = testname
1664+
16381665
self.next_id = 1
16391666
self.nodes = []
16401667
self.reserved_ports = []

‎devtools/check-bolt‎

356 KB
Binary file not shown.

0 commit comments

Comments
 (0)