Skip to content

Commit ad71790

Browse files
aduh95aduh95
committed
tools: run CI with shared libs on GHA
1 parent fba8e6a commit ad71790

File tree

6 files changed

+331
-1
lines changed

6 files changed

+331
-1
lines changed

‎.github/workflows/test-shared.yml‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
name: Test Shared librairies
2+
3+
on:
4+
pull_request:
5+
paths-ignore:
6+
- .mailmap
7+
- README.md
8+
- .github/**
9+
- '!.github/workflows/test-shared.yml'
10+
types: [opened, synchronize, reopened, ready_for_review]
11+
push:
12+
branches:
13+
- main
14+
- canary
15+
- v[0-9]+.x-staging
16+
- v[0-9]+.x
17+
paths-ignore:
18+
- .mailmap
19+
- README.md
20+
- .github/**
21+
- '!.github/workflows/test-shared.yml'
22+
23+
concurrency:
24+
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
25+
cancel-in-progress: true
26+
27+
env:
28+
FLAKY_TESTS: keep_retrying
29+
NIXPKGS_REPO: https://github.com/NixOS/nixpkgs
30+
NIXPKGS_PIN: c53baa6685261e5253a1c355a1b322f82674a824
31+
32+
permissions:
33+
contents: read
34+
35+
jobs:
36+
build:
37+
strategy:
38+
fail-fast: false
39+
matrix:
40+
include:
41+
- runner: ubuntu-24.04
42+
system: x86_64-linux
43+
- runner: ubuntu-24.04-arm
44+
system: aarch64-linux
45+
- runner: macos-13
46+
system: x86_64-darwin
47+
- runner: macos-14
48+
system: aarch64-darwin
49+
name: '${{ matrix.system }}: with shared libraries'
50+
runs-on: ${{ matrix.runner }}
51+
steps:
52+
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
53+
54+
- uses: cachix/install-nix-action@f0fe604f8a612776892427721526b4c7cfb23aba # v31
55+
with:
56+
extra_nix_config: sandbox = true
57+
github_access_token: ${{ secrets.GITHUB_TOKEN }}
58+
nix_path: nixpkgs=${{ env.NIXPKGS_REPO }}/archive/${{ env.NIXPKGS_PIN }}.tar.gz
59+
60+
- uses: cachix/cachix-action@0fc020193b5a1fa3ac4575aa3a7d3aa6a35435ad # v16
61+
with:
62+
name: nodejs
63+
authToken: ${{ secrets.CACHIX_AUTH_TOKEN }}
64+
65+
- name: Configure sccache
66+
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7.0.1
67+
with:
68+
script: |
69+
core.exportVariable('SCCACHE_GHA_VERSION', 'on');
70+
core.exportVariable('ACTIONS_CACHE_SERVICE_V2', 'on');
71+
core.exportVariable('ACTIONS_RESULTS_URL', process.env.ACTIONS_RESULTS_URL || '');
72+
core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');
73+
74+
- name: Patch npm gyp
75+
run: curl -L https://github.com/npm/cli/pull/8531.diff | git apply --directory=deps/npm --exclude=deps/npm/package-lock.json
76+
77+
- name: Build Node.js and run tests
78+
run: |
79+
nix-shell \
80+
--pure --keep FLAKY_TESTS \
81+
--keep SCCACHE_GHA_VERSION --keep ACTIONS_CACHE_SERVICE_V2 --keep ACTIONS_RESULTS_URL --keep ACTIONS_RUNTIME_TOKEN \
82+
--arg loadJSBuiltinsDynamically false \
83+
--arg ccache '(import <nixpkgs> {}).sccache' \
84+
--arg devTools '[]' \
85+
--arg benchmarkTools '[]' \
86+
--run '
87+
make run-ci -j4 V=1 TEST_CI_ARGS="-p actions --measure-flakiness 9 --skip-tests=$CI_SKIP_TESTS"
88+
'

‎.github/workflows/tools.yml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ on:
3030
- llhttp
3131
- minimatch
3232
- nbytes
33+
- nixpkgs-unstable
3334
- nghttp2
3435
- nghttp3
3536
- ngtcp2
@@ -198,6 +199,14 @@ jobs:
198199
cat temp-output
199200
tail -n1 temp-output | grep "NEW_VERSION=" >> "$GITHUB_ENV" || true
200201
rm temp-output
202+
- id: nixpkgs-unstable
203+
subsystem: tools
204+
label: tools
205+
run: |
206+
./tools/dep_updaters/update-nixpkgs-pin.sh > temp-output
207+
cat temp-output
208+
tail -n1 temp-output | grep "NEW_VERSION=" >> "$GITHUB_ENV" || true
209+
rm temp-output
201210
- id: nghttp2
202211
subsystem: deps
203212
label: dependencies

‎BUILDING.md‎

Lines changed: 70 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -239,6 +239,7 @@ Consult previous versions of this document for older versions of Node.js:
239239

240240
Installation via Linux package manager can be achieved with:
241241

242+
* Nix, NixOS: `nix-shell`
242243
* Ubuntu, Debian: `sudo apt-get install python3 g++-12 gcc-12 make python3-pip`
243244
* Fedora: `sudo dnf install python3 gcc-c++ make python3-pip`
244245
* CentOS and RHEL: `sudo yum install python3 gcc-c++ make python3-pip`
@@ -259,6 +260,75 @@ installed, you can find them under the menu `Xcode -> Open Developer Tool ->
259260
More Developer Tools...`. This step will install `clang`, `clang++`, and
260261
`make`.
261262

263+
#### Nix integration
264+
265+
If you are using Nix and direnv, you can use the following to get started:
266+
267+
```bash
268+
echo 'use_nix --arg sharedLibDeps {} --argstr icu small' > .envrc
269+
direnv allow .
270+
make build-ci -j12
271+
```
272+
273+
The use of `make build-ci` is to ensure you are using the `CONFIG_FLAGS`
274+
environment variable. You can also specify it manually:
275+
276+
```sh
277+
./configure $CONFIG_FLAGS
278+
make -j12
279+
```
280+
281+
Passing the `--arg sharedLibDeps {}` instructs direnv and Nix to generate an
282+
environment that uses the vendored-in native dependencies. Using the vendored-in
283+
dependencies result in a result closer to the official binaries, the tradeoff
284+
being the build will take longer to complete as you'd have to build those
285+
dependencies instead of using the cached ones from the Nix cache. You can omit
286+
that flag to use all the shared dependencies, or specify only some dependencies:
287+
288+
```bash
289+
cat -> .envrc <<'EOF'
290+
use nix --arg sharedLibDeps '{
291+
inherit (import <nixpkgs> {})
292+
openssl
293+
zlib
294+
;
295+
}'
296+
EOF
297+
```
298+
299+
Passing the `--argstr icu small` instructs direnv and Nix to pass `--with-intl=small` in
300+
the `CONFIG_FLAGS` environment variable. If you omit this, the prebuilt ICU from Nix cache
301+
will be used, which should speed up greatly compilation time.
302+
303+
The use of `direnv` is completely optional, you can also use `nix-shell` directly,
304+
e.g. here's a command you can use to build a binary for benchmarking purposes:
305+
306+
```bash
307+
# Passing `--arg loadJSBuiltinsDynamically false` to instruct the compiler to
308+
# embed the JS core files so it is no longer affected by local changes
309+
# (necessary for getting useful benchmark results).
310+
# Passing `--arg devTools '[]' --arg benchmarkTools '[]'` since we don't need
311+
# those to build node.
312+
nix-shell \
313+
--arg loadJSBuiltinsDynamically false \
314+
--arg devTools '[]' --arg benchmarkTools '[]' \
315+
--run 'make build-ci -j12'
316+
317+
mv out/Release/node ./node_old
318+
319+
# ...
320+
# Make your local changes, and re-build node
321+
322+
nix-shell \
323+
--arg loadJSBuiltinsDynamically false \
324+
--arg devTools '[]' --arg benchmarkTools '[]' \
325+
--run 'make build-ci -j12'
326+
327+
nix-shell --pure --run './node benchmark/compare.js --old ./node_old --new ./node http | Rscript benchmark/compare.R'
328+
```
329+
330+
There are additional attributes you can pass, see `shell.nix` file for more details.
331+
262332
#### Building Node.js
263333

264334
If the path to your build directory contains a space, the build will likely
@@ -267,7 +337,6 @@ fail.
267337
To build Node.js:
268338

269339
```bash
270-
export CXX=g++-12
271340
./configure
272341
make -j4
273342
```

‎doc/contributing/writing-and-running-benchmarks.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,10 @@ Basic Unix tools are required for some benchmarks.
2626
[Git for Windows][git-for-windows] includes Git Bash and the necessary tools,
2727
which need to be included in the global Windows `PATH`.
2828

29+
If you are using Nix, all the required tools are already listed in the
30+
`benchmarkTools` argument of the `shell.nix` file, so you can skip those
31+
prerequesites.
32+
2933
### HTTP benchmark requirements
3034

3135
Most of the HTTP benchmarks require a benchmarker to be installed. This can be

‎shell.nix‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
{
2+
pkgs ? import <nixpkgs> {},
3+
loadJSBuiltinsDynamically ? true, # Load `lib/**.js` from disk instead of embedding
4+
ncu-path ? null, # Provide this if you want to use a local version of NCU
5+
icu ? pkgs.icu,
6+
sharedLibDeps ? {
7+
inherit (pkgs)
8+
ada
9+
brotli
10+
c-ares
11+
libuv
12+
nghttp2
13+
nghttp3
14+
ngtcp2
15+
openssl
16+
simdjson
17+
sqlite
18+
zlib
19+
zstd
20+
;
21+
http-parser = pkgs.llhttp;
22+
simdutf = pkgs.simdutf.overrideAttrs {
23+
version = "6.5.0";
24+
25+
src = pkgs.fetchFromGitHub {
26+
owner = "simdutf";
27+
repo = "simdutf";
28+
tag = "v6.5.0";
29+
hash = "sha256-bZ4r62GMz2Dkd3fKTJhelitaA8jUBaDjG6jOysEg8Nk=";
30+
};
31+
};
32+
uvwasi = pkgs.uvwasi.overrideAttrs {
33+
version = "0.0.22";
34+
35+
src = pkgs.fetchFromGitHub {
36+
owner = "nodejs";
37+
repo = "uvwasi";
38+
rev = "refs/heads/v0.0.22-proposal";
39+
hash = "sha256-vrtL9HDzbbbx3QmRsifRzNTYilC4zgo9JmJoA9haUFQ=";
40+
};
41+
postPatch = ''
42+
substituteInPlace CMakeLists.txt \
43+
--replace-fail 'list(APPEND uvwasi_cflags -fvisibility=hidden --std=gnu89)' 'list(APPEND uvwasi_cflags --std=gnu89)'
44+
'';
45+
cmakeFlags = [
46+
"-DUVWASI_BUILD_TESTS=OFF"
47+
];
48+
};
49+
},
50+
ccache ? pkgs.ccache,
51+
ninja ? pkgs.ninja,
52+
devTools ? [
53+
pkgs.curl
54+
pkgs.gh
55+
pkgs.git
56+
pkgs.jq
57+
pkgs.shellcheck
58+
] ++ (if (ncu-path == null) then [
59+
pkgs.node-core-utils
60+
] else [
61+
(pkgs.writeShellScriptBin "git-node" "exec \"${ncu-path}/bin/git-node.js\" \"$@\"")
62+
(pkgs.writeShellScriptBin "ncu-ci" "exec \"${ncu-path}/bin/ncu-ci.js\" \"$@\"")
63+
(pkgs.writeShellScriptBin "ncu-config" "exec \"${ncu-path}/bin/ncu-config.js\" \"$@\"")
64+
]),
65+
benchmarkTools ? [
66+
pkgs.R
67+
pkgs.rPackages.ggplot2
68+
pkgs.rPackages.plyr
69+
pkgs.wrk
70+
],
71+
extraConfigFlags ? [
72+
"--without-npm"
73+
"--debug-node"
74+
],
75+
}:
76+
77+
let
78+
useSharedICU = if builtins.isString icu then icu == "system" else icu != null;
79+
useSharedAda = builtins.hasAttr "ada" sharedLibDeps;
80+
useSharedOpenSSL = builtins.hasAttr "openssl" sharedLibDeps;
81+
in
82+
pkgs.mkShell {
83+
inherit (pkgs.nodejs_latest) nativeBuildInputs;
84+
85+
buildInputs = builtins.attrValues sharedLibDeps ++ pkgs.lib.optionals useSharedICU [ icu ];
86+
87+
packages = [
88+
ccache
89+
] ++ devTools ++ benchmarkTools;
90+
91+
shellHook = if (ccache != null) then ''
92+
export CC="${pkgs.lib.getExe ccache} $CC"
93+
export CXX="${pkgs.lib.getExe ccache} $CXX"
94+
ls "${pkgs.lib.getLib sharedLibDeps.uvwasi}"
95+
'' else "";
96+
97+
BUILD_WITH = if (ninja != null) then "ninja" else "make";
98+
NINJA = if (ninja != null) then "${pkgs.lib.getExe ninja}" else "";
99+
CI_SKIP_TESTS = pkgs.lib.concatStringsSep "," ([
100+
] ++ pkgs.lib.optionals useSharedAda [
101+
# Different versions of Ada affect the WPT tests
102+
"test-url"
103+
] ++ pkgs.lib.optionals useSharedOpenSSL [
104+
# Path to the openssl.cnf is different from the expected one
105+
"test-strace-openat-openssl"
106+
]);
107+
CONFIG_FLAGS = builtins.toString ([
108+
(if icu == null
109+
then "--without-intl"
110+
else "--with-intl=${if useSharedICU then "system" else icu}-icu")
111+
] ++ extraConfigFlags ++ pkgs.lib.optionals (ninja != null) [
112+
"--ninja"
113+
] ++ pkgs.lib.optionals loadJSBuiltinsDynamically [
114+
"--node-builtin-modules-path=${builtins.toString ./.}"
115+
] ++ pkgs.lib.concatMap (name: [
116+
"--shared-${builtins.replaceStrings [ "c-ares" ] [ "cares" ] name}"
117+
"--shared-${builtins.replaceStrings [ "c-ares" ] [ "cares" ] name}-libpath=${
118+
pkgs.lib.getLib sharedLibDeps.${name}
119+
}/lib"
120+
"--shared-${builtins.replaceStrings [ "c-ares" ] [ "cares" ] name}-include=${
121+
pkgs.lib.getInclude sharedLibDeps.${name}
122+
}/include"
123+
]) (builtins.attrNames sharedLibDeps));
124+
}

‎tools/dep_updaters/update-nixpkgs-pin.sh‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
#!/bin/sh
2+
set -ex
3+
# Shell script to update Nixpkgs pin in the source tree to the most recent
4+
# version on the unstable channel.
5+
6+
BASE_DIR=$(cd "$(dirname "$0")/../.." && pwd)
7+
WORKFLOW_FILE="$BASE_DIR/.github/workflows/test-shared.yml"
8+
9+
NIXPKGS_REPO=$(grep NIXPKGS_REPO: "$WORKFLOW_FILE" | awk -F': ' '{ print $2 }')
10+
CURRENT_VERSION_SHA1=$(grep NIXPKGS_PIN: "$WORKFLOW_FILE" | awk -F': ' '{ print $2 }')
11+
CURRENT_VERSION=$(echo "$CURRENT_UPSTREAM_SHA1" | head -c 7)
12+
13+
NEW_UPSTREAM_SHA1=$(git ls-remote "$NIXPKGS_REPO.git" nixpkgs-unstable | awk '{print $1}')
14+
NEW_VERSION=$(echo "$NEW_UPSTREAM_SHA1" | head -c 7)
15+
16+
17+
# shellcheck disable=SC1091
18+
. "$BASE_DIR/tools/dep_updaters/utils.sh"
19+
20+
compare_dependency_version "nixpkgs-unstable" "$NEW_VERSION" "$CURRENT_VERSION"
21+
22+
TMP_FILE=$(mktemp)
23+
sed "s/$CURRENT_VERSION_SHA1/$NEW_UPSTREAM_SHA1/" "$WORKFLOW_FILE" > "$TMP_FILE"
24+
mv "$TMP_FILE" "$WORKFLOW_FILE"
25+
26+
echo "All done!"
27+
echo ""
28+
echo "Please git add and commit the new version:"
29+
echo ""
30+
echo "$ git add $WORKFLOW_FILE"
31+
echo "$ git commit -m 'tools: bump nixpkgs-unstable pin to $NEW_VERSION'"
32+
echo ""
33+
34+
# The last line of the script should always print the new version,
35+
# as we need to add it to $GITHUB_ENV variable.
36+
echo "NEW_VERSION=$NEW_VERSION"

0 commit comments

Comments
 (0)