Skip to content

Commit d40a130

Browse files
sofislmiguelvelezsa
sofisl
and
miguelvelezsa
committed
build: integrate nodejs core libraries (#848)
* build: run interdependent tests * build: run interdependent tests * fix * fix(ci): make run-interdependent-tests.sh safer The script was failing when running in a directory that does not have a 'packages' subdirectory. This commit adds a check to ensure the directory exists before trying to list its contents. * fix(deps): syncpack dependency versions This commit fixes the dependency mismatches reported by syncpack. * fix(ts): address type errors from @types/node update This commit fixes the TypeScript errors that were introduced after updating the dependencies with syncpack. The errors were related to the new definition of Uint8Array in @types/node. * ci(syncpack): automate fixing of mismatches This commit updates the syncpack workflow to automatically fix dependency mismatches and commit the changes. * Update package.json * revert(ci): remove automated fix for syncpack This reverts the previous commit that automated the fixing of syncpack mismatches. * fix(deps): update dependencies to resolve syncpack mismatches * fix(proto3-json-serializer): resolve Uint8Array and Buffer type issues * run lint * feat: add top-level lint-fix script This commit adds a top-level lint-fix script that runs in all packages. It also fixes an issue in package that was causing the linting to fail. * fix(lint): run gts fix on all packages * chore: make sure we are actually installing packages locally * remove the package-lock.json * fix * chore: fix * retry * chore: fix * Update continuous.yaml * run lint * chore: fix nodejs runtime * address comments * remove numbers * pin typescript dependency --------- Co-authored-by: miguel <miguelvelezsa@google.com>
1 parent ab5c8c9 commit d40a130

File tree

53 files changed

+902
-454
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

53 files changed

+902
-454
lines changed

‎.github/workflows/continuous.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -194,4 +194,4 @@ jobs:
194194
npm install
195195
npm test
196196
npm run system-test
197-
197+

‎.github/workflows/syncpack.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
on:
2+
push:
3+
branches:
4+
- main
5+
pull_request:
6+
7+
name: syncpack
8+
9+
jobs:
10+
syncpack:
11+
runs-on: ubuntu-latest
12+
steps:
13+
- uses: actions/checkout@v5
14+
- uses: actions/setup-node@v4
15+
with:
16+
node-version: 18
17+
- run: npm install
18+
- run: npm install -g syncpack
19+
- run: syncpack list-mismatches

‎ci/lint-fix.sh‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
#!/bin/bash
2+
#
3+
# Copyright 2023 Google LLC
4+
#
5+
# Licensed under the Apache License, Version 2.0 (the "License");
6+
# you may not use this file except in compliance with the License.
7+
# You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
17+
set -eo pipefail
18+
19+
export PROJECT_ROOT=$(realpath $(dirname "${BASH_SOURCE[0]}")/..)
20+
21+
subdirs=(
22+
dev-packages
23+
.github/scripts
24+
packages
25+
generator
26+
)
27+
28+
for subdir in ${subdirs[@]}; do
29+
for d in `ls -d ${PROJECT_ROOT}/${subdir}/*/ 2>/dev/null`; do
30+
if [ -f "${d}package.json" ]; then
31+
if grep -q '"fix"' "${d}package.json"; then
32+
echo "running lint fix in ${d}"
33+
pushd ${d}
34+
rm -rf node_modules
35+
rm -rf build
36+
npm install --ignore-scripts --engine-strict; npm install
37+
npm run fix
38+
popd
39+
fi
40+
fi
41+
done
42+
done

‎ci/run-interdependent-tests.sh‎

Lines changed: 218 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,218 @@
1+
#!/bin/bash
2+
# Copyright 2023 Google LLC
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# https://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
set -e
17+
18+
# This script is used to run tests on all of the interdependent libraries
19+
# in this monorepo. It will:
20+
#
21+
# 1. Build a dependency graph of all the libraries in the monorepo
22+
# 2. Run the tests for each library in the correct order
23+
# 3. Make sure that there are no conflicting dependencies in the monorepo
24+
#
25+
# This script is meant to be run in a CI environment.
26+
27+
# Get the root directory of the monorepo
28+
ROOT_DIR=${ROOT_DIR:-$(pwd)}
29+
30+
# Get the test type from the first argument, default to "test"
31+
TEST_COMMAND="${1:-test}"
32+
33+
# An array of all the packages in the monorepo
34+
if [ -d "$ROOT_DIR/packages/" ]; then
35+
PACKAGE_DIRS=$(ls -d "$ROOT_DIR/packages/"*/)
36+
else
37+
PACKAGE_DIRS=()
38+
fi
39+
40+
# Two arrays to hold the dependency graph
41+
PACKAGE_NAMES=()
42+
PACKAGE_DEPS=()
43+
# Arrays to simulate an associative array mapping package name to its directory name
44+
PKG_NAME_TO_DIR_MAP_KEYS=()
45+
PKG_NAME_TO_DIR_MAP_VALUES=()
46+
47+
# A function to get the package name from package.json
48+
get_package_name() {
49+
local package_dir=$1
50+
local package_json="$package_dir/package.json"
51+
if [ -f "$package_json" ]; then
52+
cat "$package_json" | jq -r '.name'
53+
fi
54+
}
55+
56+
# A function to get the dependencies of a package
57+
get_dependencies() {
58+
local package_dir=$1
59+
local package_json="$package_dir/package.json"
60+
if [ -f "$package_json" ]; then
61+
local dependencies=$(cat "$package_json" | jq -r '(.dependencies // {}) | keys | .[]' | tr '\n' ' ')
62+
local dev_dependencies=$(cat "$package_json" | jq -r '(.devDependencies // {}) | keys | .[]' | tr '\n' ' ')
63+
echo "$dependencies $dev_dependencies"
64+
fi
65+
}
66+
67+
# A function to build the dependency graph
68+
build_dependency_graph() {
69+
for package_dir in ${PACKAGE_DIRS[@]}; do
70+
package_dir=${package_dir%/}
71+
package_name=$(get_package_name "$package_dir")
72+
if [ -z "$package_name" ]; then
73+
echo "Warning: could not get package name from $package_dir/package.json. Skipping."
74+
continue
75+
fi
76+
dependencies=$(get_dependencies "$package_dir")
77+
PACKAGE_NAMES+=("$package_name")
78+
PACKAGE_DEPS+=("$dependencies")
79+
PKG_NAME_TO_DIR_MAP_KEYS+=("$package_name")
80+
PKG_NAME_TO_DIR_MAP_VALUES+=("$(basename "$package_dir")")
81+
done
82+
}
83+
84+
# Build the dependency graph
85+
build_dependency_graph
86+
87+
# A function toposort the dependency graph
88+
toposort() {
89+
local -a visited
90+
local -a recursion_stack
91+
local -a sorted_packages
92+
93+
for i in "${!PACKAGE_NAMES[@]}"; do
94+
visited[$i]=0
95+
recursion_stack[$i]=0
96+
done
97+
98+
for i in "${!PACKAGE_NAMES[@]}"; do
99+
if [ "${visited[$i]}" -eq 0 ]; then
100+
toposort_util "$i"
101+
fi
102+
done
103+
104+
echo "${sorted_packages[@]}"
105+
}
106+
107+
toposort_util() {
108+
local package_index=$1
109+
visited[$package_index]=1
110+
recursion_stack[$package_index]=1
111+
112+
local dependencies=${PACKAGE_DEPS[$package_index]}
113+
for dependency in $dependencies; do
114+
# We only care about dependencies that are in our monorepo
115+
local is_in_monorepo=0
116+
local dependency_index=-1
117+
for i in "${!PACKAGE_NAMES[@]}"; do
118+
if [ "${PACKAGE_NAMES[$i]}" = "$dependency" ]; then
119+
is_in_monorepo=1
120+
dependency_index=$i
121+
break
122+
fi
123+
done
124+
125+
if [ "$is_in_monorepo" -eq 1 ]; then
126+
if [ "${recursion_stack[$dependency_index]}" -eq 1 ]; then
127+
echo "Error: cyclic dependency detected"
128+
exit 1
129+
fi
130+
if [ "${visited[$dependency_index]}" -eq 0 ]; then
131+
toposort_util "$dependency_index"
132+
fi
133+
fi
134+
done
135+
136+
recursion_stack[$package_index]=0
137+
sorted_packages+=("${PACKAGE_NAMES[$package_index]}")
138+
}
139+
140+
# Get the sorted packages
141+
SORTED_PACKAGES=($(toposort))
142+
143+
# Arrays to map package name to its tarball path
144+
PKG_TARBALL_MAP_KEYS=()
145+
PKG_TARBALL_MAP_VALUES=()
146+
147+
# Build, pack, and test all packages in a single loop
148+
for package_name in "${SORTED_PACKAGES[@]}"; do
149+
package_dir_name=""
150+
for i in "${!PKG_NAME_TO_DIR_MAP_KEYS[@]}"; do
151+
if [ "${PKG_NAME_TO_DIR_MAP_KEYS[$i]}" = "$package_name" ]; then
152+
package_dir_name="${PKG_NAME_TO_DIR_MAP_VALUES[$i]}"
153+
break
154+
fi
155+
done
156+
157+
if [ -z "$package_dir_name" ]; then
158+
echo "Warning: could not find directory for package $package_name. Skipping."
159+
continue
160+
fi
161+
162+
echo "Processing $package_name..."
163+
cd "$ROOT_DIR/packages/$package_dir_name"
164+
165+
# Install dependencies from registry first.
166+
npm install --ignore-scripts
167+
168+
# Find monorepo dependencies and install their packed versions
169+
current_package_index=-1
170+
for i in "${!PACKAGE_NAMES[@]}"; do
171+
if [ "${PACKAGE_NAMES[$i]}" = "$package_name" ]; then
172+
current_package_index=$i
173+
break
174+
fi
175+
done
176+
177+
dependencies=${PACKAGE_DEPS[$current_package_index]}
178+
for dependency in $dependencies; do
179+
# Check if the dependency is a monorepo package that has been packed
180+
tarball_path=""
181+
for i in "${!PKG_TARBALL_MAP_KEYS[@]}"; do
182+
if [ "${PKG_TARBALL_MAP_KEYS[$i]}" = "$dependency" ]; then
183+
tarball_path="${PKG_TARBALL_MAP_VALUES[$i]}"
184+
break
185+
fi
186+
done
187+
188+
if [ -n "$tarball_path" ]; then
189+
echo "Installing monorepo dependency $dependency from $tarball_path"
190+
# Use --no-save to avoid adding the file path to package.json
191+
npm install "$tarball_path" --no-save
192+
fi
193+
done
194+
195+
if grep -q '"compile":' package.json; then
196+
echo "Compiling $package_name..."
197+
198+
# 🌟 ADDED: Explicitly delete the build directory before compilation
199+
rm -rf ./build
200+
201+
tsc -p . --skipLibCheck
202+
fi
203+
204+
# Pack the current package for downstream dependencies
205+
# The `npm pack` command outputs the filename of the tarball. We capture it.
206+
# The output might have other warnings, so we take the last line.
207+
tarball_filename=$(npm pack | tail -n 1)
208+
tarball_path="$ROOT_DIR/packages/$package_dir_name/$tarball_filename"
209+
PKG_TARBALL_MAP_KEYS+=("$package_name")
210+
PKG_TARBALL_MAP_VALUES+=("$tarball_path")
211+
echo "Packed $package_name to $tarball_path"
212+
213+
# Run tests for this package
214+
echo "Running tests for $package_name with command: npm run $TEST_COMMAND"
215+
npm run "$TEST_COMMAND"
216+
217+
cd "$ROOT_DIR"
218+
done

‎ci/run_single_test.sh‎

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -39,33 +39,42 @@ if [ ${BUILD_TYPE} != "presubmit" ]; then
3939
export MOCHA_REPORTER=xunit
4040
fi
4141

42-
# Install dependencies
43-
echo "npm install --ignore-scripts --engine-strict; npm install"
44-
npm install --ignore-scripts --engine-strict; npm install
45-
46-
4742
retval=0
4843

44+
# In this run, we're running both 1) cascading tests, as well as
45+
# 2) individual tests for a given package. The reason we run them separately
46+
# is because cascading tests need to run without compilation; so, we're
47+
# also running individual package tests to ensure compilation is done correctly
48+
4949
set +e
5050
case ${TEST_TYPE} in
5151
lint)
52+
npm install --ignore-scripts --engine-strict
5253
npm run prelint
5354
npm run lint
5455
retval=$?
5556
;;
5657
samples)
57-
npm run samples-test
58+
${PROJECT_ROOT}/ci/run-interdependent-tests.sh "${TEST_TYPE}-test"
59+
npm install --ignore-scripts --engine-strict; npm install
60+
npm run ${TEST_TYPE}-test
5861
retval=$?
5962
;;
6063
system)
61-
npm run system-test
64+
${PROJECT_ROOT}/ci/run-interdependent-tests.sh "${TEST_TYPE}-test"
65+
npm install --ignore-scripts --engine-strict; npm install
66+
npm run ${TEST_TYPE}-test
6267
retval=$?
6368
;;
6469
units)
70+
${PROJECT_ROOT}/ci/run-interdependent-tests.sh "test"
71+
npm install --ignore-scripts --engine-strict; npm install
6572
npm run test
6673
retval=$?
6774
;;
6875
*)
76+
${PROJECT_ROOT}/ci/run-interdependent-tests.sh "$TEST_TYPE"
77+
retval=$?
6978
;;
7079
esac
7180
set -e

0 commit comments

Comments
 (0)