Skip to content

Commit 240478c

Browse files
mondorealemondoreale
authored
refactor [NET-1606]: Bundle utils separately for browser and NodeJS (#3321)
This pull request introduces significant improvements to the `@streamr/utils` package, focusing on robust browser and Node.js environment support, build configuration enhancements, and codebase modernization. The main changes include a new dual build and export system for browser and Node.js, environment variable handling refactoring, and the replacement of Node.js-specific APIs with cross-platform polyfills. ### Changes **Build and packaging enhancements:** - Added a dual build system using a new `rollup.config.mts` to generate separate browser and Node.js bundles, with appropriate module resolution and aliasing for each environment. (`packages/utils/rollup.config.mts`) - Updated `package.json` to use the `exports` field for conditional exports (browser vs. Node.js), adjusted the `files` array, and added new build scripts and dependencies for Rollup and polyfills. (`packages/utils/package.json`) [[1]](diffhunk://#diff-21f8eb1d7a149d42ab0454ea905d44f624ff1d0aeb6d7a88d75b42962ca6b053L10-R36) [[2]](diffhunk://#diff-21f8eb1d7a149d42ab0454ea905d44f624ff1d0aeb6d7a88d75b42962ca6b053R47-R69) - Introduced a `prebuild` and `postbuild` step to streamline the build process and ensure clean output. (`packages/utils/package.json`) **Cross-platform and polyfill improvements:** - Replaced direct Node.js API usage (e.g., `crypto`, `os`) with cross-platform abstractions and browser polyfills (e.g., `buffer-shim`, `path-browserify`, custom `os` and `env` shims for browsers). (`packages/utils/src/browser/crypto.ts`, `packages/utils/src/browser/env.ts`, `packages/utils/src/browser/os.ts`, `packages/utils/src/exports-browser.ts`, `packages/utils/src/keyToArrayIndex.ts`, `packages/utils/karma.config.ts`) [[1]](diffhunk://#diff-285b8e8fa583afc20697242820c6adb2419720d1c2056c6ecb7d8617060c1a51R1-R23) [[2]](diffhunk://#diff-245bac5c3851ffe8bc18571e59cf298ecb92e78d093c47fb8b1cef8c740174feR1-R5) [[3]](diffhunk://#diff-0106994557e39509eb1269938f670aa46dea2f4ab6e6e0e7c06124b973cfa268R1-R6) [[4]](diffhunk://#diff-394788f4b1559a7010d4d4a1ae7f96a11cb992f358a46090cfb9c1eebfbfac05L1-R1) [[5]](diffhunk://#diff-394788f4b1559a7010d4d4a1ae7f96a11cb992f358a46090cfb9c1eebfbfac05L28-R28) F8929faeL1) - Removed the old `crossPlatformCrypto.ts` in favor of new environment-specific implementations. (`packages/utils/src/crossPlatformCrypto.ts`) **Environment variable handling:** - Refactored all environment variable access to use a unified `env` abstraction, ensuring safe access in both Node.js and browser contexts. (`packages/utils/src/Logger.ts`, `packages/utils/src/browser/env.ts`) [[1]](diffhunk://#diff-312c84735c8ba55bd72f753e86af89e6c8f1b058ac23ac2290d447c64b7fd6d4R5) [[2]](diffhunk://#diff-312c84735c8ba55bd72f753e86af89e6c8f1b058ac23ac2290d447c64b7fd6d4L21-R36) [[3]](diffhunk://#diff-312c84735c8ba55bd72f753e86af89e6c8f1b058ac23ac2290d447c64b7fd6d4L46-R45) [[4]](diffhunk://#diff-312c84735c8ba55bd72f753e86af89e6c8f1b058ac23ac2290d447c64b7fd6d4L98-R97) [[5]](diffhunk://#diff-312c84735c8ba55bd72f753e86af89e6c8f1b058ac23ac2290d447c64b7fd6d4L117-R116) [[6]](diffhunk://#diff-91ff14206456eacaae372406d303b78f2a1ca01c710b6f0f005a8ee20b2cf50eR1-R19) **Testing and configuration:** - Updated Jest and Karma configurations to support new module aliases and environment-specific code paths. (`packages/utils/jest.config.ts`, `packages/utils/karma.config.ts`) ([packages/utils/jest.config.tsL1-R11](diffhunk://#diff-138963f3d6083e08b3091ffdcd032a655c4738ba0a41476560753dd9f6b1efeaL1-R11), F8929faeL1) **Code modernization and cleanup:** - Updated cryptographic utilities to use new cross-platform imports and types, improving maintainability and browser compatibility. (`packages/utils/src/SigningUtil.ts`, `packages/utils/src/exports.ts`) [[1]](diffhunk://#diff-264c1a909477860ce2ed2c73f193874422c99397476f8d74ec602ccd11b80512L9-R9) [[2]](diffhunk://#diff-264c1a909477860ce2ed2c73f193874422c99397476f8d74ec602ccd11b80512L21-L23) [[3]](diffhunk://#diff-264c1a909477860ce2ed2c73f193874422c99397476f8d74ec602ccd11b80512L61-R57) [[4]](diffhunk://#diff-264c1a909477860ce2ed2c73f193874422c99397476f8d74ec602ccd11b80512L94-R90) [[5]](diffhunk://#diff-264c1a909477860ce2ed2c73f193874422c99397476f8d74ec602ccd11b80512L170-R166) [[6]](diffhunk://#diff-264c1a909477860ce2ed2c73f193874422c99397476f8d74ec602ccd11b80512L202-R200) [[7]](diffhunk://#diff-264c1a909477860ce2ed2c73f193874422c99397476f8d74ec602ccd11b80512L232-R231) [[8]](diffhunk://#diff-264c1a909477860ce2ed2c73f193874422c99397476f8d74ec602ccd11b80512L246-R244) [[9]](diffhunk://#diff-264c1a909477860ce2ed2c73f193874422c99397476f8d74ec602ccd11b80512L260-R258) [[10]](diffhunk://#diff-d700ca1a1d7a0254f811405cb99a0f7eba3b8ec8396aede7362c8c842fbb7a56L56-R56) **Additional minor improvements:** - Added `@streamr/utils` to the Docker build bootstrap step. (`Dockerfile.node`) These changes collectively make the `@streamr/utils` package more robust, easier to consume in different environments, and simpler to maintain. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > Modernizes `@streamr/utils` for first-class browser and Node.js support with dual bundles, conditional exports, and polyfilled APIs. > > - New Rollup config (`packages/utils/rollup.config.mts`) outputs `exports-browser.*` and `exports-nodejs.*`; `package.json` uses conditional `exports` and updated build scripts > - Introduces env/crypto platform shims: `src/nodejs/{env,crypto}.ts` and `src/browser/{env,crypto,os}.ts`; removes `src/crossPlatformCrypto.ts` > - Replaces Node-only APIs with cross-platform imports: `Logger` reads from `@/env`; hashing via `computeMd5`; updated `SigningUtil` to use `@/crypto` > - Adds browser entry `src/exports-browser.ts` (includes `buffer-shim`) and path aliases for both envs; updates Jest/Karma configs and tsconfigs (`tsconfig.{node,browser,jest,karma}.json`) > - Updates `packages/utils/package.json` fields (`main/module/types`, `files`) and dependencies (Rollup, plugins, polyfills) > - Docker build now bootstraps `@streamr/utils` in `Dockerfile.node` > > <sup>Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit b3774ce. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup> <!-- /CURSOR_SUMMARY -->
1 parent 58d67f8 commit 240478c

25 files changed

+2184
-701
lines changed

‎Dockerfile.node‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ FROM node:${NODE_VERSION}-bullseye AS build
33
WORKDIR /usr/src/network
44
COPY . .
55
RUN --mount=type=cache,target=/root/.npm \
6+
npm run bootstrap-pkg --package=@streamr/utils && \
67
npm run bootstrap-pkg --package=@streamr/proto-rpc && \
78
npm run bootstrap-pkg --package=@streamr/autocertifier-client && \
89
npm run bootstrap-pkg --package=@streamr/dht && \

‎package-lock.json‎

Lines changed: 1762 additions & 645 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/utils/jest.config.ts‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,11 @@
1-
export { default } from '../../jest.config'
1+
import type { Config } from "@jest/types"
2+
import defaultConfig from "../../jest.config"
3+
4+
const config: Config.InitialOptions = {
5+
...defaultConfig,
6+
moduleNameMapper: {
7+
"^@/(.*)$": "<rootDir>/src/nodejs/$1",
8+
},
9+
}
10+
11+
export default config

‎packages/utils/karma.config.ts‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,16 @@
11
import { createKarmaConfig, createWebpackConfig } from '@streamr/browser-test-runner'
2+
import { resolve } from 'path'
23

34
const TEST_PATHS = ['test/**/*.ts']
45

56
export default createKarmaConfig(TEST_PATHS, createWebpackConfig({
67
libraryName: 'utils',
78
fallback: {
89
module: false
9-
}
10+
},
11+
alias: {
12+
'@': resolve(__dirname, 'src/browser'),
13+
os: resolve(__dirname, 'src/browser/os.ts'),
14+
path: 'path-browserify',
15+
},
1016
}))

‎packages/utils/package.json‎

Lines changed: 36 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,17 +7,36 @@
77
"url": "git+https://github.com/streamr-dev/network.git",
88
"directory": "packages/utils"
99
},
10-
"main": "./dist/src/exports.js",
11-
"types": "./dist/src/exports.d.ts",
10+
"main": "./dist/exports-nodejs.cjs",
11+
"module": "./dist/exports-nodejs.mjs",
12+
"types": "./dist/exports-nodejs.d.ts",
13+
"exports": {
14+
".": {
15+
"browser": {
16+
"types": "./dist/exports-browser.d.ts",
17+
"import": "./dist/exports-browser.mjs",
18+
"require": "./dist/exports-browser.cjs"
19+
},
20+
"default": {
21+
"types": "./dist/exports-nodejs.d.ts",
22+
"import": "./dist/exports-nodejs.mjs",
23+
"require": "./dist/exports-nodejs.cjs"
24+
}
25+
}
26+
},
1227
"files": [
13-
"dist",
28+
"dist/exports-nodejs.*",
29+
"dist/exports-browser.*",
1430
"!*.tsbuildinfo",
1531
"README.md",
1632
"LICENSE"
1733
],
1834
"scripts": {
35+
"prebuild": "npm run reset-self",
1936
"build": "tsc -b",
20-
"check": "tsc -p tsconfig.jest.json",
37+
"postbuild": "NODE_OPTIONS=\"--import tsx\" rollup -c rollup.config.mts",
38+
"check": "tsc -b tsconfig.jest.json",
39+
"reset-self": "rimraf --glob '**/*.tsbuildinfo'",
2140
"clean": "jest --clearCache --config '{}' || true; rm -rf dist *.tsbuildinfo node_modules/.cache || true",
2241
"eslint": "eslint --cache --cache-location=node_modules/.cache/.eslintcache/ '*/**/*.{js,ts}'",
2342
"test": "jest",
@@ -28,16 +47,28 @@
2847
"dependencies": {
2948
"@noble/curves": "^1.9.7",
3049
"@noble/post-quantum": "^0.4.1",
50+
"buffer": "^6.0.3",
51+
"buffer-shim": "^1.0.1",
3152
"eventemitter3": "^5.0.0",
3253
"lodash": "^4.17.21",
54+
"md5": "^2.3.0",
55+
"path-browserify": "^1.0.1",
3356
"pino": "^10.1.0",
3457
"pino-pretty": "^13.1.2",
58+
"readable-stream": "^4.7.0",
3559
"secp256k1": "^5.0.1",
3660
"sha3": "^2.1.4"
3761
},
3862
"devDependencies": {
63+
"@rollup/plugin-alias": "^5.1.1",
64+
"@rollup/plugin-node-resolve": "^16.0.3",
3965
"@streamr/browser-test-runner": "^0.0.1",
4066
"@types/lodash": "^4.17.21",
41-
"@types/secp256k1": "^4.0.7"
67+
"@types/md5": "^2.3.6",
68+
"@types/secp256k1": "^4.0.7",
69+
"rimraf": "^6.1.2",
70+
"rollup": "^4.53.3",
71+
"rollup-plugin-dts": "^6.3.0",
72+
"tsx": "^4.21.0"
4273
}
4374
}

‎packages/utils/rollup.config.mts‎

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
import { defineConfig, type RollupOptions } from 'rollup'
2+
import { dts } from 'rollup-plugin-dts'
3+
import alias, { type Alias } from '@rollup/plugin-alias'
4+
import { nodeResolve } from '@rollup/plugin-node-resolve'
5+
import { fileURLToPath } from 'node:url'
6+
7+
const nodejsAliases: Alias[] = [
8+
{
9+
find: /^@\//,
10+
replacement: fileURLToPath(
11+
new URL('./dist/nodejs/src/nodejs/', import.meta.url)
12+
),
13+
},
14+
]
15+
16+
const browserAliases: Alias[] = [
17+
{
18+
find: /^@\//,
19+
replacement: fileURLToPath(
20+
new URL('./dist/browser/src/browser/', import.meta.url)
21+
),
22+
},
23+
{
24+
find: 'os',
25+
replacement: fileURLToPath(
26+
new URL('./dist/browser/src/browser/os.js', import.meta.url)
27+
),
28+
},
29+
{
30+
find: 'path',
31+
replacement: 'path-browserify',
32+
},
33+
{
34+
find: 'stream',
35+
replacement: 'readable-stream',
36+
},
37+
{
38+
find: /^pino$/,
39+
replacement: 'pino/browser',
40+
},
41+
]
42+
43+
export default defineConfig([
44+
nodejs(),
45+
nodejsTypes(),
46+
browser(),
47+
browserTypes(),
48+
])
49+
50+
function nodejs(): RollupOptions {
51+
return {
52+
input: './dist/nodejs/src/exports.js',
53+
output: [
54+
{
55+
format: 'es',
56+
file: './dist/exports-nodejs.mjs',
57+
sourcemap: true,
58+
},
59+
{
60+
format: 'cjs',
61+
file: './dist/exports-nodejs.cjs',
62+
sourcemap: true,
63+
},
64+
],
65+
plugins: [
66+
alias({
67+
entries: nodejsAliases,
68+
}),
69+
nodeResolve({
70+
preferBuiltins: true,
71+
}),
72+
],
73+
external: [
74+
/node_modules/,
75+
/@streamr\//,
76+
],
77+
}
78+
}
79+
80+
function browser(): RollupOptions {
81+
return {
82+
input: './dist/browser/src/exports-browser.js',
83+
output: [
84+
{
85+
format: 'es',
86+
file: './dist/exports-browser.mjs',
87+
sourcemap: true,
88+
},
89+
{
90+
format: 'cjs',
91+
file: './dist/exports-browser.cjs',
92+
sourcemap: true,
93+
},
94+
],
95+
plugins: [
96+
alias({
97+
entries: browserAliases,
98+
}),
99+
nodeResolve({
100+
browser: true,
101+
preferBuiltins: false,
102+
}),
103+
],
104+
external: [
105+
/node_modules/,
106+
/@streamr\//,
107+
],
108+
}
109+
}
110+
111+
function nodejsTypes(): RollupOptions {
112+
return {
113+
input: './dist/nodejs/src/exports.d.ts',
114+
output: [
115+
{
116+
file: './dist/exports-nodejs.d.ts',
117+
},
118+
],
119+
plugins: [
120+
alias({
121+
entries: nodejsAliases,
122+
}),
123+
nodeResolve(),
124+
dts(),
125+
],
126+
external: [
127+
/node_modules/,
128+
/@streamr\//,
129+
],
130+
}
131+
}
132+
133+
function browserTypes(): RollupOptions {
134+
return {
135+
input: './dist/browser/src/exports-browser.d.ts',
136+
output: [
137+
{
138+
file: './dist/exports-browser.d.ts',
139+
},
140+
],
141+
plugins: [
142+
alias({
143+
entries: browserAliases,
144+
}),
145+
nodeResolve({
146+
browser: true,
147+
preferBuiltins: false,
148+
}),
149+
dts(),
150+
],
151+
external: [
152+
/node_modules/,
153+
/@streamr\//,
154+
],
155+
}
156+
}

‎packages/utils/src/Logger.ts‎

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ import pino from 'pino'
22
import path from 'path'
33
import without from 'lodash/without'
44
import padEnd from 'lodash/padEnd'
5+
import { env } from '@/env'
56

67
export type LogLevel = 'silent' | 'fatal' | 'error' | 'warn' | 'info' | 'debug' | 'trace'
78

@@ -18,23 +19,21 @@ const parseBoolean = (value: string | undefined) => {
1819
}
1920
}
2021

21-
declare let window: any
22-
2322
/**
24-
* Disabled when in browser or when environment variable DISABLE_PRETTY_LOG is set to true.
23+
* Disabled when environment variable DISABLE_PRETTY_LOG is set to true.
2524
*/
2625
function isPrettyPrintDisabled(): boolean {
27-
return typeof window === 'object' || (parseBoolean(process.env.DISABLE_PRETTY_LOG) ?? false)
26+
return parseBoolean(env.DISABLE_PRETTY_LOG) ?? false
2827
}
2928

3029
function isJestRunning(): boolean {
31-
return process.env.JEST_WORKER_ID !== undefined
30+
return env.JEST_WORKER_ID !== undefined
3231
}
3332

3433
const rootLogger = pino({
3534
name: 'rootLogger',
36-
enabled: !process.env.NOLOG,
37-
level: process.env.LOG_LEVEL ?? 'info',
35+
enabled: !env.NOLOG,
36+
level: env.LOG_LEVEL ?? 'info',
3837
formatters: {
3938
level: (label) => {
4039
return { level: label } // log level as string instead of number
@@ -43,7 +42,7 @@ const rootLogger = pino({
4342
transport: isPrettyPrintDisabled() ? undefined : {
4443
target: 'pino-pretty',
4544
options: {
46-
colorize: parseBoolean(process.env.LOG_COLORS) ?? true,
45+
colorize: parseBoolean(env.LOG_COLORS) ?? true,
4746
singleLine: true,
4847
translateTime: 'yyyy-mm-dd"T"HH:MM:ss.l',
4948
ignore: 'pid,hostname',
@@ -95,7 +94,7 @@ export class Logger {
9594
name: Logger.createName(loggerModule),
9695
...contextBindings
9796
}, {
98-
level: process.env.LOG_LEVEL ?? defaultLogLevel
97+
level: env.LOG_LEVEL ?? defaultLogLevel
9998
})
10099
this.fatal = wrappedMethodCall(this.logger.fatal.bind(this.logger))
101100
this.error = wrappedMethodCall(this.logger.error.bind(this.logger))
@@ -114,7 +113,7 @@ export class Logger {
114113
const parts = parsedPath.dir.split(path.sep)
115114
fileId = parts[parts.length - 1]
116115
}
117-
const longName = without([process.env.STREAMR_APPLICATION_ID, fileId], undefined).join(':')
116+
const longName = without([env.STREAMR_APPLICATION_ID, fileId], undefined).join(':')
118117
return isPrettyPrintDisabled() ?
119118
longName : padEnd(longName.substring(0, this.NAME_LENGTH), this.NAME_LENGTH, ' ')
120119
}

0 commit comments

Comments
 (0)