Skip to content

Commit 6210e6c

Browse files
NPM to Bun; Docker Build Caching; Fix Tests; Windows Build and Test Improvements (#620)
* Update dependencies and Dockerfile command - Bump version of `@malloy-publisher/app`, `@malloy-publisher/sdk`, and `@malloy-publisher/server` to 0.0.165 in `bun.lock` and `package.json`. - Add `@opentelemetry/exporter-prometheus` version 0.212.0 to dependencies. - Change the command in the Dockerfile to use `bun` instead of `node` for starting the server. These changes enhance the project by updating to the latest versions of dependencies and optimizing the Docker command for better performance. Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com> * Update k6 Tests workflow to trigger on pull requests - Added a trigger for the k6 tests workflow to run on pull requests targeting the main branch, in addition to the existing push trigger. This change enhances the testing process by ensuring that tests are executed for pull requests, improving code quality before merging. Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com> * Update cross-platform tests workflow to enhance caching - Modified the caching configuration in the GitHub Actions workflow to include `node_modules` and its subdirectories, improving dependency management during builds. - Added a condition to only install dependencies if the cache is not hit, optimizing the workflow's efficiency. This update aims to streamline the testing process and reduce build times by leveraging caching more effectively. Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com> * Refactor caching in cross-platform tests workflow - Simplified the caching configuration by removing redundant paths for `node_modules`, focusing on the primary cache directory. - Streamlined the dependency installation step to eliminate unnecessary echo statements, enhancing clarity and efficiency. This update aims to improve the performance and maintainability of the GitHub Actions workflow for cross-platform testing. Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com> * Update cross-platform tests workflow to use actions/cache@v5 - Upgraded the caching action from v4 to v5 for improved performance and features. - Removed the condition to install dependencies only if the cache is not hit, simplifying the installation step. This change aims to enhance the efficiency and maintainability of the GitHub Actions workflow for cross-platform testing. Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com> * Refactor dependency installation in cross-platform tests workflow - Split the dependency installation step into separate jobs for Unix and Windows environments, enhancing clarity and maintainability. - Removed the caching step for Bun dependencies to streamline the workflow. This update aims to improve the cross-platform testing process by providing clearer installation steps for different operating systems. Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com> * Enhance cross-platform tests workflow with OS-specific build steps - Introduced separate build steps for Unix and Windows environments to improve clarity and ensure compatibility. - Updated the environment variable handling for dependency installation to maintain consistency across platforms. This change aims to streamline the build process in the cross-platform testing workflow, enhancing maintainability and clarity. Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com> * Update Windows dependency installation in cross-platform tests workflow - Modified the Bun installation command for Windows to use the `--linker=isolated` option, enhancing the dependency management process. - This change aims to improve compatibility and performance during the installation of dependencies in the cross-platform testing workflow. Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com> * Refactor cross-platform tests workflow for improved dependency management - Consolidated the dependency installation step for Unix and Windows into a single job, enhancing clarity and reducing redundancy. - Introduced caching for Bun dependencies to optimize installation times and improve workflow efficiency. - Updated environment variable handling to ensure consistent behavior across different operating systems. This update aims to streamline the cross-platform testing process and enhance maintainability of the workflow. Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com> * Add ShikiJS languages and themes dependencies, optimize Dockerfile caching - Introduced `@shikijs/langs` and `@shikijs/themes` as dependencies in both `bun.lock` and `packages/sdk/package.json`. - Enhanced the Dockerfile by implementing caching for Bun installations and builds, improving build efficiency and reducing installation times. These updates aim to enrich the SDK with additional language and theme support while optimizing the build process in the Docker environment. Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com> * Refactor dependency installation in cross-platform tests workflow - Updated the Bun installation command to include `--prefer-offline` and `--ignore-scripts`, enhancing efficiency and reducing unnecessary script executions. - Removed the environment variable handling for post-install scripts, simplifying the workflow. This change aims to streamline the dependency management process in the cross-platform testing workflow, improving overall performance and maintainability. Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com> * Enhance cross-platform tests workflow with Windows-specific configurations - Added environment variables for Windows user profile and Bun cache directory to improve compatibility. - Introduced steps to configure temporary directories and disable Windows Defender during the workflow execution. - Updated caching for Bun dependencies to differentiate between Windows and Unix-like systems, optimizing installation times. These changes aim to enhance the cross-platform testing process by ensuring a smoother experience on Windows environments while maintaining efficiency across all platforms. Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com> * Enhance cross-platform compatibility in E2E test setup - Introduced `fileURLToPath` and `import.meta.url` to resolve file paths, ensuring compatibility across different operating systems, particularly Windows. - Updated the method of determining `__dirname` and `__filename` to improve path resolution in the test environment. These changes aim to streamline the E2E testing process by enhancing cross-platform functionality and reliability. Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com> * Remove pull request trigger from k6 tests workflow - Eliminated the pull_request trigger for the k6 tests workflow, streamlining the workflow to only run on pushes to the main branch and manual dispatch. This change aims to simplify the workflow configuration and focus on direct pushes to the main branch for testing. Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com> * Remove redundant step for rebuilding DuckDB native bindings in cross-platform tests workflow - Eliminated the step that rebuilt DuckDB native bindings during the dependency installation process, streamlining the workflow. This change aims to simplify the cross-platform testing process by removing unnecessary steps, enhancing overall efficiency. Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com> * Implement Windows-specific workarounds in cross-platform tests workflow - Added environment variables for custom user profile and Bun cache directory to address known issues with Bun on Windows. - Removed the caching step for Bun dependencies on Windows to streamline the workflow while maintaining efficiency. These changes aim to enhance the cross-platform testing process by providing necessary adjustments for Windows environments, ensuring faster CI runs. Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com> * Update Docker configuration for improved command execution - Removed the version specification from `docker-compose.yml` for simplification. - Updated the CMD instruction in the Dockerfile to preload the instrumentation script before starting the server, enhancing performance and monitoring capabilities. These changes aim to streamline the Docker setup and improve the application's startup behavior. Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com> * Refactor Dockerfile to streamline base image setup - Removed comments regarding the pre-built base image for clarity. - Maintained the focus on the Java base image and the base dependencies setup. These changes aim to simplify the Dockerfile and enhance readability for future modifications. Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com> * Update Dockerfile CMD instruction and enhance Docker image workflow caching - Modified the CMD instruction in the Dockerfile to include 'run' for improved command execution. - Added caching configuration in the GitHub Actions workflow for Docker image builds to optimize build times. These changes aim to enhance the application's startup behavior and improve the efficiency of the Docker image build process. Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com> * Enhance Docker image workflow with cache testing and configuration updates - Added a pull request trigger for the main branch to improve workflow responsiveness. - Updated the Docker image build outputs to disable automatic pushing, allowing for manual control. - Introduced a cache testing step to verify cache functionality and ensure successful cache pushes to the registry. These changes aim to optimize the Docker image workflow by enhancing cache management and improving build verification processes. Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com> * Update Docker image workflow to enable automatic pushing and remove cache testing steps - Removed the pull request trigger for the main branch to simplify the workflow. - Updated Docker image build outputs to enable automatic pushing to the registry. - Eliminated the cache testing step to streamline the workflow process. These changes aim to enhance the efficiency of the Docker image publishing process by reducing unnecessary steps and ensuring automatic updates to the registry. Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com> * Enhance server instrumentation and metrics handling (#622) - Updated the Dockerfile to simplify the CMD instruction for starting the server. - Integrated Prometheus metrics handling into the server, including a new endpoint for metrics and middleware for HTTP request metrics. - Implemented a graceful shutdown for the OpenTelemetry SDK during server termination. - Refactored instrumentation to support both Prometheus and OTLP exporters, improving observability. These changes aim to enhance monitoring capabilities and streamline server performance during operation and shutdown. Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com> --------- Signed-off-by: Sagar Swami Rao Kulkarni <sagarswamirao@gmail.com>
1 parent 241095c commit 6210e6c

File tree

12 files changed

+231
-73
lines changed

12 files changed

+231
-73
lines changed

.github/workflows/cross-platform-tests.yml

Lines changed: 34 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,16 @@ name: Cross Platform Tests
33
on:
44
workflow_call:
55

6+
permissions: {}
7+
8+
# Bun bug in Windows: https://github.com/oven-sh/bun/issues/17011#issuecomment-2769196629
9+
# We need to use a custom userprofile and cache directory for Windows. And also need to disable the Defender real-time monitoring.
10+
# There is no solution for this yet (but a couple of active issues), so we need to use this workaround for faster CI runs.
11+
12+
env:
13+
WINDOWS_USERPROFILE: 'D:'
14+
WINDOWS_BUN_CACHE_DIR: 'D:\.bun\install\cache'
15+
616
jobs:
717
cross_platform_test:
818
name: >-
@@ -14,31 +24,45 @@ jobs:
1424
runs-on: ${{ matrix.os }}
1525

1626
steps:
27+
- name: Configure Temp Directory (Windows)
28+
if: runner.os == 'Windows'
29+
shell: pwsh
30+
run: |
31+
$TEMP_DIR = "$env:WINDOWS_USERPROFILE\Temp"
32+
New-Item -ItemType Directory -Force -Path $TEMP_DIR | Out-Null
33+
"TEMP=$TEMP_DIR" >> $env:GITHUB_ENV
34+
"TMP=$TEMP_DIR" >> $env:GITHUB_ENV
35+
36+
- name: Disable Windows Defender (Windows)
37+
if: runner.os == 'Windows'
38+
shell: powershell
39+
run: Set-MpPreference -DisableRealtimeMonitoring $true
40+
1741
- uses: actions/checkout@v4
1842
with:
1943
submodules: "true"
2044

2145
- name: Setup Bun
2246
uses: oven-sh/setup-bun@v2
47+
env:
48+
USERPROFILE: ${{ runner.os == 'Windows' && env.WINDOWS_USERPROFILE || '' }}
2349
with:
2450
bun-version: 1.2.23
2551

26-
- name: Cache Bun dependencies
27-
id: bun-cache
28-
uses: actions/cache@v4
52+
- name: Cache Bun dependencies (Linux/macOS)
53+
if: runner.os != 'Windows'
54+
uses: actions/cache@v5
2955
with:
3056
path: ~/.bun/install/cache
3157
key: ${{ runner.os }}-bun-${{ hashFiles('**/bun.lock') }}
32-
restore-keys: |
33-
${{ runner.os }}-bun-
58+
restore-keys: ${{ runner.os }}-bun-
3459

3560
- name: Install dependencies
36-
run: |
37-
echo "Bun cache hit: ${{ steps.bun-cache.outputs.cache-hit }}"
38-
bun install --frozen-lockfile
39-
shell: bash
4061
env:
41-
BUN_CONFIG_SKIP_POSTINSTALL_SCRIPTS: ${{ matrix.os == 'windows-latest' && '1' || '' }}
62+
BUN_INSTALL_CACHE_DIR: ${{ runner.os == 'Windows' && env.WINDOWS_BUN_CACHE_DIR || '' }}
63+
BUN_CONFIG_SKIP_POSTINSTALL_SCRIPTS: "1"
64+
run: bun install --frozen-lockfile --prefer-offline
65+
shell: bash
4266

4367
- name: Generate API types
4468
run: bun run generate-api-types
@@ -47,8 +71,6 @@ jobs:
4771
- name: Build
4872
run: bun run build
4973
shell: bash
50-
env:
51-
BUN_CONFIG_SKIP_POSTINSTALL_SCRIPTS: ${{ matrix.os == 'windows-latest' && '1' || '' }}
5274

5375
- name: Run tests
5476
run: bun run test

.github/workflows/docker-image.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,8 @@ jobs:
6060
labels: ${{ steps.meta.outputs.labels }}
6161
tags: ${{ env.REGISTRY_IMAGE }}
6262
outputs: type=image,push-by-digest=true,name-canonical=true,push=true
63+
cache-from: type=registry,ref=${{ env.REGISTRY_IMAGE }}:buildcache-${{ env.PLATFORM_PAIR }}
64+
cache-to: type=registry,ref=${{ env.REGISTRY_IMAGE }}:buildcache-${{ env.PLATFORM_PAIR }},mode=max
6365

6466
- name: Export digest
6567
run: |

Dockerfile

Lines changed: 17 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,16 @@
1-
# The generate-api-types scripts require Java.
1+
# syntax=docker/dockerfile:1.4
2+
3+
# Java for generate-api-types scripts
24
FROM amazoncorretto:21.0.8 AS java-base
35

4-
# Production runtime — stable tooling layers first for max caching
5-
FROM oven/bun:1.2.23-slim AS runner
6+
FROM oven/bun:1.2.23-slim AS base-deps
67

7-
# All system dependencies in a single layer
88
RUN apt-get update && apt-get install -y --no-install-recommends \
99
curl ca-certificates unzip git \
1010
openssl libcurl4 libssl3 dnsutils iputils-ping file && \
1111
update-ca-certificates && \
1212
rm -rf /var/lib/apt/lists/*
1313

14-
# DuckDB CLI + Snowflake driver + Node 20 LTS
1514
RUN curl -L https://install.duckdb.org | bash && \
1615
ln -s /root/.duckdb/cli/latest/duckdb /usr/local/bin/duckdb && \
1716
curl -sSL https://raw.githubusercontent.com/iqea-ai/duckdb-snowflake/main/scripts/install-adbc-driver.sh | bash && \
@@ -37,25 +36,29 @@ COPY packages/app/package.json ./packages/app/package.json
3736
COPY packages/sdk/package.json ./packages/sdk/package.json
3837

3938
# Install all workspace dependencies once (cached across builds)
40-
RUN bun install
39+
RUN --mount=type=cache,target=/root/.bun/install/cache \
40+
bun install
4141

4242
# Build SDK first
4343
COPY packages/sdk/ ./packages/sdk/
4444
WORKDIR /publisher/packages/sdk
45-
RUN bun run build
45+
RUN --mount=type=cache,target=/root/.bun \
46+
bun run build
4647

4748
# Build app
4849
WORKDIR /publisher/packages/app
4950
COPY packages/app/ ./
50-
RUN NODE_OPTIONS='--max-old-space-size=4096' bun run build:server
51+
RUN --mount=type=cache,target=/root/.bun \
52+
NODE_OPTIONS='--max-old-space-size=4096' bun run build:server
5153

5254
# Build server
5355
WORKDIR /publisher/packages/server
5456
COPY packages/server/ ./
55-
RUN bun run build:server-only
57+
RUN --mount=type=cache,target=/root/.bun \
58+
bun run build:server-only
5659

57-
# Final image — continue from the cached runner base
58-
FROM runner AS final
60+
# Final image
61+
FROM base-deps AS final
5962
WORKDIR /publisher
6063

6164
# Copy built artifacts from builder
@@ -68,12 +71,13 @@ COPY --from=builder /publisher/packages/sdk/dist/ /publisher/packages/sdk/dist/
6871
COPY --from=builder /publisher/packages/sdk/package.json /publisher/packages/sdk/package.json
6972

7073
# Install production-only deps
71-
RUN bun install --production
74+
RUN --mount=type=cache,target=/root/.bun/install/cache \
75+
bun install --production
7276

7377
# Runtime config
7478
ENV NODE_ENV=production
7579
ENV PATH="/root/.duckdb/cli/latest:$PATH"
7680
RUN mkdir -p /etc/publisher
7781
EXPOSE 4000
7882

79-
CMD ["node", "--require", "./packages/server/dist/instrumentation.js", "./packages/server/dist/server.js"]
83+
CMD ["bun", "run", "./packages/server/dist/server.js"]

bun.lock

Lines changed: 12 additions & 8 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docker-compose.yml

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,3 @@
1-
version: "3.9"
2-
31
services:
42
publisher.ci:
53
container_name: publisher-ci

packages/sdk/package.json

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,8 @@
108108
"eslint-plugin-react-hooks": "5.0.0",
109109
"eslint-plugin-storybook": "^0.11.4",
110110
"shiki": "^1.16.3",
111+
"@shikijs/langs": "^1.16.3",
112+
"@shikijs/themes": "^1.16.3",
111113
"vite": "^6.3.2",
112114
"vite-plugin-dts": "^4.5.3",
113115
"vite-plugin-svgr": "^4.3.0"

packages/server/package.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@
4242
"@modelcontextprotocol/sdk": "^1.13.2",
4343
"@opentelemetry/api": "^1.9.0",
4444
"@opentelemetry/auto-instrumentations-node": "^0.57.0",
45+
"@opentelemetry/exporter-prometheus": "^0.212.0",
4546
"@opentelemetry/sdk-metrics": "^2.0.0",
4647
"@opentelemetry/sdk-node": "^0.200.0",
4748
"@opentelemetry/sdk-trace-node": "^2.0.0",

packages/server/src/health.ts

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313
import { Express, NextFunction, Request, Response } from "express";
1414
import { Server } from "http";
1515
import { components } from "./api";
16+
import { shutdownSDK } from "./instrumentation";
1617
import { logger } from "./logger";
1718
export type OperationalState =
1819
components["schemas"]["ServerStatus"]["operationalState"];
@@ -112,6 +113,13 @@ export function registerSignalHandlers(
112113
closeServer(mcpServer, "MCP server"),
113114
]);
114115

116+
try {
117+
await shutdownSDK();
118+
logger.info("OpenTelemetry SDK shut down");
119+
} catch (_error) {
120+
/* do nothing */
121+
}
122+
115123
try {
116124
logger.close();
117125
} catch (_error) {

0 commit comments

Comments
 (0)