Skip to content

Commit d67c8f4

Browse files
author
Your Name
committed
release: v1.5.7
BullMQ enrichment migration - Rewrote enrichment pipeline on BullMQ v5: artist, track, podcast workers with pause/resume/stop support and Bull Board visibility - Essentia publishes audio:analysis:complete events; CLAP subscribes reactively instead of polling (eliminates scan delay between phases) - Thread-safe DB pool in CLAP worker, all DB calls in run_in_executor - Fixed psycopg2.pool submodule import crash on BullMQ vibe worker startup - Silenced EnrichmentStateService disconnect error on already-closed Redis conn Enrichment fixes - lastfmTags NULL caused mood-tags phase to silently skip all tracks; migration backfills NULL -> '{}' and sets column default - Cover art fetch errors for temp-MBID albums (temp-* passed to Cover Art Archive) - VIBE-VOCAB vocabulary JSON not copied to Docker image (TypeScript omits .json) - Wired cleanupOldResolved() to run daily; added missing enrichment status indexes - Fixed circuit breaker reset, orphan cleanup, podcast entityType, hang detection Soulseek - Track search-page downloads in activity tracker - Use async fs.promises.access instead of synchronous existsSync - Verify file exists on disk before emitting download:complete (#110) Docker image size: 28.4 GB -> 12.2 GB - Removed all CUDA/NVIDIA dependencies; switched to CPU-only PyTorch/TensorFlow - tensorflow-cpu + essentia-tensorflow --no-deps (avoids GPU TF transitive dep) - Fixed .dockerignore: **/node_modules and **/.next now excluded from build context PWA / mobile - Background audio session loss on iOS/Android: SilenceKeepalive singleton, tryResume() in MediaSession play handler, direct track load on 'ended' event, visibilitychange/pageshow foreground recovery - Lock orientation to portrait for Android device lock (#117) Discovery - Retry All re-importing albums already in library: apply same three-level filter as GET /current before creating download jobs; delete stale UnavailableAlbum records for already-present albums. Closes #34 CI / release - linux/arm64 added to release and nightly Docker builds (#87) - Isolated release CI from nightly GHA cache (cache-from/cache-to removed from docker-publish.yml to guarantee clean release builds) - Redis vm.overcommit_memory=1 sysctl added to prod and server compose files Other fixes - Cross-artist album fallback by title+year prevents library splitting (#50) - Retry temp-MBID artists after 24h not 7 days; hide temp MBIDs from API (#112) - 3-attempt ECONNRESET retry on all Deezer getPlaylist call sites (#119) - check response.ok on health probe — fetch does not throw on 5xx (#104) - Z-index stacking hierarchy established (MiniPlayer through OverlayPlayer) - API token display overflow on iPhone (min-w-0/overflow-hidden on flex container)
1 parent 9ffad66 commit d67c8f4

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

62 files changed

+1723
-978
lines changed

.dockerignore

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
.git
2+
.worktrees
3+
.claude
4+
.serena
5+
.aider-desk
6+
.aider.tags.cache.v4
7+
.roo
8+
.ruff_cache
9+
.vscode
10+
context_portal
11+
logs
12+
docs
13+
scripts
14+
*.md
15+
*.log
16+
**/node_modules
17+
**/.next
18+
.env*
19+
coverage
20+
*.test.ts
21+
__tests__
22+
android
23+
ios

.github/workflows/docker-nightly.yml

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,9 @@ jobs:
2525
sudo rm -rf /usr/local/share/boost
2626
sudo rm -rf "$AGENT_TOOLSDIRECTORY"
2727
28+
- name: Set up QEMU
29+
uses: docker/setup-qemu-action@v3
30+
2831
- name: Set up Docker Buildx
2932
uses: docker/setup-buildx-action@v3
3033

@@ -52,5 +55,4 @@ jobs:
5255
org.opencontainers.image.version=nightly-${{ steps.sha.outputs.short }}
5356
cache-from: type=gha
5457
cache-to: type=gha,mode=max
55-
# ARM64 disabled due to QEMU emulation issues with npm packages
56-
platforms: linux/amd64
58+
platforms: linux/amd64,linux/arm64

.github/workflows/docker-publish.yml

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -60,9 +60,7 @@ jobs:
6060
${{ env.IMAGE_NAME }}:latest
6161
cache-from: type=gha
6262
cache-to: type=gha,mode=max
63-
# Note: ARM64 removed due to QEMU emulation issues with npm packages
64-
# Can be re-added when using native ARM64 runners
65-
platforms: linux/amd64
63+
platforms: linux/amd64,linux/arm64
6664

6765
create-release:
6866
needs: [build]

CHANGELOG.md

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,12 +5,28 @@ All notable changes to Kima will be documented in this file.
55
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
66
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
77

8-
## [1.5.6] - 2026-02-22
8+
## [1.5.7] - 2026-02-23
9+
10+
### Added
11+
12+
- **BullMQ enrichment infrastructure**: Rewrote the entire enrichment pipeline on top of BullMQ v5, replacing the custom BLPOP/Redis queue loops. Artist, track, and podcast enrichment all run as proper BullMQ Worker instances with job-level pause, resume, and stop support. All queues are visible in the Bull Board admin dashboard. The orchestrator pushes jobs into BullMQ and uses a sentinel pattern to track when all jobs in a phase have completed before advancing.
13+
- **Reactive vibe queuing**: The Essentia audio analyzer now publishes an `audio:analysis:complete` event to Redis when each track finishes. The CLAP service subscribes and immediately queues a vibe embedding job for that track — eliminating the previous polling-based approach where CLAP scanned the database on a fixed interval looking for newly-completed Essentia tracks.
914

1015
### Fixed
1116

17+
- **PWA background audio session lost on iOS and Android**: Pausing from lock-screen / notification controls while the app was backgrounded caused iOS to reclaim the audio session, blocking any subsequent `audio.play()` call until the app was foregrounded. Fixes two related symptoms: (1) resuming from lock-screen controls appeared to do nothing until the app was opened, (2) music stopped after extended background playback during track transitions. Fixed by: calling `audioEngine.tryResume()` synchronously inside the MediaSession `play` handler (within the user-activation window iOS grants to MediaSession callbacks); adding a silent looping audio keepalive (`silence-keepalive.ts`) that holds the OS audio session while user audio is paused and the app is backgrounded; loading the next track directly from the `ended` event handler to eliminate the inter-track silence gap that triggered session reclaim; and adding `visibilitychange` / `pageshow` foreground recovery to retry playback if the engine is paused when the app returns to the foreground.
18+
- **Discovery "Retry All" importing entire albums already in library**: The `POST /discover/retry-unavailable` endpoint fetched all raw `UnavailableAlbum` records for the week without applying the same three-level filter the `GET /current` endpoint uses before displaying them. As a result, clicking "Retry All" triggered full re-downloads of albums that were already present in the library (matched by discovery MBID, library MBID, or fuzzy title+artist). The retry handler now applies all three filters before creating download jobs, and deletes stale `UnavailableAlbum` records for albums already in the library so they do not reappear. Closes #34.
19+
- **Mood-tags phase silently skipping all tracks**: `lastfmTags` was `NULL` for tracks that had been enriched before the column was added. The mood-tags enrichment phase queries `WHERE lastfmTags != '{}'`, which never matches `NULL` — so every track was silently skipped every cycle. Migration backfills all `NULL` values to `'{}'` and sets the column default, so newly enriched tracks are never NULL.
20+
- **Docker image size (28.4 GB → 12.2 GB)**: Removed all CUDA and NVIDIA dependencies from the Docker image. The `audio-analyzer` and `audio-analyzer-clap` services now run on CPU-only PyTorch and TensorFlow. Changed pip installs to use the CPU-only PyTorch wheel index (`--index-url https://download.pytorch.org/whl/cpu`), replaced `tensorflow` with `tensorflow-cpu`, and installed `essentia-tensorflow --no-deps` to prevent pip from pulling the GPU TensorFlow variant as a transitive dependency. Removed `nvidia-cudnn-cu12`, `torchvision` (not imported), the `/opt/cudnn8` CUDA layer, and all NVIDIA library paths from the supervisor `LD_LIBRARY_PATH`. No regressions: TensorFlow confirmed running on CPU, all 9 MusiCNN classification heads load normally.
21+
- **Docker build context bloat**: `frontend/node_modules/` (598 MB) and `frontend/.next/` (313 MB) were not excluded from the Docker build context. The `.dockerignore` `node_modules` pattern only matched root-level; changed to `**/node_modules`. Added `**/.next`. Combined these reduced the `COPY frontend/ ./` layer from 946 MB to ~50 MB.
22+
- **Cover art fetch errors for temp-MBID albums**: Albums with temporary MBIDs (temp-*) were being passed to the Cover Art Archive API, causing 400 errors. Added validation to skip temp-MBIDs in artist enrichment and data cache.
23+
- **VIBE-VOCAB vocabulary file missing**: The vocabulary JSON file wasn't being copied to the Docker image because TypeScript doesn't copy .json files automatically. Added explicit import to force tsc to copy it.
24+
- **Redis memory overcommit warning**: Added `vm.overcommit_memory=1` sysctl to docker-compose.prod.yml and docker-compose.server.yml.
1225
- **Z-index stacking order**: MiniPlayer was z-50 (same tier as modals), causing it to appear above open dialogs due to DOM ordering. Established a consistent stacking hierarchy: MiniPlayer z-[45] → TopBar z-50 → VibeOverlay/toasts z-[55] → MobileSidebar backdrop z-[60] / drawer z-[70] → all modals z-[80] → nested confirm z-[85] → toast z-[100] → OverlayPlayer z-[9999]. MobileSidebar was also using non-standard `z-100` which is not a valid Tailwind class.
1326
- **API token display overflowing viewport on iPhone**: The newly-generated token `<code>` block extended beyond the screen on narrow viewports due to missing `min-w-0` / `overflow-hidden` on its flex container; added both.
27+
- **CLAP BullMQ worker crash on startup**: `import psycopg2` does not implicitly import `psycopg2.pool`; the BullMQ vibe worker was crashing immediately because `psycopg2.pool.ThreadedConnectionPool` was referenced without the submodule being imported. Added explicit `import psycopg2.pool`.
28+
- **EnrichmentStateService Redis disconnect error**: Calling `disconnect()` on an already-closed Redis connection raised an unhandled error. The disconnect is now silenced when the connection is already in a closed state.
29+
- **CLAP worker thread-safety**: All PostgreSQL calls in the CLAP BullMQ worker are now wrapped in `run_in_executor` so they execute on a thread-pool thread rather than blocking the asyncio event loop. Connection pool is initialized once per process and shared safely across concurrent jobs.
1430

1531
## [1.5.5] - 2026-02-21
1632

Dockerfile

Lines changed: 33 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -42,25 +42,38 @@ RUN mkdir -p /app/backend /app/frontend /app/audio-analyzer /app/models \
4242
# ============================================
4343
WORKDIR /app/audio-analyzer
4444

45-
# Install Python dependencies for audio analysis
46-
# Note: TensorFlow must be installed explicitly for Python 3.11+ compatibility
45+
# Install all Python dependencies in a single layer to minimize image size
46+
# CPU-only torch/torchaudio: install first via the CPU index so downstream
47+
# packages (laion-clap, transformers) reuse the already-installed CPU wheels.
48+
# tensorflow-cpu replaces tensorflow to avoid pulling in CUDA runtime libs.
49+
# essentia-tensorflow declares a dependency on `tensorflow` (not tensorflow-cpu)
50+
# so we install it with --no-deps after tensorflow-cpu is already present.
4751
RUN pip3 install --no-cache-dir --break-system-packages \
48-
'tensorflow>=2.13.0,<2.16.0' \
52+
torch torchaudio torchvision \
53+
--index-url https://download.pytorch.org/whl/cpu \
54+
&& pip3 install --no-cache-dir --break-system-packages \
55+
'tensorflow-cpu>=2.13.0,<2.14.0' \
56+
&& pip3 install --no-cache-dir --break-system-packages --no-deps \
4957
essentia-tensorflow \
58+
&& pip3 install --no-cache-dir --break-system-packages \
5059
redis \
51-
psycopg2-binary
52-
53-
# Install cuDNN 8 for TensorFlow GPU (separate from PyTorch's cuDNN 9)
54-
# TF 2.15 needs cuDNN 8, PyTorch needs cuDNN 9 -- installed to isolated path to avoid conflicts
55-
RUN pip3 install --no-cache-dir --break-system-packages --target=/opt/cudnn8 'nvidia-cudnn-cu12==8.9.7.29'
60+
psycopg2-binary \
61+
'laion-clap>=1.1.4' \
62+
'librosa>=0.10.0' \
63+
'transformers>=4.30.0' \
64+
'pgvector>=0.2.0' \
65+
'python-dotenv>=1.0.0' \
66+
'requests>=2.31.0' \
67+
'bullmq==2.19.5' \
68+
&& pip cache purge \
69+
&& find /usr -name "*.pyc" -delete \
70+
&& find /usr -name "__pycache__" -type d -exec rm -rf {} + 2>/dev/null || true
5671

57-
# Download Essentia ML models (~200MB total) - these enable Enhanced vibe matching
72+
# Download all ML models in a single layer (~800MB total)
5873
# IMPORTANT: Using MusiCNN models to match analyzer.py expectations
59-
RUN echo "Downloading Essentia ML models for Enhanced vibe matching..." && \
60-
# Base MusiCNN embedding model (required for all predictions)
74+
RUN echo "Downloading ML models..." && \
6175
curl -L --retry 3 --retry-delay 5 --connect-timeout 30 --max-time 300 -o /app/models/msd-musicnn-1.pb \
6276
"https://essentia.upf.edu/models/autotagging/msd/msd-musicnn-1.pb" && \
63-
# Mood classification heads (using MusiCNN architecture)
6477
curl -L --retry 3 --retry-delay 5 --connect-timeout 30 --max-time 300 -o /app/models/mood_happy-msd-musicnn-1.pb \
6578
"https://essentia.upf.edu/models/classification-heads/mood_happy/mood_happy-msd-musicnn-1.pb" && \
6679
curl -L --retry 3 --retry-delay 5 --connect-timeout 30 --max-time 300 -o /app/models/mood_sad-msd-musicnn-1.pb \
@@ -75,46 +88,26 @@ RUN echo "Downloading Essentia ML models for Enhanced vibe matching..." && \
7588
"https://essentia.upf.edu/models/classification-heads/mood_acoustic/mood_acoustic-msd-musicnn-1.pb" && \
7689
curl -L --retry 3 --retry-delay 5 --connect-timeout 30 --max-time 300 -o /app/models/mood_electronic-msd-musicnn-1.pb \
7790
"https://essentia.upf.edu/models/classification-heads/mood_electronic/mood_electronic-msd-musicnn-1.pb" && \
78-
# Other classification heads
7991
curl -L --retry 3 --retry-delay 5 --connect-timeout 30 --max-time 300 -o /app/models/danceability-msd-musicnn-1.pb \
8092
"https://essentia.upf.edu/models/classification-heads/danceability/danceability-msd-musicnn-1.pb" && \
8193
curl -L --retry 3 --retry-delay 5 --connect-timeout 30 --max-time 300 -o /app/models/voice_instrumental-msd-musicnn-1.pb \
8294
"https://essentia.upf.edu/models/classification-heads/voice_instrumental/voice_instrumental-msd-musicnn-1.pb" && \
83-
echo "ML models downloaded successfully" && \
95+
curl -L --retry 3 --retry-delay 5 --connect-timeout 30 --max-time 300 -o /app/models/music_audioset_epoch_15_esc_90.14.pt \
96+
"https://huggingface.co/lukewys/laion_clap/resolve/main/music_audioset_epoch_15_esc_90.14.pt" && \
97+
echo "All ML models downloaded" && \
8498
ls -lh /app/models/
8599

86-
# Copy audio analyzer script
100+
# Copy audio analyzer scripts
87101
COPY services/audio-analyzer/analyzer.py /app/audio-analyzer/
88102

89103
# ============================================
90104
# CLAP ANALYZER SETUP (Vibe Similarity)
91105
# ============================================
92106
WORKDIR /app/audio-analyzer-clap
93107

94-
# Install CLAP Python dependencies
95-
# Note: torch is large (~2GB) but required for CLAP embeddings
96-
RUN pip3 install --no-cache-dir --break-system-packages \
97-
'laion-clap>=1.1.4' \
98-
'torch>=2.0.0' \
99-
'torchaudio>=2.0.0' \
100-
'torchvision>=0.15.0' \
101-
'librosa>=0.10.0' \
102-
'transformers>=4.30.0' \
103-
'pgvector>=0.2.0' \
104-
'python-dotenv>=1.0.0' \
105-
'requests>=2.31.0'
106-
107108
# Copy CLAP analyzer script
108109
COPY services/audio-analyzer-clap/analyzer.py /app/audio-analyzer-clap/
109110

110-
# Pre-download CLAP model (~600MB) during build to avoid runtime download
111-
# The analyzer expects the model at /app/models/music_audioset_epoch_15_esc_90.14.pt
112-
RUN echo "Downloading CLAP model for vibe similarity..." && \
113-
curl -L --retry 3 --retry-delay 5 --connect-timeout 30 --max-time 300 -o /app/models/music_audioset_epoch_15_esc_90.14.pt \
114-
"https://huggingface.co/lukewys/laion_clap/resolve/main/music_audioset_epoch_15_esc_90.14.pt" && \
115-
echo "CLAP model downloaded successfully" && \
116-
ls -lh /app/models/music_audioset_epoch_15_esc_90.14.pt
117-
118111
# Create database readiness check script
119112
RUN cat > /app/wait-for-db.sh << 'EOF'
120113
#!/bin/bash
@@ -166,7 +159,9 @@ RUN npx prisma generate
166159
# Copy backend source and build
167160
COPY backend/src ./src
168161
COPY backend/tsconfig.json ./
169-
RUN npm run build
162+
RUN npm run build && \
163+
npm prune --production && \
164+
rm -rf src tests __tests__ tsconfig*.json
170165

171166
COPY backend/docker-entrypoint.sh ./
172167
COPY backend/healthcheck.js ./healthcheck-backend.js
@@ -275,7 +270,7 @@ stdout_logfile=/dev/stdout
275270
stdout_logfile_maxbytes=0
276271
stderr_logfile=/dev/stderr
277272
stderr_logfile_maxbytes=0
278-
environment=DATABASE_URL="postgresql://kima:kima@localhost:5432/kima",REDIS_URL="redis://localhost:6379",MUSIC_PATH="/music",BATCH_SIZE="10",SLEEP_INTERVAL="5",MAX_ANALYZE_SECONDS="90",BRPOP_TIMEOUT="30",MODEL_IDLE_TIMEOUT="300",NUM_WORKERS="2",THREADS_PER_WORKER="1",LD_LIBRARY_PATH="/opt/cudnn8/nvidia/cudnn/lib:/usr/local/lib/python3.11/dist-packages/nvidia/cublas/lib:/usr/local/lib/python3.11/dist-packages/nvidia/cufft/lib:/usr/local/lib/python3.11/dist-packages/nvidia/cuda_runtime/lib:/usr/local/lib/python3.11/dist-packages/nvidia/cuda_nvrtc/lib:/usr/local/lib/python3.11/dist-packages/nvidia/cusolver/lib:/usr/local/lib/python3.11/dist-packages/nvidia/cusparse/lib:/usr/local/lib/python3.11/dist-packages/nvidia/nccl/lib"
273+
environment=DATABASE_URL="postgresql://kima:kima@localhost:5432/kima",REDIS_URL="redis://localhost:6379",MUSIC_PATH="/music",BATCH_SIZE="10",SLEEP_INTERVAL="5",MAX_ANALYZE_SECONDS="90",BRPOP_TIMEOUT="30",MODEL_IDLE_TIMEOUT="300",NUM_WORKERS="2",THREADS_PER_WORKER="1"
279274
priority=50
280275

281276
[program:audio-analyzer-clap]

0 commit comments

Comments
 (0)