[GUIDE] Immich v1.137.3 - Upgrade Recovery Process

[Gamechiefx]

Immich Upgrade Recovery Guide - Complete Troubleshooting Documentation

Date: 2025-08-06
System: Proxmox LXC Container
Immich Version: 1.137.3
Issue: Immich upgrade broke application - blank page and missing build artifacts
Status: ✅ FULLY RESOLVED

---

📋 Problem Summary

After upgrading Immich using the Proxmox LXC community scripts, the application failed to start properly with multiple critical errors:

1. Backend Error: Cannot find module '/opt/immich/app/dist/main'
2. Frontend Error: Blank page displayed in browser
3. Reference Issue: GitHub Discussion #5264

---

🔍 Initial Diagnosis

Step 1: Analyzed Startup Script

• Command: ./start-web.sh
• Error: Error: Cannot find module '/opt/immich/app/dist/main'
• Root Cause: Missing compiled build artifacts after upgrade

Step 2: Directory Structure Assessment

• Issue Found: /opt/immich/app/dist/main.js was missing
• Directory Status: Only machine-learning directory present in /opt/immich/app/
• Source Code: Available in /opt/immich/source/ but not compiled

Step 3: Referenced GitHub Discussion

• Warning: Community scripts discussion warned about breaking changes
• Issue: Upgrades can break directory structure and require rebuilds

---

🛠️ Resolution Process

Phase 1: Backend Server Recovery

Issue: Node.js Build Dependencies Missing

• Error: node-gyp failure for canvas package
• Missing: pangocairo, libcairo2-dev, and graphics libraries

Solution: Install System Dependencies

apt update && apt install -y pkg-config libcairo2-dev libpango1.0-dev libjpeg-dev libgif-dev librsvg2-dev

• Result: ✅ Canvas compilation dependencies resolved
• Time: ~2 minutes

Issue: Missing Server Build Artifacts

• Missing: /opt/immich/app/dist/main.js and related compiled files
• Status: Source code present but not built

Solution: Rebuild Server from Source

cd /opt/immich/source/server
npm ci                    # Install dependencies (2m)
npm run build            # Compile TypeScript (1m)

• Result: ✅ dist/ directory created with all build artifacts
• Files: main.js, main.js.map, and full compiled application

Issue: Incorrect Application Structure

• Problem: Built files in source directory, not where runtime expects them
• Expected: /opt/immich/app/dist/main

Solution: Copy Build Artifacts to Runtime Location

cp -r /opt/immich/source/server/dist /opt/immich/app/
cp -r /opt/immich/source/server/node_modules /opt/immich/app/
cp /opt/immich/source/server/package.json /opt/immich/app/

• Result: ✅ Runtime can find all required modules and files

Issue: Package.json Path Issues

• Error: Error: ENOENT: no such file or directory, open './package.json'
• Problem: Different scripts expect package.json in different locations

Solution: Copy Package.json to Multiple Locations

cp /opt/immich/source/server/package.json /opt/immich/

• Result: ✅ Both start-web.sh and start.sh can find package.json

Phase 2: Geodata and Resource Recovery

Issue: Reverse Geocoding Failure

• Error: Error: ENOENT: no such file or directory, open '/build/geodata/geodata-date.txt'
• Problem: Missing geodata files for map functionality

Solution: Copy and Symlink Geodata

cp -r /opt/immich/source/server/resources /opt/immich/app/
mkdir -p /build && ln -sf /opt/immich/geodata /build/geodata

• Result: ✅ Reverse geocoding functionality restored
• Features: Map functionality and location tagging working

Phase 3: Frontend Recovery

Issue: Blank Page in Browser

• Symptoms: HTTP 200 response with 0 bytes content
• Problem: Missing web frontend files (/build/www/index.html)
• Log Warning: Unable to open /build/www/index.html, skipping SSR

Solution 1: Build TypeScript SDK Dependency

cd /opt/immich/source/open-api/typescript-sdk
npm ci          # Install SDK dependencies
npm run build   # Build TypeScript SDK

• Reason: Web frontend depends on @immich/sdk local package
• Result: ✅ SDK built successfully

Solution 2: Build Web Frontend

cd /opt/immich/source/web
npm ci                    # Install web dependencies (2m)
npm run build            # Build Svelte/Vite frontend (1m 37s)

• Technologies: Svelte + Vite build system
• Output: build/ directory with complete web application
• Files: index.html (6,875 bytes), assets, service worker, etc.

Solution 3: Deploy Frontend Files

mkdir -p /build/www
cp -r /opt/immich/source/web/build/* /build/www/

• Result: ✅ Web frontend files deployed to expected location

Phase 4: Process Management and Startup

Issue: Port Conflicts

• Error: Error: listen EADDRINUSE: address already in use :::2283
• Problem: Old Immich process still running on port 2283

Solution: Clean Process Management

# Find processes using port 2283
netstat -tlnp | grep :2283
# Kill specific process
kill -9 23468
# Verify port is free
netstat -tlnp | grep :2283  # Should return nothing

• Result: ✅ Port 2283 available for new Immich instance

Issue: Script Selection

• Discovery: Multiple startup scripts available
• Options: start-web.sh, start.sh
• Optimal: start.sh starts both API and microservices workers

Solution: Use Proper Startup Script

cd /opt/immich
nohup ./start.sh > /tmp/immich-start.log 2>&1 &

• Result: ✅ Both API and microservices workers started successfully

---

🎯 Final Verification

Backend Verification

curl -s -w "HTTP Status: %{http_code}\nResponse Time: %{time_total}s\n" http://127.0.0.1:2283/api/server-info/ping

• Result: HTTP 200, ~3ms response time ✅

Frontend Verification

curl -s -w "HTTP Status: %{http_code}\nContent Length: %{size_download} bytes\n" http://127.0.0.1:2283

• Before Fix: HTTP 200, 0 bytes (blank page) ❌
• After Fix: HTTP 200, 6,875 bytes (full HTML) ✅

Service Status

• API Worker: "Nest application successfully started" ✅
• Microservices Worker: "Immich Microservices is running [v1.137.3]" ✅
• Database: 28,395 clip index + 38,000 face index connected ✅
• Storage: All mount folders verified ✅
• Features: Smart search, facial recognition, maps all enabled ✅

---

📊 Summary of Changes Made

Files Created/Modified:

1. Backend Build Artifacts:

   • /opt/immich/app/dist/ - Complete compiled application
   • /opt/immich/app/node_modules/ - Runtime dependencies
   • /opt/immich/app/package.json - Package manifest

2. Frontend Build Artifacts:

   • /build/www/index.html - Main web interface (6,875 bytes)
   • /build/www/_app/ - Svelte application assets
   • /build/www/*.js - Service workers and JavaScript

3. Dependencies Built:

   • /opt/immich/source/open-api/typescript-sdk/build/ - TypeScript SDK
   • System packages: libcairo2-dev, libpango1.0-dev, etc.

4. Resource Links:

   • /build/geodata/ → /opt/immich/geodata/ (symlink)

System Commands Used:

• Package Management: apt install, npm ci, npm run build
• File Operations: cp -r, mkdir -p, ln -sf
• Process Management: pkill, kill -9, netstat, nohup
• Verification: curl, head, tail, ls -la

---

⚠️ Key Lessons Learned

1. Immich Upgrade Risks

• Issue: Community script upgrades can break build artifacts
• Prevention: Always backup before upgrading
• Recovery: Rebuild from source is reliable solution

2. Dependency Chain

• Backend: Server requires compiled TypeScript + Node modules
• Frontend: Web interface requires TypeScript SDK + Svelte build
• Order: SDK → Web Frontend → Server deployment

3. Multiple Startup Methods

• start-web.sh: API worker only (partial functionality)
• start.sh: Both API + microservices (full functionality)
• Recommendation: Use start.sh for complete service

4. Port Management

• Issue: Old processes can persist after failed starts
• Solution: Always verify port availability before restart
• Tool: netstat -tlnp | grep :2283

5. File Path Expectations

• Server Expects: /opt/immich/app/dist/main.js
• Frontend Expects: /build/www/index.html
• Resources Expect: /build/geodata/geodata-date.txt

---

🚀 Usage Instructions Post-Recovery

Start Immich

cd /opt/immich
./start.sh

Stop Immich

pkill -f "node.*dist/main"
# OR use Ctrl+C if running in foreground

Check Status

# Check if services are running
netstat -tlnp | grep :2283

# Test web interface
curl -I http://127.0.0.1:2283

# Check logs
tail -f /tmp/immich-start.log

Access Web Interface

• Local: http://127.0.0.1:2283
• Network: http://[your-lxc-ip]:2283

---

🛡️ Prevention for Future Upgrades

Pre-Upgrade Checklist

1. Backup LXC container (Proxmox snapshot)
2. Export configuration (/opt/immich/.env)
3. Document current working state
4. Test backup restoration procedure

Post-Upgrade Verification

1. Test backend: curl http://127.0.0.1:2283/api/server-info/ping
2. Test frontend: Check browser loads interface
3. Verify database: Ensure asset counts match
4. Check logs: Monitor for errors in startup

Recovery Procedure Summary

1. Install system build dependencies
2. Rebuild server: npm ci && npm run build in /opt/immich/source/server/
3. Rebuild TypeScript SDK in /opt/immich/source/open-api/typescript-sdk/
4. Rebuild web frontend: npm ci && npm run build in /opt/immich/source/web/
5. Deploy build artifacts to runtime locations
6. Clean restart with proper startup script

---

📞 Troubleshooting Quick Reference

Error: "Cannot find module '/opt/immich/app/dist/main'"

• Fix: Rebuild server backend from source
• Commands: See "Backend Server Recovery" section

Error: Blank page in browser

• Fix: Rebuild and deploy web frontend
• Commands: See "Frontend Recovery" section

Error: "EADDRINUSE: address already in use"

• Fix: Kill existing processes on port 2283
• Command: netstat -tlnp | grep :2283 then kill -9 [PID]

Error: "ENOENT: no such file or directory, open './package.json'"

• Fix: Copy package.json to working directory
• Command: cp /opt/immich/source/server/package.json /opt/immich/

Error: Missing geodata files

• Fix: Create symlink to geodata directory
• Command: mkdir -p /build && ln -sf /opt/immich/geodata /build/geodata

---

✅ Final Status - Complete Success

Resolution Time: ~45 minutes
Current Status: ✅ FULLY OPERATIONAL
Version: Immich v1.137.3
All Features Working: Photo management, AI search, facial recognition, maps, etc.
Performance: <3ms response time, all original optimizations intact

This comprehensive recovery demonstrates that Immich upgrade issues can be resolved by rebuilding the application components from source when build artifacts are lost during the upgrade process.

[Gamechiefx]

As an added update to the guide, most of the broken upgrades I've seen are the result of npm not completing its build process due to an error or the fact that source files were not copied due to a script error. This guide essentially copies files from the source to the app directory and does an npm rebuild to complete the upgrade. all while maintaining immich data.