Skip to content

INSTALL.md

Latest commit

 

History

History
478 lines (343 loc) · 12.4 KB

INSTALL.md

File metadata and controls

478 lines (343 loc) · 12.4 KB

Z-Push Zimbra Shim - Complete Installation Guide

Transform Z-Push from SOAP-based to direct Zimbra internal API calls for 20-60x performance improvements and critical bug fixes.

This guide provides a complete recipe for installing the Z-Push Zimbra Shim on a new Zimbra mailbox node, combining both the Java shim backend and the containerized Z-Push frontend.

Table of Contents

  1. Overview
  2. Prerequisites
  3. Installation Recipe
  4. Advanced Manual Installation
  5. Verification
  6. Troubleshooting
  7. Performance Validation

Overview

This installation provides:

  • 🚀 20-60x Performance Improvement - Direct API calls bypass SOAP/XML overhead
  • 🐛 Folder Space Bug Fixed - Folders like "0-Zimbra Security" work perfectly
  • 🔐 Proper 2FA Support - App passwords with EAS authentication context
  • 📧 Full Email Sync - Send and receive emails through optimized pipeline
  • 🔄 Automatic Fallback - SOAP fallback ensures production reliability

Architecture:

Mobile Device → Nginx → Z-Push Container → Java Shim → Zimbra Internal APIs
                ↑ Port 443/80      ↑ Port 9073    ↑ Port 8080

Quick Start Summary

TL;DR - For experienced administrators:

# 1. Build shim on external machine (required once)
git clone https://github.com/JimDunphy/build_zimbra.sh.git
cd build_zimbra.sh && ./build_zimbra.sh --init
git clone <zpush-shim-repository> && cd zpush-shim
./deploy-shim.sh --build

# 2. Deploy to mailbox node
scp dist/zpush-shim.jar deploy-shim.sh root@zimbra-host:/tmp/
ssh root@zimbra-host "cd /tmp && ./deploy-shim.sh --all"

# 3. Deploy Z-Push container
git clone <zpush-container-repository> && cd nginx
./manage.sh --up

# 4. Configure Zimbra routing
./manage.sh --patch-templates-clean && zmproxyctl restart

Result: ActiveSync working with 20-60x performance improvement. See detailed instructions below for troubleshooting and verification.

Prerequisites

System Requirements

On Zimbra Mailbox Node:

  • Zimbra FOSS Edition 8.8.15+ (Network Edition already has ActiveSync)
  • Docker and Docker Compose installed
  • Administrative access (root)
  • Java 11+ (included with Zimbra)
  • 4GB+ RAM recommended
  • 10GB+ free disk space

For Shim Compilation (External Machine):

  • Java 11+ JDK
  • Maven 3.6+ or Ant
  • Docker (recommended for build environment)

Network Requirements

  • Port 9073 available for Z-Push container
  • Port 8080 used by Zimbra (should be running)
  • Internet access for Docker image building

Verify Prerequisites

# Check Zimbra edition (must be FOSS)
zmcontrol -v
# Should NOT show "NETWORK" in the license type

# Check Docker installation
docker --version && docker-compose --version

# Check available ports
ss -tlnp | grep -E ":9073|:8080"
# 8080 should show Zimbra, 9073 should be free

# Check disk space
df -h /opt/zimbra /var/lib/docker

Installation Recipe

Step 1: Compile Z-Push Java Shim (External Build Machine)

⚠️ IMPORTANT: Never build on production mailbox nodes - use external build environment only

Option A: Using DockerZimbraRHEL8 Build Environment (Recommended)

Use the DockerZimbraRHEL8 project for a clean, isolated build environment:

# On your build machine (NOT the mailbox node)
git clone https://github.com/JimDunphy/DockerZimbraRHEL8.git
cd DockerZimbraRHEL8

# Follow the build_zimbra.sh project setup
# This provides a complete Zimbra development environment

# Clone the shim source
git clone <zpush-shim-repository>
cd zpush-shim

# Build using Ant (Zimbra's official build system)
./deploy-shim.sh --build

# The result should be: dist/zpush-shim.jar
ls -la dist/zpush-shim.jar

Option B: Using build_zimbra.sh Project (Simplified)

# On your build machine (NOT the mailbox node)
# Clone the build_zimbra.sh project - handles all dependencies automatically
git clone https://github.com/JimDunphy/build_zimbra.sh.git
cd build_zimbra.sh

# Setup complete build environment (installs Java, Ant, all dependencies)
./build_zimbra.sh --init

# Clone the shim source
git clone <zpush-shim-repository>
cd zpush-shim

# Build using Ant (environment already configured)
./deploy-shim.sh --build
# or manually: ant clean jar

# Verify build
ls -la dist/zpush-shim.jar

Step 2: Deploy Java Shim to Zimbra Mailbox Node

# Copy JAR and deploy script to mailbox node
scp dist/zpush-shim.jar deploy-shim.sh root@zimbra-host:/tmp/

# On the Zimbra mailbox node
cd /tmp

# Make deploy script executable
chmod +x deploy-shim.sh

# Deploy using the automated script
./deploy-shim.sh --deploy

# Restart Zimbra to load the extension
./deploy-shim.sh --restart

# Verify deployment
./deploy-shim.sh --verify

# Check status anytime
./deploy-shim.sh --status

What deploy-shim.sh --deploy does:

  • Creates /opt/zimbra/lib/ext/zpush-shim/ directory
  • Copies JAR with correct permissions (root:root 444)
  • Uses Zimbra's extension loading mechanism
  • No development tools needed on production server

Step 3: Deploy Z-Push Container with Shim Integration

# Clone the Z-Push container project
git clone <zpush-container-repository>
cd nginx

# Configure environment
cp env.example .env

# Edit .env for your environment
nano .env

Essential .env settings:

# Zimbra Configuration
ZIMBRA_URL=https://your-zimbra-host.com:8443  # Your Zimbra admin URL
ZIMBRA_USE_JAVA_SHIM=true
ZIMBRA_SHIM_URL=http://localhost:8080/service/extension/zpush-shim

# Z-Push Container Settings  
ZPUSH_HTTP_PORT=9073
PHP_MEMORY_LIMIT=256M
ZPUSH_IMAGE=local/zpush:latest

Step 4: Build and Deploy Z-Push Container

# Build the container with integrated shim patch
docker-compose build

# Start the Z-Push container
./manage.sh --up

# Verify container health
./manage.sh --health
# Should show: Container is healthy

Step 5: Configure Nginx Routing

# Backup existing templates
./manage.sh --backup-templates

# Apply nginx routing configuration
./manage.sh --patch-templates-clean

# Restart Zimbra proxy to apply changes
zmproxyctl restart

# Verify nginx configuration
nginx -t

Step 6: Verify Complete Installation

# Verify Java shim deployment and endpoint
./deploy-shim.sh --status
./deploy-shim.sh --verify

# Check container status
docker ps | grep zpush

# Test Z-Push container endpoint
curl -I http://127.0.0.1:9073/Microsoft-Server-ActiveSync

# Test via Zimbra proxy (end-to-end)
curl -I https://your-zimbra-host/Microsoft-Server-ActiveSync

# Check logs for shim activity (shows ultra-fast performance)
tail -f /opt/zimbra/log/mailbox.log | grep "ZPZB/74-SHIM"

Advanced Manual Installation

For environments where the automated method isn't suitable:

Manual Z-Push Backend Integration

# If you have an existing Z-Push installation
cd /var/www/zpush/backend/zimbra

# Backup original
cp zimbra.php zimbra.php.backup

# Apply shim integration patch
patch -p1 < /path/to/patches/zimbra.patch

# Verify patch applied
grep "74-SHIM" zimbra.php

Manual Nginx Template Configuration

# Get manual instructions
./manage.sh --print-manual-instructions

# Follow the displayed steps to manually modify:
# - /opt/zimbra/conf/nginx/templates/nginx.conf.web.template
# - /opt/zimbra/conf/nginx/templates/nginx.conf.web.https.default.template

# Generate new configuration
zmproxyconfgen && zmproxyctl restart

Manual Environment Configuration

Set these variables in your Z-Push backend configuration:

# In /var/www/zpush/backend/zimbra/config.php
define('ZIMBRA_USE_JAVA_SHIM', true);
define('ZIMBRA_SHIM_URL', 'http://localhost:8080/service/extension/zpush-shim');
define('ZIMBRA_URL', 'https://your-zimbra-host:8443');

Verification

Step 1: Health Checks

# Zimbra services
su - zimbra -c "zmcontrol status"

# Java shim endpoint
curl -X POST -d "action=ping" http://localhost:8080/service/extension/zpush-shim

# Z-Push container
./manage.sh --health

# Nginx routing
curl -I https://your-zimbra-host/Microsoft-Server-ActiveSync

Step 2: Test ActiveSync Device

  1. Add Exchange Account:

    • Server: your-zimbra-host.com
    • Username: your-email@domain.com
    • Password: your-password (or app password if 2FA enabled)

  2. Monitor Sync Activity:

    # Watch shim activity
tail -f /opt/zimbra/log/mailbox.log | grep "ZPZB/74-SHIM"

# Watch Z-Push container logs
./manage.sh --logs

Step 3: Performance Validation

# Check for ultra-fast response times (2-5ms)
tail -f /opt/zimbra/log/mailbox.log | grep "elapsed="

# Should see entries like:
# soap - GetFolderRequest elapsed=0
# soap - SearchRequest elapsed=2

Expected Performance:

  • Before (SOAP): 100-200ms response times
  • After (Shim): 2-5ms response times
  • Improvement: 20-60x faster

Troubleshooting

Container Issues

# Container won't start
./manage.sh --logs

# Port conflicts
ss -tlnp | grep 9073

# DNS resolution issues
echo '{"dns": ["8.8.8.8", "1.1.1.1"]}' > /etc/docker/daemon.json
systemctl restart docker

Shim Issues

# Shim not responding
curl -v http://localhost:8080/service/extension/zpush-shim

# Check Zimbra extension loading
grep -i "zpush\|shim" /opt/zimbra/log/mailbox.log

# Verify JAR deployment
ls -la /opt/zimbra/lib/ext/zpush-shim/zpush-shim.jar

Nginx Issues

# Check nginx configuration
nginx -t

# Verify upstream exists
grep -r "upstream zpush" /opt/zimbra/conf/nginx/

# Check template generation
ls -la /opt/zimbra/conf/nginx/includes/

Authentication Issues

# Test Zimbra authentication
curl -k -d "loginOp=login&username=test&password=test" \
     https://your-zimbra-host/service/preauth

# For 2FA users, ensure app passwords are used
# Check Z-Push authentication logs
grep -i "auth" /data/log/z-push-error.log

Performance Issues

# Verify shim is being used (not falling back to SOAP)
grep -c "falling back to SOAP" /data/log/z-push-error.log

# If count > 0, check shim connectivity
curl -X POST -d "action=getfolders&username=test&password=test" \
     http://localhost:8080/service/extension/zpush-shim

# Monitor actual response times
tail -f /opt/zimbra/log/mailbox.log | grep "elapsed="

Performance Validation

Before and After Comparison

Operation Before (SOAP) After (Shim) Improvement
Folder List 100-500ms 2-8ms 20-60x
Message List 200-1000ms 3-15ms 50-200x
Folder w/Spaces ❌ 404 Error ✅ Works Bug Fixed
Large Sync 5-30 seconds 200ms-2s 25-150x
Memory Usage 200-500MB 50-100MB 5-10x less

Monitoring Commands

# Real-time performance monitoring
tail -f /opt/zimbra/log/mailbox.log | grep -E "elapsed=|ZPZB/74-SHIM"

# Container resource usage
docker stats nginx-zpush-1

# Zimbra memory usage
ps aux | grep java | grep mailboxd

Success Indicators

Logs showing shim success:

INFO  [qtp-13661] [name=user@domain.com;ua=...ZPZB/74-SHIM;] soap - GetFolderRequest elapsed=0
INFO  [qtp-13661] [name=user@domain.com;ua=...ZPZB/74-SHIM;] soap - SearchRequest elapsed=2

Performance targets:

  • Initial folder sync: < 1 second (was 10+ seconds)
  • Individual operations: < 10ms (was 100+ ms)
  • Memory per sync: < 100MB (was 300+ MB)
  • No "404" errors for folders with spaces

Summary

This installation provides a complete, production-ready Z-Push Zimbra Shim deployment that:

✅ Dramatically improves performance (20-60x faster)
✅ Fixes critical bugs (folder spaces, 2FA support)
✅ Maintains reliability (automatic SOAP fallback)
✅ Uses automated deployment (Docker + management scripts)
✅ Provides comprehensive monitoring (logs and health checks)

The combination of the Java shim backend and containerized Z-Push frontend creates a powerful, maintainable solution for high-performance ActiveSync on Zimbra FOSS installations.

Next Steps:

  1. Monitor performance and logs for the first week
  2. Set up log rotation for Z-Push container logs
  3. Document any environment-specific configurations
  4. Plan regular maintenance and updates

For ongoing support and maintenance, refer to the individual README files in the repository for detailed operational procedures.