Skip to content

INTERNALS-SHIM.md

Latest commit

 

History

History
1101 lines (904 loc) · 33.7 KB

INTERNALS-SHIM.md

File metadata and controls

1101 lines (904 loc) · 33.7 KB

Z-Push Java Shim - Developer Internals Guide

This document explains the technical implementation of the Z-Push Java Shim for developers who need to modify, compile, or extend the code. Assumes familiarity with C programming but new to Java and Java toolchains.

Table of Contents

  1. Architecture Overview
  2. Java vs C: Key Differences
  3. Project Structure
  4. Core Components
  5. Build System (Ant)
  6. Compilation Process
  7. Integration with Z-Push
  8. Development Workflow
  9. Debugging and Testing
  10. Common Modifications
  11. Troubleshooting

Architecture Overview

High-Level Flow

Mobile Device (ActiveSync)
    ↓ HTTP/ActiveSync Protocol
Z-Push PHP Backend (zimbra.php)
    ↓ HTTP/JSON (NEW!)
Java Shim Servlet (ZPushShim.java)
    ↓ Direct Java Method Calls
Zimbra Internal APIs
    ↓ Database/File Operations
Zimbra Mailbox Data

What Problem We're Solving

Before (SOAP):

PHP → XML/SOAP → HTTP → Zimbra SOAP Parser → Internal APIs
  • Slow: XML parsing overhead
  • Buggy: URL encoding issues with folder names containing spaces
  • Complex: Multiple protocol layers

After (Shim):

PHP → JSON → HTTP → Java Servlet → Internal APIs (direct call)
  • Fast: Direct API calls, no XML parsing
  • Reliable: No URL encoding issues
  • Simple: Single HTTP call with JSON response

Performance Impact

  • 15,000 folders: 5-15 seconds → 50-200ms (30-300x faster)
  • Memory usage: 200-500MB → 20-50MB (10x reduction)
  • Folder names with spaces: 404 errors → Works perfectly

Java vs C: Key Differences

Since you're coming from C, here are the key differences to understand:

Memory Management

// C - Manual memory management
char* buffer = malloc(1024);
strcpy(buffer, "Hello");
free(buffer);  // Must remember to free!
// Java - Automatic garbage collection
String buffer = new String("Hello");
// No free() needed - garbage collector handles it

Compilation Process

# C - Direct compilation to machine code
gcc -o myprogram myprogram.c
./myprogram

# Java - Compile to bytecode, run on JVM
javac MyProgram.java    # Creates MyProgram.class
java MyProgram          # JVM interprets bytecode

Error Handling

// C - Return codes and errno
int result = some_function();
if (result < 0) {
    printf("Error: %s\n", strerror(errno));
    return -1;
}
// Java - Exceptions (like structured error handling)
try {
    someFunction();
} catch (ServiceException e) {
    System.out.println("Error: " + e.getMessage());
    throw new RuntimeException(e);
}

Strings

// C - Null-terminated character arrays
char str[256];
strcpy(str, "Hello");
strcat(str, " World");
// Java - String objects (immutable)
String str = "Hello";
str = str + " World";  // Creates new string object

Project Structure

zpushshim/
├── build.xml                         # Build configuration (like Makefile)
├── com/                              # Java package structure
│   └── zimbra/
│       └── zpush/
│           └── shim/
│               ├── ZPushShimCompat.java     # Cross-platform main implementation
│               ├── ZPushShimExtension.java  # Extension wrapper (cross-platform)
│               └── ZPushShim.java           # Original Zimbra-only implementation (excluded from dev builds)
├── build/                            # Build output (like obj/ in C)
│   ├── classes/                      # Compiled .class files
│   └── META-INF/                     # Extension metadata
├── dist/                             # Final output
│   └── zpush-shim.jar               # Zimbra extension JAR
├── lib/                              # Downloaded dependencies
│   └── gson-2.10.1.jar              # JSON processing library
├── docs/                             # Documentation
├── config-shim.php                  # Z-Push configuration
├── deploy-shim.sh                   # Deployment script
└── zimbra.php                       # Modified Z-Push backend

File Comparison to C Projects

Java File C Equivalent Purpose
build.xml Makefile Build configuration
ZPushShimCompat.java main.c Cross-platform main logic
ZPushShimExtension.java wrapper.c Extension interface
ZPushShim.java main_full.c Full Zimbra implementation (production only)
zpush-shim.jar Executable binary Deployable application

Zimbra Extension Model

Unlike a standalone servlet, this is a Zimbra Extension that:

  • Lives in /opt/zimbra/lib/ext/zpush-shim/
  • Gets loaded by Zimbra's extension framework
  • Registers servlet endpoints automatically
  • Uses Zimbra's existing authentication and APIs

Standalone Dev Server (No Zimbra Required)

For development on machines without Zimbra, a lightweight HTTP server mirrors the shim’s action API using mock data.

  • Class: com.zimbra.zpush.shim.DevServer
  • Core logic (shared with servlet): com.zimbra.zpush.shim.CompatCore
  • Default endpoint: http://127.0.0.1:8081/service/extension/zpush-shim
  • Ping contract: returns { "status": "ok", ... } (matches deploy verification)

Run it

ant run-dev           # starts on 127.0.0.1:8081
# or
make run-dev

Change port/bind

# Defaults: 127.0.0.1:8081
# Override via args: -b <bind> -p <port>
java -cp build:lib/gson-2.10.1.jar com.zimbra.zpush.shim.DevServer -b 0.0.0.0 -p 9090

Test it quickly

curl -X POST -d "action=ping" http://127.0.0.1:8081/service/extension/zpush-shim

Run tests against it

make test-dev         # uses test/shim-tests-dev.yml (base_url=http://127.0.0.1:8081)

Note

  • The standalone dev server intentionally does not expose /service/soap. Test runs in dev mode will log a 404 warning for SOAP login and proceed with shim actions — this is expected.

Real vs Compat Modes

  • Inside mailboxd (Jetty): The handler prefers the real Zimbra implementation and uses internal APIs to fetch actual data:
    • authenticate → Provisioning.authAccount(...) with EAS context; returns an encoded authToken and account info.
    • getfolders → Mailbox.getFolderList(...) mapped to JSON.
    • getmessages → Mailbox.search(SearchParams) returning message metadata.
    • getmessage → Mailbox.getMessageById(...) returning headers/fragment (plain/html).
  • Standalone dev mode: Falls back to CompatCore mock data for fast, dependency-free iteration and unit-style testing.
  • Tests: test/test_shim_endpoints.py works in both modes (dev: mocks; mailboxd: real data). No test code changes are required to switch modes.

Core Components

1. Cross-Platform Architecture Overview

The shim now uses a dual-implementation architecture for maximum compatibility:

Development Machine          Zimbra Server
      ↓                           ↓
ZPushShimCompat.java  →   ZPushShimCompat.java
   (Mock Mode)               (Zimbra Mode)
      ↓                           ↓
   Mock Data              Real Zimbra APIs

Key Files:

  • ZPushShimCompat.java: Cross-platform implementation with runtime detection
  • ZPushShimExtension.java: Extension wrapper that works everywhere
  • ZPushShim.java: Original full Zimbra implementation (excluded from dev builds)

2. ZPushShimCompat.java - Cross-Platform Main Implementation

Think of this like a CGI program in C:

// C CGI equivalent
int main() {
    char* query = getenv("QUERY_STRING");
    parse_parameters(query);
    
    if (strcmp(action, "authenticate") == 0) {
        handle_auth();
    } else if (strcmp(action, "getfolders") == 0) {
        handle_folders();
    }
    
    printf("Content-Type: application/json\r\n\r\n");
    printf("{\"success\": true}\n");
    return 0;
}

Cross-Platform Java Implementation:

public class ZPushShimCompat {
    private boolean isZimbraAvailable() {
        try {
            Class.forName("com.zimbra.cs.account.Provisioning");
            return true;  // On Zimbra server
        } catch (ClassNotFoundException e) {
            return false; // On development machine
        }
    }
    
    protected void doPost(HttpServletRequest request, HttpServletResponse response) {
        String action = request.getParameter("action");
        
        if ("authenticate".equals(action)) {
            handleAuth(request, response);
        } else if ("getfolders".equals(action)) {
            handleGetFolders(request, response);
        }
        
        response.setContentType("application/json");
        response.getWriter().write("{\"success\": true}");
    }
}

Benefits of Cross-Platform Design:

  • Compiles anywhere: No Zimbra dependencies at compile time
  • Works everywhere: Detects environment and adapts at runtime
  • Development friendly: Provides meaningful mock data for testing
  • Production ready: Will use real Zimbra APIs when deployed (via reflection)

Key Methods Breakdown

Cross-Platform Authentication Handler

private void handleAuth(HttpServletRequest request, HttpServletResponse response) {
    String username = request.getParameter("username");
    String password = request.getParameter("password");
    
    if (!isZimbraAvailable()) {
        // Development mode: return mock authentication
        Map<String, Object> result = new HashMap<>();
        result.put("success", true);
        result.put("authToken", "dev-token-" + System.currentTimeMillis());
        result.put("accountId", "dev-account");
        result.put("displayName", "Development User");
        result.put("email", username);
        writeJson(response, result);
        return;
    }
    
    // Zimbra mode: use reflection to call actual Zimbra APIs
    // TODO: Implement real Zimbra authentication via reflection
    writeError(response, "Zimbra authentication not implemented", 501);
}

How It Works:

  • Development: Returns mock authentication data for testing
  • Production: Will use reflection to call actual Zimbra authentication APIs
  • Safe: Never breaks, always provides meaningful responses

Folder Handler (Solves the Space Bug)

private void handleGetFolders(HttpServletRequest request, HttpServletResponse response) {
    // Direct API call - no SOAP/XML parsing!
    List<Folder> allFolders = mbox.getFolderList(octxt, MailItem.Type.UNKNOWN);
    
    for (Folder folder : allFolders) {
        // Handle folder names with spaces properly
        String folderName = folder.getName();
        if (folderName != null) {
            folderName = folderName.trim(); // No URL encoding issues!
        }
        folderData.put("name", folderName);
    }
}

2. Integration with Z-Push (zimbra.php)

Modified Authentication Flow:

public function Logon($username, $domain, $password) {
    // Try Java Shim first
    if ($this->_shimEnabled) {
        if ($this->_shimAuth($username, $password)) {
            $this->_connected = true;
            return true; // Skip SOAP entirely!
        }
    }
    
    // Fallback to original SOAP method
    $body = '<AuthRequest xmlns="urn:zimbraAccount">...';
    $response = $this->SoapRequest($body, $header, true, $returnJSON);
}

Shim Call Function:

private function _callShim($action, $params = array()) {
    // Prepare HTTP POST (like curl in C)
    $postData = http_build_query($params);
    curl_setopt($this->_shimCurl, CURLOPT_POSTFIELDS, $postData);
    
    $response = curl_exec($this->_shimCurl);
    $result = json_decode($response, true);
    
    if ($result['success']) {
        return $result; // Use shim result
    } else {
        return false;   // Fall back to SOAP
    }
}

Build System (Apache Ant)

Zimbra uses Apache Ant as their build system for extensions. Think of build.xml as a sophisticated Makefile with XML syntax:

build.xml Structure

<project name="zpush-shim" default="jar" basedir=".">
    <!-- Project properties (like #define in C) -->
    <property name="extension.name" value="zpush-shim"/>
    <property name="extension.version" value="1.0.0"/>
    <property name="src.dir" value="."/>
    <property name="build.dir" value="build"/>
    
    <!-- Classpath (like -I and -L flags in gcc) -->
    <path id="compile.classpath">
        <fileset dir="${zimbra.lib.dir}">
            <include name="**/*.jar"/>
        </fileset>
        <fileset dir="${zimbra.home}/jetty/webapps/service/WEB-INF/lib">
            <include name="zimbra-store.jar"/>
            <include name="zimbra-common.jar"/>
        </fileset>
    </path>
    
    <!-- Build targets (like Makefile targets) -->
    <target name="compile" depends="init">
        <javac srcdir="${src.dir}" destdir="${build.dir}" 
               classpathref="compile.classpath"/>
    </target>
</project>

Build Modes (compile-compat vs compile-full)

The build now supports two modes, selected automatically based on Zimbra presence at /opt/zimbra:

  • compile-compat (no Zimbra detected):
    • Compiles CompatCore and DevServer only (no servlet-api dependency).
    • Use ant run-dev or make run-dev to start the mock shim at 127.0.0.1:8081.
  • compile-full (Zimbra detected):
    • Compiles the full extension: ZPushShimExtension, ZPushShimCompat, and CompatCore.
    • ant jar packages dist/zpush-shim.jar for deployment under /opt/zimbra/lib/ext/zpush-shim/.

JDK Setup (RHEL 8 and similar)

Use the Zimbra-bundled JDK for building/deploying the extension on a Zimbra host, and any modern JDK (8+/11/17) for the standalone dev server.

Zimbra server build/deploy (recommended)

  • Switch to zimbra user (PATH is pre-set correctly on typical installs): 
    su - zimbra
java -version         # verify Java; typically OpenJDK 17 on modern Zimbra
ant -version          # sanity check Ant sees the JDK
ant clean jar         # builds the extension (compile-full)
  • Optional explicit env: 
    export JAVA_HOME=~/common/lib/jvm/java
export PATH="$JAVA_HOME/bin:$PATH"

Standalone dev server (no Zimbra required)

  • Ensure a modern JDK is first on PATH (JDK 11 or 17 preferred, 8 supported): 
    # Example paths vary by distro; use the JDK you installed
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk
export PATH="$JAVA_HOME/bin:$PATH"
java -version
  • Run the server and tests: 
    make run-dev     # starts http://127.0.0.1:8081/service/extension/zpush-shim
make test-dev    # runs tests against the dev server

Notes

  • Older JDKs may not support --version; use java -version instead.
  • The build targets use source/target=1.8; compiling with JDK 11/17 is fine.
  • If commands pick up the wrong JDK, check: 
    which java; which javac; echo $JAVA_HOME; echo $PATH

Zimbra Java Environment Helpers

On Zimbra hosts, prefer Zimbra’s own environment helpers so the correct Java and paths are set:

  • Source Zimbra env and export variables (fast path used by Zimbra scripts):

    # Much faster; just call zmlocalconfig once
source /opt/zimbra/bin/zmshutil || exit 1
zmsetvars -f

    This populates variables like zimbra_java_home, zimbra_home, etc., and updates PATH to include the bundled JDK at /opt/zimbra/common/lib/jvm/java.

  • Use the zmjava wrapper to launch Java with proper Zimbra options (ext dirs, policies):

    /opt/zimbra/bin/zmjava -version
# or, to run a standalone class/jar if needed
/opt/zimbra/bin/zmjava -Dfile.encoding=UTF-8 com.example.Main

    Internally, zmjava computes JRE_EXT_DIR based on zimbra_java_home:

    • ${zimbra_java_home}/jre/lib/ext if ${zimbra_java_home}/jre exists
    • otherwise ${zimbra_java_home}/lib/ext

For this project, you usually don’t need to invoke zmjava directly because the shim runs inside mailboxd. But if you add any helper tools that run on a Zimbra host, prefer zmshutil/zmsetvars -f and the zmjava wrapper to ensure consistency with Zimbra’s runtime.

Dependency Management

C approach:

# Manual library management
gcc -I/usr/include/zimbra -L/usr/lib/zimbra -lzimbra-store myprogram.c

Ant approach (Updated for jetty_base):

<!-- Direct reference to existing Zimbra JARs -->
<path id="compile.classpath">
    <fileset dir="${zimbra.home}/lib" erroronmissingdir="false">
        <include name="**/*.jar"/>
    </fileset>
    <!-- Corrected paths for modern Zimbra installations -->
    <fileset dir="${zimbra.home}/jetty_base/webapps/service/WEB-INF/lib" erroronmissingdir="false">
        <include name="zimbra-store.jar"/>
    </fileset>
    <fileset dir="${zimbra.home}/jetty_base/common/lib" erroronmissingdir="false">
        <include name="**/servlet-api*.jar"/>
    </fileset>
</path>

Important Path Changes:

  • Old: /opt/zimbra/jetty/lib → New: /opt/zimbra/jetty_base/common/lib
  • Old: /opt/zimbra/jetty/webapps → New: /opt/zimbra/jetty_base/webapps
  • The erroronmissingdir="false" allows building on development machines without Zimbra

Key Differences from Maven:

  • No automatic downloads: Uses existing Zimbra JARs on the system
  • Explicit paths: You specify exactly where to find libraries
  • Simpler: Less magic, more direct control
  • Zimbra-native: This is how all Zimbra extensions are built
  • Cross-machine development: Can build on machines without Zimbra installed

Cross-Machine Development Support

The build system now supports development on machines that don't have Zimbra installed:

Development vs Deployment Machines

Development Machine (this machine):

  • No Zimbra installation required
  • Can compile for syntax checking (with warnings)
  • Creates JAR files that can be copied to Zimbra servers
  • Uses erroronmissingdir="false" to handle missing paths gracefully

Deployment Machine (Zimbra server):

  • Has full Zimbra installation at /opt/zimbra/
  • All classpath dependencies available
  • Can compile without warnings
  • Supports direct deployment via ant deploy

Cross-Machine Workflow

# On development machine (no Zimbra)
ant clean compile  # May show missing JAR warnings - this is normal
ant jar           # Creates dist/zpush-shim.jar

# Copy JAR to Zimbra server
scp dist/zpush-shim.jar zimbra-server:/tmp/

# On Zimbra server
sudo mkdir -p /opt/zimbra/lib/ext/zpush-shim/
sudo cp /tmp/zpush-shim.jar /opt/zimbra/lib/ext/zpush-shim/
sudo chown -R zimbra:zimbra /opt/zimbra/lib/ext/zpush-shim/
su - zimbra -c "zmmailboxdctl restart"

Understanding Build Warnings

Normal on development machines:

WARNING: Zimbra installation not found at /opt/zimbra
         Compilation may fail. This is normal on development machines.
         Deploy and test on actual Zimbra server.

What this means:

  • Syntax checking works (Java compiler validates code structure)
  • Missing Zimbra JARs prevent full compilation verification
  • Runtime functionality must be tested on actual Zimbra server

Compilation Process

Command Comparison

Operation C Ant (Zimbra)
Clean rm -rf obj/ ant clean
Compile gcc -c *.c ant compile
Link gcc -o prog *.o ant jar
Install cp prog /usr/bin/ ant deploy

Build Targets (like Makefile targets)

# 1. Clean previous builds
ant clean

# 2. Download dependencies (like wget for libraries)
ant deps

# 3. Compile Java source to bytecode (cross-platform compatible)
ant compile
# Creates: build/com/zimbra/zpush/shim/ZPushShimCompat.class
# Creates: build/com/zimbra/zpush/shim/ZPushShimExtension.class
# Note: ZPushShim.java is excluded for cross-platform compatibility

# 4. Package into JAR (like creating .a archive)
ant jar
# Creates: dist/zpush-shim.jar (works on dev machine AND Zimbra server)

# 5. Deploy to Zimbra extension directory (only on Zimbra server)
ant deploy
# Copies to: /opt/zimbra/lib/ext/zpush-shim/

Cross-Platform Build Process

Development Machine (no Zimbra):

  1. ant clean compile - Compiles successfully with warnings about missing Zimbra
  2. ant jar - Creates deployable JAR with mock functionality
  3. JAR contains cross-platform code that detects runtime environment

Zimbra Server:

  1. Same build commands work without warnings
  2. Runtime detection automatically uses real Zimbra APIs
  3. Full production functionality available

What Happens During Build

# Target: clean
[echo] Cleaning build artifacts...
# rm -rf build/ dist/

# Target: deps  
[echo] Downloading Gson dependency...
# wget https://repo1.maven.org/maven2/.../gson-2.10.1.jar

# Target: compile
[echo] Compiling Z-Push Shim Extension...
# javac -cp /opt/zimbra/lib/*.jar com/zimbra/zpush/shim/*.java -d build/

# Target: jar
[echo] Creating JAR file: dist/zpush-shim.jar
# jar cf dist/zpush-shim.jar -C build . -C . META-INF/MANIFEST.MF

JAR File Structure

A JAR is like a tar.gz archive containing compiled classes:

# View JAR contents (like tar -tf)
jar -tf dist/zpush-shim.jar

META-INF/MANIFEST.MF                           # Extension metadata
com/zimbra/zpush/shim/ZPushShim.class         # Main shim logic
com/zimbra/zpush/shim/ZPushShimExtension.class # Extension wrapper
com/google/gson/                              # Bundled JSON library

Zimbra Extension MANIFEST.MF

The MANIFEST.MF file tells Zimbra which class to load:

Manifest-Version: 1.0
Zimbra-Extension-Class: com.zimbra.zpush.shim.ZPushShimExtension
Implementation-Title: Z-Push Zimbra Java Shim
Implementation-Version: 1.0.0

Integration with Z-Push

How PHP Calls Java

Traditional SOAP Flow:

// Build XML
$soap = '<AuthRequest xmlns="urn:zimbraAccount">...';
// Send HTTP request with XML body
$response = curl_exec($curl);
// Parse XML response
$xml = simplexml_load_string($response);

New Shim Flow:

// Build simple parameters
$params = array('action' => 'authenticate', 'username' => $user);
// Send HTTP POST
curl_setopt($curl, CURLOPT_POSTFIELDS, http_build_query($params));
$response = curl_exec($curl);
// Parse JSON response
$result = json_decode($response, true);

Fallback Mechanism

private function _callShim($action, $params) {
    try {
        $response = curl_exec($this->_shimCurl);
        if ($response === false) {
            return false; // Triggers SOAP fallback
        }
        return json_decode($response, true);
    } catch (Exception $e) {
        ZLog::Write(LOGLEVEL_WARN, "Shim failed: " . $e->getMessage());
        return false; // Automatic fallback to SOAP
    }
}

Development Workflow

1. Setting Up Development Environment

Install Java Development Kit:

# Check if Java is installed
java -version
javac -version

# Install if needed (Ubuntu/Debian)
sudo apt-get install openjdk-11-jdk ant

# Install if needed (CentOS/RHEL)
sudo yum install java-11-openjdk-devel ant

Verify Ant Installation:

ant -version
# Should show Apache Ant 1.9+ and Java 8+

Note for C Programmers:

  • Ant is like make but uses XML instead of Makefiles
  • Java compiles to bytecode, not machine code
  • Extensions are like shared libraries (.so) that get loaded into Zimbra

2. Code Modification Workflow

Step 1: Edit Java Code

# Edit the main shim logic
vim com/zimbra/zpush/shim/ZPushShim.java

# Or edit the extension wrapper
vim com/zimbra/zpush/shim/ZPushShimExtension.java

Step 2: Compile and Test

# Compile only (faster for syntax checking)
ant compile

# Full build with packaging
ant clean jar

Step 3: Deploy and Test

# Deploy as Zimbra extension
ant deploy
# OR use the script:
./deploy-shim.sh --deploy

# Restart Zimbra mailbox service (loads extensions)
su - zimbra -c "zmmailboxdctl restart"

# Test the change
curl -X POST -d "action=ping" http://localhost:8080/service/extension/zpush-shim

Key Differences from C Development:

  • No linking step: Java bytecode is "linked" at runtime
  • Extension loading: Zimbra loads extensions automatically on restart
  • Classpath: Like LD_LIBRARY_PATH but for Java classes

3. Common Development Tasks

Adding a New API Endpoint:

  1. Add method to ZPushShim.java:
private void handleNewOperation(HttpServletRequest request, HttpServletResponse response) 
        throws ServiceException, IOException {
    
    String param = request.getParameter("someParam");
    
    // Call Zimbra internal API
    SomeResult result = mbox.someZimbraMethod(octxt, param);
    
    // Convert to JSON and return
    Map<String, Object> data = new HashMap<>();
    data.put("result", result.getValue());
    writeJson(response, data);
}
  1. Add to action dispatcher:
switch (action.toLowerCase()) {
    case "authenticate":
        handleAuth(request, response);
        break;
    case "newoperation":  // Add this case
        handleNewOperation(request, response);
        break;
    // ... other cases
}
  1. Add PHP integration to zimbra.php:
private function _shimNewOperation($param) {
    return $this->_callShim('newoperation', array('someParam' => $param));
}

4. IDE Setup (Optional but Helpful)

IntelliJ IDEA (Recommended):

# Import Maven project
File → Open → select zpushshim/ directory
# IntelliJ automatically recognizes Maven structure

Eclipse:

# Import Maven project
File → Import → Existing Maven Projects → Browse to zpushshim/

VS Code:

# Install Java Extension Pack
# Open zpushshim/ directory
# VS Code automatically detects Maven project

Debugging and Testing

1. Enabling Debug Output

Java Side (add to ZPushShim.java):

// Add debug logging
ZimbraLog.misc.debug("Shim received action: %s with params: %s", action, params);

PHP Side (zimbra.php):

// Enable debug in Z-Push config
define('LOGLEVEL', LOGLEVEL_DEBUG);

// Check for shim debug messages
ZLog::Write(LOGLEVEL_DEBUG, 'Shim call successful: ' . $action);

2. Testing Individual Components

Test Shim Directly:

# Test ping
curl -X POST -d "action=ping" http://localhost:8080/service/extension/zpush-shim

# Test authentication
curl -X POST -d "action=authenticate&username=test&password=test&protocol=eas" \
     http://localhost:8080/service/extension/zpush-shim

# Test with debugging
curl -v -X POST -d "action=getfolders&authToken=xyz" \
     http://localhost:8080/service/extension/zpush-shim

Test Z-Push Integration:

# Check Z-Push logs
tail -f /var/log/z-push/z-push.log | grep -i shim

# Check Zimbra logs
tail -f /opt/zimbra/log/mailbox.log | grep -i "zpush\|shim"

3. Common Debug Scenarios

Shim Not Responding:

# Check if JAR is deployed
ls -la /opt/zimbra/lib/ext/zpush-shim/zpush-shim.jar

# Check Zimbra service status
su - zimbra -c "zmcontrol status"

# Check extension registration (logs)
grep -i "zpush-shim" /opt/zimbra/log/mailbox.log | tail -n 50

Authentication Failing:

# Test Zimbra auth directly
curl -k -d "loginOp=login&username=test&password=test" \
     https://mail.domain.com/service/preauth

Performance Issues:

// Add timing to Java methods
long startTime = System.currentTimeMillis();
// ... method logic ...
long elapsed = System.currentTimeMillis() - startTime;
ZimbraLog.misc.info("Operation took %dms", elapsed);

Common Modifications

1. Adding New Zimbra API Calls

Pattern to Follow:

private void handleNewZimbraCall(HttpServletRequest request, HttpServletResponse response) 
        throws ServiceException, IOException {
    
    // 1. Validate auth token
    AuthToken authToken = getValidatedAuthToken(request.getParameter("authToken"));
    if (authToken == null) {
        writeError(response, "Invalid auth token", 401);
        return;
    }
    
    // 2. Get required parameters
    String requiredParam = request.getParameter("requiredParam");
    if (requiredParam == null) {
        writeError(response, "Missing requiredParam", 400);
        return;
    }
    
    // 3. Set up Zimbra context
    Account account = authToken.getAccount();
    Mailbox mbox = MailboxManager.getInstance().getMailboxByAccount(account);
    OperationContext octxt = new OperationContext(account);
    
    // 4. Call Zimbra internal API
    try {
        SomeZimbraResult result = mbox.someZimbraMethod(octxt, requiredParam);
        
        // 5. Convert to simple format and return
        Map<String, Object> data = new HashMap<>();
        data.put("result", result.getSomeValue());
        writeJson(response, data);
        
    } catch (NoSuchItemException e) {
        writeError(response, "Item not found", 404);
    }
}

2. Performance Optimization

Caching Results:

private final Map<String, Object> cache = new ConcurrentHashMap<>();

private Object getCachedResult(String key, Supplier<Object> supplier) {
    return cache.computeIfAbsent(key, k -> supplier.get());
}

Batch Operations:

// Instead of multiple individual calls
for (String folderId : folderIds) {
    getMessages(folderId); // Multiple round trips
}

// Use batch API
List<String> allFolderIds = Arrays.asList(folderIds);
Map<String, List<Message>> allMessages = mbox.getMessagesByFolders(octxt, allFolderIds);

3. Error Handling Improvements

Structured Error Responses:

private void writeError(HttpServletResponse response, String error, int statusCode, String errorCode) {
    response.setStatus(statusCode);
    Map<String, Object> errorResponse = new HashMap<>();
    errorResponse.put("success", false);
    errorResponse.put("error", error);
    errorResponse.put("errorCode", errorCode);
    errorResponse.put("timestamp", System.currentTimeMillis());
    writeJson(response, errorResponse);
}

Troubleshooting

Common Compilation Errors

Missing Dependencies:

[javac] package com.zimbra.cs.account does not exist

Solution: Check Zimbra is installed and paths are correct in build.xml:

export ZIMBRA_HOME=/opt/zimbra
ant clean jar

Wrong Java Version:

[javac] invalid target release: 11

Solution: Update Java or modify build.xml:

<javac ... source="1.8" target="1.8">

Classpath Issues:

[javac] cannot find symbol: class ZimbraExtension

Solution: Verify Zimbra JARs are accessible (updated paths):

ls -la /opt/zimbra/lib/*.jar
ls -la /opt/zimbra/jetty_base/webapps/service/WEB-INF/lib/zimbra-*.jar
ls -la /opt/zimbra/jetty_base/common/lib/servlet-api*.jar

Cross-Machine Development Issues:

WARNING: Zimbra installation not found at /opt/zimbra

Solution: This is normal on development machines. Either:

  • Ignore the warning and create the JAR for deployment elsewhere
  • Install Zimbra on your development machine
  • Use the cross-machine workflow documented above

Runtime Issues

ClassNotFoundException:

java.lang.ClassNotFoundException: com.zimbra.zpush.shim.ZPushShimExtension

Solution: Verify extension deployment:

# Check JAR contains the extension class
jar -tf dist/zpush-shim.jar | grep ZPushShimExtension

# Verify deployment location (extension directory)
ls -la /opt/zimbra/lib/ext/zpush-shim/zpush-shim.jar

# Check Zimbra logs for extension loading
tail -f /opt/zimbra/log/mailbox.log | grep -i "zpush\|extension"

Extension Not Loading:

[main] WARN - Extension zpush-shim failed to load

Solution: Check MANIFEST.MF and extension class:

# Verify MANIFEST.MF has correct extension class
jar -xf dist/zpush-shim.jar META-INF/MANIFEST.MF
cat META-INF/MANIFEST.MF

# Should contain:
# Zimbra-Extension-Class: com.zimbra.zpush.shim.ZPushShimExtension

OutOfMemoryError:

java.lang.OutOfMemoryError: Java heap space

Solution: Increase Zimbra JVM memory:

# Edit Zimbra JVM settings
zmlocalconfig -e zimbra_server_jvm_heap_size=2048m
zmcontrol restart

Performance Debugging

Slow Response Times:

// Add method timing
private void logMethodTiming(String method, long startTime) {
    long elapsed = System.currentTimeMillis() - startTime;
    if (elapsed > 1000) { // Log if > 1 second
        ZimbraLog.misc.warn("Slow method %s took %dms", method, elapsed);
    }
}

Memory Leaks:

// Monitor cache sizes
private void logCacheStats() {
    ZimbraLog.misc.info("Token cache size: %d", tokenCache.size());
    // Clear old entries
    tokenCache.entrySet().removeIf(entry -> entry.getValue().isExpired());
}

Summary for C Programmers

Think of this Java project as:

  • ZPushShim.java = Your main application logic
  • ZPushShimExtension.java = Your library interface (like .so exports)
  • build.xml = Your Makefile
  • zpush-shim.jar = Your compiled shared library (.so)
  • Ant = Your build system (like make, but XML-based)
  • Extension = Plugin/module that gets loaded into Zimbra
  • Zimbra APIs = System libraries you're linking against

Key Differences from C:

  • Memory management: Automatic garbage collection
  • Compilation: Source → bytecode → JVM execution
  • Linking: Runtime linking via classpath
  • Deployment: Copy JAR to extension directory, restart service
  • Debugging: JVM provides stack traces and runtime info

Advantages over C for this project:

  • Memory safety: No segfaults, buffer overflows, or memory leaks
  • Rich libraries: Excellent HTTP/JSON support built-in
  • Zimbra integration: Direct access to Zimbra's Java APIs
  • Platform independence: Same JAR runs on any OS with JVM

The build process is simpler than modern C++ (no autotools/cmake complexity), and Ant is straightforward once you understand the XML syntax. The extension model makes deployment and updates much safer than traditional C shared libraries.