ESA Euclid Mission: Cluster Detection Visualization

An advanced interactive web-based visualization platform for astronomical cluster detection data from the ESA Euclid Mission. This sophisticated Dash application provides real-time analysis capabilities with comprehensive data integration, advanced filtering controls, interactive cluster analysis tools, and seamless remote access support.

📑 Table of Contents

Quick Remote Access Setup
Overview
Environment Requirements
Key Features
Project Structure
Quick Start
Configuration
Command-Line Options
Feature Usage Guide
Architecture & Technical Specifications
Development Environment
Advanced Capabilities & Data Analysis
Performance & Scalability
Troubleshooting & Support
Enterprise Benefits & Comparison
Recent Development Milestones
Technical Specifications & Data Insights

🔗 Quick Remote Access Setup

Access the application on a remote server (location of the stored data) using SSH port forwarding:

Step 1: Connect with Port Forwarding

From your local machine, run:

# Connect to remote server with port forwarding
ssh -L 8050:localhost:8050 username@remote-server.domain

# Example for CC-IN2P3 systems:
ssh -L 8050:localhost:8050 vnistane@cca.in2p3.fr

Step 2: Clone the Repository

Once connected to the remote server, clone the repository:

# Clone the repository
git clone https://github.com/virajnistane/EuclidClusterViz.git
cd EuclidClusterViz

# Or use SSH if you have access:
# git clone git@github.com:virajnistane/EuclidClusterViz.git

Step 3: Launch the Application

Navigate to the project directory and launch:

cd /path/to/ClusterViz  # or just 'cd EuclidClusterViz' if you just cloned
./launch.sh

Step 4: Access in Your Browser

Open your web browser on your local machine and navigate to:

http://localhost:8050

Important: Keep the SSH connection alive while using the application.

Alternative Ports (if 8050 is in use)

# Try alternative ports
ssh -L 8051:localhost:8050 username@remote-server.domain  # Access: http://localhost:8051
ssh -L 8052:localhost:8050 username@remote-server.domain  # Access: http://localhost:8052
ssh -L 8053:localhost:8050 username@remote-server.domain  # Access: http://localhost:8053

Connection Verification

When successfully connected, you'll see:

✓ User successfully connected at 16:21:57
  ✓ SSH tunnel appears to be working correctly
  Browser: Mozilla/5.0 (...)
  Connection from: 127.0.0.1

🌌 Overview

This tool provides a professional-grade visualization solution for Euclid cluster detection algorithms (PZWAV/AMICO) with integrated support for:

Interactive Cluster Analysis Tab with cutout generation, CATRED box views, and mask overlays
High-resolution CATRED data with masked HEALPix processing and PHZ analysis
Interactive mosaic imaging with MER tile integration and trace management
Real-time filtering by SNR and redshift with client-side performance
Advanced UI controls with dynamic visibility and responsive design
Professional remote access with SSH tunnel monitoring and automation
Trace management with hide/show and clear controls for all overlay types

🔧 Environment Requirements

Modern Package Management: Uses pyproject.toml with optional uv for 10-100x faster installations

The EDEN-3.1 environment lacks several critical modules (healpy, dash, plotly, etc.), so a virtual environment is required:

# Recommended: One-command setup with automatic uv integration
./setup_venv.sh

# The script will:
# 1. Create virtual environment (.venv)
# 2. Auto-install uv for faster package management
# 3. Install cluster-visualization package from pyproject.toml
# 4. Install all dependencies (10-100x faster with uv)

Package Configuration: Modern pyproject.toml setup with:

Build system: Hatchling (fast, modern) with setuptools fallback
CLI commands: cluster-viz and cluster-viz-test after installation
Development tools: pytest, black, mypy, pylint (optional)
Optional uv: Automatic bootstrapping for faster installations

Core Dependencies: plotly, pandas, numpy, astropy, healpy, shapely, dash, dash-bootstrap-components, psutil

🎯 Key Features

🔬 Advanced Data Analysis

Algorithm Comparison: Real-time switching between PZWAV, AMICO, and BOTH algorithms
Cluster Matching: Visual overlay showing matched PZWAV-AMICO cluster pairs with connecting ovals (BOTH mode only)
Interactive Cluster Analysis: Dedicated tab with cutout generation, CATRED box views, and mask cutouts
Smart Filtering: Client-side SNR and redshift filtering with preserved zoom states
CATRED Integration: High-resolution masked data with effective coverage thresholding
Mosaic Visualization: Dynamic MER tile mosaic loading with opacity controls
HEALPix Mask Overlay: Effective coverage footprint visualization with configurable opacity
PHZ Analysis: Interactive photometric redshift probability distribution plots with improved click detection

🎨 Cluster Analysis Tools

Cutout Generation: Create MER mosaic cutouts centered on selected clusters with configurable size and opacity
CATRED Box Views: Load high-resolution catalog data in a box around clusters with customizable parameters
Mask Cutouts: Generate HEALPix mask cutouts showing coverage around selected clusters
Trace Management: Independent hide/show and clear controls for cutouts, CATRED boxes, and mask overlays
Parameter Synchronization: Unified controls between sidebar and cluster analysis tab
Single-Section Expansion: Only one options section visible at a time for cleaner interface

🖥️ Professional UI Controls

Tabbed Interface: Separate tabs for main visualization and cluster analysis
Highlighted Section Headers: Clear visual hierarchy with Bootstrap styling
Dynamic Visibility: Context-aware control hiding/showing based on user selections
Algorithm-Based Toggle Control: Matching clusters toggle enabled only in BOTH mode
Real-time Updates: Live button text updates showing click counts and status
Responsive Design: Optimized layout for different screen sizes and zoom levels
Intuitive Workflow: Guided user experience with helpful tooltips and status indicators
Mosaic & Mask Management: Separate controls for background images and HEALPix footprint overlays
Collapsible Sections: Organized controls with expandable/collapsible cards

🌐 Enterprise Remote Access

SSH Tunnel Monitoring: Automatic detection and setup guidance for remote connections
Connection Validation: Real-time feedback on tunnel status and user connectivity
Multi-port Support: Automatic fallback to available ports (8050, 8051, 8052, 8053)
Production Ready: Robust error handling and connection management

⚡ Performance Optimization

Client-side Filtering: Real-time SNR/redshift filtering without server round-trips
Layered Rendering: Optimized trace ordering (polygons → mosaics → mask overlays → CATRED → clusters)
Preserved State: Zoom levels and filter settings maintained during updates
Efficient Caching: Smart data caching with trace preservation for smooth interactions
Trace Preservation: Mosaic and mask overlay traces retained across data updates
Color-coded tiles for easy identification
Aspect ratio controls for optimal viewing
Plot size adjustment for different screen sizes
Responsive web interface that works reliably across different browsers

📁 Project Structure

Modular Architecture

cluster_visualization/
├── src/
│   ├── cluster_dash_app.py          # 🚀 Main application entry point
│   ├── config.py                    # ⚙️ Configuration management
│   ├── data/
│   │   ├── loader.py               # 📊 Data loading and caching
│   │   ├── catred_handler.py       # 🔬 CATRED data processing
│   │   └── mermosaic.py            # 🖼️ Mosaic image handling
│   └── visualization/
│       ├── traces.py               # 📈 Plotly trace creation
│       └── figures.py              # 🎨 Figure layout management
├── callbacks/
│   ├── main_plot.py                # 🎯 Core plotting callbacks
│   ├── catred_callbacks.py         # 🔬 CATRED-specific callbacks
│   ├── mosaic_callback.py          # 🖼️ Mosaic & mask overlay callbacks
│   ├── ui_callbacks.py             # 🎛️ UI control callbacks
│   ├── phz_callbacks.py            # 📊 PHZ analysis callbacks
│   └── cluster_modal_callbacks.py  # 🔍 Cluster analysis tab callbacks
├── ui/
│   └── layout.py                   # 🖥️ Dash layout components
├── core/
│   └── app.py                      # 🏗️ Core application management
└── utils/
    ├── myutils.py                  # 🛠️ Utility functions
    └── colordefinitions.py         # 🎨 Color schemes

Launch Scripts & Configuration

📁 Root Directory/
├── launch.sh                       # 🚀 Universal launcher (recommended)
├── setup_venv.sh                   # 🔧 Virtual environment setup with uv
├── pyproject.toml                  # 📦 Modern Python package config
├── config.ini                      # ⚙️ Default configuration
├── config_local.ini                # 🔒 Personal config (gitignored)
├── requirements.txt                # 📋 Legacy dependencies (backup)
└── README.md                       # 📖 This documentation

Package Management (pyproject.toml)

Modern Python packaging with pyproject.toml:

# Build system: Hatchling (fast) with setuptools fallback
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

# CLI commands available after installation
[project.scripts]
cluster-viz = "cluster_visualization.src.cluster_dash_app:main"
cluster-viz-test = "cluster_visualization.tests.run_all_tests:main"

# Dependencies managed in pyproject.toml
[project]
dependencies = ["plotly==5.17.0", "pandas==2.1.4", ...]

# Optional development dependencies
[project.optional-dependencies]
dev = ["pytest>=7.0", "black>=22.0", "mypy>=1.0", ...]

Installation Commands:

# Standard installation (uses uv if available)
pip install -e .

# With uv (10-100x faster, auto-installed by setup_venv.sh)
uv pip install -e .

# Install with development dependencies
pip install -e ".[dev]"
uv pip install -e ".[dev]"

# Install specific dependency groups
pip install -e ".[test]"   # Testing tools
pip install -e ".[docs]"   # Documentation tools
pip install -e ".[all]"    # Everything

CLI Commands (available after installation):

cluster-viz              # Launch the Dash application
cluster-viz-test         # Run all tests

🚀 Quick Start

1. One-Time Setup (Fast with uv!)

# Clone the repository
git clone https://github.com/virajnistane/EuclidClusterViz.git
cd EuclidClusterViz

# Run setup (installs uv automatically for 10-100x faster installation)
./setup_venv.sh

# Setup process:
# ✓ Creates .venv with EDEN integration
# ✓ Installs uv package manager (automatic)
# ✓ Installs cluster-visualization package
# ✓ Installs all dependencies (blazing fast with uv)
# ✓ Optional: dev dependencies (pytest, black, mypy)

2. Launch Application (Multiple Options)

# 🎯 Option 1: Use CLI command (after setup)
cluster-viz

# 🚀 Option 2: Universal launcher (auto-activates venv)
./launch.sh

# 🔧 Option 3: With custom configuration
./launch.sh --config /path/to/custom_config.ini
cluster-viz --config /path/to/custom_config.ini

# 🌐 Option 4: Enable external access
cluster-viz --external

# 📝 Option 5: Direct Python execution
python cluster_visualization/src/cluster_dash_app.py

# View all available options
python cluster_visualization/src/cluster_dash_app.py --help

💡 Tip: Use --config to quickly switch between different datasets or testing/production environments without modifying your default configuration.

3. Application Interface

The app opens with a tabbed interface:

Main Visualization Tab

Highlighted control sections:

🔵 Algorithm: Switch between PZWAV/AMICO/BOTH
🔵 Cluster Matching: Enable matched cluster visualization (available in BOTH mode only)
🔵 SNR Filtering: Real-time signal-to-noise filtering with separate controls for PZWAV/AMICO
🔵 Redshift Filtering: Photometric redshift constraints
🔵 Display Options: Polygon fills, MER tiles, aspect ratio
🔵 High-res CATRED data: Advanced catalog integration with dynamic controls
🔵 Mosaic Image Controls: Background image overlays with opacity control
🔵 HEALPix Mask Overlay: Effective coverage footprint visualization

Cluster Analysis Tab

Interactive cluster-specific analysis:

🎯 Click-to-Select: Click any cluster point on the main plot to select it
🔬 Generate Cutouts: Create MER mosaic cutouts around selected clusters
- Configurable size (arcmin), opacity, and colorscale
- Hide/Show and Clear controls for trace management
🔍 CATRED Box Views: Load high-resolution catalog data in a box
- Box size, redshift bin width, mask threshold, magnitude limit
- Marker size (constant or KRON radius) and color customization
- Independent trace management controls
🗺️ Mask Cutouts: Generate HEALPix coverage cutouts
- Configurable size and opacity
- Separate trace visibility controls
📊 Analysis Results: Display analysis outcomes and statistics

Configuration

The application uses an INI-based configuration system for easy customization. The configuration specifies all data paths and settings.

Configuration Priority

The application loads configuration files in the following priority order:

Custom config file (if specified via --config argument)
config_local.ini (personal configuration, gitignored)
config.ini (default configuration, tracked in git)

Quick Setup

Automatic setup (recommended):
```
./setup_config.sh
```
This script will detect common paths and create a personalized config_local.ini file.

Manual setup:

cp config_example.ini config_local.ini
# Edit config_local.ini with your specific paths

Test configuration:

python config.py

# Or test with custom config
python cluster_visualization/src/cluster_dash_app.py --config test_config.ini

Using Custom Configuration Files

You can specify a custom configuration file when launching the application:

# Using root launcher (recommended)
./launch.sh --config /path/to/custom_config.ini

# Show help
./launch.sh --help

# Test dependencies
./launch.sh --test-dependencies

# Using scripts directory launcher
./cluster_visualization/scripts/launch.sh --config my_config.ini

# Using virtual environment script
./cluster_visualization/scripts/run_dash_app_venv.sh --config custom.ini

# Using remote access script
./cluster_visualization/scripts/run_remote_dash.sh --config custom.ini

# Direct Python execution
python cluster_visualization/src/cluster_dash_app.py --config custom.ini

# Show all Python command-line options
python cluster_visualization/src/cluster_dash_app.py --help

Benefits of custom config files:

📁 Multiple datasets: Easily switch between different data directories
🧪 Testing: Use test datasets without modifying main configuration
👥 Team sharing: Share project-specific configs via version control
🔄 Quick switching: Rapidly change between production/development/test setups

Configuration Files

config.ini - Default configuration (tracked in git)
config_local.ini - Your personal configuration (gitignored, takes precedence)
config_example.ini - Example with common configuration patterns
Custom configs - Any INI file can be specified via --config argument

Key Configuration Sections

[paths] - All data directories and file locations
[files] - Specific file and directory names for each algorithm

pip install -r requirements.txt

Dependencies

plotly - Interactive plotting library
pandas - Data manipulation
numpy - Numerical computations
astropy - FITS file handling
shapely - Geometric operations
healpy - HEALPix operations for masked CATRED data
dash - Web application framework
dash-bootstrap-components - Enhanced UI components

🎛️ Command-Line Options

The application supports several command-line arguments for flexible deployment:

Available Options

./launch.sh [OPTIONS]

Options:
  --config FILE            Use custom configuration file
  --test-dependencies      Test all dependencies and exit
  --help, -h               Show help message

Direct Python execution:
  python cluster_visualization/src/cluster_dash_app.py [OPTIONS]
  
  Options:
    --config PATH    Path to custom configuration file (default: auto-detect config_local.ini or config.ini)
    --external       Allow external access (binds to 0.0.0.0 instead of 127.0.0.1)
    --remote         Alias for --external (for backward compatibility)
    --help           Show help message and exit

Usage Examples

# Launch with default configuration
./launch.sh

# Show help message
./launch.sh --help

# Custom configuration file
./launch.sh --config /path/to/my_config.ini

# Test dependencies without launching
./launch.sh --test-dependencies

# Direct Python execution examples:
# Default configuration with local access
python cluster_visualization/src/cluster_dash_app.py

# Custom configuration file
python cluster_visualization/src/cluster_dash_app.py --config /path/to/my_config.ini

# External access (for network deployment)
python cluster_visualization/src/cluster_dash_app.py --external

# Combined: custom config with external access
python cluster_visualization/src/cluster_dash_app.py --config production.ini --external

When to Use Each Option

--config:
- Testing with different datasets
- Switching between production/development environments
- Using team-specific configuration files
- Running multiple instances with different data
--test-dependencies:
- Verify all required Python packages are installed
- Check project structure and configuration files
- Troubleshoot environment issues before launching
--help:
- View detailed usage information
- See all available options and examples
--external (Python only):
- Accessing from other machines on the network
- Accessing from other machines on the network
- Running on a server accessible to multiple users
- Container/cloud deployments
- Note: Use SSH tunneling for secure remote access (recommended)

🎮 Feature Usage Guide

Cluster Analysis Workflow

To analyze a specific cluster in detail:

Select a Cluster: Click on any cluster point in the main visualization
Open Cluster Analysis Tab: Switch to the "Cluster Analysis" tab
View Cluster Info: See selected cluster's RA, Dec, Redshift, SNR details
Choose Analysis Type:
- Cutout: Generate mosaic images centered on the cluster
- CATRED Box: Load high-resolution catalog data around the cluster
- Mask Cutout: Visualize coverage in the cluster region
Configure Parameters: Expand the options section by clicking the action button
Generate: Click the generate/view button to create the visualization
Manage Traces: Use Hide/Show and Clear buttons to control visibility

Smart UI Features:

Only one options section expands at a time (cutout, CATRED box, or mask)
Parameters sync between sidebar and cluster analysis tab
Trace management buttons enable automatically when traces exist
Hide button toggles between "Hide" and "Show" text

PHZ (Photometric Redshift) Analysis

To view PHZ probability distributions:

Click on a CATRED Point: Click any CATRED data point (when CATRED data is rendered)
View PHZ PDF Plot: The PHZ-PDF panel updates showing the probability distribution
Analyze Redshift:
- Blue curve shows the probability distribution
- Red dashed line indicates PHZ_MODE_1 (most probable redshift)
- Green dotted line shows PHZ_MEDIAN
Coordinate Matching: Uses pointNumber from clickData for accurate point identification

Technical Note: PHZ callback uses pointNumber instead of customdata for reliable point indexing, as customdata may contain coverage values in masked mode.

Cluster Matching Visualization

To visualize matched PZWAV-AMICO cluster pairs:

Select BOTH Algorithm: Set the algorithm dropdown to "BOTH"
Enable Matching: The "Show matched clusters (CAT-CL)" switch becomes enabled automatically
Toggle On: Activate the switch to see green ovals connecting matched pairs
Visual Indicators:
- 🟦 Square markers: PZWAV detected clusters
- 🔷 Diamond markers: AMICO detected clusters
- 🟢 Green ovals: Visual connections between matched pairs
Filter & Zoom: Use SNR/redshift filters and zoom - matching ovals update in real-time

Note: The matching switch is automatically disabled when using PZWAV or AMICO individually.

HEALPix Mask Overlay

To visualize the effective survey coverage:

Zoom In: Zoom to a region smaller than 2° × 2° (button becomes enabled)
Click "Render HEALPix Mask Overlay": Loads footprint data for visible tiles
Adjust Opacity: Use the opacity slider to control mask transparency (0.0-1.0)
Interpret Colors:
- Yellow/Green: High coverage (weight ≥ 0.95)
- Blue/Purple: Lower coverage (weight 0.80-0.95)
Independent Control: Mask overlay is independent of mosaic images

Performance: Limited to 5 tiles per zoom with 30-second timeout for responsiveness.

Mosaic Image Background

To add astronomical background images:

Enable Mosaic: Activate the "Enable MER-MOSAIC loading" switch
Zoom In: Zoom to a region smaller than 2° × 2°
Click "Render MER-MOSAIC images": Loads background images for visible tiles
Adjust Opacity: Use the mosaic opacity slider (0.0-1.0)
Multiple Layers: Mosaics and masks can be displayed simultaneously

Layer Management

The application maintains proper layering automatically:

Bottom → Top Layer Order:
1. Tile Polygons (CORE/LEV1 boundaries)
2. Mosaic Images (background astronomy)
3. HEALPix Mask Overlay (coverage footprint)
4. Mosaic Cutouts (cluster-centered images)
5. Mask Cutouts (cluster-centered coverage)
6. CATRED Data Points (high-res catalog)
7. CATRED Box Data (cluster-centered catalog)
8. Cluster Markers & Matching Ovals (detections)

Smart Preservation: All overlay layers are retained when:

Switching algorithms (PZWAV ↔ AMICO ↔ BOTH)
Applying SNR/redshift filters
Rendering/clearing CATRED data
Zooming or panning the view

Trace Management: Independent controls for:

Mosaic cutouts: Hide/Show and Clear buttons in Cluster Analysis tab
CATRED boxes: Hide/Show and Clear buttons in Cluster Analysis tab
Mask cutouts: Hide/Show and Clear buttons in Cluster Analysis tab
Global mosaics: Controls in main sidebar
Global masks: Controls in main sidebar
Global CATRED: Controls in main sidebar

🏗️ Architecture & Technical Specifications

Modular System Design

The application follows a sophisticated modular architecture enabling clean separation of concerns:

# Core Application Architecture
ClusterVisualizationApp
├── DataLoader           # 📊 FITS/HDF5 data processing with caching
├── CATREDHandler       # 🗺️  HEALPix masked catalog integration  
├── MOSAICHandler       # 🖼️  Background image overlays and HEALPix mask visualization
├── TraceCreator        # 📈 Plotly trace generation and optimization
└── FigureManager       # 🎨 Layout composition and client-side callbacks

Configuration Management

# config.py - Centralized configuration system
DATA_DIRECTORIES = {
    'PZWAV': '/sps/euclid/OU-LE3/CL/ial_workspace/workdir/MergeDetCat/RR2_south/',
    'AMICO': '/sps/euclid/OU-LE3/CL/ial_workspace/workdir/RR2_downloads/',
    'CATRED': '/sps/euclid/OU-LE3/CL/ial_workspace/workdir/catred_data/',
    'MOSAIC': '/sps/euclid/OU-LE3/CL/ial_workspace/workdir/mosaic_images/'
}

UI_CONFIG = {
    'DEFAULT_ALGORITHM': 'PZWAV',
    'ASPECT_RATIO': 'free',
    'DEFAULT_SNR_THRESHOLD': 4.0,
    'CATRED_COVERAGE_THRESHOLD': 0.05,
    'TRACE_LAYER_ORDER': ['polygons', 'mosaics', 'mask_overlays', 'catred', 'clusters']
}

Performance Optimizations

Client-side Filtering: Real-time SNR/redshift updates without server round-trips
Lazy Loading: CATRED, MOSAIC, and HEALPix mask data loaded on-demand with progress indicators
Optimized Trace Layering: Strategic rendering order (polygons → mosaics → mask overlays → CATRED → clusters)
Memory Management: Efficient HEALPix processing with masked arrays (NSIDE=16384)
Smart Caching: Intelligent data caching for algorithm switching and view changes
Trace Preservation: Mosaic and mask overlay traces retained across CATRED/filter updates

Advanced Features

SSH Tunnel Monitoring: Automatic connection detection with real-time guidance
Dynamic UI Controls: CATRED controls auto-hide/show based on switch state
Cluster Matching Visualization: Oval shapes connecting matched PZWAV-AMICO cluster pairs (BOTH mode)
HEALPix Mask Overlay: Effective coverage footprint visualization with configurable opacity
PHZ PDF Integration: Interactive photometric redshift probability plots
Responsive Layout: Bootstrap-styled UI with highlighted section organization
Multi-algorithm Support: Seamless PZWAV ↔ AMICO ↔ BOTH switching with data preservation
Trace Management: Intelligent preservation of mosaic and mask overlay layers across updates

🛠️ Development Environment

Modern Package Management

# 🚀 Fast Setup with uv (10-100x faster than pip)
./setup_venv.sh  # Auto-installs uv in venv

# Package installed in editable mode
pip install -e .          # Standard (slower)
uv pip install -e .       # With uv (faster)

# Install development dependencies
pip install -e ".[dev]"   # pytest, black, mypy, pylint
uv pip install -e ".[dev]" # Same, but faster

# Run tests
cluster-viz-test          # CLI command
pytest cluster_visualization/tests/

# Code formatting
black cluster_visualization/

# Type checking  
mypy cluster_visualization/

Supported Deployments

# Production Environment (EUCLID systems)
source /cvmfs/euclid-dev.in2p3.fr/EDEN-3.1/bin/activate
source .venv/bin/activate  # Virtual env with additional packages

# Development Environment (Universal)
./setup_venv.sh && source .venv/bin/activate

# Quick Launch
cluster-viz                # CLI command (after installation)
./launch.sh                # Shell launcher (auto-activates venv)

# Container Deployment (Future)
docker build -t euclid-cluster-viz .

Code Organization

cluster_visualization/src/
├── cluster_dash_app.py     # 🎯 Main application entry point
├── callbacks/              # 📞 Modular callback system
│   ├── main_plot.py       #     Primary plot generation
│   ├── ui_callbacks.py    #     UI control management  
│   ├── catred_callbacks.py#     CATRED data handling
│   ├── mosaic_callback.py #     Mosaic & HEALPix mask overlay
│   └── phz_callbacks.py   #     PHZ PDF visualization
├── components/            # 🧩 Reusable UI components
│   └── layout.py         #     Main layout with highlighted sections
├── data/                 # 📊 Data handlers
│   ├── loader.py        #     FITS/HDF5 data loading
│   ├── catred_handler.py#     CATRED catalog processing
│   └── mermosaic.py     #     Mosaic & mask visualization
├── visualization/        # 📈 Plotting components
│   ├── traces.py        #     Trace creation (clusters, ovals, overlays)
│   └── figures.py       #     Figure layout management
└── utils/                # 🔧 Core utilities
    ├── myutils.py        #     Data processing utilities
    └── colordefinitions.py#     Color scheme management

Key Technologies & Dependencies

Package Management: Modern pyproject.toml with Hatchling build backend
Fast Installation: Optional uv for 10-100x faster package installation
Core Framework: Dash 2.14+ with Plotly 5.17 for high-performance visualization
Astronomical Libraries: astropy, healpy for FITS/HEALPix data processing
Performance: pandas, numpy for efficient data manipulation, psutil for monitoring
Spatial Analysis: shapely for polygon operations and coordinate transformations
UI Framework: Bootstrap 5 for responsive design with custom styling
Development Tools: pytest, black, mypy, pylint (optional [dev] dependencies)

🎯 Advanced Capabilities & Data Analysis

Multi-Algorithm Cluster Detection

PZWAV Algorithm: 7,437 merged clusters across 11 individual tiles
AMICO Algorithm: 25,843 merged clusters with enhanced detection sensitivity
Real-time Comparison: Instant algorithm switching with preserved view settings
Statistical Analysis: Automatic cluster count summaries and detection rate comparisons

CATRED High-Resolution Catalog Integration

# Advanced HEALPix Processing (NSIDE=16384)
- Sparse format support for efficient memory usage
- Masked vs unmasked data comparison modes
- Coverage threshold filtering (configurable, default: 5%)
- Interactive PHZ PDF visualization on cluster click

Interactive Data Exploration

Smart Filtering: Client-side SNR and redshift filtering without server delays
Dynamic Layering: Optimized trace rendering (polygons → mosaics → mask overlays → CATRED → clusters)
Spatial Navigation: Advanced zoom/pan with coordinate system preservation
Hover Analytics: Detailed cluster properties, tile information, and metadata
Cluster Matching: Visual indication of PZWAV-AMICO cross-matched clusters with connecting ovals
Coverage Visualization: HEALPix footprint overlays showing effective survey coverage

Remote Collaboration Features

SSH Tunnel Auto-Setup: Intelligent connection monitoring with real-time guidance
Connection Validation: Automatic detection of proper SSH tunnel configuration
Multi-user Support: Concurrent access capability with connection tracking
Cross-platform Access: Works on any system with SSH and web browser

Professional Visualization Controls

Aspect Ratio Management: Equal vs free aspect ratio with proper coordinate scaling
Polygon Fill Toggle: Dynamic CORE region visibility control
Mosaic Image Overlays: Background astronomical images with opacity control
HEALPix Mask Overlays: Effective coverage footprint with independent opacity settings
MER Tile Visualization: 1,935 tile polygons with unique color coding
Cluster Matching Toggle: Enable/disable matched cluster pair visualization (BOTH mode only)

🚀 Performance & Scalability

Build & Installation Performance

uv Package Manager: 10-100x faster than pip for package installation
- Automatic installation in virtual environment by setup_venv.sh
- Parallel downloads and installations
- Shared package cache across environments
- Example: Full dependency install ~10s with uv vs ~5min with pip
Hatchling Build Backend: Modern, fast Python package builds
Editable Install: Development mode with instant code changes

Runtime Optimization Features

Client-side Processing: Real-time filtering without server round-trips
Lazy Loading Architecture: On-demand data loading for CATRED, MOSAIC, and HEALPix mask components
Memory Efficiency: Smart caching system for algorithm switching with psutil monitoring
Trace Management: Optimized layer ordering and intelligent trace preservation across updates
Independent Overlays: Separate control of mosaic images and HEALPix mask layers
Startup Timing: Performance instrumentation shows bottlenecks (EDEN check, venv activation, imports)

Large Dataset Handling

Sparse HEALPix Support: Efficient processing of NSIDE=16384 astronomical data
Progressive Loading: Staged data presentation for improved user experience
Background Processing: Non-blocking data operations with progress indicators
Scalable Architecture: Modular design supports future data expansion
Viewport-based Rendering: Only render visible data (zoom-based oval filtering)

Performance Benchmarks

Package Installation: 10s (uv) vs 5min (pip) for full setup
Virtual Environment: ~0.15s activation time
Dependency Checks: ~1.1s for all imports (healpy, plotly, pandas, etc.)
Startup Time: ~1.3s total (EDEN + venv + deps)

🔧 Troubleshooting & Support

Connection Issues

If you can't access the application:

Verify SSH port forwarding is active (see Quick Remote Access Setup above)
Keep the SSH connection alive while using the app
Access via http://localhost:8050 (not the server IP)
If port 8050 is in use, try alternative ports: 8051, 8052, 8053

Environment & Dependencies

# Full reinstallation (recommended)
./setup_venv.sh  # Recreates .venv with uv and all dependencies

# Manual dependency installation
pip install -e .              # Install from pyproject.toml
uv pip install -e .           # Faster with uv

# Specific packages (if needed)
pip install healpy astropy
uv pip install healpy astropy  # Faster

# Development tools
pip install -e ".[dev]"       # pytest, black, mypy, pylint

# uv not working? Fall back to pip
pip install -e .              # Works without uv

Package Installation Issues

# pyproject.toml build errors
pip install --upgrade pip setuptools wheel
pip install hatchling  # Install build backend

# Fallback to setuptools (if hatchling fails)
# Edit pyproject.toml: uncomment setuptools, comment hatchling

# uv installation failed?
# Script automatically falls back to pip - no action needed

# Check installed package
pip show cluster-visualization

Data Access & Configuration

# Verify Data Paths (check config.py)
ls /sps/euclid/OU-LE3/CL/ial_workspace/workdir/MergeDetCat/RR2_south/
ls /sps/euclid/OU-LE3/CL/ial_workspace/workdir/RR2_downloads/

# Custom Module Path Issues
export PYTHONPATH="${PYTHONPATH}:/path/to/cluster_visualization"

Performance Optimization

Slow Installation: Install uv: pip install uv, then reinstall: uv pip install -e .
Slow Loading: Start with Basic View, enable Detailed View only when needed
Memory Issues: Use client-side filtering instead of server-side processing
Large Datasets: Enable CATRED sparse mode for NSIDE=16384 data
Network Latency: Use local SSH tunnel, avoid direct server access

💼 Enterprise Benefits & Comparison

New Visualization Capabilities

🔗 Cluster Matching Visualization (BOTH Mode)

When using the "BOTH" algorithm mode, the application can display matched PZWAV-AMICO cluster pairs:

Visual Matching: Semi-transparent green ovals connect each PZWAV cluster (square marker) with its matched AMICO cluster (diamond marker)
Smart Activation: The matching clusters toggle is automatically enabled only when algorithm is set to "BOTH"
Cross-Identification: Uses CROSS_ID_CLUSTER field to link detections between algorithms
Interactive Overlay: Ovals are rendered with proper layering and can be toggled on/off without losing mosaic or mask data

🗺️ HEALPix Mask Overlay

Visualize the effective survey coverage using HEALPix footprint data:

Coverage Visualization: Display HEALPix pixels (NSIDE=16384) showing effective coverage weight
Independent Control: Separate button and opacity slider for mask overlays (independent of mosaic images)
Color-Coded Weights: Viridis colormap showing coverage quality (0.8-1.0 weight range)
Zoom-Dependent Loading: Automatically loads mask data for visible tiles when zoomed in
Performance Optimized: Limits to 5 tiles per zoom with 30-second timeout for responsiveness
Trace Preservation: Mask overlays are retained when switching algorithms or updating CATRED data

🖼️ Multi-Layer Overlay System

The application now supports independent control of multiple overlay layers:

Base Layer: Tile polygons (CORE/LEV1 regions)
Mosaic Layer: Background astronomical images with opacity control
Mask Layer: HEALPix effective coverage footprint with separate opacity
CATRED Layer: High-resolution catalog data points
Cluster Layer: Detection markers with matching ovals (BOTH mode)

Layer Management:

Each layer can be independently toggled on/off
Opacity controls for mosaic and mask layers
Intelligent trace preservation across data updates
Optimized rendering order for proper visual stacking

Advantages over Traditional Jupyter Notebooks

🔒 Production Reliability: No widget display issues in VS Code or remote environments
⚡ Enhanced Performance: Optimized for web browsers with client-side processing
🌐 Enterprise Sharing: Easy collaboration via URL sharing with SSH tunnel support
📱 Responsive Design: Adaptive interface works across devices and screen sizes
📦 Self-contained Deployment: Standalone HTML exports work without server dependencies
🎯 Superior Interactivity: Real-time zoom, pan, and filtering operations
🔄 Algorithm Comparison: Seamless switching between PZWAV, AMICO, and BOTH with preserved settings
🔗 Visual Cross-Matching: Geometric overlay showing matched cluster pairs across algorithms
🗺️ Multi-Layer Visualization: Independent control of mosaics, masks, CATRED, cutouts, and clusters
🔬 Cluster Analysis Tools: Dedicated interface for cutout generation, CATRED boxes, and mask overlays
🎛️ Trace Management: Granular hide/show and clear controls for all overlay types
🚀 Production Ready: Scalable web service deployment with monitoring capabilities
🔐 Secure Remote Access: Built-in SSH tunnel monitoring and connection validation
📊 Advanced Analytics: CATRED masked data integration with interactive PHZ visualization
🔍 Intelligent Monitoring: Automatic detection and resolution of connectivity issues
📋 Professional UI: Bootstrap-styled tabbed interface with organized sections
💾 State Preservation: Smart trace management retains overlays across all data operations
🎨 Smart UI Sections: Single-expansion sections for cleaner, more intuitive workflows

Professional Development Features

Modular Architecture: Clean separation of concerns with maintainable codebase
Configuration Management: Centralized settings with environment-specific configurations
Error Handling: Comprehensive fallback mechanisms and user guidance
Performance Monitoring: Built-in connection tracking and performance optimization
Extensible Design: Plugin-ready architecture for future enhancements

🔄 Recent Development Milestones

Q4 2024: Core Infrastructure

✅ Modular Architecture Implementation: Complete separation of data, UI, and callback layers
✅ Configuration System: Centralized path management and settings organization
✅ Performance Optimization: Client-side filtering and optimized trace layering
✅ SSH Tunnel Monitoring: Automatic connection detection with real-time user guidance

Q1 2025: Advanced Features

✅ CATRED Data Enhancement: Masked HEALPix support with sparse format (NSIDE=16384)
✅ Interactive PHZ Visualization: Click-to-view photometric redshift probability plots
✅ Dynamic UI Controls: CATRED section auto-hide/show based on switch state
✅ UI Layout Refactoring: Professional highlighting with modular section organization

November 2025: Mosaic & Matching Enhancements

✅ HEALPix Mask Overlay: Effective coverage footprint visualization with independent controls
✅ Cluster Matching Visualization: Oval shapes connecting matched PZWAV-AMICO pairs (BOTH mode)
✅ Algorithm-Based Toggle Control: Matching clusters switch enabled only in BOTH mode
✅ Trace Preservation System: Intelligent retention of mosaic and mask overlay layers
✅ Optimized Layer Management: Refined rendering order (polygons → mosaics → masks → CATRED → clusters)

November 2025 (Late): Cluster Analysis & Trace Management

✅ Cluster Analysis Tab: Dedicated interface for cluster-specific analysis with tabbed layout
✅ Cutout Generation: MER mosaic cutouts with configurable parameters and trace management
✅ CATRED Box Views: High-resolution catalog boxes around clusters with customization
✅ Mask Cutouts: HEALPix coverage cutouts centered on selected clusters
✅ Trace Management System: Hide/Show and Clear controls for all cutout types
✅ PHZ Callback Improvements: Fixed point detection using pointNumber instead of customdata
✅ Smart UI Sections: Only one options section expands at a time for cleaner interface
✅ Parameter Synchronization: Unified controls between sidebar and cluster analysis tab

Current State: Enterprise-Grade Platform

✅ Multi-Layer Visualization: Independent control of mosaics, masks, CATRED, cutouts, and cluster overlays
✅ Advanced Matching Analysis: Visual cluster cross-matching with geometric overlays
✅ Interactive Cluster Analysis: Comprehensive tools for detailed cluster investigation
✅ Robust State Management: Preserved traces and settings across all data operations
✅ Professional Documentation: Comprehensive README reflecting sophisticated architecture
✅ Trace Management: Granular control over visibility and clearing of all overlay types

📊 Technical Specifications & Data Insights

Dataset Statistics

PZWAV Detection: 7,437 clusters across 11 tiles with optimized SNR filtering
AMICO Detection: 25,843 clusters with enhanced sensitivity and validation
Spatial Coverage: 1,935 MER tile polygons with LEV1 and CORE region definitions
CATRED Integration: High-resolution catalog with coverage threshold filtering

Performance Metrics

Rendering Speed: Client-side filtering enables real-time updates (< 100ms)
Memory Efficiency: HEALPix sparse format reduces memory usage by ~80%
Connection Monitoring: SSH tunnel validation within 1-2 seconds
Data Loading: Progressive loading with visual progress indicators

Technology Stack

Core Framework: Dash 2.17+ with Plotly high-performance visualization
Data Processing: astropy, healpy, pandas, numpy for astronomical data
Spatial Analysis: shapely for coordinate transformations and polygon operations
UI Framework: Bootstrap 5 with custom responsive styling
Performance: Client-side callbacks, lazy loading, smart caching
Monitoring: Flask middleware for SSH tunnel validation and user tracking

Name		Name	Last commit message	Last commit date
Latest commit History 200 Commits
cluster_visualization		cluster_visualization
.gitignore		.gitignore
README.md		README.md
config.ini		config.ini
launch.sh		launch.sh
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt
setup_config.sh		setup_config.sh
setup_venv.sh		setup_venv.sh
uv.lock		uv.lock

Folders and files

Latest commit

History

Repository files navigation