diff --git a/scripts/coyote_protector_beamtime_cds_to_sdf/README.md b/scripts/coyote_protector_beamtime_cds_to_sdf/README.md deleted file mode 100644 index fd118a7..0000000 --- a/scripts/coyote_protector_beamtime_cds_to_sdf/README.md +++ /dev/null @@ -1,59 +0,0 @@ -## RUNNING COYOTE FOR AN EXPERIMENT - -As of January 2026, the proposed solution for detector protection is a pre-scan of the chip followed by a post-process of the images using the coyote algorithm. The output of the algorithm is a .csv file with the positions, in pixel coordinates, of each cristal whose size is above `alert_um`, for each frame that compose the pre-scanning. - -## Key Highlights: -- Fine-tuned YOLOv11 model for accurate crystal detection -- Pixel-to-micron size conversion and threshold alert system -- GPU Inference script that save results with bounding boxes and size info -- Dataflow management between different clusters - -## Overview of the scripts - -To use the routine, the user has to understand the different scripts in the coyote_protector_beamtime folder. -This README.md is made according to SLAC's infrastructure : The data will be collected on `cds/` and exported on `s3df/` for processing. The results will then be imported to `cds/`. - -On one hand, there is the main algorithm to run `coyote_detection_routine_v2.sh` has to be located on `cds/.../`or wherever the images will be located after there acquisition. This algorithm deals with the export, post-processing via SLURM and the import of the results. - -On the other hand, the user has : -- inference_coyote.py -- run_inference.sh -- weight_yolov11n_150epochs. - -Those three algorithms are in charge of the inference and the generation of the .csv file. In particular, the .pt (PyTorch) file, corresponds to the weights of the YOLO model used. It can be changed with any other weight file. - - -## Setting up the coyote protector - - -### 1. On `/sdf` - -In your workspace (needs a lot of available quota), clone the repo coyote_protector/scripts/coyote_protector_beamtime_cds_to_sdf - -In `inference_coyote.py` change : -- weights_path -> path to the weights of the algo, (for instance '/sdf/data/lcls/ds/prj/prjlumine22/results/pmonteil/coyote_beamtime_19jan/weight_yolov11n_150epochs.pt') -- mag_factor -> magnification factor (for instance 1) -- px_size -> size of the pixels in um, (for instance 0.5) -- alert_um -> threshold for alert, (for instance 100.0 um) - -`run_inference.sh` should not be changed - - -### 2. On `/cds` (or where the data are stored) - -Clone the repo coyote_protector/scripts/coyote_protector_beamtime_cds_to_sdf - -In `coyote_detection_routine_v2.sh`, change : -- SOURCE_PATH -> put the path where the images are located, (for instance "/cds/data/iocData/ioc-icl-alvium05/logs") -- DEST_BASE -> put the path of the base folder on S3DF where runs are created. THIS HAS TO BE THE FOLDER WHERE the repo coyote_protector/scripts/coyote_protector_beamtime_cds_to_sdf has been cloned. (for instance "/sdf/data/lcls/ds/prj/prjlumine22/results/pmonteil/coyote_beamtime_19jan") -- RESULTS_BACK_BASE -> put the path where you want results to be copied back on CDS, (for instance "/cds/home/p/pmonteil") - - -## Using the coyote detector - -On `/cds`, run - -``` bash -./coyote_detection_routine_v2.sh --user=name_of_the_user -``` -If the user wants a passwordless experience, refer to the issue related to key creations. diff --git a/scripts/coyote_protector_beamtime_cds_to_sdf/coyote_detection_routine_v2.sh b/scripts/coyote_protector_beamtime_cds_to_sdf/coyote_detection_routine_v2.sh deleted file mode 100755 index 5b9ff56..0000000 --- a/scripts/coyote_protector_beamtime_cds_to_sdf/coyote_detection_routine_v2.sh +++ /dev/null @@ -1,108 +0,0 @@ -#!/bin/bash - -#Default username -USER="pmonteil" - -# ---------------------------- -# Parse user namearguments -# ---------------------------- -for arg in "$@"; do - case $arg in - --user=*) - USER="${arg#*=}" - shift - ;; - *) - ;; - esac -done - - -DEST_HOST="psana.sdf.slac.stanford.edu" - -# Source images on psdev (CDS mount) -SOURCE_PATH="/cds/data/iocData/ioc-icl-alvium05/logs" - -# Base folder on S3DF where runs are created -DEST_BASE="/sdf/data/lcls/ds/prj/prjlumine22/results/pmonteil/coyote_beamtime_19jan" - -# Where you want results copied back on CDS -RESULTS_BACK_BASE="/cds/home/p/pmonteil" - -echo "[INFO] Source images : ${SOURCE_PATH}" -echo "[INFO] Remote base : ${DEST_HOST}:${DEST_BASE}" -echo "[INFO] Results back to : ${RESULTS_BACK_BASE}" -echo - -# ------------------------------------------------------------ -# 1) Create a new detection_N/images folder on psana -# ------------------------------------------------------------ -RUN_DIR="$(ssh "${USER}@${DEST_HOST}" "bash -lc ' - set -euo pipefail - mkdir -p \"${DEST_BASE}\" - - i=1 - while [ -d \"${DEST_BASE}/detection_\$i\" ]; do - i=\$((i+1)) - done - - RUN=\"${DEST_BASE}/detection_\$i\" - mkdir -p \"\$RUN/images\" - echo \"\$RUN\" -'")" - -echo "[INFO] Created remote run dir: ${RUN_DIR}" - -# ------------------------------------------------------------ -# 2) Copy images into detection_N/images/ -# ------------------------------------------------------------ -echo "[INFO] Copying images to ${RUN_DIR}/images/ ..." -rsync -avr --delete \ - "${SOURCE_PATH}/" \ - "${USER}@${DEST_HOST}:${RUN_DIR}/images/" - -# ------------------------------------------------------------ -# 3) Launch run_inference.sh from inside detection_N -# Assumption: run_inference.sh exists on psana at ../RUN_DIR/run_inference.sh -# If it lives elsewhere, see note below. -# ------------------------------------------------------------ -echo "[INFO] Launching inference job..." -JOB_ID="$(ssh "${USER}@${DEST_HOST}" "bash -lc ' - set -euo pipefail - cd \"${RUN_DIR}\" - jid=\$(sbatch --parsable ../run_inference.sh) - echo \$jid -'")" - -echo "[INFO] Submitted job: ${JOB_ID}" - -# ------------------------------------------------------------ -# 4) Wait for CSV output -# ------------------------------------------------------------ -REMOTE_CSV="${RUN_DIR}/runs/size_measurements/measurements_complete.csv" -echo "[INFO] Waiting for output CSV: ${REMOTE_CSV}" - -while true; do - if ssh "${USER}@${DEST_HOST}" "bash -lc 'test -f \"${REMOTE_CSV}\"'"; then - echo "[INFO] CSV detected." - break - fi - sleep 0.5 -done - -# ------------------------------------------------------------ -# 5) Copy results back to CDS -# ------------------------------------------------------------ -RUN_NAME="$(basename "${RUN_DIR}")" -LOCAL_DEST_DIR="${RESULTS_BACK_BASE}/${RUN_NAME}" -mkdir -p "${LOCAL_DEST_DIR}" - -echo "[INFO] Copying CSV back to: ${LOCAL_DEST_DIR}/" -rsync -avr \ - "${USER}@${DEST_HOST}:${REMOTE_CSV}" \ - "${LOCAL_DEST_DIR}/" - -echo -echo "[DONE] Results available at:" -echo " ${LOCAL_DEST_DIR}/measurements_complete.csv" - diff --git a/scripts/coyote_protector_beamtime_cds_to_sdf/inference_coyote.py b/scripts/coyote_protector_beamtime_cds_to_sdf/inference_coyote.py deleted file mode 100755 index f9ec694..0000000 --- a/scripts/coyote_protector_beamtime_cds_to_sdf/inference_coyote.py +++ /dev/null @@ -1,109 +0,0 @@ -""" -YOLOv8 Inference Script → CSV (sizes) + YOLO-rendered images ------------------------------------------------------------- -- Uses YOLO's own annotated images (save=True). -- Computes longest side per detection in px and μm. -- If size > alert_um (default 100 μm), prints "Stop the beam" and marks it in CSV. - -EDIT THESE BEFORE RUNNING: - - weights_path: Path to your trained YOLO weights (.pt) - - chip_pic_dir: Folder containing images to run inference on - - px_to_um: Pixel-to-micron conversion factor for your setup -""" - -import os -import csv -from pathlib import Path -import numpy as np -from ultralytics import YOLO - -# ---------- Paths (edit these) ---------- -weights_path = '/sdf/home/p/pmonteil/prjlumine22/results/pmonteil/coyote_beamtime_19jan/weight_yolov11n_150epochs.pt' # trained weights -mag_factor = 1 #magnification factor (i.e the zoom of the optics used) -px_size = 0.5 #pixel size -px_to_um = px_size/mag_factor # pixel -> micron conversion -alert_um = 100.0 # threshold for alert - -# --------------------------------------- - -chip_pic_dir = 'images' # folder of images to test - -# Where to write the CSV -out_dir = Path("runs/size_measurements") -out_dir.mkdir(parents=True, exist_ok=True) -csv_path = out_dir / "measurements.csv" - -# Load model -model = YOLO(weights_path) - -# Run prediction directly on the whole directory -results = model.predict( - source=chip_pic_dir, # directly pass the folder - save=False, # YOLO draws/saves annotated images - verbose=True -) - -# Write CSV -with open(csv_path, "w", newline="") as f: - writer = csv.writer(f) - writer.writerow([ - "image", - "det_idx", - "class_id", - "class_name", - "confidence", - "x_center_px", - "y_center_px", - "width_px", - "height_px", - "longest_px", - "longest_um", - "alert" - ]) - - for r in results: - img_name = os.path.basename(r.path) - - if r.boxes is None or len(r.boxes) == 0: - continue - - xywh = r.boxes.xywh.cpu().numpy() # [x_c, y_c, w, h] in px - confs = r.boxes.conf.cpu().numpy() if r.boxes.conf is not None else np.array([]) - clses = r.boxes.cls.cpu().numpy().astype(int) if r.boxes.cls is not None else np.array([], dtype=int) - - for i, (x_c, y_c, bw, bh) in enumerate(xywh): - longest_px = float(max(bw, bh)) - longest_um = longest_px * px_to_um - - cls_id = int(clses[i]) if clses.size > i else -1 - cls_name = model.names.get(cls_id, str(cls_id)) if hasattr(model, "names") else str(cls_id) - conf = float(confs[i]) if confs.size > i else float("nan") - - alert_flag = "STOP" if longest_um > alert_um else "" - if alert_flag: - print(f"[STOP] {img_name} — det {i+1}: {longest_um:.2f} μm > {alert_um} μm → Stop the beam") - - writer.writerow([ - img_name, - i + 1, - cls_id, - cls_name, - f"{conf:.4f}", - f"{x_c:.2f}", - f"{y_c:.2f}", - f"{bw:.2f}", - f"{bh:.2f}", - f"{longest_px:.2f}", - f"{longest_um:.2f}", - alert_flag - ]) - -print(f"\nCSV saved to: {csv_path}") -print("YOLO's annotated images are under runs/detect/predict*/") - -# Rename CSV after completion -final_csv_path = out_dir / "measurements_complete.csv" -csv_path.rename(final_csv_path) - -print(f"CSV renamed to: {final_csv_path}") - diff --git a/scripts/coyote_protector_beamtime_cds_to_sdf/run_inference.sh b/scripts/coyote_protector_beamtime_cds_to_sdf/run_inference.sh deleted file mode 100755 index b7f0f38..0000000 --- a/scripts/coyote_protector_beamtime_cds_to_sdf/run_inference.sh +++ /dev/null @@ -1,29 +0,0 @@ -#!/bin/bash -#SBATCH --partition=turing -#SBATCH --account=lcls:prjlumine22 -#SBATCH --job-name=yolo_detection -#SBATCH --nodes=1 -#SBATCH --ntasks=1 -#SBATCH --cpus-per-task=4 # CPU mostly for I/O + Python -#SBATCH --gpus=1 # request 1 GPU -#SBATCH --time=04:00:00 -#SBATCH --output=logs_detection/detection_turing_%j.out -#SBATCH --error=logs_detection/detection_turing_%j.err - -# ==== Thread / OMP hygiene ==== -export OMP_PROC_BIND=close -export OMP_PLACES=cores -export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK}" -export MKL_NUM_THREADS="${SLURM_CPUS_PER_TASK}" - -# ==== CUDA visibility for multi-GPU ==== -export CUDA_DEVICE_ORDER=PCI_BUS_ID -export CUDA_VISIBLE_DEVICES=0,1,2,3 - -echo "[SLURM] Allocated GPUs: $CUDA_VISIBLE_DEVICES" - -echo "[SLURM] Running inference_coyote.py on $(hostname)" - -# /sdf/home/p/pmonteil/miniconda3/envs/coyote/bin/python ../inference_coyote.py - -/sdf/data/lcls/ds/prj/prjlumine22/results/coyote_protector/miniconda3_coyote/envs/env_coyote/bin/python ../inference_coyote.py diff --git a/scripts/coyote_protector_beamtime_cds_to_sdf/weight_yolov11n_150epochs.pt b/scripts/coyote_protector_beamtime_cds_to_sdf/weight_yolov11n_150epochs.pt deleted file mode 100755 index 20bf6d5..0000000 Binary files a/scripts/coyote_protector_beamtime_cds_to_sdf/weight_yolov11n_150epochs.pt and /dev/null differ diff --git a/scripts/coyote_protector_xtc_gui_ready_test/README.md b/scripts/coyote_protector_xtc_gui_ready_test/README.md deleted file mode 100644 index 6868caa..0000000 --- a/scripts/coyote_protector_xtc_gui_ready_test/README.md +++ /dev/null @@ -1,225 +0,0 @@ -## COYOTE PROTECTOR XTC GUI READY - -This folder contains a complete workflow for processing XTC data from LCLS experiments, extracting images, running YOLO-based crystal detection, and generating comprehensive CSV reports with trajectory information. - -## Key Highlights: -- Export raw XTC detector data to PNG images with normalized versions -- Fine-tuned YOLOv11 model for accurate crystal detection -- Pixel-to-micron size conversion and threshold alert system -- CSV merging to combine trajectory and crystal detection data -- Organized results in `results_csv/` folder within run directories - -## Overview of the Scripts - -This workflow consists of two main processing stages: - -### Stage 1: XTC Data Export -**Script:** `export_xtc_normalized_args.py` -- Extracts images from LCLS XTC files using psana -- Saves raw and normalized (contrast-enhanced) PNG images -- Records trajectory coordinates (XTRAJ, YTRAJ) for each event -- Outputs: `run_{run_number}_png/`, `run_{run_number}_png_norm/`, `event_data.csv` - -### Stage 2: Crystal Detection (YOLO Inference) -**Script:** `inference_coyote_xtc.py` -- Runs YOLOv11 inference on exported PNG images -- Detects crystals and computes size in pixels and microns -- Generates two CSV files: - - `measurements_complete.csv` - All detections - - `measurements_above_threshold_complete.csv` - Only detections exceeding size threshold -- Outputs: Both CSVs in `results_csv/` folder - -### Stage 3: Data Merging -**Script:** `merge_crystals_data.py` -- Merges event trajectory data with crystal detection results -- Creates one line per crystal with all relevant information -- Output: `merged_crystals.csv` in `results_csv/` folder with columns: image, xtraj, ytraj, and all crystal characteristics - -### Orchestration Script -**Script:** `run_export_infer_xtc.sh` -- Bash wrapper that runs both export and inference scripts sequentially -- Simplifies execution of the complete workflow - -## Setting up the Coyote Protector XTC Workflow - -### 1. Prerequisites -- psana environment (for XTC data access): `source /sdf/group/lcls/ds/ana/sw/conda2/manage/bin/psconda.sh` -- YOLO weights file (`.pt` model) -- LCLS experiment identifier and run number - -### 2. Configure `inference_coyote_xtc.py` -Edit the script to set detection parameters: -```python -weights_path = "/path/to/your/weights/best.pt" # Path to YOLO model weights -mag_factor = 5.56 # Optical magnification -px_size = 3.45 # Pixel size in microns -dowsamp_factor = 2 # Downsampling factor (if applied) -px_to_um = px_size * dowsamp_factor / mag_factor # Computed pixel-to-micron conversion -alert_um = 50.0 # Size threshold in microns -``` - -**Output Structure:** -``` -results_csv/ -├── measurements_complete.csv # All detections -└── measurements_above_threshold_complete.csv # Detections > alert_um threshold -``` - -### 4. Configure `merge_crystals_data.py` -No configuration needed — it automatically reads from the results_csv folder and merges the data. - - -## Running the Complete Workflow - -### Using sbatch with Environment Variables - -The scripts are designed to be submitted via SLURM's `sbatch` command with environment variables passed as key=value pairs. - -#### Syntax: -```bash -sbatch RUN_NUMBER= EXP_NUMBER= MAX_EVENTS= -``` - -**Parameter Descriptions:** -- `RUN_NUMBER`: LCLS run number to process (e.g., 61) -- `EXP_NUMBER`: Experiment identifier (e.g., mfx101346325, xpp) -- `MAX_EVENTS`: Maximum number of events to process (e.g., 80000) - -#### Examples: - -**Complete Workflow (Export + Inference):** -```bash -sbatch run_export_infer_xtc.sh RUN_NUMBER=61 EXP_NUMBER=mfx101346325 MAX_EVENTS=80000 -``` - -**Export Only:** -Parameters have to be changed into the script directly (no parsing) -```bash -sbatch run_export_xtc.sh -``` - -**Inference Only:** -Parameters have to be changed into the script directly (no parsing) -```bash -sbatch run_inference_xtc.sh RUN_NUMBER=61 -``` - -**Check Job Status:** -```bash -# View all your jobs -squeue -u $(whoami) - -# View specific job details -squeue -j - -# Check job logs -tail -f logs_export/export_turing_.out -``` - -### Running Scripts Interactively (Alternative) - -If you prefer not to use sbatch or need to run interactively: - -**Step 1: Export XTC Data** -```bash -# Must source psana environment first! -source /sdf/group/lcls/ds/ana/sw/conda2/manage/bin/psconda.sh - -python export_xtc_normalized_args.py - -# Example: -python export_xtc_normalized_args.py 61 mfx101346325 1 80000 -``` - -**Step 2: Run Crystal Detection** -```bash -python inference_coyote_xtc.py - -# Example (using normalized images): -python inference_coyote_xtc.py run_61/run_61_png_norm -``` - -**Step 3: Merge Results** -```bash -python merge_crystals_data.py - -# Example: -python merge_crystals_data.py 61 -``` - -## Output Files Description - -### From `export_xtc_normalized_args.py` -- `run_{run_number}/results_csv/event_data.csv` - - Columns: `event_id, png_file, xtraj, ytraj` - - One row per event/frame processed - -### From `inference_coyote_xtc.py` -- `results_csv/measurements_complete.csv` - - Columns: `image, det_idx, class_id, class_name, confidence, x_center_px, y_center_px, width_px, height_px, longest_px, longest_um, alert` - - All detected crystals - -- `results_csv/measurements_above_threshold_complete.csv` - - Same columns as above - - Only crystals with `longest_um > alert_um` - -### From `merge_crystals_data.py` -- `run_{run_number}/results_csv/merged_crystals.csv` - - Columns: `image, xtraj, ytraj, det_idx, class_id, class_name, confidence, x_center_px, y_center_px, width_px, height_px, longest_px, longest_um, alert` - - One row per crystal detection - - Includes trajectory coordinates matched by image name - -## Typical Workflow Example - -```bash -# Example: Process run 61 with 80,000 events -sbatch run_export_infer_xtc.sh RUN_NUMBER=61 EXP_NUMBER=mfx101346325MAX_EVENTS=80000 - -# Monitor job status -squeue -u $(whoami) - -# Check output (once job is running or complete) -tail -f logs_export/export_turing_.out - -# Once complete, verify results -ls -lh run_61/results_csv/ -# Should contain: -# event_data.csv -# measurements_complete.csv -# measurements_above_threshold_complete.csv -# merged_crystals.csv -``` - -## Result Summary - -After running the complete workflow, you will have: - -1. **Extracted Images:** Raw and normalized PNG files from XTC detector data -2. **Event Data:** CSV with trajectory coordinates for each event -3. **Crystal Detections:** Two CSVs with detection results (all and filtered by size threshold) -4. **Merged Dataset:** Single comprehensive CSV combining all information, ready for analysis - -All results are organized in the `run_{run_number}/results_csv/` directory for easy access and further analysis. - -## Troubleshooting - -### psana import error -Make sure to source the psana environment before running export script: -```bash -source /sdf/group/lcls/ds/ana/sw/conda2/manage/bin/psconda.sh -``` - -### YOLO weights not found -Verify the path in `inference_coyote_xtc.py`: -```python -weights_path = "/path/to/your/weights.pt" -``` - -### No images detected during inference -- Check that images were successfully saved in `run_{run_number}_png/` or `run_{run_number}_png_norm/` -- Verify the image directory path passed to inference script is correct -- Check image file format (should be `.png`) - -### CSV merge fails -- Ensure both `event_data.csv` and `measurements_complete.csv` exist in `run_{run_number}/results_csv/` -- Verify image filenames match between the two CSVs (event filenames should match YOLO output filenames) diff --git a/scripts/coyote_protector_xtc_gui_ready_test/weights_yolov11n_150epochs_merged_dataset.pt b/scripts/coyote_protector_xtc_gui_ready_test/weights_yolov11n_150epochs_merged_dataset.pt deleted file mode 100644 index 11a2198..0000000 Binary files a/scripts/coyote_protector_xtc_gui_ready_test/weights_yolov11n_150epochs_merged_dataset.pt and /dev/null differ diff --git a/scripts/production/inference_xtc_serial_pipeline/README.md b/scripts/production/inference_xtc_serial_pipeline/README.md new file mode 100644 index 0000000..d354c06 --- /dev/null +++ b/scripts/production/inference_xtc_serial_pipeline/README.md @@ -0,0 +1,485 @@ +# NOT RECOMMENDED : OBSOLETE VERSION KEPT ONLY FOR LEGACY PURPOSE + +# Coyote Protector Serial Workflow + +This folder contains the end-to-end serial workflow used to process LCLS runs, export detector PNGs, run YOLO inference, and merge trajectory + crystal detection results into final CSV files. + +The general workflow is the following: +- a script is launched from CDS to launch processing on SDF +- images and metadata are processed on SDF +- relevant CSV outputs can be copied back to CDS + +Unlike the parallel pipeline, this version does not split timestamps into chunks and does not submit worker jobs per part. + +## Quick setup on SDF and sync to CDS + +1) Create a folder on SDF in your experiment results area. + +This folder hosts the code and output of processing (images, logs, and CSV files). + +- SSH to psana and run: +```bash +mkdir -p "/sdf/data/lcls/ds/mfx/${EXP_NUMBER}/results/" +``` + +2) Clone the full repo, then navigate to the serial production folder: + +```bash +git clone git@github.com:lcls-mlcv/coyote_protector.git +cd /sdf/data/lcls/ds/mfx/${EXP_NUMBER}/results/coyote_protector/scripts/production/inference_xtc_serial_pipeline/ +``` + +3) If launching from CDS, update `routine_detection_v3.sh` before syncing: + +- `DEST_HOST="psana.sdf.slac.stanford.edu"` +- `SDF_BASE` should point to something similar to `/sdf/data/lcls/ds/mfx/${EXP_NUMBER}/results/coyote_protector/scripts/production/inference_xtc_serial_pipeline/` +- `RESULTS_BACK_BASE` should point to your CDS destination folder + +These are user-specific and should be changed for production. + +4) Rsync that SDF folder to CDS (optional, if orchestrating from CDS): + +```bash +rsync -av "${USER_NAME}@${DEST_HOST}:/sdf/data/lcls/ds/mfx/${EXP_NUMBER}/results/coyote_protector/scripts/production/inference_xtc_serial_pipeline/" ./ +``` + +5) (Optional) Passwordless SSH setup (CDS -> SDF) + +Some steps in this pipeline connect multiple times to SDF (`psana.sdf.slac.stanford.edu`) using `ssh`. +If SSH keys are not configured, you will be prompted for your password repeatedly. + +To avoid repeated authentication, configure passwordless SSH from CDS to SDF before production runs. + +- Generate an SSH key on CDS: + +```bash +ssh-keygen -t ed25519 +``` + +- Authorize the public key on SDF: + +```bash +ssh-copy-id @psana.sdf.slac.stanford.edu +``` + +- Test from CDS: + +```bash +ssh @psana.sdf.slac.stanford.edu +``` + +6) Run from CDS: + +```bash +source /cds/group/pcds/pyps/conda/pcds_conda +python bash_launcher.py --user= --run_number= --exp_number= --max_events= --camera_name= > quick_run.log & +``` + +## Required configuration before production + +Review these hardcoded values before production runs. + +### 1) SLURM resources/accounts + +- `run_export_infer_xtc.sh` + - `#SBATCH --account=lcls:prjlumine22` + - `#SBATCH --partition=turing` + - `#SBATCH --gpus=1` + +- `run_export_xtc.sh` + - `#SBATCH --account=lcls:prjlumine22` + - `#SBATCH --partition=turing` + +- `run_inference_xtc.sh` + - `#SBATCH --account=lcls:prjlumine22` + - `#SBATCH --partition=turing` + - `#SBATCH --gpus=1` + +You must have access to these resources and projects. + +### 2) Python environments + +- psana env sourced by export scripts: + - `/sdf/group/lcls/ds/ana/sw/conda2/manage/bin/psconda.sh` + +- YOLO Python binary used by inference/merge: + - `YOLO_PYTHON=/sdf/data/lcls/ds/prj/prjlumine22/results/coyote_protector/miniconda3_coyote/envs/env_coyote/bin/python` + +Note : you must have the access to prjlumine22 to use this environment, will be build in lcls-tools soon. + +### 3) YOLO model + calibration + +In `inference_coyote_xtc.py` on SDF, verify: +- `weights_path` +- `mag_factor` # magnification factor from the microscope +- `px_size` # real pixel size (in micro meters) +- `downsamp_factor` # camera downsampling factor (usually 2) +- `alert_um` # threshold above which the crystal is considerd to be dangerous for the dector (set artifically low for development purpose) + +Note : the last 4 values are is setup dependant, please use the operators for the values. + +### 4) Orchestrator paths (CDS ↔ SDF) + +In `routine_detection_v3.sh`, verify: +- `SDF_BASE` +- `RESULTS_BACK_BASE` +- `DEST_HOST` + +## Run options + +## A) Launch from CDS with Python wrapper + +```bash +python bash_launcher.py \ + --user= \ + --run_number=61 \ + --exp_number=mfx101346325 \ + --save_normalized=1 \ + --max_events=80000 \ + --use_normalized=1 \ + --camera_name=inline_alvium +``` + +Dry-run (prints command only): + +```bash +python bash_launcher.py --dry_run +``` + +This calls `routine_detection_v3.sh`, which submits `run_export_infer_xtc.sh` on SDF and copies back `run_/results_csv/`. + +## B) Recommended: full serial workflow on SDF + +```bash +sbatch run_export_infer_xtc.sh \ + RUN_NUMBER=61 \ + EXP_NUMBER=mfx101346325 \ + SAVE_NORMALIZED=1 \ + MAX_EVENTS=80000 \ + USE_NORMALIZED=1 \ + CAMERA_NAME=inline_alvium +``` + +What it does: +- creates `run_/` +- `cd` into `run_/` +- exports images + `results_csv/event_data.csv` +- runs inference and writes `results_csv/measurements_*.csv` +- merges to `results_csv/merged_crystals.csv` + +## C) Export only + +```bash +sbatch run_export_xtc.sh \ + RUN_NUMBER=61 \ + EXP_NUMBER=mfx101346325 \ + SAVE_NORMALIZED=1 \ + MAX_EVENTS=80000 \ + CAMERA_NAME=inline_alvium +``` + +Note: this script does not create `run_/` before export. It writes image folders (`run__png*`) and `results_csv/event_data.csv` relative to the directory where the job runs. + +## D) Inference + merge only (after export) + +```bash +sbatch run_inference_xtc.sh RUN_NUMBER=61 USE_NORMALIZED=1 +``` + +Important behavior: +- `run_inference_xtc.sh` expects images under `run_/run__png_norm` (or `_png`) +- `inference_coyote_xtc.py` and `merge_crystals_data.py` write/read `results_csv` in the current working directory + +So this script is only safe if your current directory and exported image layout are consistent with those assumptions. + + +## Monitoring and logs + +Check jobs: + +```bash +squeue -u $(whoami) +``` + +Master sequential logs: + +```bash +tail -f logs_export/export_infer_turing_.out +tail -f logs_export/export_infer_turing_.err +``` + +Export-only logs: + +```bash +tail -f logs_export/export_turing_.out +tail -f logs_export/export_turing_.err +``` + +Inference-only logs: + +```bash +tail -f logs_export/inference_turing_.out +tail -f logs_export/inference_turing_.err +``` + +## Output layout + +### Full sequential run (`run_export_infer_xtc.sh`) + +Inside `run_/`: + +- `run__png/` (raw write currently disabled in export script) +- `run__png_norm/` (when `SAVE_NORMALIZED=1`) +- `results_csv/` + - `event_data.csv` + - `measurements_complete.csv` + - `measurements_above_threshold.csv` + - `merged_crystals.csv` + +### Orchestrated run from CDS (`routine_detection_v3.sh`) + +On CDS destination: +- `run__results/` + - synchronized copy of SDF `run_/results_csv/` + +## Common issues + +- `psana` import fails + Source psana env before export scripts: + ```bash + source /sdf/group/lcls/ds/ana/sw/conda2/manage/bin/psconda.sh + ``` + +- YOLO inference fails to start + Validate `YOLO_PYTHON` in shell scripts and `weights_path` in `inference_coyote_xtc.py`. + +- Missing images + `export_xtc_normalized_args.py` processes one event every 3 events (`eid % 3 == 0`). Also, raw PNG write is commented out, so typically only normalized images are written. + +- Merge file-not-found + `merge_crystals_data.py` reads from `results_csv/event_data.csv` and `results_csv/measurements_above_threshold.csv` in the current working directory. + +- Routine output lists a wrong file name + `routine_detection_v3.sh` prints `measurements_above_threshold_complete.csv`, but inference writes `measurements_above_threshold.csv`. + +## Minimal reproducible command + +```bash +sbatch run_export_infer_xtc.sh RUN_NUMBER=61 EXP_NUMBER=mfx101346325 SAVE_NORMALIZED=1 MAX_EVENTS=100 USE_NORMALIZED=1 CAMERA_NAME=inline_alvium +``` + +When complete: + +```bash +ls -lh run_61/results_csv/ +``` + +## What this pipeline does in detail + +1. Export XTC detector frames and trajectory metadata (`export_xtc_normalized_args.py`) +2. Run YOLO on exported image folder (`inference_coyote_xtc.py`) +3. Merge detections with XTRAJ/YTRAJ by image filename (`merge_crystals_data.py`) +4. Optional CDS orchestrator submits on SDF and pulls CSVs back (`routine_detection_v3.sh`) + +## Folder scripts + +- `bash_launcher.py` + Python launcher for `routine_detection_v3.sh` with CLI args. + +- `routine_detection_v3.sh` + Runs from CDS, SSHs to SDF, submits sequential SLURM job, waits for merged CSV, and rsyncs results back. + +- `run_export_infer_xtc.sh` + Single sequential SLURM job: export -> inference -> merge, inside `run_/`. + +- `run_export_xtc.sh` + Export-only SLURM job. + +- `run_inference_xtc.sh` + Inference + merge SLURM job. + +- `export_xtc_normalized_args.py` + Exports normalized PNGs and `event_data.csv` from XTC. + +- `inference_coyote_xtc.py` + Runs YOLO and writes `measurements_complete.csv` and `measurements_above_threshold.csv`. + +- `merge_crystals_data.py` + Merges trajectory data and above-threshold detections into `merged_crystals.csv`. + +## CSV schema summary + +### `event_data.csv` + +Columns: +- `event_id` +- `png_file` +- `xtraj` +- `ytraj` + +### `measurements_complete.csv` and `measurements_above_threshold.csv` + +Columns: +- `image` +- `det_idx` +- `class_id` +- `class_name` +- `confidence` +- `x_center_px` +- `y_center_px` +- `width_px` +- `height_px` +- `longest_px` +- `longest_um` +- `alert` + +### `merged_crystals.csv` + +Columns: +- `image` +- `xtraj` +- `ytraj` +- `det_idx` +- `class_id` +- `class_name` +- `confidence` +- `x_center_px` +- `y_center_px` +- `width_px` +- `height_px` +- `longest_px` +- `longest_um` +- `alert` + +Note: current merge uses `measurements_above_threshold.csv` (not all detections). + +## Precise inputs/outputs by algorithm (reference) + +This section is the contract for each script: exact runtime inputs, what it reads, and what it writes. + +### 1) `export_xtc_normalized_args.py` + +CLI input: + +```bash +python export_xtc_normalized_args.py [run_number] [exp_number] [save_normalized] [max_events] [camera_name] +``` + +Defaults in script: +- `run_number=61` +- `exp="mfx "` (must usually be overridden) +- `save_normalized=True` +- `max_events=10000` +- `camera_name="inline_alvium"` + +Reads: +- psana datasource from `exp/run/max_events` +- detectors: ``, `XTRAJ`, `YTRAJ` + +Writes: +- `run__png/event_.png` (raw write currently commented out) +- `run__png_norm/event_.png` (if `save_normalized` true) +- `results_csv/event_data.csv` + +Specific behavior: +- processes only events where `event_id % 3 == 0` +- stores CSV `png_file` as raw filename pattern `event_.png` + +### 2) `inference_coyote_xtc.py` + +CLI input: + +```bash +python inference_coyote_xtc.py +``` + +Reads: +- image files from `` +- YOLO weights from hardcoded `weights_path` + +Writes: +- `results_csv/measurements_complete.csv` +- `results_csv/measurements_above_threshold.csv` + +Thresholding behavior: +- computes `longest_um = max(width_px, height_px) * px_to_um` +- flags row with `alert="STOP"` when `longest_um > alert_um` +- writes all rows to `measurements_complete.csv` and only flagged rows to `measurements_above_threshold.csv` + +### 3) `merge_crystals_data.py` + +CLI input: + +```bash +python merge_crystals_data.py [run_number] +``` + +Reads: +- `results_csv/event_data.csv` +- `results_csv/measurements_above_threshold.csv` + +Writes: +- `results_csv/merged_crystals.csv` + +Merge logic: +- renames `event_data.csv` column `png_file` -> `image` +- left-merges measurements with `xtraj,ytraj` on `image` + +### 4) `run_export_infer_xtc.sh` + +Runtime key=value inputs: +- `RUN_NUMBER` +- `EXP_NUMBER` +- `SAVE_NORMALIZED` +- `MAX_EVENTS` +- `USE_NORMALIZED` +- `CAMERA_NAME` + +Behavior: +- creates and enters `run_/` +- runs export (`../export_xtc_normalized_args.py`) +- runs inference (`../inference_coyote_xtc.py`) +- runs merge (`../merge_crystals_data.py`) + +Writes inside `run_/`: +- `run__png/` +- `run__png_norm/` +- `results_csv/event_data.csv` +- `results_csv/measurements_complete.csv` +- `results_csv/measurements_above_threshold.csv` +- `results_csv/merged_crystals.csv` + +### 5) `run_export_xtc.sh` + +Runtime key=value inputs: +- `RUN_NUMBER` +- `EXP_NUMBER` +- `SAVE_NORMALIZED` +- `MAX_EVENTS` +- `CAMERA_NAME` + +Behavior: +- runs `./export_xtc_normalized_args.py` in current folder (no automatic `run_/` `cd`) + +Writes in current folder: +- `run__png/` +- `run__png_norm/` +- `results_csv/event_data.csv` + +### 6) `run_inference_xtc.sh` + +Runtime key=value inputs: +- `RUN_NUMBER` +- `USE_NORMALIZED` + +Reads image folder: +- `run_/run__png_norm` or `run_/run__png` + +Runs: +- `./inference_coyote_xtc.py ` +- `./merge_crystals_data.py ` + +Writes in current folder: +- `results_csv/measurements_complete.csv` +- `results_csv/measurements_above_threshold.csv` +- `results_csv/merged_crystals.csv` diff --git a/scripts/production/inference_xtc_serial_pipeline/bash_launcher.py b/scripts/production/inference_xtc_serial_pipeline/bash_launcher.py new file mode 100644 index 0000000..b85c72f --- /dev/null +++ b/scripts/production/inference_xtc_serial_pipeline/bash_launcher.py @@ -0,0 +1,141 @@ +#!/usr/bin/env python3 +""" +bash_launcher.py +Launches routine_detection_v3.sh with specified arguments. +Can be run from CDS to orchestrate the complete XTC workflow. + +USAGE: + python bash_launcher.py [OPTIONS] + python bash_launcher.py --help + +EXAMPLES: + # Run with defaults + python bash_launcher.py + + # Run with custom parameters + python bash_launcher.py --run_number=61 --exp_number=mfx101346325 --max_events=80000 + + # Run with specific user + python bash_launcher.py --user=pmonteil --run_number=100 +""" + +import argparse +import subprocess +import sys +from pathlib import Path + + +def eprint(msg): + """Write message to stderr.""" + sys.stderr.write(str(msg) + "\n") + + +def main(): + parser = argparse.ArgumentParser( + description="Launch coyote XTC detection routine via bash", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=__doc__ + ) + + parser.add_argument( + "--user", + default="pmonteil", + help="Username for SSH to SDF (default: pmonteil)" + ) + parser.add_argument( + "--run_number", + type=int, + default=61, + help="LCLS run number to process (default: 61)" + ) + parser.add_argument( + "--exp_number", + default="mfx101346325", + help="Experiment identifier (default: mfx101346325)" + ) + parser.add_argument( + "--save_normalized", + type=int, + choices=[0, 1], + default=1, + help="Save normalized images (1=yes, 0=no, default: 1)" + ) + parser.add_argument( + "--max_events", + type=int, + default=10000, + help="Maximum events to process (default: 10000)" + ) + parser.add_argument( + "--use_normalized", + type=int, + choices=[0, 1], + default=1, + help="Use normalized images for inference (1=yes, 0=no, default: 1)" + ) + parser.add_argument( + "--camera_name", + default="inline_alvium", + help="Psana detector name for the inline camera (default: inline_alvium)" + ) + parser.add_argument( + "--dry_run", + action="store_true", + help="Print command without executing" + ) + + args = parser.parse_args() + + # Get the script path (should be in same directory) + script_path = Path(__file__).parent / "routine_detection_v3.sh" + + if not script_path.exists(): + eprint("[ERROR] Script not found: {}".format(script_path)) + sys.exit(1) + + # Build command + cmd = [ + "bash", + str(script_path), + "--user={}".format(args.user), + "RUN_NUMBER={}".format(args.run_number), + "EXP_NUMBER={}".format(args.exp_number), + "SAVE_NORMALIZED={}".format(args.save_normalized), + "MAX_EVENTS={}".format(args.max_events), + "USE_NORMALIZED={}".format(args.use_normalized), + "CAMERA_NAME={}".format(args.camera_name), + ] + + print("[INFO] ============================================") + print("[INFO] COYOTE XTC DETECTION LAUNCHER") + print("[INFO] ============================================") + print("[INFO] User: {}".format(args.user)) + print("[INFO] Run Number: {}".format(args.run_number)) + print("[INFO] Experiment: {}".format(args.exp_number)) + print("[INFO] Save Normalized: {}".format(args.save_normalized)) + print("[INFO] Max Events: {}".format(args.max_events)) + print("[INFO] Use Normalized: {}".format(args.use_normalized)) + print("[INFO] Camera Name: {}".format(args.camera_name)) + print("[INFO] ============================================") + print() + + print("[INFO] Command: {}".format(" ".join(cmd))) + print() + + if args.dry_run: + print("[INFO] DRY RUN MODE - Command not executed") + return 0 + + try: + result = subprocess.run(cmd, check=False) + return result.returncode + except KeyboardInterrupt: + eprint("\n[INFO] Interrupted by user") + return 130 + except Exception as e: + eprint("[ERROR] Failed to execute command: {}".format(e)) + return 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/coyote_protector_xtc_gui_ready_test/export_xtc_normalized_args.py b/scripts/production/inference_xtc_serial_pipeline/export_xtc_normalized_args.py similarity index 72% rename from scripts/coyote_protector_xtc_gui_ready_test/export_xtc_normalized_args.py rename to scripts/production/inference_xtc_serial_pipeline/export_xtc_normalized_args.py index 7d91c7a..0e66a41 100644 --- a/scripts/coyote_protector_xtc_gui_ready_test/export_xtc_normalized_args.py +++ b/scripts/production/inference_xtc_serial_pipeline/export_xtc_normalized_args.py @@ -27,6 +27,7 @@ def parse_bool(s: str) -> bool: DEFAULT_EXP = "mfx " DEFAULT_SAVE_NORM = True DEFAULT_MAX_EVENTS = 10000 +DEFAULT_CAMERA = "inline_alvium" argc = len(sys.argv) @@ -35,11 +36,13 @@ def parse_bool(s: str) -> bool: exp = DEFAULT_EXP SAVE_NORMALIZED = DEFAULT_SAVE_NORM max_events = DEFAULT_MAX_EVENTS -elif argc in (2, 3, 4, 5): + camera_name = DEFAULT_CAMERA +elif argc in (2, 3, 4, 5, 6): run_number = int(sys.argv[1]) exp = sys.argv[2] if argc >= 3 else DEFAULT_EXP SAVE_NORMALIZED = parse_bool(sys.argv[3]) if argc >= 4 else DEFAULT_SAVE_NORM max_events = int(sys.argv[4]) if argc >= 5 else DEFAULT_MAX_EVENTS + camera_name = sys.argv[5] if argc >= 6 else DEFAULT_CAMERA else: print( "Usage:\n" @@ -48,21 +51,21 @@ def parse_bool(s: str) -> bool: " python export_xtc_normalized.py \n" " python export_xtc_normalized.py \n" " python export_xtc_normalized.py \n" + " python export_xtc_normalized.py \n" ) sys.exit(1) -print(f"[CONFIG] exp={exp} run={run_number} save_normalized={SAVE_NORMALIZED} max_events={max_events}") +print(f"[CONFIG] exp={exp} run={run_number} save_normalized={SAVE_NORMALIZED} max_events={max_events} camera_name={camera_name}") # ------------------------- # psana setup # ------------------------- #print(run_number, exp, max_events, SAVE_NORMALIZED) ds = DataSource(exp=exp, run=[run_number], max_events=max_events) -#ds = DataSource(exp='mfx101346325', run=59, max_events=100) myrun = next(ds.runs()) -cam = myrun.Detector("inline_alvium") +cam = myrun.Detector(camera_name) xtraj = myrun.Detector("XTRAJ") ytraj = myrun.Detector("YTRAJ") @@ -92,27 +95,29 @@ def parse_bool(s: str) -> bool: eid = 0 for evt in myrun.events(): - img = cam.raw.value(evt) - if img is None: - continue + if eid % 3 == 0 : + img = cam.raw.value(evt) + if img is None: + eid += 1 + continue - x = xtraj(evt) - y = ytraj(evt) + x = xtraj(evt) + y = ytraj(evt) - # Save raw - fname = out_dir / f"event_{eid:06d}.png" - cv2.imwrite(str(fname), img) + # Save raw + fname = out_dir / f"event_{eid:06d}.png" + #cv2.imwrite(str(fname), img) - # Save normalized - if SAVE_NORMALIZED: - img8 = cv2.normalize(img, None, 0, 255, cv2.NORM_MINMAX).astype(np.uint8) - fname_norm = norm_dir / f"event_{eid:06d}.png" - cv2.imwrite(str(fname_norm), img8) + # Save normalized + if SAVE_NORMALIZED: + img8 = cv2.normalize(img, None, 0, 255, cv2.NORM_MINMAX).astype(np.uint8) + fname_norm = norm_dir / f"event_{eid:06d}.png" + cv2.imwrite(str(fname_norm), img8) - # CSV entry (same as your second script) - writer.writerow([eid, fname.name, x[eid], y[eid]]) + # CSV entry (same as your second script) + writer.writerow([eid, fname.name, x[eid], y[eid]]) - print(f"[EVENT {eid}] Saved {fname.name} | X={x[eid]}, Y={y[eid]}") + print(f"[EVENT {eid}] Saved {fname.name} | X={x[eid]}, Y={y[eid]}") eid += 1 diff --git a/scripts/coyote_protector_xtc_gui_ready_test/inference_coyote_xtc.py b/scripts/production/inference_xtc_serial_pipeline/inference_coyote_xtc.py similarity index 58% rename from scripts/coyote_protector_xtc_gui_ready_test/inference_coyote_xtc.py rename to scripts/production/inference_xtc_serial_pipeline/inference_coyote_xtc.py index 1972353..8d31f9b 100644 --- a/scripts/coyote_protector_xtc_gui_ready_test/inference_coyote_xtc.py +++ b/scripts/production/inference_xtc_serial_pipeline/inference_coyote_xtc.py @@ -1,18 +1,16 @@ - -## same as test_xtc/inference_coyote.py but with argument parsing +#!/usr/bin/env python3 """ -YOLOv8 Inference Script → CSV (sizes) +YOLOv8 Inference Script → CSV (sizes) + above-threshold CSV ------------------------------------------------------------ - Runs YOLO inference on a folder of images - Computes longest side per detection in px and μm - Flags detections larger than a given μm threshold +- Writes: + 1) results_csv/measurements_complete.csv (ALL detections) + 2) results_csv/measurements_above_threshold.csv (ONLY detections above threshold) USAGE: python inference_coyote_xtc.py - -EDIT THESE BEFORE RUNNING: - - weights_path: Path to your trained YOLO weights (.pt) - - px_to_um: Pixel-to-micron conversion factor for your setup """ import os @@ -37,27 +35,22 @@ # Paths / parameters # ------------------------- weights_path = ( - "/sdf/home/p/pmonteil/coyote_protector_test_PL_labeling_tries_v3_run61_mfx101346325_200_random/scripts/runs/detect/train_150epochs_v11_merged/weights/best.pt" + "/sdf/data/lcls/ds/prj/prjlumine22/results/pmonteil/coyote/latest_weights/weights_yolov11n_150epochs_merged_dataset.pt" ) -''' -weights_path = ( - "/sdf/home/p/pmonteil/prjlumine22/results/pmonteil/" - "coyote_beamtime_19jan/weight_yolov11n_150epochs.pt" -) -''' - mag_factor = 5.56 # optical magnification px_size = 3.45 # pixel size (μm) -dowsamp_factor = 2 # if images were downsampled before inference -px_to_um = px_size*dowsamp_factor / mag_factor -alert_um = 50.0 #threshold, in μm, above which to flag detections +dowsamp_factor = 2 # if images were downsampled before inference +px_to_um = px_size * dowsamp_factor / mag_factor +alert_um = 50.0 # threshold, in μm, above which to flag detections # Output CSV directory -#out_dir = Path("runs/size_measurements") out_dir = Path("results_csv") out_dir.mkdir(parents=True, exist_ok=True) -csv_path = out_dir / "measurements.csv" + +csv_all_tmp = out_dir / "measurements.csv" +csv_all_final = out_dir / "measurements_complete.csv" +csv_above = out_dir / "measurements_above_threshold.csv" # ------------------------- # Load model @@ -74,24 +67,32 @@ ) # ------------------------- -# Write CSV +# CSV header # ------------------------- -with open(csv_path, "w", newline="") as f: - writer = csv.writer(f) - writer.writerow([ - "image", - "det_idx", - "class_id", - "class_name", - "confidence", - "x_center_px", - "y_center_px", - "width_px", - "height_px", - "longest_px", - "longest_um", - "alert" - ]) +header = [ + "image", + "det_idx", + "class_id", + "class_name", + "confidence", + "x_center_px", + "y_center_px", + "width_px", + "height_px", + "longest_px", + "longest_um", + "alert" +] + +# ------------------------- +# Write BOTH CSVs +# ------------------------- +with open(csv_all_tmp, "w", newline="") as f_all, open(csv_above, "w", newline="") as f_above: + w_all = csv.writer(f_all) + w_above = csv.writer(f_above) + + w_all.writerow(header) + w_above.writerow(header) for r in results: img_name = os.path.basename(r.path) @@ -111,14 +112,16 @@ cls_name = model.names.get(cls_id, str(cls_id)) conf = float(confs[i]) if confs.size > i else float("nan") - alert_flag = "STOP" if longest_um > alert_um else "" - if alert_flag: + is_above = longest_um > alert_um + alert_flag = "STOP" if is_above else "" + + if is_above: print( f"[STOP] {img_name} — det {i+1}: " f"{longest_um:.2f} μm > {alert_um} μm → Stop the beam" ) - writer.writerow([ + row = [ img_name, i + 1, cls_id, @@ -131,12 +134,21 @@ f"{longest_px:.2f}", f"{longest_um:.2f}", alert_flag - ]) + ] + + # Write to "all detections" + w_all.writerow(row) + + # Write to "above threshold only" + if is_above: + w_above.writerow(row) -print(f"\nCSV saved to: {csv_path}") +print(f"\nCSV (all) saved to: {csv_all_tmp}") +print(f"CSV (above threshold only) saved to: {csv_above}") -# Rename CSV after completion -final_csv_path = out_dir / "measurements_complete.csv" -csv_path.rename(final_csv_path) +# Rename "all" CSV after completion +if csv_all_final.exists(): + csv_all_final.unlink() +csv_all_tmp.rename(csv_all_final) -print(f"CSV renamed to: {final_csv_path}") +print(f"CSV (all) renamed to: {csv_all_final}") diff --git a/scripts/coyote_protector_xtc_gui_ready_test/merge_crystals_data.py b/scripts/production/inference_xtc_serial_pipeline/merge_crystals_data.py similarity index 97% rename from scripts/coyote_protector_xtc_gui_ready_test/merge_crystals_data.py rename to scripts/production/inference_xtc_serial_pipeline/merge_crystals_data.py index 0bf6cea..342450c 100644 --- a/scripts/coyote_protector_xtc_gui_ready_test/merge_crystals_data.py +++ b/scripts/production/inference_xtc_serial_pipeline/merge_crystals_data.py @@ -31,7 +31,7 @@ results_csv_dir = Path(f"run_{run_number}/results_csv") results_csv_dir = Path("results_csv") event_data_csv = results_csv_dir / "event_data.csv" -measurements_csv = results_csv_dir / "measurements_complete.csv" +measurements_csv = results_csv_dir / "measurements_above_threshold.csv" # Check if files exist if not event_data_csv.exists(): diff --git a/scripts/production/inference_xtc_serial_pipeline/routine_detection_v3.sh b/scripts/production/inference_xtc_serial_pipeline/routine_detection_v3.sh new file mode 100644 index 0000000..7ae78cf --- /dev/null +++ b/scripts/production/inference_xtc_serial_pipeline/routine_detection_v3.sh @@ -0,0 +1,138 @@ +#!/bin/bash +# +# routine_detection_v3.sh +# Orchestrates the XTC export and inference workflow +# Runs on CDS and launches export_infer_xtc.sh on SDF via SSH +# + +# ------- DEFAULT INPUTS --------- +USER="pmonteil" +RUN_NUMBER=61 +EXP_NUMBER="mfx101346325" +SAVE_NORMALIZED=1 +MAX_EVENTS=10000 +USE_NORMALIZED=1 +CAMERA_NAME="inline_alvium" +# -------------------------------- + +DEST_HOST="psana.sdf.slac.stanford.edu" +SDF_BASE="/sdf/data/lcls/ds/mfx/${EXP_NUMBER}/results/coyote_protector/scripts/production/inference_xtc_parallel_pipeline" # <-- UPDATE THIS PATH TO YOUR SDF WORKING DIRECTORY +RESULTS_BACK_BASE="/cds/data/iocDatas/..../..../" # <-- UPDATE THIS PATH TO YOUR LOCAL RESULTS DIRECTORY + +# ------- PARSE key=value arguments --------- +for arg in "$@"; do + case $arg in + --user=*) + USER="${arg#*=}" + ;; + RUN_NUMBER=*) + RUN_NUMBER="${arg#*=}" + ;; + EXP_NUMBER=*) + EXP_NUMBER="${arg#*=}" + ;; + SAVE_NORMALIZED=*) + SAVE_NORMALIZED="${arg#*=}" + ;; + MAX_EVENTS=*) + MAX_EVENTS="${arg#*=}" + ;; + USE_NORMALIZED=*) + USE_NORMALIZED="${arg#*=}" + ;; + CAMERA_NAME=*) + CAMERA_NAME="${arg#*=}" + ;; + *) + echo "[WARN] Unknown argument: $arg" + ;; + esac +done + +echo "[INFO] ==============================================" +echo "[INFO] COYOTE PROTECTOR XTC DETECTION ROUTINE v3" +echo "[INFO] ==============================================" +echo "[INFO] User: ${USER}" +echo "[INFO] SDF Host: ${DEST_HOST}" +echo "[INFO] SDF Base: ${SDF_BASE}" +echo "[INFO] Run Number: ${RUN_NUMBER}" +echo "[INFO] Experiment: ${EXP_NUMBER}" +echo "[INFO] Save Normalized: ${SAVE_NORMALIZED}" +echo "[INFO] Max Events: ${MAX_EVENTS}" +echo "[INFO] Use Normalized for Inference: ${USE_NORMALIZED}" +echo "[INFO] Camera Name: ${CAMERA_NAME}" +echo "[INFO] Results back to: ${RESULTS_BACK_BASE}" +echo + +# ==================================== +# STEP 1: Launch export_infer_xtc.sh on SDF via SSH +# ==================================== +echo "[STEP 1/3] Launching export_infer_xtc.sh on SDF..." + +JOB_ID="$(ssh "${USER}@${DEST_HOST}" "bash -lc ' + set -euo pipefail + cd \"${SDF_BASE}\" + jid=\$(sbatch --parsable run_export_infer_xtc.sh RUN_NUMBER=${RUN_NUMBER} EXP_NUMBER=${EXP_NUMBER} SAVE_NORMALIZED=${SAVE_NORMALIZED} MAX_EVENTS=${MAX_EVENTS} USE_NORMALIZED=${USE_NORMALIZED} CAMERA_NAME=${CAMERA_NAME}) + echo \$jid +'")" + +echo "[INFO] Submitted job on SDF: ${JOB_ID}" +echo + +# ==================================== +# STEP 2: Wait for results on SDF +# ==================================== +echo "[STEP 2/3] Waiting for merged results on SDF..." + +REMOTE_CSV="${SDF_BASE}/run_${RUN_NUMBER}/results_csv/merged_crystals.csv" + +max_attempts=360 # 3 hours with 30-second intervals +attempt=0 + +while [ $attempt -lt $max_attempts ]; do + if ssh "${USER}@${DEST_HOST}" "bash -lc 'test -f \"${REMOTE_CSV}\"'"; then + echo "[INFO] Merged CSV detected on SDF." + break + fi + attempt=$((attempt + 1)) + if [ $((attempt % 4)) -eq 0 ]; then + echo "[INFO] Waiting... (attempt $attempt/$max_attempts)" + fi + sleep 30 +done + +if [ $attempt -eq $max_attempts ]; then + echo "[ERROR] Timeout waiting for results after 3 hours" + exit 1 +fi + +echo + +# ==================================== +# STEP 3: Copy all results back to CDS +# ==================================== +echo "[STEP 3/3] Copying results back to CDS..." + +RESULTS_DIR="run_${RUN_NUMBER}_results" +LOCAL_DEST_DIR="${RESULTS_BACK_BASE}/${RESULTS_DIR}" +mkdir -p "${LOCAL_DEST_DIR}" + +echo "[INFO] Copying from: ${DEST_HOST}:${SDF_BASE}/run_${RUN_NUMBER}/results_csv/" +echo "[INFO] Copying to: ${LOCAL_DEST_DIR}/" + +rsync -avr \ + "${USER}@${DEST_HOST}:${SDF_BASE}/run_${RUN_NUMBER}/results_csv/" \ + "${LOCAL_DEST_DIR}/" + +echo +echo "[DONE] ==============================================" +echo "[INFO] Complete workflow finished successfully." +echo "[INFO] Results available at:" +echo "[INFO] ${LOCAL_DEST_DIR}/" +echo "[INFO] ==============================================" +echo "[INFO] Key files:" +echo "[INFO] - merged_crystals.csv (all data merged)" +echo "[INFO] - measurements_complete.csv (all detections)" +echo "[INFO] - measurements_above_threshold_complete.csv (above threshold only)" +echo "[INFO] - event_data.csv (trajectory data)" +echo "[DONE]" diff --git a/scripts/production/inference_xtc_serial_pipeline/run_export_infer_xtc.sh b/scripts/production/inference_xtc_serial_pipeline/run_export_infer_xtc.sh new file mode 100644 index 0000000..679f589 --- /dev/null +++ b/scripts/production/inference_xtc_serial_pipeline/run_export_infer_xtc.sh @@ -0,0 +1,147 @@ +#!/bin/bash +#SBATCH --partition=turing +#SBATCH --account=lcls:prjlumine22 +#SBATCH --job-name=export_infer_xtc +#SBATCH --nodes=1 +#SBATCH --ntasks=1 +#SBATCH --cpus-per-task=4 +#SBATCH --gpus=1 +#SBATCH --time=04:00:00 +#SBATCH --output=logs_export/export_infer_turing_%j.out +#SBATCH --error=logs_export/export_infer_turing_%j.err + +set -euo pipefail + +export OMP_PROC_BIND=close +export OMP_PLACES=cores +export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK}" +export MKL_NUM_THREADS="${SLURM_CPUS_PER_TASK}" +export CUDA_DEVICE_ORDER=PCI_BUS_ID + +echo "[SLURM] Host: $(hostname)" +echo "[SLURM] CUDA_VISIBLE_DEVICES: ${CUDA_VISIBLE_DEVICES:-"(not set)"}" +echo "[SLURM] Starting job..." + +PSCONDA_SH="/sdf/group/lcls/ds/ana/sw/conda2/manage/bin/psconda.sh" +YOLO_PYTHON="/sdf/data/lcls/ds/prj/prjlumine22/results/coyote_protector/miniconda3_coyote/envs/env_coyote/bin/python" + +# ------- DEFAULT INPUTS --------- +RUN_NUMBER=61 +EXP_NUMBER="mfx101346325" +SAVE_NORMALIZED=1 +MAX_EVENTS=10000 +USE_NORMALIZED=1 +CAMERA_NAME="inline_alvium" +# -------------------------------- + +# ------- PARSE key=value arguments --------- +for arg in "$@"; do + case $arg in + RUN_NUMBER=*) + RUN_NUMBER="${arg#*=}" + ;; + EXP_NUMBER=*) + EXP_NUMBER="${arg#*=}" + ;; + SAVE_NORMALIZED=*) + SAVE_NORMALIZED="${arg#*=}" + ;; + MAX_EVENTS=*) + MAX_EVENTS="${arg#*=}" + ;; + USE_NORMALIZED=*) + USE_NORMALIZED="${arg#*=}" + ;; + CAMERA_NAME=*) + CAMERA_NAME="${arg#*=}" + ;; + *) + echo "[WARN] Unknown argument: $arg" + ;; + esac +done + +echo "[ARGS] run=${RUN_NUMBER} exp=${EXP_NUMBER} save_norm=${SAVE_NORMALIZED} max_events=${MAX_EVENTS} use_normalized=${USE_NORMALIZED} camera=${CAMERA_NAME}" +echo + +# ==================================== +# NEW STEP 0: Create run folder and cd into it +# ==================================== +RUN_DIR="run_${RUN_NUMBER}" +mkdir -p "${RUN_DIR}" +cd "${RUN_DIR}" + +echo "[STEP 0/3] Created and entered: $(pwd)" +echo + +# ==================================== +# STEP 1: Export XTC Data +# ==================================== +echo "[STEP 1/3] Running export_xtc_normalized_args.py" +START_STEP1=$(date +%s.%N) +set +u +source "${PSCONDA_SH}" +set -u + +# run from parent directory (scripts live one level up) +python ../export_xtc_normalized_args.py "${RUN_NUMBER}" "${EXP_NUMBER}" "${SAVE_NORMALIZED}" "${MAX_EVENTS}" "${CAMERA_NAME}" +END_STEP1=$(date +%s.%N) +DURATION_STEP1=$(echo "$END_STEP1 - $START_STEP1" | bc -l) + +# Count the number of images processed +if [ "${SAVE_NORMALIZED}" = "1" ]; then + IMAGE_DIR="run_${RUN_NUMBER}_png_norm" +else + IMAGE_DIR="run_${RUN_NUMBER}_png" +fi +NUM_IMAGES=$(find "${IMAGE_DIR}" -name "*.png" | wc -l) + +echo "[STEP 1/3] Export completed. Processed ${NUM_IMAGES} images." +echo + +# ==================================== +# STEP 2: Run YOLO Inference +# ==================================== +echo "[STEP 2/3] Running inference_coyote_xtc.py" +START_STEP2=$(date +%s.%N) + +# Determine which image directory to use (now relative to run_${RUN_NUMBER}/) +if [ "${USE_NORMALIZED}" = "1" ]; then + IMAGE_DIR="run_${RUN_NUMBER}_png_norm" +else + IMAGE_DIR="run_${RUN_NUMBER}_png" +fi + +"${YOLO_PYTHON}" ../inference_coyote_xtc.py "${IMAGE_DIR}" +END_STEP2=$(date +%s.%N) +DURATION_STEP2=$(echo "$END_STEP2 - $START_STEP2" | bc -l) + +echo "[STEP 2/3] Inference completed." +echo + +# ==================================== +# STEP 3: Merge Results +# ==================================== +echo "[STEP 3/3] Running merge_crystals_data.py" +START_STEP3=$(date +%s.%N) +"${YOLO_PYTHON}" ../merge_crystals_data.py "${RUN_NUMBER}" +END_STEP3=$(date +%s.%N) +DURATION_STEP3=$(echo "$END_STEP3 - $START_STEP3" | bc -l) + +echo "[STEP 3/3] Merging completed." +echo + +echo "[SLURM] Complete workflow finished successfully." +echo "Results saved to: ${RUN_DIR}/results_csv/" +echo + +# Calculate mean times per image +MEAN_STEP1=$(echo "$DURATION_STEP1 / $NUM_IMAGES" | bc -l) +MEAN_STEP2=$(echo "$DURATION_STEP2 / $NUM_IMAGES" | bc -l) +MEAN_STEP3=$(echo "$DURATION_STEP3 / $NUM_IMAGES" | bc -l) + +echo "=== TIMING PROFILE ===" +echo "Total images processed: ${NUM_IMAGES}" +echo "Step 1 (Export XTC Data): ${DURATION_STEP1} seconds total, ${MEAN_STEP1} seconds per image" +echo "Step 2 (YOLO Inference): ${DURATION_STEP2} seconds total, ${MEAN_STEP2} seconds per image" +echo "Step 3 (Merge Results): ${DURATION_STEP3} seconds total, ${MEAN_STEP3} seconds per image" diff --git a/scripts/production/inference_xtc_serial_pipeline/run_export_xtc.sh b/scripts/production/inference_xtc_serial_pipeline/run_export_xtc.sh new file mode 100644 index 0000000..48e69c9 --- /dev/null +++ b/scripts/production/inference_xtc_serial_pipeline/run_export_xtc.sh @@ -0,0 +1,83 @@ +#!/bin/bash +#SBATCH --partition=turing +#SBATCH --account=lcls:prjlumine22 +#SBATCH --job-name=export_xtc +#SBATCH --nodes=1 +#SBATCH --ntasks=1 +#SBATCH --cpus-per-task=4 +#SBATCH --time=02:00:00 +#SBATCH --output=logs_export/export_turing_%j.out +#SBATCH --error=logs_export/export_turing_%j.err + +set -euo pipefail + +export OMP_PROC_BIND=close +export OMP_PLACES=cores +export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK}" +export MKL_NUM_THREADS="${SLURM_CPUS_PER_TASK}" + +echo "[SLURM] Host: $(hostname)" +echo "[SLURM] Starting job..." + +PSCONDA_SH="/sdf/group/lcls/ds/ana/sw/conda2/manage/bin/psconda.sh" + +# ------- DEFAULT INPUTS --------- +RUN_NUMBER=61 +EXP_NUMBER="mfx101346325" +SAVE_NORMALIZED=1 +MAX_EVENTS=10000 +CAMERA_NAME="inline_alvium" +# -------------------------------- + +# ------- PARSE key=value arguments --------- +for arg in "$@"; do + case $arg in + RUN_NUMBER=*) + RUN_NUMBER="${arg#*=}" + ;; + EXP_NUMBER=*) + EXP_NUMBER="${arg#*=}" + ;; + SAVE_NORMALIZED=*) + SAVE_NORMALIZED="${arg#*=}" + ;; + MAX_EVENTS=*) + MAX_EVENTS="${arg#*=}" + ;; + CAMERA_NAME=*) + CAMERA_NAME="${arg#*=}" + ;; + *) + echo "[WARN] Unknown argument: $arg" + ;; + esac +done + +echo "[ARGS] run=${RUN_NUMBER} exp=${EXP_NUMBER} save_norm=${SAVE_NORMALIZED} max_events=${MAX_EVENTS} camera=${CAMERA_NAME}" +echo + +echo "[STEP 1/1] Running export_xtc_normalized_args.py" +START_EXPORT=$(date +%s.%N) +set +u +source "${PSCONDA_SH}" +set -u + +python ./export_xtc_normalized_args.py "${RUN_NUMBER}" "${EXP_NUMBER}" "${SAVE_NORMALIZED}" "${MAX_EVENTS}" "${CAMERA_NAME}" +END_EXPORT=$(date +%s.%N) +DURATION_EXPORT=$(echo "$END_EXPORT - $START_EXPORT" | bc -l) + +# Count the number of images processed +if [ "${SAVE_NORMALIZED}" = "1" ]; then + IMAGE_DIR="run_${RUN_NUMBER}_png_norm" +else + IMAGE_DIR="run_${RUN_NUMBER}_png" +fi +NUM_IMAGES=$(find "${IMAGE_DIR}" -name "*.png" | wc -l) + +echo "[STEP 1/1] Export completed. Processed ${NUM_IMAGES} images." +echo + +echo "[SLURM] Export completed successfully." +echo "=== TIMING PROFILE ===" +echo "Total images processed: ${NUM_IMAGES}" +echo "Export XTC Data: ${DURATION_EXPORT} seconds total, $(echo "$DURATION_EXPORT / $NUM_IMAGES" | bc -l) seconds per image" diff --git a/scripts/production/inference_xtc_serial_pipeline/run_inference_xtc.sh b/scripts/production/inference_xtc_serial_pipeline/run_inference_xtc.sh new file mode 100644 index 0000000..16e5c8e --- /dev/null +++ b/scripts/production/inference_xtc_serial_pipeline/run_inference_xtc.sh @@ -0,0 +1,65 @@ +#!/bin/bash +#SBATCH --partition=turing +#SBATCH --account=lcls:prjlumine22 +#SBATCH --job-name=inference_xtc +#SBATCH --nodes=1 +#SBATCH --ntasks=1 +#SBATCH --cpus-per-task=4 +#SBATCH --gpus=1 +#SBATCH --time=02:00:00 +#SBATCH --output=logs_export/inference_turing_%j.out +#SBATCH --error=logs_export/inference_turing_%j.err + +set -euo pipefail + +export OMP_PROC_BIND=close +export OMP_PLACES=cores +export OMP_NUM_THREADS="${SLURM_CPUS_PER_TASK}" +export MKL_NUM_THREADS="${SLURM_CPUS_PER_TASK}" +export CUDA_DEVICE_ORDER=PCI_BUS_ID + +echo "[SLURM] Host: $(hostname)" +echo "[SLURM] CUDA_VISIBLE_DEVICES: ${CUDA_VISIBLE_DEVICES:-"(not set)"}" +echo "[SLURM] Starting job..." + +YOLO_PYTHON="/sdf/data/lcls/ds/prj/prjlumine22/results/coyote_protector/miniconda3_coyote/envs/env_coyote/bin/python" + +# ------- DEFAULT INPUTS --------- +RUN_NUMBER=61 +USE_NORMALIZED=1 +# -------------------------------- + +# ------- PARSE key=value arguments --------- +for arg in "$@"; do + case $arg in + RUN_NUMBER=*) + RUN_NUMBER="${arg#*=}" + ;; + USE_NORMALIZED=*) + USE_NORMALIZED="${arg#*=}" + ;; + *) + echo "[WARN] Unknown argument: $arg" + ;; + esac +done + +echo "[ARGS] run=${RUN_NUMBER} use_normalized=${USE_NORMALIZED}" +echo + +# Determine which image directory to use +if [ "${USE_NORMALIZED}" = "1" ]; then + IMAGE_DIR="run_${RUN_NUMBER}/run_${RUN_NUMBER}_png_norm" +else + IMAGE_DIR="run_${RUN_NUMBER}/run_${RUN_NUMBER}_png" +fi + +echo "[STEP 1/2] Running inference_coyote_xtc.py" +"${YOLO_PYTHON}" ./inference_coyote_xtc.py "${IMAGE_DIR}" + +echo +echo "[STEP 2/2] Running merge_crystals_data.py" +"${YOLO_PYTHON}" ./merge_crystals_data.py "${RUN_NUMBER}" + +echo +echo "[SLURM] Inference and merging completed successfully."