Closed
Conversation
Features: - Snakemake metadata parsing (base64 filenames, JSON content) - SLURM integration via squeue and sacct - LSF integration via bjobs and bhist - Interactive TUI with ratatui: - Progress bar header - Status count bar (pending/running/done/failed) - Filterable job list (All/Running/Failed/Pending/Completed) - Sortable job list (Status/Rule/Time) - Job detail panel - Vim-style navigation (j/k/g/G) - Help overlay (?) - Unified job state merging from multiple data sources - Test pipeline infrastructure with pixi.toml 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Core Features: - Live polling service for SLURM (squeue/sacct) and LSF (bjobs/bhist) - Auto-detects scheduler at runtime - Configurable poll intervals - Background tokio tasks for non-blocking updates - File watching for snakemake metadata directory - Uses notify crate for cross-platform support - Debounced events to handle rapid file changes - Log viewer TUI component - Scrolling navigation (j/k, g/G) - Follow mode for real-time log tailing - Graceful error handling for missing files Infrastructure: - README with installation, usage, and keyboard shortcuts - mkdocs documentation site with material theme - GitHub Actions workflows for CI, release, and docs - VHS demo tapes for terminal recording Updated Dependencies: - notify = "7.0" for file watching - tokio with full features for async runtime - camino for UTF-8 path handling - chrono for timestamp handling 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- ratatui 0.30.0-beta.0 (from 0.29) - crossterm 0.29 (from 0.28) - Updated run_app to use concrete CrosstermBackend type (required due to Backend trait changes in ratatui 0.30) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Format all code with cargo fmt - Add #[allow(dead_code)] for utility functions in charmer-lsf/bhist.rs - Remove redundant trim() before split_whitespace() 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Create a new charmer-parsers crate to consolidate duplicated parsing code: - time.rs: Shared timestamp parsing (SLURM and LSF formats), duration parsing - memory.rs: Memory string parsing with format enum (Slurm, SlurmSacct, Lsf) - command.rs: Async command execution wrapper with error handling - lib.rs: Common utilities (non_empty_string, split_delimited) Refactor charmer-slurm and charmer-lsf to use the shared parsers: - Reduces code duplication by ~150 lines - Consistent error handling across scheduler modules - Unified memory format handling 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add ToJobStatus trait to types.rs for converting scheduler states - Implement ToJobStatus for SlurmJobState and LsfJobState - Split merge.rs (425 lines) into focused submodules: - merge/comment.rs - Comment/description field parsing - merge/slurm.rs - SLURM job merging using ToJobStatus - merge/lsf.rs - LSF job merging using ToJobStatus - merge/snakemake.rs - Snakemake metadata merging - merge/correlation.rs - Cross-source job correlation - merge/mod.rs - Module exports Benefits: - State conversion logic is now explicit and testable via trait - Clearer separation of concerns in merge logic - Easier to maintain and extend individual merge strategies 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Scan existing .snakemake/metadata/ files on startup - Filter jobs to show only incomplete or recently started (within history_hours) - Update clean-test task to remove .snakemake directory - Add monitor-test task for TUI development - Update test pipeline with longer delays for TUI testing (~2 min runtime) - Add .gitignore for test pipeline outputs 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- JobList now uses filtered job IDs from App instead of raw state.jobs - Watcher now watches working directory for .snakemake creation - Detect .snakemake and .snakemake/metadata directory creation events 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
File watching can be unreliable when directories don't exist at startup or are recreated. Added a 2-second periodic re-scan of the metadata directory as a robust fallback mechanism. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Job Detail Panel: - Colorful status display with symbols - Resources section (queue, node, CPUs, memory, time limit) - Timing section (wait time, runtime, start time) - Error details for failed jobs (exit code, message) - Output file preview with truncation - Shell command preview Job List: - Inline progress bar with percentage - Color-coded status counts (R=running, C=completed, F=failed, Q=queued) - Truncated job labels for long names Wildcard Extraction: - Auto-extract sample/chrom wildcards from output paths - Heuristic parsing for common patterns (sample1_chr1.vcf) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Press 'l' or Enter to open log viewer for selected job
- Log viewer opens as overlay (90% width, 85% height)
- Keybindings in log viewer:
- j/k: scroll up/down
- g/G: top/bottom
- F: toggle follow mode (auto-scroll)
- r: refresh log content
- q/Esc: close log viewer
- Auto-refresh in follow mode when state updates
- Falls back to .snakemake/log/{rule}.log if no log path in metadata
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Pressing 'l' now toggles the log viewer (open/close) - Disabled mouse capture to allow trackpad text selection for copy/paste - Terminal now allows normal text selection behavior 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Automatically detect and classify job failures (OOM, timeout, exit code, cancelled, node failure) and provide actionable suggestions for fixes. - Add charmer-slurm/src/failure.rs: Query sacct for detailed failure info - Add charmer-lsf/src/failure.rs: Query bhist -l for LSF failure analysis - Add FailureAnalysis and FailureMode types to charmer-state - Update job detail panel to display failure analysis with suggestions - Enrich failed jobs with analysis during polling (limited to 5 at a time) For OOM failures, shows memory used vs limit and suggests increased allocation. For timeout failures, shows runtime vs limit and suggests time increase. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Log panel now appears at the bottom instead of blocking the window - Shows tailed output with auto-refresh when navigating jobs - j/k keys still navigate jobs (log panel updates automatically) - F toggles follow mode - l toggles log panel open/close - Added syntax highlighting for ERROR/WARN/INFO in log output 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Double all sleep times for longer job observation - Add 2 more samples (6 total) and 1 more chromosome (4 total) - sample3 fails during alignment with exit 137 (OOM simulation) - Added log files to each rule for log panel testing - Error messages include realistic OOM output 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Use working directory to resolve relative log paths from snakemake
- Try multiple log locations in order:
1. Snakemake metadata log files (relative to working dir)
2. SLURM log path format (.snakemake/slurm_logs/rule_{rule}/{jobid}.log)
3. Common pattern logs/{rule}/{wildcard}.log
4. Fallback logs/{rule}.log
- Shows helpful path in error if log not found
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
When job-specific logs aren't found, fall back to the most recent .snakemake/log/*.snakemake.log file which contains output for all jobs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add working directory and date/time to header with colors - Add version to footer (bottom right) - Split job list into responsive columns (rule, sample, slurm id) - Columns appear/disappear based on panel width - Improve color contrast for dark mode (DarkGray → Gray for labels) - Change row highlight from background to bold text - Open logs panel by default - Add snake emoji to header 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…d ETA Major TUI improvements: - Column headers for jobs panel with responsive layout - Smart Job ID column (shows SLURM ID, "local" for localrules, or Runtime) - Environment detection (pixi, conda, container) from shellcmd and metadata - Structured error parsing with categorized types and icons - Rule summary view (toggle with 'r' key) showing per-rule statistics - Time estimates with ETA display in header 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Move demo.tape and quickstart.tape from vhs/ to docs/tapes/ - Create generate.sh that launches test pipeline and records tapes - Output GIFs to docs/images/ instead of docs/assets/ - Update README.md and docs/index.md to reference new image paths - Update pixi.toml demo task to use generate.sh 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add MAX_RULE_WIDTH (24 chars) to prevent rule column from being too wide - Add is_target field to Job struct to identify target rules (like "all") - Create synthetic jobs for target rules from snakemake log jobs_by_rule - Display target rules with 🎯 emoji in job list Target rules are rules with no output files that don't create metadata. They're detected as rules with count=1 in the log that have no jobs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
VHS runs in a fresh shell without PATH to cargo binaries. - Build release binary first in generate.sh - Use ../../target/release/charmer path in tapes 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This file was created earlier but never added to git. It contains the snakemake log parser used throughout the codebase. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Install charmer to ~/.cargo/bin so tapes can use simple `charmer` command instead of ugly relative paths. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Run `pixi run docs-deploy` to deploy mkdocs to gh-pages branch. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add 'c' keybinding to copy shell command to clipboard - Add 'e' keybinding to expand/collapse command in details panel - Fix target job detection: only create synthetic jobs for rules without output files (parsed from log), not all rules with count=1 - Fix target job status: update status on every log parse - Fix target job sorting: always sort to bottom of job list - Add arboard dependency for cross-platform clipboard support 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The log panel wasn't showing by default because open_log_viewer() was called before update_job_list(). Without update_job_list(), the MAIN_PIPELINE_JOB_ID wasn't in the job list, causing open_log_viewer() to return early without setting show_log_viewer = true. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Simplify header to single line with abbreviated status counts (Pend/Run/Done/Fail) and colored separators - Remove duplicate progress bar from header, keep only in Jobs panel - Move Filter/Sort display into Jobs panel title - Update progress bar style to [▮▮▮▮────](n/m) with bold text - Add status message feedback in footer (shows copy confirmations) - Fix row numbering to start at 1 (excluding main pipeline job) - Fix visible job count to exclude main pipeline pseudo-job 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Replace simple 'r' toggle with proper Tabs widget at top of left panel - Tabs widget shows visual indication of current view (Jobs/Rules) - Keyboard shortcut 'r' still works to toggle between views - Update borders on job_list and rule_summary to work with tabs 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Show sparkline visualization of last 20 completed job runtimes - Jobs are shown in chronological order (oldest to newest) - Sparkline only displayed when rule has 2+ completed jobs with timing data - Helps identify runtime trends and performance patterns for rules 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Show bar chart in rule detail panel displaying job status breakdown - Four bars show Running, Done, Failed, and Pending jobs - Color-coded bars match status colors (yellow, green, red, blue) - Bar chart dynamically appears when rule has jobs - Works in combination with sparkline to provide comprehensive visualization 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- New DAG view accessible via 'd' key showing job dependency graph - Uses Canvas widget to render visual representation of pipeline - Rules displayed in circular layout with color-coded status - Individual jobs shown as small circles around their parent rule - Rule connections shown with lines - Color coding: Green=completed, Yellow=running, Red=failed, Blue=pending - Added DAG tab to tabs widget - Updated help screen to include 'd' key - Press 'd' again to return to jobs view This is the most exciting feature - provides visual overview of entire pipeline! 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Convert header from Gauge to single-line bordered Paragraph - Show app name, status, cwd, ETA, datetime, and abbreviated counts - Update progress bar style to [▮▮▮▮────](n/m) with bold text - Move filter/sort labels to Jobs panel title - Add status_message to footer for copy feedback (bold purple) - Fix row numbering to exclude MAIN_PIPELINE_JOB_ID - Change target job symbol from 🎯 to ◎ (single-width) - Update target job status when pipeline completes - Remove status bar (consolidated into header) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Build actual dependency graph from job inputs/outputs - Use topological sort (Kahn's algorithm) for layer assignment - Position nodes left-to-right by layer, spread vertically within layers - Node size reflects job count, color reflects status - Draw edges between dependent rules 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Use Unicode box-drawing characters instead of Braille canvas - Render rules as colored boxes with status icons - Show job counts (completed/total) in each box - Draw vertical arrows between dependency layers - Add legend for status symbols - Much cleaner and more readable than Canvas circles 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add Scrollbar widget to job list when items exceed viewport - Add Scrollbar widget to log viewer (both full and panel modes) - Remove grey border styling from log panel (was causing color issues) - Scrollbars show ↑/↓ symbols at ends with █ thumb 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Move tab selection into block title line: "[Jobs] Rules DAG" - Remove separate 2-line tabs row, reclaim vertical space - Soften rule highlighting: use DarkGray background instead of REVERSED - Add scrollbar to rule summary table - Show filter/sort info in job list title_bottom 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Replace barchart with compact text status: ▶3 ✓12 ✗1 ○5 - Require 3+ data points for sparkline (prevents big yellow box) - Change sparkline color from yellow to cyan for better visibility - Simplify layout without separate chart sections 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove separate DAG view (was buggy with large job lists) - Add inline dependency visualization in job list: - ● white dot for selected job - ○─ cyan for upstream dependencies (jobs selected depends on) - ○─ magenta for downstream dependents (jobs that depend on selected) - Compute dependencies from job inputs/outputs - Remove 'd' keybinding, simplify to Jobs/Rules tabs only 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Change from horizontal dashes to vertical bars: │○│ Shows the dependency chain as a connected vertical line. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- ○ (cyan) = upstream dependency - ● (white) = selected job - ○ (magenta) = downstream dependent - │ (gray) = connecting line through unrelated jobs Example: 1 ✓ fastqc sample1 ○ <- upstream (cyan) 2 ✓ other sample2 │ <- unrelated but in chain range 3 ▶ align sample1 ● <- selected (white) 4 ○ count sample1 ○ <- downstream (magenta) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Show dependencies as a proper tree with horizontal connectors:
○─┐ upstream (cyan dot, white connector)
│ trunk through unrelated jobs (white)
●─┤ selected (white bold dot)
│ trunk
○─┘ downstream (magenta dot, white connector)
Colored dots indicate relationship, white lines show tree structure.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add full transitive dependency chain visualization - Show status-based dots in chain (○ pending, ● completed, ◐ running) - Display all wildcards with colored pipe separators in job list - Add actual ratatui Gauge widget for progress bar - Update progress header layout: Filter/Sort | Gauge | (count) - Color filter values (cyan) and sort values (yellow) - Match wildcard colors in Job Details panel (key=value format) - Remove Runtime Trend sparkline from Rules view - Fix scrollbar positioning and thumb size consistency - Soften rule list highlighting (bold only, no background) - Rename Details panels to "Job Details" consistently 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Show runtime in human-readable format (1h 23m 45s) - Include date in Started timestamp (2025-12-18 14:30:00) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Resolves conflicts by taking feature branch changes which include: - Full transitive dependency chain visualization - Colored wildcards with pipe separators - Ratatui Gauge widget for progress bar - Scrollbars with proper positioning - Human-readable time formatting 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix generate.sh to run from repo root with absolute paths - Update tape files to use repo-relative paths instead of ../.. - Remove webm output from demo tape - Update notify dependency from 7.0 to 8.2 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Use contains_key instead of get().is_none() - Simplify int comparison (total_len < max_len) - Allow clippy::too_many_arguments for build_job_item - Allow clippy::needless_range_loop for chain position marking - Escape brackets in doc comment to fix rustdoc link warning 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Create tests/pipelines/demo with 12 samples, 9 rules - Staggered timing shows job accumulation and progression - Fan-out pattern (3 regions per sample) demonstrates dependency chains - Update demo.tape with longer recording showing Jobs/Rules views - Update generate.sh to use demo pipeline with 4 cores - Add clean-demo task to pixi.toml 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Deploy to gh-pages branch instead of GitHub Pages artifacts. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Member
Author
|
Closing due to merge conflicts. Will cherry-pick the cluster pipeline commit instead. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Sync develop branch to main, bringing in the cluster test pipeline feature.
🤖 Generated with Claude Code