Add Claude-powered Slack bot for data transfer requests

CHANGELOG.md

@@ -0,0 +1,39 @@
+# Changelog
+
+## Unreleased (feature/slack-claude)
+
+### Parallel Manifest Build
+
+- Parallelize manifest generation into up to 4 concurrent `rclone lsjson` workers, reducing listing time for large datasets with many top-level subdirectories (`slurm_tools.py`)
+- Add `manifest combine` CLI command to merge parallel lsjson part files (with `.prefix` sidecars) into a unified `manifest.jsonl` (`cli.py`)
+- Bump prepare job memory from 16 GB to 250 GB to accommodate large listings (`slurm_tools.py`)
+- Add `--max-backlog=1000000` to `rclone lsjson` calls to prevent the walker from stalling on large buckets (`cli.py`, `slurm_tools.py`)
+- Report manifest build progress (files listed, bytes listed) via `manifest.jsonl.progress` sidecar file (`cli.py`)
+- Track prepare job phases (`listing_source`, `combining_manifest`, `analyzing`, `sharding`, `rendering`, `submitting`) in `progress.json` (`slurm_tools.py`)
+
+### Claude-Powered Slack Bot
+
+- Add Claude-powered Slack bot for interactive data transfer requests via Slack threads
+- Add intelligent rclone flag selection based on file size distribution analysis
+- Add `check_path_exists` tool to validate source paths before submitting jobs
+- Add `list_buckets` tool to enumerate buckets at remote endpoints
+- Add `read_job_logs` tool to access job analysis data, prepare logs, and shard transfer logs
+- Add lightweight Haiku triage to filter thread messages and skip unrelated chatter
+- Add per-user job ownership so only the submitting user can cancel their jobs
+- Restore thread context from Slack API after bot restarts
+- Report manifest listing progress (files_listed, bytes_listed) in job status during `building_manifest` phase
+
+### Slurm Robustness
+
+- Increase prepare job time limit to 4 days for very large datasets
+- Unset conflicting `SLURM_MEM_*` environment variables in prepare.sh
+- Add `--export=NONE` to all `sbatch` calls to prevent environment leakage
+- Allow users to set lower array concurrency (max 64)
+- Verify source path exists before creating run directory and submitting jobs
+- Reduce thread history limits to mitigate Slack rate limits
+
+### Bug Fixes
+
+- Fix `run_id` format to be filename-safe (no colons)
+- Fix markdown rendering in Slack responses
+- Improve error logging for manifest build failures (write to `xfer-err/` with full context)

README.md

@@ -150,7 +150,35 @@ run/
 
 ---
 
-### 2. Shard the manifest
+### 2. Analyze file size distribution (optional)
+
+Analyze the manifest to see file size statistics and get suggested rclone flags:
+
+```bash
+uv run xfer manifest analyze \
+  --in run/manifest.jsonl \
+  --out run/analysis.json
+```
+
+Output (`analysis.json`):
+```json
+{
+  "status": "ok",
+  "total_files": 125000,
+  "total_bytes": 5368709120,
+  "total_bytes_human": "5.00 GiB",
+  "profile": "small_files",
+  "profile_explanation": "Optimized for small files (82% < 1MB, median 256 KiB)",
+  "suggested_flags": "--transfers 64 --checkers 128 --fast-list --retries 10 ...",
+  "histogram": [...]
+}
+```
+
+The Slack bot runs this analysis automatically on every transfer to select optimal flags.
+
+---
+
+### 3. Shard the manifest
 
 Splits the manifest into balanced shards (by total bytes):
 
@@ -174,7 +202,7 @@ run/
 
 ---
 
-### 3. Render Slurm scripts
+### 4. Render Slurm scripts
 
 Creates:
 
@@ -199,7 +227,7 @@ uv run xfer slurm render \
 
 ---
 
-### 4. Submit the job
+### 5. Submit the job
 
 ```bash
 uv run xfer slurm submit --run-dir run
@@ -264,7 +292,21 @@ sbatch run/sbatch_array.sh
 
 ## Recommended rclone flags (starting points)
 
-### High-throughput S3↔S3
+### Intelligent Auto-Selection (Slack Bot)
+
+When using the Slack bot, rclone flags are **automatically selected** based on file size distribution analysis:
+
+| Profile | Condition | Flags |
+|---------|-----------|-------|
+| **Small files** | >70% files < 1MB | `--transfers 64 --checkers 128 --fast-list` |
+| **Large files** | >50% files > 100MB | `--transfers 16 --checkers 32 --buffer-size 256M` |
+| **Mixed** | Default | `--transfers 32 --checkers 64 --fast-list` |
+
+All profiles include `--retries 10 --low-level-retries 20 --stats 600s --progress` for reliability and logging.
+
+### Manual Flag Selection (CLI)
+
+#### High-throughput S3↔S3
 
 ```text
 --transfers 32
@@ -275,15 +317,23 @@ sbatch run/sbatch_array.sh
 --stats 30s
 ```
 
-### Small objects (metadata heavy)
+#### Small objects (metadata heavy)
 
 ```text
---transfers 16
+--transfers 64
 --checkers 128
 --fast-list
 ```
 
-### Track progress
+#### Large objects
+
+```text
+--transfers 16
+--checkers 32
+--buffer-size 256M
+```
+
+#### Track progress (always recommended)
 
 ```text
 --progress --stats 600s
@@ -296,6 +346,7 @@ sbatch run/sbatch_array.sh
 ```
 run/
   manifest.jsonl
+  analysis.json          # File size analysis and suggested flags
   shards/
     shard_000123.jsonl
     shards.meta.json