Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 30 additions & 7 deletions cmd/scaffold.go
Original file line number Diff line number Diff line change
Expand Up @@ -178,13 +178,36 @@ var scaffoldCmd = &cobra.Command{
Use: "scaffold",
Short: "Scaffold a new Sietch vault",
Long: `Scaffold a new Sietch vault with secure encryption and configurable options.
This creates the necessary directory structure and configuration files for your vault.

Examples:
sietch scaffold --template photoVault
sietch scaffold --template photoVault --name "My Photo Vault"
sietch scaffold --template photoVault --name "My Photo Vault" --path /path/to/vault
sietch scaffold --template photoVault --name "My Photo Vault" --path /path/to/vault --force`,
This creates the necessary directory structure and configuration files for your vault
using pre-configured templates optimized for different use cases.

Available Templates:
Photos & Media:
photoVault - Photo storage with strong dedup and high compression
videoVault - Video storage with large chunks and light compression
audioLibrary - Audio/podcast storage with balanced settings

Documents & Knowledge:
documentsVault - Office/PDF documents with aggressive compression
codeVault - Code repositories with fine-grained deduplication
reporterVault - Journalism/sensitive documents with manual sync

Backups & Archives:
systemBackup - System backups optimized for performance
coldArchive - Long-term archival with maximum compression

Examples:
List all available templates:
sietch scaffold --list

Create a vault from a template:
sietch scaffold --template photoVault
sietch scaffold --template videoVault --name "My Movies"
sietch scaffold --template documentsVault --name "Work Docs" --path ~/Documents
sietch scaffold --template codeVault --name "Projects" --path ~/Code --force

Learn more about templates:
See ~/.config/sietch/templates/README.md for detailed comparison`,

RunE: func(cmd *cobra.Command, args []string) error {
// Check if user wants to list templates
Expand Down
74 changes: 67 additions & 7 deletions template/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Each template is defined in a JSON file with the following structure:

### Basic Information
- **`name`**: Display name for the template (required)
- **`description`**: Human-readable description of the template's purpose (required)
- **`description`**: Human readable description of the template's purpose (required)
- **`version`**: Template version (required)
- **`author`**: Who created this template (required)
- **`tags`**: Array of tags for categorization and filtering (optional)
Expand All @@ -53,7 +53,7 @@ Defines the default vault configuration that will be applied when using this tem
- **`dedup_max_size`**: Maximum file size for deduplication (e.g., `"64MB"`)
- **`dedup_gc_threshold`**: Garbage collection threshold (number)
- **`dedup_index_enabled`**: Enable deduplication index (`true`/`false`)
- **`dedup_cross_file`**: Allow cross-file deduplication (`true`/`false`)
- **`dedup_cross_file`**: Allow cross file deduplication (`true`/`false`)

### Directory Structure (`directories`)
Array of directories to create in the vault. These are created relative to the vault root:
Expand Down Expand Up @@ -225,10 +225,70 @@ Templates are validated when loaded. Common issues:
6. **Version your templates**
7. **Document any special requirements**

## Examples
## Available Templates

### Photos and Media
- **`photoVault`** - Photo storage with fixed chunking, strong dedup, high compression
- **`videoVault`** - Video storage with larger chunks, lighter compression, tuned hashing
- **`audioLibrary`** - Audio/podcast storage with balanced compression and indexing

### Documents and Knowledge
- **`documentsVault`** - Office/PDF documents with higher compression, content-dedup, index enabled
- **`codeVault`** - Code repositories/artifacts with fingerprint dedup, fast hashing, moderate chunks
- **`reporterVault`** - Journalism/reporting with secure defaults, manual sync, metadata emphasis

### Backups and Archives
- **`systemBackup`** - System backups with large chunks, parallel sync, conservative dedup
- **`coldArchive`** - Long-term archival with maximum compression, minimal write amplification

## Template Comparison Matrix

| Template | Chunk Size | Compression | Hash | Dedup Min/Max | GC Threshold | Index | Cross File Dedup | Use Case |
|----------|-----------|-------------|------|---------------|--------------|-------|------------------|----------|
| **photoVault** | 8MB | gzip | sha256 | 1MB / 64MB | 500 | ✓ | ✓ | Photos, RAW images, high-res media |
| **videoVault** | 32MB | lz4 | sha256 | 16MB / 256MB | 100 | ✓ | ✗ | Large video files, movies, recordings |
| **audioLibrary** | 8MB | gzip | sha256 | 2MB / 128MB | 300 | ✓ | ✓ | Music, podcasts, audio collections |
| **documentsVault** | 2MB | gzip | sha256 | 512KB / 32MB | 1500 | ✓ | ✓ | Office docs, PDFs, text files |
| **codeVault** | 4MB | gzip | sha256 | 256KB / 16MB | 2000 | ✓ | ✓ | Source code, repos, build artifacts |
| **reporterVault** | 4MB | gzip | sha256 | 1KB / 32MB | 1000 | ✓ | ✓ | Journalism, sensitive documents |
| **systemBackup** | 16MB | lz4 | sha256 | 8MB / 128MB | 500 | ✓ | ✓ | Full system backups, disaster recovery |
| **coldArchive** | 16MB | gzip | sha512 | 4MB / 256MB | 200 | ✓ | ✓ | Long term storage, archival data |

### Understanding the Settings

**Chunk Size:**
- **Small (2-4MB)**: Better dedup, slower for large files → Documents, Code
- **Medium (8MB)**: Balanced performance → Photos, Audio
- **Large (16-32MB)**: Faster processing, less dedup overhead → Videos, Backups

**Compression:**
- **gzip**: Higher compression ratio, slower → Documents, Photos, Archives
- **lz4**: Faster compression, lower ratio → Videos, Backups
- **none**: No compression overhead → Already compressed formats

**Hash Algorithm:**
- **sha256**: Standard security, fast
- **sha512**: Higher security, slightly slower → Cold Archives

**Deduplication:**
- **Min/Max Size**: Range of chunk sizes eligible for dedup
- **GC Threshold**: Number of unreferenced chunks before cleanup suggestion
- **Index**: Enables faster chunk lookups (recommended: enabled)
- **Cross File**: Dedup across different files (disable for independent files like videos)

**When to Choose Each Template:**

| If you're storing... | Use this template | Why |
|---------------------|-------------------|-----|
| Family photos, vacation pictures | `photoVault` | Optimized for JPEG/PNG with good dedup for similar images |
| Movies, screen recordings | `videoVault` | Large chunks, fast compression, minimal dedup overhead |
| Music library, podcasts | `audioLibrary` | Balanced for MP3/FLAC with moderate dedup |
| Word docs, PDFs, spreadsheets | `documentsVault` | Aggressive compression + dedup for text content |
| Git repos, npm packages | `codeVault` | Fine grained dedup for similar source files |
| Sensitive documents, sources | `reporterVault` | Security focused with manual sync control |
| Full system snapshots | `systemBackup` | Performance optimized for large backups |
| Old files, compliance data | `coldArchive` | Maximum compression for rarely accessed data |

See the existing templates in this directory:
- `photoVault.json` - Photo storage template
- `reporterVault.json` - Reporting template
## Examples

These serve as examples of how to structure your own templates.
See the templates in this directory for reference implementations. Each template serves as an example of how to structure your own custom templates.
21 changes: 21 additions & 0 deletions template/audioLibrary.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"name": "Audio Library",
"description": "Balanced compression and indexing for audio files, podcasts, and music collections",
"version": "1.0.0",
"author": "Sietch Team",
"tags": ["audio", "music", "podcast", "media"],
"config": {
"chunking_strategy": "fixed",
"chunk_size": "8MB",
"hash_algorithm": "sha256",
"compression": "gzip",
"sync_mode": "manual",
"enable_dedup": true,
"dedup_strategy": "content",
"dedup_min_size": "2MB",
"dedup_max_size": "128MB",
"dedup_gc_threshold": 300,
"dedup_index_enabled": true,
"dedup_cross_file": true
}
}
21 changes: 21 additions & 0 deletions template/codeVault.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"name": "Code Vault",
"description": "Optimized for code repositories and build artifacts with fingerprint deduplication, fast hashing, and moderate chunk sizes",
"version": "1.0.0",
"author": "Sietch Team",
"tags": ["code", "source", "repository", "artifacts"],
"config": {
"chunking_strategy": "fixed",
"chunk_size": "4MB",
"hash_algorithm": "sha256",
"compression": "gzip",
"sync_mode": "manual",
"enable_dedup": true,
"dedup_strategy": "content",
"dedup_min_size": "256KB",
"dedup_max_size": "16MB",
"dedup_gc_threshold": 2000,
"dedup_index_enabled": true,
"dedup_cross_file": true
}
}
21 changes: 21 additions & 0 deletions template/coldArchive.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"name": "Cold Archive",
"description": "Optimized for long-term archival storage with maximum compression and minimal write amplification",
"version": "1.0.0",
"author": "Sietch Team",
"tags": ["archive", "cold-storage", "long-term", "compression"],
"config": {
"chunking_strategy": "fixed",
"chunk_size": "16MB",
"hash_algorithm": "sha512",
"compression": "gzip",
"sync_mode": "manual",
"enable_dedup": true,
"dedup_strategy": "content",
"dedup_min_size": "4MB",
"dedup_max_size": "256MB",
"dedup_gc_threshold": 200,
"dedup_index_enabled": true,
"dedup_cross_file": true
}
}
21 changes: 21 additions & 0 deletions template/documentsVault.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"name": "Documents Vault",
"description": "Optimized for Office documents and PDFs with high compression, content deduplication, and indexing enabled",
"version": "1.0.0",
"author": "Sietch Team",
"tags": ["documents", "office", "pdf", "text"],
"config": {
"chunking_strategy": "fixed",
"chunk_size": "2MB",
"hash_algorithm": "sha256",
"compression": "gzip",
"sync_mode": "manual",
"enable_dedup": true,
"dedup_strategy": "content",
"dedup_min_size": "512KB",
"dedup_max_size": "32MB",
"dedup_gc_threshold": 1500,
"dedup_index_enabled": true,
"dedup_cross_file": true
}
}
21 changes: 21 additions & 0 deletions template/systemBackup.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"name": "System Backup",
"description": "Optimized for system backups with large chunks, parallel sync support, and conservative deduplication",
"version": "1.0.0",
"author": "Sietch Team",
"tags": ["backup", "system", "recovery", "disaster-recovery"],
"config": {
"chunking_strategy": "fixed",
"chunk_size": "16MB",
"hash_algorithm": "sha256",
"compression": "lz4",
"sync_mode": "manual",
"enable_dedup": true,
"dedup_strategy": "content",
"dedup_min_size": "8MB",
"dedup_max_size": "128MB",
"dedup_gc_threshold": 500,
"dedup_index_enabled": true,
"dedup_cross_file": true
}
}
21 changes: 21 additions & 0 deletions template/videoVault.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
{
"name": "Video Vault",
"description": "Optimized for video storage with large chunks, lighter compression, and tuned hashing for video files",
"version": "1.0.0",
"author": "Sietch Team",
"tags": ["video", "media", "movies", "large-files"],
"config": {
"chunking_strategy": "fixed",
"chunk_size": "32MB",
"hash_algorithm": "sha256",
"compression": "lz4",
"sync_mode": "manual",
"enable_dedup": true,
"dedup_strategy": "content",
"dedup_min_size": "16MB",
"dedup_max_size": "256MB",
"dedup_gc_threshold": 100,
"dedup_index_enabled": true,
"dedup_cross_file": false
}
}
Loading