Skip to content

Commit 9f30b31

Browse files
committed
docs: enhance compression documentation and update links
- Added a new compression guide detailing the process and requirements for compressing downloaded media files. - Updated links in the FAQ and troubleshooting sections to point to the new compression guide for better user navigation and clarity.
1 parent 3930b57 commit 9f30b31

File tree

3 files changed

+239
-2
lines changed

3 files changed

+239
-2
lines changed

docs/src/content/docs/help/faq.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -90,7 +90,7 @@ This converts:
9090
- JPG/JPEG → JPEG XL (30-50% smaller)
9191
- MP4 → AV1 (30-50% smaller)
9292

93-
See the [README](https://github.com/3dnsfw/kemono-scraper#compression) for requirements.
93+
See the [Compression guide](/Kemono-Scraper/usage/compression/) for detailed instructions and requirements.
9494

9595
---
9696

docs/src/content/docs/help/troubleshooting.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -243,7 +243,7 @@ Large creators can have hundreds of gigabytes of content. Monitor your disk spac
243243

244244
1. Using `--maxPosts` to limit downloads
245245
2. Downloading to an external drive
246-
3. Compressing files after download
246+
3. [Compressing files after download](/Kemono-Scraper/usage/compression/) to save 30-50% disk space
247247

248248
---
249249

Lines changed: 237 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,237 @@
1+
---
2+
title: Compression
3+
description: Compress downloaded media files to save disk space
4+
sidebar:
5+
order: 5
6+
---
7+
8+
After downloading files, you can compress them to save significant disk space. The scraper includes a built-in compression script that converts images and videos to more efficient formats.
9+
10+
## What Gets Compressed
11+
12+
The compression script automatically processes:
13+
14+
- **Images**: JPG/JPEG → JPEG XL (typically 30-50% smaller)
15+
- **Videos**: MP4 → AV1 (typically 30-50% smaller)
16+
17+
The scraper automatically detects compressed files on subsequent runs, so you won't re-download files that have already been compressed.
18+
19+
## Requirements
20+
21+
Before you can use compression, you need to install the required tools:
22+
23+
### libjxl (for JPEG XL)
24+
25+
**Linux:**
26+
27+
```bash
28+
# Arch Linux / Manjaro
29+
paru -S libjxl
30+
# or
31+
sudo pacman -S libjxl
32+
33+
# Debian / Ubuntu
34+
sudo apt install libjxl-tools
35+
```
36+
37+
**Windows:**
38+
39+
```powershell
40+
winget install --id=libjxl.libjxl -e
41+
```
42+
43+
**macOS:**
44+
45+
```bash
46+
brew install jpeg-xl
47+
```
48+
49+
### ffmpeg (with SVT-AV1 support)
50+
51+
**Linux:**
52+
53+
```bash
54+
# Arch Linux / Manjaro
55+
paru -S ffmpeg
56+
# or
57+
sudo pacman -S ffmpeg
58+
59+
# Debian / Ubuntu
60+
sudo apt install ffmpeg
61+
```
62+
63+
**Windows:**
64+
65+
```powershell
66+
winget install ffmpeg
67+
```
68+
69+
**macOS:**
70+
71+
```bash
72+
brew install ffmpeg
73+
```
74+
75+
:::tip[Verify Installation]
76+
Check that the tools are installed correctly:
77+
78+
```bash
79+
cjxl --version
80+
ffmpeg -version
81+
```
82+
83+
:::
84+
85+
## Running Compression
86+
87+
### Basic Usage
88+
89+
Navigate to your project directory and run:
90+
91+
```bash
92+
bun run compress
93+
```
94+
95+
This will compress all eligible files in your download directories.
96+
97+
### On Windows PowerShell
98+
99+
You can also use the PowerShell script:
100+
101+
```powershell
102+
.\compress.ps1
103+
```
104+
105+
### On Windows/Linux/Mac (Shell Script)
106+
107+
```bash
108+
./compress.sh
109+
```
110+
111+
## Configuration
112+
113+
You can customize compression settings using environment variables:
114+
115+
### Environment Variables
116+
117+
```bash
118+
JPEG_XL_QUALITY=95 AV1_CRF=28 bun run compress
119+
```
120+
121+
| Variable | Default | Description |
122+
|----------|---------|-------------|
123+
| `JPEG_XL_QUALITY` | 90 | JPEG XL quality (1-100, higher = better quality) |
124+
| `JPEG_XL_EFFORT` | 7 | Encoding effort (1-9, higher = slower but smaller files) |
125+
| `AV1_CRF` | 30 | AV1 quality (lower = better, 18-35 typical range) |
126+
| `AV1_PRESET` | 6 | SVT-AV1 preset (0-13, lower = slower but better quality) |
127+
128+
### Windows PowerShell Parameters
129+
130+
On Windows PowerShell, you can pass parameters directly:
131+
132+
```powershell
133+
.\compress.ps1 -JpegXlQuality 95 -Av1Crf 28
134+
```
135+
136+
## Understanding Quality Settings
137+
138+
### JPEG XL Quality
139+
140+
- **Lower values (70-85)**: Smaller files, slight quality loss
141+
- **Default (90)**: Good balance of quality and size
142+
- **Higher values (95-100)**: Near-lossless, larger files
143+
144+
### JPEG XL Effort
145+
146+
- **Lower (1-3)**: Fast encoding, larger files
147+
- **Default (7)**: Good balance
148+
- **Higher (8-9)**: Slow encoding, smallest files
149+
150+
### AV1 CRF (Constant Rate Factor)
151+
152+
- **Lower (18-25)**: Higher quality, larger files
153+
- **Default (30)**: Good balance
154+
- **Higher (31-35)**: Lower quality, smaller files
155+
156+
### AV1 Preset
157+
158+
- **Lower (0-3)**: Slowest encoding, best quality
159+
- **Default (6)**: Good balance
160+
- **Higher (10-13)**: Fast encoding, slightly lower quality
161+
162+
## Examples
163+
164+
### High Quality Compression
165+
166+
For maximum quality (larger files):
167+
168+
```bash
169+
JPEG_XL_QUALITY=98 JPEG_XL_EFFORT=9 AV1_CRF=25 AV1_PRESET=4 bun run compress
170+
```
171+
172+
### Maximum Compression
173+
174+
For smallest file sizes (some quality loss):
175+
176+
```bash
177+
JPEG_XL_QUALITY=85 JPEG_XL_EFFORT=9 AV1_CRF=32 AV1_PRESET=8 bun run compress
178+
```
179+
180+
### Balanced (Default)
181+
182+
The default settings provide a good balance:
183+
184+
```bash
185+
bun run compress
186+
```
187+
188+
## How It Works
189+
190+
1. The script scans your download directories for uncompressed files
191+
2. Images (JPG/JPEG) are converted to `.jxl` format
192+
3. Videos (MP4) are converted to `_av1.mp4` format
193+
4. Original files are preserved alongside compressed versions
194+
5. On subsequent scraper runs, compressed files are detected and skipped
195+
196+
:::note
197+
The original files are kept. You can delete them manually after verifying the compressed versions work correctly.
198+
:::
199+
200+
## Troubleshooting
201+
202+
### "Command not found: cjxl"
203+
204+
Make sure `libjxl` is installed and in your PATH. Verify with:
205+
206+
```bash
207+
which cjxl
208+
```
209+
210+
### "ffmpeg: command not found"
211+
212+
Install `ffmpeg` and ensure it's in your PATH:
213+
214+
```bash
215+
which ffmpeg
216+
```
217+
218+
### Compression is very slow
219+
220+
- Lower `JPEG_XL_EFFORT` (try 5-6)
221+
- Increase `AV1_PRESET` (try 8-10)
222+
- Process files in smaller batches
223+
224+
### Files are too large after compression
225+
226+
- Lower `JPEG_XL_QUALITY` (try 85-90)
227+
- Increase `AV1_CRF` (try 32-35)
228+
229+
### Files are too small / quality is poor
230+
231+
- Increase `JPEG_XL_QUALITY` (try 95-98)
232+
- Lower `AV1_CRF` (try 25-28)
233+
234+
## Next Steps
235+
236+
- Learn about [basic usage](/Kemono-Scraper/usage/basic/) for downloading files
237+
- See [command line options](/Kemono-Scraper/usage/cli-options/) for more control

0 commit comments

Comments
 (0)