Skip to content

Commit 3958ac5

Browse files
josecelanojosecelano
committed
docs: add console application output patterns research
Add comprehensive research document analyzing how popular console applications handle stdout/stderr separation and logging patterns. Includes analysis of: - Traditional Unix conventions - Modern application patterns (Docker, Cargo, Git, npm, curl, rsync, Terraform) - Pattern analysis and best practices - Specific recommendations for Torrust Deployer implementation This research informs design decisions for console output and logging architecture in the deployment tool.
1 parent d8a890d commit 3958ac5

File tree

2 files changed

+377
-0
lines changed

2 files changed

+377
-0
lines changed

‎docs/research/UX/console-app-output-patterns.md‎

Lines changed: 375 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,375 @@
1+
# Console Application Output Patterns Research
2+
3+
> **📋 Research Document Status**
4+
> This document contains research findings on how popular console applications handle stdout/stderr separation and logging. These findings inform design decisions but do not represent current implementation choices.
5+
6+
## Overview
7+
8+
This research examines how popular console applications handle the challenge of separating **user-facing output** from **internal logging/diagnostics**. Console applications face a fundamental challenge: both user output and application logs typically compete for the same output channels (stdout and stderr), which can create confusion and poor user experience.
9+
10+
## The Core Problem
11+
12+
Console applications need to handle multiple types of output:
13+
14+
1. **User output**: Results, data, final outcomes meant for end users
15+
2. **Progress information**: Status updates, progress indicators
16+
3. **Diagnostic logs**: Debug info, internal operations, detailed execution traces
17+
4. **Error messages**: User-facing errors vs. technical diagnostic errors
18+
5. **Warnings and info**: Various levels of informational messages
19+
20+
The challenge is mapping these to the two available output channels (stdout and stderr) in a way that:
21+
22+
- Follows Unix conventions and user expectations
23+
- Allows proper piping and redirection
24+
- Provides clear separation for different use cases
25+
- Supports various verbosity levels
26+
27+
## Traditional Unix Convention
28+
29+
The traditional Unix approach is simple and clear:
30+
31+
- **stdout**: Primary output, the "result" of the command
32+
- **stderr**: Error messages and diagnostics
33+
34+
### Examples of Traditional Pattern
35+
36+
```bash
37+
# ls - directory listing to stdout, errors to stderr
38+
ls /some/directory > files.txt # Results go to file
39+
ls /nonexistent 2> errors.txt # Errors go to file
40+
41+
# grep - matches to stdout, errors to stderr
42+
cat file.txt | grep "pattern" | sort # Clean pipeline
43+
grep "pattern" /nonexistent 2>/dev/null # Suppress errors
44+
45+
# find - results to stdout, errors to stderr
46+
find /etc -name "*.conf" 2>/dev/null | head -10
47+
```
48+
49+
## Popular Application Patterns
50+
51+
### 1. Docker - Structured Logging with User Output
52+
53+
Docker uses a sophisticated approach that separates user output from logs:
54+
55+
**Normal Operations:**
56+
57+
```bash
58+
# User output (container output) goes to stdout
59+
docker run nginx
60+
# Logs from container appear on stdout (container's stdout/stderr)
61+
62+
# Docker's own messages go to stderr
63+
docker pull ubuntu # Progress/status info to stderr
64+
docker build . # Build logs to stdout, Docker messages to stderr
65+
```
66+
67+
**With verbosity:**
68+
69+
```bash
70+
docker run --log-driver=json-file nginx # Structured logging
71+
docker logs container_name # Retrieve logs separately
72+
docker run -d nginx # Daemon mode, logs separate
73+
```
74+
75+
**Key insights from Docker:**
76+
77+
- Container output (the actual result) goes to stdout
78+
- Docker's operational messages go to stderr
79+
- Supports structured logging with different drivers
80+
- Daemon mode completely separates logging from user interaction
81+
82+
### 2. Cargo (Rust) - Clean User Experience
83+
84+
Cargo provides one of the cleanest console UX patterns:
85+
86+
**Default mode (clean):**
87+
88+
```bash
89+
cargo build
90+
Compiling hello-world v0.1.0
91+
Finished dev [unoptimized + debuginfo] target(s) in 0.75s
92+
93+
cargo run
94+
Finished dev [unoptimized + debuginfo] target(s) in 0.02s
95+
Running `target/debug/hello-world`
96+
Hello, world! # <- This is the program output (stdout)
97+
```
98+
99+
**Verbose mode (detailed):**
100+
101+
```bash
102+
cargo build -v # Much more detailed compilation info
103+
cargo run -v # Shows all the underlying commands being executed
104+
```
105+
106+
**Key insights from Cargo:**
107+
108+
- Progress/status messages go to stderr (so they don't interfere with piping)
109+
- Final program output goes to stdout
110+
- Verbose mode shows all the underlying operations
111+
- Clean separation allows: `cargo run | grep something` to work properly
112+
113+
### 3. Git - Mixed Approach with Context
114+
115+
Git uses a more complex pattern that varies by subcommand:
116+
117+
**Status and informational commands:**
118+
119+
```bash
120+
git status # All output to stdout (it's the result)
121+
git log # All output to stdout (it's the result)
122+
git diff # All output to stdout (it's the result)
123+
```
124+
125+
**Operational commands:**
126+
127+
```bash
128+
git clone https://... # Progress to stderr, allows redirection
129+
git push # Status messages to stderr
130+
git pull # Progress/status to stderr
131+
```
132+
133+
**Error handling:**
134+
135+
```bash
136+
git checkout nonexistent-branch # Error to stderr
137+
git commit # Prompts/messages to stderr, success to stdout
138+
```
139+
140+
**Key insights from Git:**
141+
142+
- Context matters - "query" commands put results on stdout
143+
- "Action" commands put progress on stderr, results on stdout
144+
- Interactive features (prompts, editor) use stderr
145+
146+
### 4. npm/yarn - Progress and Results Separation
147+
148+
Package managers show interesting patterns:
149+
150+
**npm:**
151+
152+
```bash
153+
npm install # Progress/warnings to stderr, allows clean piping
154+
npm list # Results to stdout
155+
npm run test # Test output to stdout, npm messages to stderr
156+
```
157+
158+
**yarn:**
159+
160+
```bash
161+
yarn install # Progress indicators to stderr
162+
yarn add pkg # Success messages to stderr, allows piping
163+
```
164+
165+
**Key insights:**
166+
167+
- Package manager operations put progress on stderr
168+
- Actual command output (like test results) goes to stdout
169+
- This allows piping package manager output: `npm test | grep "failed"`
170+
171+
### 5. curl - Data vs. Progress Separation
172+
173+
curl provides a clear example of data vs. metadata separation:
174+
175+
**Default:**
176+
177+
```bash
178+
curl https://api.example.com/data
179+
{"result": "data"} # Response body to stdout
180+
181+
curl https://api.example.com/data -o file.json # Body to file, progress to stderr
182+
```
183+
184+
**Verbose:**
185+
186+
```bash
187+
curl -v https://api.example.com/data
188+
* Connected to api.example.com # Debug info to stderr
189+
* SSL connection established # Debug info to stderr
190+
{"result": "data"} # Response body still to stdout
191+
```
192+
193+
**Key insights:**
194+
195+
- The actual data (purpose of the command) goes to stdout
196+
- All metadata, progress, debug info goes to stderr
197+
- Verbose mode adds more stderr output but doesn't change stdout
198+
199+
### 6. rsync - Progress and Results
200+
201+
rsync shows how to handle long-running operations:
202+
203+
**Default:**
204+
205+
```bash
206+
rsync -av source/ dest/
207+
building file list ... done # Progress to stderr
208+
file1 # File being transferred (stderr)
209+
file2
210+
sent 1,234 bytes received 56 bytes 2,580.00 bytes/sec # Summary to stderr
211+
```
212+
213+
**Quiet mode:**
214+
215+
```bash
216+
rsync -av --quiet source/ dest/ # Only errors to stderr, clean for scripts
217+
```
218+
219+
**Key insights:**
220+
221+
- Progress information goes to stderr
222+
- Allows scripting: `rsync --quiet ... && echo "success"`
223+
- Summary and statistics are considered progress, not results
224+
225+
### 7. Terraform/OpenTofu - State and Logs
226+
227+
Infrastructure tools show complex logging patterns:
228+
229+
**Terraform:**
230+
231+
```bash
232+
terraform apply
233+
# Plan output to stdout (it's a result the user needs to see)
234+
# Progress/status to stderr
235+
# TF_LOG=DEBUG environment variable controls detailed logs
236+
```
237+
238+
**Key insights:**
239+
240+
- Plan output is treated as "result" (stdout)
241+
- Execution progress goes to stderr
242+
- Debug logs are controlled via environment variables
243+
- Supports structured logging for advanced use cases
244+
245+
## Pattern Analysis
246+
247+
### Common Strategies Identified
248+
249+
1. **Pure Unix Convention** (ls, cat, grep)
250+
- stdout: Command results/output
251+
- stderr: Errors only
252+
- Simple and predictable
253+
254+
2. **Progress/Results Separation** (curl, cargo, npm)
255+
- stdout: Final results/data
256+
- stderr: Progress, status, metadata
257+
- Excellent for piping and automation
258+
259+
3. **Context-Dependent** (git)
260+
- Query commands: Results to stdout
261+
- Action commands: Progress to stderr, results to stdout
262+
- More complex but intuitive per command
263+
264+
4. **Structured Logging** (docker, terraform)
265+
- Support for log drivers/files
266+
- Environment variable control
267+
- Separate logging infrastructure
268+
269+
5. **Verbosity-Controlled** (most modern tools)
270+
- Default: Minimal, user-focused output
271+
- -v: More operational details
272+
- -vv/-vvv: Debug/trace information
273+
274+
### Verbosity Patterns
275+
276+
Most modern CLI tools follow this pattern:
277+
278+
- **Default**: Essential user information only
279+
- **-v/--verbose**: Operational details, progress info
280+
- **-vv**: Debug information, internal operations
281+
- **-vvv**: Trace-level, all internal details
282+
- **-q/--quiet**: Suppress all non-essential output
283+
284+
## Recommendations for Console Applications
285+
286+
Based on this research, here are the identified best practices:
287+
288+
### 1. Follow the "Results vs. Progress" Pattern
289+
290+
- **stdout**: Final results, data that users want to pipe/redirect
291+
- **stderr**: Progress updates, status messages, operational info
292+
293+
### 2. Implement Graduated Verbosity
294+
295+
```bash
296+
myapp command # Essential info only
297+
myapp command -v # + Progress/status
298+
myapp command -vv # + Debug info
299+
myapp command -vvv # + Trace/all details
300+
myapp command -q # Minimal output
301+
```
302+
303+
### 3. Consider Structured Logging for Complex Apps
304+
305+
- Log files for persistent diagnostics
306+
- JSON logs for machine parsing
307+
- Separate debug logs controlled by environment variables
308+
309+
### 4. Make Errors Actionable
310+
311+
- Error messages should go to stderr
312+
- Include context and suggested fixes
313+
- Distinguish between user errors and system errors
314+
315+
### 5. Support Automation Use Cases
316+
317+
- Quiet modes for scripting
318+
- Machine-readable output options (JSON, CSV)
319+
- Exit codes that reflect operation success/failure
320+
321+
## Application to Torrust Deployer
322+
323+
For the Torrust Tracker Deploy application, these patterns suggest:
324+
325+
### Recommended Approach
326+
327+
1. **User Output** (stdout):
328+
- Deployment results
329+
- Configuration summaries
330+
- Final status reports
331+
332+
2. **Progress/Operational** (stderr):
333+
- Step progress ("Provisioning instance...")
334+
- Status updates ("Instance ready")
335+
- Non-critical warnings
336+
337+
3. **Debug Logs** (stderr + optional files):
338+
- Detailed operation logs
339+
- Ansible/Terraform output
340+
- System diagnostics
341+
342+
4. **Errors** (stderr):
343+
- User errors with actionable guidance
344+
- System errors with recovery instructions
345+
346+
### Example Implementation
347+
348+
```bash
349+
# Default - clean user experience
350+
torrust-deploy provision env1
351+
✓ Instance provisioned successfully # to stderr (progress)
352+
Instance: env1-tracker (192.168.1.100) # to stdout (result)
353+
354+
# Verbose - show operational details
355+
torrust-deploy provision env1 -v
356+
→ Creating LXD instance... # to stderr
357+
→ Configuring network... # to stderr
358+
→ Installing packages... # to stderr
359+
✓ Instance provisioned successfully # to stderr
360+
Instance: env1-tracker (192.168.1.100) # to stdout
361+
362+
# Piping works cleanly
363+
torrust-deploy provision env1 | jq .ip_address
364+
torrust-deploy provision env1 -q > deployment.txt
365+
```
366+
367+
This approach follows the successful patterns used by cargo, docker, and other modern CLI tools while maintaining Unix convention compatibility.
368+
369+
## References
370+
371+
- [UNIX Philosophy - Rule of Silence](http://www.catb.org/~esr/writings/taoup/html/ch01s06.html#id2877917)
372+
- [GNU Coding Standards - Writing Robust Programs](https://www.gnu.org/prep/standards/html_node/Errors.html)
373+
- [Cargo's output handling](https://doc.rust-lang.org/cargo/reference/build-scripts.html#outputs-of-the-build-script)
374+
- [Docker logging drivers](https://docs.docker.com/config/containers/logging/)
375+
- [The Art of Command Line](https://github.com/jlevy/the-art-of-command-line)

‎project-words.txt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ cpus
1515
Crossplane
1616
dearmor
1717
debootstrap
18+
debuginfo
1819
distutils
1920
Dockerfiles
2021
dpkg
@@ -47,6 +48,7 @@ maxbytes
4748
mgmt
4849
mocksecret
4950
MVVM
51+
myapp
5052
myenv
5153
nameof
5254
newgrp

0 commit comments

Comments
 (0)