diff --git a/.changeset/literal-apostrophe-fix.md b/.changeset/literal-apostrophe-fix.md new file mode 100644 index 0000000..44e0a9f --- /dev/null +++ b/.changeset/literal-apostrophe-fix.md @@ -0,0 +1,22 @@ +--- +'command-stream': minor +--- + +Add `literal()` function to preserve apostrophes in shell arguments + +When passing text containing apostrophes to programs that store it literally (like API calls via CLI tools), apostrophes would appear corrupted as triple quotes (`'''`). The new `literal()` function uses double-quote escaping which preserves apostrophes while still escaping shell-dangerous characters. + +**New features:** + +- `literal(value)` - Mark text for double-quote escaping, preserving apostrophes +- `quoteLiteral(value)` - Low-level function for manual command building + +**Usage:** + +```javascript +import { $, literal } from 'command-stream'; + +// Apostrophes now preserved for API storage +const notes = "Dependencies didn't exist"; +await $`gh release create --notes ${literal(notes)}`; +``` diff --git a/README.md b/README.md index 26ba11b..7348ed1 100644 --- a/README.md +++ b/README.md @@ -254,6 +254,60 @@ await $`echo ${userInput}`; // ✅ Safe - auto-escaped - ❌ Any untrusted source - ❌ When you're unsure - use normal interpolation instead +### Preserving Apostrophes with `literal()` (Advanced) + +When passing text to programs that store it literally (like API calls via CLI tools), apostrophes can appear corrupted as triple quotes (`'''`). This happens because the default `quote()` function uses Bash's `'\''` escaping pattern for apostrophes. + +Use `literal()` to preserve apostrophes when the receiving program stores text literally: + +```javascript +import { $, literal } from 'command-stream'; + +// Problem: Default escaping uses '\'' pattern for apostrophes +const releaseNotes = "Fix bug when dependencies didn't exist"; +await $`gh release create v1.0.0 --notes ${releaseNotes}`; +// Apostrophe may appear as ''' in GitHub if the API stores it literally + +// Solution: Use literal() to preserve apostrophes +await $`gh release create v1.0.0 --notes ${literal(releaseNotes)}`; +// Apostrophe stays as ' - text displays correctly on GitHub + +// literal() still escapes shell-dangerous characters +const text = "User's input with $variable and `backticks`"; +await $`command ${literal(text)}`; +// $ and ` are escaped, but apostrophe stays as-is +``` + +**How `literal()` differs from `raw()` and default quoting:** + +| Function | Apostrophe `'` | Shell Chars `$ \` "` | Safety Level | +| ----------- | -------------- | -------------------- | ------------ | +| Default | `'\''` escaped | Safely quoted | ✅ Maximum | +| `literal()` | Preserved | Escaped | ✅ Safe | +| `raw()` | Preserved | NOT escaped | ⚠️ Dangerous | + +**When to use `literal()`:** + +- ✅ Text for APIs via CLI tools (GitHub, cloud CLIs) +- ✅ Release notes, commit messages, descriptions +- ✅ Any text that will be stored/displayed literally +- ✅ When apostrophes appear as `'''` in the output + +**Recommended Alternative: Use stdin for APIs** + +For API calls, the safest approach is to pass data via stdin: + +```javascript +const payload = JSON.stringify({ + tag_name: 'v1.0.0', + body: releaseNotes, // No escaping issues! +}); + +await $`gh api repos/owner/repo/releases -X POST --input -`.run({ + stdin: payload, +}); +``` + ## Usage Patterns ### Classic Await (Backward Compatible) @@ -1115,6 +1169,36 @@ await $`${raw(trustedCommand)}`; // ⚠️ NEVER use with untrusted input - shell injection risk! ``` +#### literal() - Preserve Apostrophes for Literal Storage + +Use when passing text to programs that store it literally (APIs, databases). Preserves apostrophes while still escaping shell-dangerous characters. + +```javascript +import { $, literal } from 'command-stream'; + +// Apostrophes preserved for API storage +const notes = "Dependencies didn't exist"; +await $`gh release create v1.0.0 --notes ${literal(notes)}`; +// → Apostrophe displays correctly on GitHub + +// Still safe: $ ` \ " are escaped +const text = "User's $variable"; +await $`command ${literal(text)}`; +// → $ is escaped, apostrophe preserved +``` + +#### quoteLiteral() - Low-level Double-Quote Escaping + +Low-level function for manual command building. Uses double quotes, preserving apostrophes. + +```javascript +import { quoteLiteral } from 'command-stream'; + +quoteLiteral("didn't"); // → "didn't" +quoteLiteral('say "hello"'); // → "say \"hello\"" +quoteLiteral('$100'); // → "\$100" +``` + ### Built-in Commands 18 cross-platform commands that work identically everywhere: diff --git a/docs/case-studies/issue-141/README.md b/docs/case-studies/issue-141/README.md new file mode 100644 index 0000000..c390fd4 --- /dev/null +++ b/docs/case-studies/issue-141/README.md @@ -0,0 +1,169 @@ +# Case Study: Apostrophe Over-Escaping (Issue #141) + +## Overview + +This case study documents the issue where apostrophes in text arguments are over-escaped when passed through command-stream, causing them to appear as triple quotes (`'''`) when the text is stored or displayed literally by receiving programs. + +## Problem Statement + +When passing text containing apostrophes through command-stream in double-quoted template literals, apostrophes are escaped using Bash's `'\''` pattern. When the receiving command (like `gh` CLI) passes this text to an API that stores it literally, the escape sequences appear as visible characters. + +### Example + +```javascript +const releaseNotes = "Fix bug when dependencies didn't exist"; +await $`gh release create v1.0.0 --notes "${releaseNotes}"`; +// GitHub receives: "Fix bug when dependencies didn'\''t exist" +// GitHub displays: "Fix bug when dependencies didn'''t exist" +``` + +## Timeline of Investigation + +1. **Initial report**: Issue observed in production with GitHub release notes +2. **Related issue**: First documented in test-anywhere repository (issue #135) +3. **Workaround implemented**: Using `gh api` with JSON stdin instead of shell arguments +4. **Root cause identified**: Double-quoting when users add quotes around interpolated values + +## Root Cause Analysis + +### The Escaping Mechanism + +The `quote()` function in command-stream (js/src/$.mjs:1056-1100) handles shell escaping: + +```javascript +function quote(value) { + // ... null/array handling ... + + // Default: wrap in single quotes, escape internal single quotes + return `'${value.replace(/'/g, "'\\''")}'`; +} +``` + +For input `didn't`, this produces `'didn'\''t'`, which is the correct Bash escaping for a single quote inside a single-quoted string. + +### The Double-Quoting Issue + +When users write: + +```javascript +await $`command "${text}"`; +``` + +The template literal contains `"` characters as static strings. The `buildShellCommand()` function then: + +1. Adds the static string `command "` +2. Calls `quote(text)` which wraps in single quotes +3. Adds the closing static string `"` + +Result: `command "'escaped'\''text'"` + +This creates double-quoting - the user's `"..."` plus the library's `'...'`. + +### Experimental Evidence + +``` +Test: Direct shell (baseline) +Command: /tmp/show-args.sh "Text with apostrophe's" +Result: [Text with apostrophe's] ✅ + +Test: With user-provided quotes (the bug) +Command: $`/tmp/show-args.sh "${testText}"` +Result: ['Dependencies didn'\''t exist'] ❌ + +Test: Without user quotes (correct usage) +Command: $`/tmp/show-args.sh ${testText}` +Result: [Dependencies didn't exist] ✅ +``` + +### Why Triple Quotes Appear + +When a program receives `'didn'\''t'`: + +1. If it **interprets** as shell → expands to `didn't` ✅ +2. If it **stores literally** → keeps as `'didn'\''t'` which displays as `didn'''t` ❌ + +The `gh` CLI passes arguments to GitHub's API, which stores them literally without shell interpretation. + +## Solutions + +### Solution 1: Correct Usage (No User Quotes) + +The simplest solution is to not add quotes around interpolated values: + +```javascript +// ❌ Wrong - double quoting +await $`gh release create --notes "${text}"`; + +// ✅ Correct - let command-stream quote +await $`gh release create --notes ${text}`; +``` + +### Solution 2: literal() Function (Proposed) + +Add a `literal()` function for cases where text should not be shell-escaped: + +```javascript +import { $, literal } from 'command-stream'; + +// Mark text as literal - minimal escaping for argument boundary only +await $`gh release create --notes ${literal(releaseNotes)}`; +``` + +This would: + +- Apply only the minimal escaping needed for argument boundaries +- Not apply Bash-specific patterns like `'\''` +- Be useful when the receiving program stores text literally + +### Solution 3: Use stdin with JSON (Recommended for APIs) + +For API calls, pass data via stdin: + +```javascript +const payload = JSON.stringify({ + tag_name: 'v1.0.0', + body: releaseNotes, +}); + +await $`gh api repos/owner/repo/releases -X POST --input -`.run({ + stdin: payload, +}); +``` + +This completely bypasses shell escaping issues. + +## Implementation Decision + +Based on the issue suggestions and analysis: + +1. **Implement `literal()` function** - For marking text that should not be shell-escaped +2. **Improve documentation** - Clarify when and how shell escaping occurs +3. **Add examples** - Show correct usage patterns and workarounds + +## Related Issues and References + +- **Issue #141**: This issue - Apostrophe over-escaping +- **Issue #45**: Automatic quote addition in interpolation +- **Issue #49**: Complex shell commands with nested quotes +- **test-anywhere #135**: First observed occurrence +- **test-anywhere PR #136**: Workaround using stdin/JSON + +## Lessons Learned + +1. Shell escaping and literal text storage are incompatible +2. Users should not add quotes around interpolated values +3. For API calls, JSON/stdin is the safest approach +4. Clear documentation and examples are essential + +## Test Cases + +A proper fix should handle: + +| Input | Expected Output | +| -------------------- | -------------------- | +| `didn't` | `didn't` | +| `it's user's choice` | `it's user's choice` | +| `text is "quoted"` | `text is "quoted"` | +| `it's "great"` | `it's "great"` | +| `` use `npm` `` | `` use `npm` `` | +| `Line 1\nLine 2` | `Line 1\nLine 2` | diff --git a/eslint.config.js b/eslint.config.js index 45be7aa..efa51a2 100644 --- a/eslint.config.js +++ b/eslint.config.js @@ -144,12 +144,14 @@ export default [ }, }, { - // Example and debug files are more lenient + // Example, experiment, and debug files are more lenient files: [ 'examples/**/*.js', 'examples/**/*.mjs', 'js/examples/**/*.js', 'js/examples/**/*.mjs', + 'experiments/**/*.js', + 'experiments/**/*.mjs', 'claude-profiles.mjs', ], rules: { diff --git a/experiments/test-apostrophe-escaping.mjs b/experiments/test-apostrophe-escaping.mjs new file mode 100755 index 0000000..c418e7f --- /dev/null +++ b/experiments/test-apostrophe-escaping.mjs @@ -0,0 +1,93 @@ +#!/usr/bin/env node +/** + * Experiment to test apostrophe escaping issue (#141) + * + * This script demonstrates the problem: + * - Apostrophes in text are escaped using Bash's '\'' pattern + * - When the text is passed to APIs that store it literally, the escape sequence appears + * - Result: "didn't" becomes "didn'''t" + */ + +import { $, raw } from '../js/src/$.mjs'; + +console.log('=== Apostrophe Escaping Issue (#141) ===\n'); + +// Test cases from the issue +const testCases = [ + { input: "didn't", description: 'Basic apostrophe' }, + { input: "it's user's choice", description: 'Multiple apostrophes' }, + { input: 'text is "quoted"', description: 'Double quotes' }, + { input: 'it\'s "great"', description: 'Mixed quotes' }, + { input: 'use `npm install`', description: 'Backticks' }, + { input: 'Line 1\nLine 2', description: 'Newlines' }, +]; + +console.log('Testing echo command with interpolated text:\n'); + +for (const { input, description } of testCases) { + console.log(`--- ${description} ---`); + console.log(`Input: "${input}"`); + + try { + // Test with standard interpolation (shows the escaping issue) + const result = await $`echo "${input}"`.run({ + capture: true, + mirror: false, + }); + console.log(`Output: "${result.stdout.trim()}"`); + + // Check if output matches input + const matches = result.stdout.trim() === input; + console.log(`Matches: ${matches ? '✅ YES' : '❌ NO'}`); + + if (!matches) { + console.log(`Expected: "${input}"`); + console.log(`Got: "${result.stdout.trim()}"`); + } + } catch (err) { + console.log(`Error: ${err.message}`); + } + + console.log(''); +} + +// Now let's see what the actual shell command looks like +console.log('\n=== Internal Command Analysis ===\n'); + +// We can trace to see the actual command being built +const testText = "didn't exist"; +console.log(`Test text: "${testText}"`); + +// Check what happens with different quoting approaches +console.log( + '\n--- Using double-quoted template literal: $`echo "${text}"` ---' +); +const result1 = await $`echo "${testText}"`.run({ + capture: true, + mirror: false, +}); +console.log(`Result: "${result1.stdout.trim()}"`); + +console.log('\n--- Using raw(): $`echo ${raw(text)}` ---'); +const result2 = await $`echo ${raw(testText)}`.run({ + capture: true, + mirror: false, +}); +console.log(`Result: "${result2.stdout.trim()}"`); + +console.log('\n--- Using plain interpolation: $`echo ${text}` ---'); +const result3 = await $`echo ${testText}`.run({ capture: true, mirror: false }); +console.log(`Result: "${result3.stdout.trim()}"`); + +console.log('\n=== Summary ===\n'); +console.log('The issue occurs because:'); +console.log('1. Text with apostrophes is passed to command-stream'); +console.log("2. command-stream uses single-quote escaping: ' → '\\''"); +console.log('3. The shell correctly interprets this for echo'); +console.log( + '4. But when passed to APIs (like gh CLI), the API receives/stores' +); +console.log(' the escaped form, not the interpreted result'); +console.log( + '\nWorkaround: Use stdin with JSON for API calls (see issue for details)' +); diff --git a/experiments/test-gh-simulation-v2.mjs b/experiments/test-gh-simulation-v2.mjs new file mode 100644 index 0000000..523064e --- /dev/null +++ b/experiments/test-gh-simulation-v2.mjs @@ -0,0 +1,63 @@ +#!/usr/bin/env node +/** + * Simulate the gh CLI receiving arguments to understand the issue + */ + +import { $, raw } from '../js/src/$.mjs'; +import fs from 'fs/promises'; + +console.log('=== Simulating What gh CLI Would Receive ===\n'); + +const testText = "Dependencies didn't exist"; + +// Create a script that echoes its arguments +const scriptPath = '/tmp/show-args.sh'; +await fs.writeFile( + scriptPath, + `#!/bin/bash +echo "Number of args: $#" +for arg in "$@"; do + echo "Arg: [$arg]" +done +` +); +await fs.chmod(scriptPath, '755'); + +console.log('1. Direct shell command (baseline - no interpolation):'); +const result1 = await $`/tmp/show-args.sh "Text with apostrophe's"`.run({ + capture: true, + mirror: false, +}); +console.log(result1.stdout); + +console.log('2. Using interpolation WITH user-provided quotes (the bug):'); +const result2 = await $`/tmp/show-args.sh "${testText}"`.run({ + capture: true, + mirror: false, +}); +console.log(result2.stdout); + +console.log('3. Using interpolation WITHOUT user quotes (correct usage):'); +const result3 = await $`/tmp/show-args.sh ${testText}`.run({ + capture: true, + mirror: false, +}); +console.log(result3.stdout); + +console.log('4. Using raw() function:'); +const result4 = await $`/tmp/show-args.sh ${raw(`"${testText}"`)}`.run({ + capture: true, + mirror: false, +}); +console.log(result4.stdout); + +// Cleanup +await fs.unlink(scriptPath); + +console.log('=== Analysis ==='); +console.log('Test 1 shows expected output - shell correctly handles quotes'); +console.log( + 'Test 2 shows double-quoting issue when user adds quotes + library adds quotes' +); +console.log('Test 3 shows correct usage - let the library handle quoting'); +console.log("Test 4 shows raw() preserves the user's exact text"); diff --git a/experiments/test-gh-simulation.mjs b/experiments/test-gh-simulation.mjs new file mode 100644 index 0000000..09a2afc --- /dev/null +++ b/experiments/test-gh-simulation.mjs @@ -0,0 +1,95 @@ +#!/usr/bin/env node +/** + * Simulate the gh CLI receiving arguments to understand the issue + * + * When you run: gh release create v1.0.0 --notes "text with apostrophe's" + * The shell expands it, and gh receives the expanded text + * + * But when you run via command-stream: + * $`gh release create v1.0.0 --notes "${text}"` + * + * What does gh actually receive? + */ + +import { $, raw } from '../js/src/$.mjs'; +import { spawn } from 'child_process'; +import { promisify } from 'util'; + +console.log('=== Simulating What gh CLI Would Receive ===\n'); + +const testText = "Dependencies didn't exist"; + +// Method 1: Create a script that echoes its arguments +console.log('1. Creating argument inspection script...'); + +// Write a simple script that shows exactly what arguments it receives +const scriptContent = `#!/bin/bash +echo "Number of args: $#" +for arg in "$@"; do + echo "Arg: [$arg]" +done +`; + +await $`echo ${raw(`'${scriptContent}'`)} > /tmp/show-args.sh && chmod +x /tmp/show-args.sh`.run( + { capture: true, mirror: false } +); + +console.log('\n2. Testing direct shell (how user expects it to work):'); +const result1 = + await $`/tmp/show-args.sh "This is ${raw("apostrophe's")} text"`.run({ + capture: true, + mirror: false, + }); +console.log(result1.stdout); + +console.log('3. Testing with interpolation (what actually happens):'); +const result2 = await $`/tmp/show-args.sh "This is ${testText}"`.run({ + capture: true, + mirror: false, +}); +console.log(result2.stdout); + +console.log('4. Testing proper usage WITHOUT user quotes:'); +const result3 = await $`/tmp/show-args.sh ${testText}`.run({ + capture: true, + mirror: false, +}); +console.log(result3.stdout); + +console.log('5. Testing with raw():'); +const result4 = await $`/tmp/show-args.sh ${raw(`"${testText}"`)}`.run({ + capture: true, + mirror: false, +}); +console.log(result4.stdout); + +console.log('\n=== Key Finding ==='); +console.log("The issue is NOT in command-stream's escaping mechanism itself."); +console.log( + 'The quote() function correctly escapes single quotes for the shell.' +); +console.log('\nThe issue is that when users write:'); +console.log(' $`gh release create --notes "${text}"`'); +console.log(''); +console.log('They are DOUBLE-quoting:'); +console.log(' 1. Their " " quotes are in the template string'); +console.log( + " 2. command-stream adds ' ' quotes around the interpolated value" +); +console.log(''); +console.log('So the command becomes:'); +console.log(" gh release create --notes \"'escaped'\\''text'\""); +console.log(''); +console.log('The correct usage is:'); +console.log(' $`gh release create --notes ${text}`'); +console.log('(Let command-stream handle the quoting!)'); + +console.log('\n=== When Does Triple Quote Appear? ==='); +console.log( + "If the shell command is passed to a program that doesn't interpret" +); +console.log('the escaping, but stores/forwards the text literally, then the'); +console.log("escape sequence '\\'' appears as '''."); + +// Cleanup +await $`rm /tmp/show-args.sh`.run({ capture: true, mirror: false }); diff --git a/experiments/test-literal-function.mjs b/experiments/test-literal-function.mjs new file mode 100644 index 0000000..cb6a0d0 --- /dev/null +++ b/experiments/test-literal-function.mjs @@ -0,0 +1,123 @@ +#!/usr/bin/env node +/** + * Test the new literal() function for preserving apostrophes + */ + +import { $, literal, quoteLiteral } from '../js/src/$.mjs'; +import fs from 'fs/promises'; + +console.log('=== Testing literal() Function ===\n'); + +// Test cases from the issue +const testCases = [ + { input: "didn't", description: 'Basic apostrophe' }, + { input: "it's user's choice", description: 'Multiple apostrophes' }, + { input: 'text is "quoted"', description: 'Double quotes' }, + { input: 'it\'s "great"', description: 'Mixed quotes' }, + { input: 'use `npm install`', description: 'Backticks' }, + { input: 'Line 1\nLine 2', description: 'Newlines' }, + { input: 'price is $100', description: 'Dollar sign' }, + { input: 'path\\to\\file', description: 'Backslashes' }, +]; + +// Create a script that echoes its arguments exactly +const scriptPath = '/tmp/show-args.sh'; +await fs.writeFile( + scriptPath, + `#!/bin/bash +for arg in "$@"; do + echo "$arg" +done +` +); +await fs.chmod(scriptPath, '755'); + +console.log('Testing quoteLiteral() function directly:\n'); +for (const { input, description } of testCases) { + const quoted = quoteLiteral(input); + console.log(`${description}:`); + console.log(` Input: "${input}"`); + console.log(` Quoted: ${quoted}`); + console.log(''); +} + +console.log('\n=== Testing with shell execution ===\n'); + +let passCount = 0; +let failCount = 0; + +for (const { input, description } of testCases) { + console.log(`--- ${description} ---`); + console.log(`Input: "${input.replace(/\n/g, '\\n')}"`); + + try { + // Test with literal() function + const result = await $`/tmp/show-args.sh ${literal(input)}`.run({ + capture: true, + mirror: false, + }); + + const output = result.stdout.trim(); + const matches = output === input; + + console.log(`Output: "${output.replace(/\n/g, '\\n')}"`); + console.log(`Match: ${matches ? '✅ PASS' : '❌ FAIL'}`); + + if (matches) { + passCount++; + } else { + failCount++; + console.log(`Expected: "${input.replace(/\n/g, '\\n')}"`); + console.log(`Got: "${output.replace(/\n/g, '\\n')}"`); + } + } catch (err) { + console.log(`Error: ${err.message}`); + failCount++; + } + + console.log(''); +} + +// Cleanup +await fs.unlink(scriptPath); + +console.log('=== Summary ==='); +console.log(`Passed: ${passCount}/${testCases.length}`); +console.log(`Failed: ${failCount}/${testCases.length}`); + +// Compare with regular quote() behavior +console.log('\n=== Comparison: quote() vs literal() ===\n'); + +const comparisonText = "Dependencies didn't exist"; +console.log(`Text: "${comparisonText}"`); + +// Create script again for comparison +await fs.writeFile( + scriptPath, + `#!/bin/bash +for arg in "$@"; do + echo "$arg" +done +` +); +await fs.chmod(scriptPath, '755'); + +const regularResult = await $`/tmp/show-args.sh ${comparisonText}`.run({ + capture: true, + mirror: false, +}); +console.log(`\nWith regular quote() (default):`); +console.log(` Result: "${regularResult.stdout.trim()}"`); + +const literalResult = await $`/tmp/show-args.sh ${literal(comparisonText)}`.run( + { + capture: true, + mirror: false, + } +); +console.log(`\nWith literal():`); +console.log(` Result: "${literalResult.stdout.trim()}"`); + +await fs.unlink(scriptPath); + +process.exit(failCount > 0 ? 1 : 0); diff --git a/experiments/test-shell-escaping-detailed.mjs b/experiments/test-shell-escaping-detailed.mjs new file mode 100644 index 0000000..7962f5c --- /dev/null +++ b/experiments/test-shell-escaping-detailed.mjs @@ -0,0 +1,90 @@ +#!/usr/bin/env node +/** + * Detailed investigation of shell escaping behavior + */ + +import { $, raw } from '../js/src/$.mjs'; + +console.log('=== Detailed Shell Escaping Investigation ===\n'); + +const testText = "didn't"; + +// Test 1: Direct shell command (baseline) +console.log('1. Direct shell command without interpolation:'); +const result1 = await $`echo "didn't"`.run({ capture: true, mirror: false }); +console.log(` Result: "${result1.stdout.trim()}"`); +console.log(` Expected: "didn't"`); +console.log(` Correct: ${result1.stdout.trim() === "didn't" ? '✅' : '❌'}`); + +// Test 2: With interpolation and user-provided double quotes +console.log( + '\n2. Interpolation with user-provided double quotes: $`echo "${text}"`' +); +const result2 = await $`echo "${testText}"`.run({ + capture: true, + mirror: false, +}); +console.log(` Result: "${result2.stdout.trim()}"`); + +// Test 3: With interpolation (no quotes around variable) +console.log('\n3. Interpolation without quotes: $`echo ${text}`'); +const result3 = await $`echo ${testText}`.run({ capture: true, mirror: false }); +console.log(` Result: "${result3.stdout.trim()}"`); + +// Test 4: Using printf to see the exact bytes +console.log('\n4. Using sh -c to verify shell interpretation:'); +const result4 = await $`sh -c 'echo "didn'"'"'t"'`.run({ + capture: true, + mirror: false, +}); +console.log(` Result: "${result4.stdout.trim()}"`); + +// Test 5: Let's manually test the quote function logic +console.log('\n5. Understanding the escaping chain:'); +console.log(` Input text: "${testText}"`); +console.log( + ` The quote() function produces: '${testText.replace(/'/g, "'\\''")}'` +); +console.log( + ` This should expand to: ${testText} when interpreted by the shell` +); + +// Test 6: Test echo with properly escaped single quote +console.log('\n6. Test if the shell correctly expands the escape:'); +const shellCmd = `echo '${testText.replace(/'/g, "'\\''")}'`; +console.log(` Command: ${shellCmd}`); +const result6 = await $`sh -c ${shellCmd}`.run({ + capture: true, + mirror: false, +}); +console.log(` Result: "${result6.stdout.trim()}"`); + +// Test 7: What command is actually being built? +console.log('\n7. Inspecting the actual command being built:'); +const verbose = process.env.COMMAND_STREAM_VERBOSE; +process.env.COMMAND_STREAM_VERBOSE = 'true'; +// Just note: we can't easily inspect the built command without modifying the library +// but we know from the code that buildShellCommand is called +console.log( + ` Note: quote("${testText}") returns: '${testText.replace(/'/g, "'\\''")}' (single-quoted with escaped apostrophe)` +); +process.env.COMMAND_STREAM_VERBOSE = verbose; + +// Test 8: Direct execution of expected result +console.log('\n8. Direct execution with raw:'); +const result8 = await $`${raw(`echo '${testText}'`)}`.run({ + capture: true, + mirror: false, +}); +console.log(` Using raw("echo \'${testText}\'"): "${result8.stdout.trim()}"`); + +console.log('\n=== Analysis ===\n'); +console.log('The key insight is:'); +console.log('- When we use $`echo "${testText}"`, command-stream:'); +console.log(' 1. Sees the " as part of the template string'); +console.log(' 2. Quotes the interpolated value with single quotes'); +console.log(" 3. The resulting command is: echo \"'didn'\\''t'\""); +console.log(' 4. This has DOUBLE quoting: user"s quotes + library\'s quotes'); +console.log(''); +console.log('The real issue is that the user\'s " quotes are part of the'); +console.log('static template string, and then the library adds MORE quotes!'); diff --git a/js/src/$.mjs b/js/src/$.mjs index 445264d..5b9e982 100755 --- a/js/src/$.mjs +++ b/js/src/$.mjs @@ -1099,6 +1099,52 @@ function quote(value) { return `'${value.replace(/'/g, "'\\''")}'`; } +/** + * Quote a value using double quotes - preserves apostrophes as-is. + * + * Use this when the text will be passed to programs that store it literally + * (like API calls via CLI tools) rather than interpreting it as shell commands. + * + * In double quotes, we only need to escape: $ ` \ " and newlines + * Apostrophes (') are preserved without escaping. + * + * @param {*} value - The value to quote + * @returns {string} - The double-quoted string with proper escaping + */ +function quoteLiteral(value) { + if (value == null) { + return '""'; + } + if (Array.isArray(value)) { + return value.map(quoteLiteral).join(' '); + } + if (typeof value !== 'string') { + value = String(value); + } + if (value === '') { + return '""'; + } + + // Check if the string needs quoting at all + // Safe characters: alphanumeric, dash, underscore, dot, slash, colon, equals, comma, plus + const safePattern = /^[a-zA-Z0-9_\-./=,+@:]+$/; + + if (safePattern.test(value)) { + // The string is safe and doesn't need quoting + return value; + } + + // Escape characters that are special inside double quotes: \ $ ` " + // Apostrophes (') do NOT need escaping in double quotes + const escaped = value + .replace(/\\/g, '\\\\') // Escape backslashes first + .replace(/\$/g, '\\$') // Escape dollar signs (prevent variable expansion) + .replace(/`/g, '\\`') // Escape backticks (prevent command substitution) + .replace(/"/g, '\\"'); // Escape double quotes + + return `"${escaped}"`; +} + function buildShellCommand(strings, values) { trace( 'Utils', @@ -1151,6 +1197,19 @@ function buildShellCommand(strings, values) { `BRANCH: buildShellCommand => RAW_VALUE | ${JSON.stringify({ value: String(v.raw) }, null, 2)}` ); out += String(v.raw); + } else if ( + v && + typeof v === 'object' && + Object.prototype.hasOwnProperty.call(v, 'literal') + ) { + // Use double-quote escaping which preserves apostrophes + const literalQuoted = quoteLiteral(v.literal); + trace( + 'Utils', + () => + `BRANCH: buildShellCommand => LITERAL_VALUE | ${JSON.stringify({ original: v.literal, quoted: literalQuoted }, null, 2)}` + ); + out += literalQuoted; } else { const quoted = quote(v); trace( @@ -6474,6 +6533,34 @@ function raw(value) { return { raw: String(value) }; } +/** + * Mark a value as literal text that should use double-quote escaping. + * + * Use this when passing text to programs that store it literally (like API calls + * via CLI tools) rather than interpreting it as shell commands. This preserves + * apostrophes as-is instead of using the '\'' escape pattern. + * + * Unlike raw(), literal() still provides proper shell escaping for special + * characters like $, `, \, and ", but apostrophes pass through unchanged. + * + * @example + * // Problem: apostrophe gets escaped as '\'' + * await $`gh release create --notes ${text}`; // "didn't" → "didn'\''t" → appears as "didn'''t" + * + * // Solution: use literal() to preserve apostrophes + * await $`gh release create --notes ${literal(text)}`; // "didn't" stays "didn't" + * + * @param {*} value - The value to mark as literal + * @returns {{ literal: string }} - Object with literal property for buildShellCommand + */ +function literal(value) { + trace( + 'API', + () => `literal() called with value: ${String(value).slice(0, 50)}` + ); + return { literal: String(value) }; +} + function set(option) { trace('API', () => `set() called with option: ${option}`); const mapping = { @@ -6744,8 +6831,10 @@ export { exec, run, quote, + quoteLiteral, create, raw, + literal, ProcessRunner, shell, set, diff --git a/js/tests/$.test.mjs b/js/tests/$.test.mjs index 6552298..8ed3461 100644 --- a/js/tests/$.test.mjs +++ b/js/tests/$.test.mjs @@ -6,8 +6,10 @@ import { exec, run, quote, + quoteLiteral, create, raw, + literal, ProcessRunner, shell, disableVirtualCommands, @@ -200,6 +202,59 @@ describe('Utility Functions', () => { expect(raw(123)).toEqual({ raw: '123' }); }); }); + + describe('quoteLiteral', () => { + test('should preserve apostrophes', () => { + expect(quoteLiteral("didn't")).toBe('"didn\'t"'); + expect(quoteLiteral("it's user's")).toBe('"it\'s user\'s"'); + }); + + test('should escape double quotes', () => { + expect(quoteLiteral('say "hello"')).toBe('"say \\"hello\\""'); + }); + + test('should escape dollar signs', () => { + expect(quoteLiteral('price $100')).toBe('"price \\$100"'); + }); + + test('should escape backticks', () => { + expect(quoteLiteral('use `npm`')).toBe('"use \\`npm\\`"'); + }); + + test('should escape backslashes', () => { + expect(quoteLiteral('path\\to\\file')).toBe('"path\\\\to\\\\file"'); + }); + + test('should handle empty string', () => { + expect(quoteLiteral('')).toBe('""'); + }); + + test('should handle null/undefined', () => { + expect(quoteLiteral(null)).toBe('""'); + expect(quoteLiteral(undefined)).toBe('""'); + }); + + test('should not quote safe strings', () => { + expect(quoteLiteral('hello')).toBe('hello'); + expect(quoteLiteral('file.txt')).toBe('file.txt'); + expect(quoteLiteral('/path/to/file')).toBe('/path/to/file'); + }); + + test('should handle arrays', () => { + expect(quoteLiteral(["it's", 'hello'])).toBe('"it\'s" hello'); + }); + }); + + describe('literal', () => { + test('should create literal object', () => { + const result = literal("didn't"); + expect(result).toEqual({ literal: "didn't" }); + }); + + test('should convert to string', () => { + expect(literal(123)).toEqual({ literal: '123' }); + }); + }); }); describe('ProcessRunner - Classic Await Pattern', () => { @@ -234,6 +289,37 @@ describe('ProcessRunner - Classic Await Pattern', () => { expect(result.stdout.trim()).toBe('raw test'); }); + test('should handle literal interpolation - preserves apostrophes', async () => { + // This is the key test for issue #141 + // literal() should preserve apostrophes without '\'' escaping + const text = "Dependencies didn't exist"; + const result = await $`echo ${literal(text)}`; + + expect(result.stdout.trim()).toBe(text); + }); + + test('should handle literal interpolation - multiple apostrophes', async () => { + const text = "it's the user's choice"; + const result = await $`echo ${literal(text)}`; + + expect(result.stdout.trim()).toBe(text); + }); + + test('should handle literal interpolation - mixed quotes', async () => { + const text = 'it\'s "great"'; + const result = await $`echo ${literal(text)}`; + + expect(result.stdout.trim()).toBe(text); + }); + + test('should handle literal interpolation - special characters', async () => { + // Dollar signs and backticks should be escaped to prevent shell expansion + const text = 'price is $100'; + const result = await $`echo ${literal(text)}`; + + expect(result.stdout.trim()).toBe(text); + }); + test('should quote dangerous characters', async () => { const dangerous = "'; rm -rf /; echo '"; const result = await $`echo ${dangerous}`; diff --git a/temp-unicode-test.txt b/temp-unicode-test.txt deleted file mode 100644 index e69de29..0000000