feat(pii): add deterministic redaction pipeline with gated overrides

Implement a full PII redaction system across CLI read outputs, with configurable modes, per-request override controls, deterministic identity replacement, and comprehensive docs/tests.

Highlights:

- Add new redaction engine in internal/pii:

  - policy handling (off|customers|all)

  - deterministic pseudonymization (with optional HS_PII_SECRET)

  - layered free-text regex redaction

  - recursive JSON walker for structured + text fields

- Add config/env support:

  - pii_mode / HS_PII_MODE

  - pii_allow_unredacted / HS_PII_ALLOW_UNREDACTED

- Add global inbox flag:

  - --unredacted (strictly gated by allow setting when redaction is enabled)

- Integrate redaction into command output paths:

  - conversations, threads, customers, users, teams, organizations, tools

  - table/csv/json/json-full

  - protect threads source and threads source-rfc822

- Add command helpers for effective mode resolution and safe raw output wrapping

- Expand tests:

  - new internal/pii unit tests

  - config tests for new keys/validation

  - cmd tests for redaction behavior and override gating

- Update README:

  - dedicated PII Redaction Pipeline section

  - trust/security callout near top

  - usage + disable/unset examples (--pii-mode off, --pii-allow-unredacted=false)

Author: rmorse

README.md

@@ -2,6 +2,10 @@
 
 A command-line interface for the [HelpScout](https://www.helpscout.com/) API. Manage mailboxes, conversations, customers, tags, users, workflows, and webhooks from the terminal.
 
+> **Built for shared and AI-assisted workflows**
+> hs-cli ships with a deterministic, layered PII redaction pipeline (structured fields + free-text + source payload protection), plus strict per-command override controls.  
+> See [PII Redaction Pipeline](#pii-redaction-pipeline).
+
 ## Install
 
 ```bash
@@ -49,13 +53,16 @@ This prompts for your Client ID and Secret, validates them against the API, and
 ```bash
 hs inbox config set --client-id your-client-id --client-secret your-client-secret
 hs inbox config set --default-mailbox 12345 --format json
+hs inbox config set --pii-mode customers --pii-allow-unredacted
 ```
 
 ### Environment variables
 
 ```bash
 export HS_CLIENT_ID=your-client-id
 export HS_CLIENT_SECRET=your-client-secret
+export HS_PII_MODE=customers
+export HS_PII_ALLOW_UNREDACTED=1
 ```
 
 ### Config file
@@ -67,6 +74,8 @@ client_id: your-client-id
 client_secret: your-client-secret
 default_mailbox: 12345
 format: table
+pii_mode: off
+pii_allow_unredacted: false
 ```
 
 **Credential resolution order:** environment variables > OS keyring > config file.
@@ -93,9 +102,79 @@ hs inbox auth logout
 | `--page` | Page number | `1` |
 | `--per-page` | Results per page | `25` |
 | `--debug` | Log HTTP requests/responses to `hs-debug.log` | `false` |
+| `--unredacted` | Disable PII redaction for this command when allowed by config/env | `false` |
 
 The `--format` flag can also be set permanently via the `format` key in the config file, or the `HS_FORMAT` environment variable.
 
+## PII Redaction Pipeline
+
+hs-cli includes a production-focused PII redaction system designed for shared terminals, MCP/LLM workflows, and incident-safe exports.
+
+### Why this matters
+
+- Prevents accidental exposure of customer/agent data in terminal output.
+- Keeps output useful for debugging and triage by preserving structure and readability.
+- Gives operators explicit control: strict defaults with an auditable per-command escape hatch.
+
+### Depth of protection
+
+Redaction is applied in layered stages:
+
+1. Structured identity redaction
+- Redacts known person/customer/user fields (names, emails, phones) across table, csv, json, and json-full outputs.
+- Covers nested payloads through a JSON walker.
+
+2. Free-text redaction
+- Scans thread/content text (`body`, `action`, `subject`, `preview`, source payloads) with a broad regex pipeline.
+- Detects and replaces common PII classes such as emails, phones, SSNs, card-like values, addresses, IPs, URLs, and person-name patterns.
+
+3. Raw source protection
+- `threads source` and `threads source-rfc822` are redacted when PII mode is enabled.
+
+### Deterministic anonymization and cache behavior
+
+- Replacements are deterministic: the same input maps to the same pseudonym across commands and runs.
+- `HS_PII_SECRET` (optional) adds a secret salt for stronger pseudonym generation.
+- Without `HS_PII_SECRET`, deterministic hashing is still used (stable fallback).
+- The engine also keeps an in-memory per-command cache so repeated values in the same output are co-derived consistently and processed efficiently.
+
+### Modes and override controls
+
+PII mode:
+
+- `off`: no redaction (default)
+- `customers`: redact customer identities
+- `all`: redact customer + user identities
+
+Override policy:
+
+- `--unredacted` disables redaction for a single command
+- It is only allowed when `pii_allow_unredacted: true` or `HS_PII_ALLOW_UNREDACTED=1`
+- If overrides are disallowed and redaction is enabled, the command fails fast with a clear error
+
+### Quick-start examples
+
+```bash
+# Enable customer-only redaction globally
+hs inbox config set --pii-mode customers
+
+# Allow temporary per-command bypasses (for incident response/debugging)
+hs inbox config set --pii-allow-unredacted
+
+# Run safely redacted output
+hs inbox conversations list --format json
+hs inbox conversations threads source-rfc822 12345 67890
+
+# Temporarily bypass redaction for one command (only if allowed)
+hs inbox --unredacted conversations get 12345 --format json-full
+
+# Disable per-command bypass again
+hs inbox config set --pii-allow-unredacted=false
+
+# Fully disable redaction
+hs inbox config set --pii-mode off --pii-allow-unredacted=false
+```
+
 ## Commands
 
 Inbox API commands are namespaced under `hs inbox ...`.
@@ -107,6 +186,12 @@ Inbox API commands are namespaced under `hs inbox ...`.
 hs inbox config set --client-id xxx --client-secret yyy
 hs inbox config set --default-mailbox 12345
 hs inbox config set --format json
+hs inbox config set --pii-mode customers
+hs inbox config set --pii-allow-unredacted
+
+# Disable/unset PII features explicitly
+hs inbox config set --pii-allow-unredacted=false
+hs inbox config set --pii-mode off --pii-allow-unredacted=false
 
 # View all config values
 hs inbox config get
@@ -126,6 +211,8 @@ hs inbox config path
 | `--client-secret` | string | HelpScout Client Secret |
 | `--default-mailbox` | int | Default mailbox ID |
 | `--format` | string | Output format: table, json, json-full, csv |
+| `--pii-mode` | string | PII redaction mode: off, customers, all |
+| `--pii-allow-unredacted` | bool | Allow per-request `--unredacted` override |
 
 ### Self-update
 
@@ -934,6 +1021,8 @@ Use `hs inbox config set` to write values, `hs inbox config get` to read them, a
 | `client_secret` | `--client-secret` | HelpScout Client Secret |
 | `default_mailbox` | `--default-mailbox` | Auto-filter conversations to this mailbox |
 | `format` | `--format` | Output format: `table`, `json`, `json-full`, or `csv` |
+| `pii_mode` | `--pii-mode` | PII redaction mode: `off`, `customers`, `all` |
+| `pii_allow_unredacted` | `--pii-allow-unredacted` | Allow `--unredacted` to bypass redaction per request |
 
 ### Environment variables
 
@@ -942,6 +1031,9 @@ Use `hs inbox config set` to write values, `hs inbox config get` to read them, a
 | `HS_CLIENT_ID` | `client_id` |
 | `HS_CLIENT_SECRET` | `client_secret` |
 | `HS_FORMAT` | `format` |
+| `HS_PII_MODE` | `pii_mode` |
+| `HS_PII_ALLOW_UNREDACTED` | `pii_allow_unredacted` |
+| `HS_PII_SECRET` | Optional secret salt for deterministic pseudonyms |
 | `HS_INBOX_PERMISSIONS` | `inbox_permissions` |
 | `HS_NO_UPDATE_CHECK` | Disable daily update check (`1`) |
 

internal/cmd/cmd_test.go

@@ -43,6 +43,7 @@ func saveRestore(t *testing.T) {
 	origCfgPath := cfgPath
 	origApiClient := apiClient
 	origFormat := format
+	origUnredacted := unredacted
 	origNoPaginate := noPaginate
 	origPage := page
 	origPerPage := perPage
@@ -54,6 +55,8 @@ func saveRestore(t *testing.T) {
 	origSetClientSecret := setClientSecret
 	origSetDefaultMailbox := setDefaultMailbox
 	origSetFormat := setFormat
+	origSetPIIMode := setPIIMode
+	origSetPIIAllowRaw := setPIIAllowRaw
 
 	selfupdate.DirOverride = t.TempDir()
 
@@ -62,6 +65,7 @@ func saveRestore(t *testing.T) {
 		cfgPath = origCfgPath
 		apiClient = origApiClient
 		format = origFormat
+		unredacted = origUnredacted
 		noPaginate = origNoPaginate
 		page = origPage
 		perPage = origPerPage
@@ -73,6 +77,8 @@ func saveRestore(t *testing.T) {
 		setClientSecret = origSetClientSecret
 		setDefaultMailbox = origSetDefaultMailbox
 		setFormat = origSetFormat
+		setPIIMode = origSetPIIMode
+		setPIIAllowRaw = origSetPIIAllowRaw
 		configSetCmd.Flags().VisitAll(func(f *pflag.Flag) {
 			f.Changed = false
 		})

internal/cmd/config.go

@@ -6,13 +6,16 @@ import (
 	"github.com/spf13/cobra"
 
 	"github.com/operator-kit/hs-cli/internal/config"
+	"github.com/operator-kit/hs-cli/internal/pii"
 )
 
 var (
 	setClientID       string
 	setClientSecret   string
 	setDefaultMailbox int
 	setFormat         string
+	setPIIMode        string
+	setPIIAllowRaw    bool
 
 	configCmd    *cobra.Command
 	configSetCmd *cobra.Command
@@ -42,6 +45,8 @@ func newConfigSetCmd() *cobra.Command {
 	cmd.Flags().StringVar(&setClientSecret, "client-secret", "", "HelpScout Client Secret")
 	cmd.Flags().IntVar(&setDefaultMailbox, "default-mailbox", 0, "default mailbox ID")
 	cmd.Flags().StringVar(&setFormat, "format", "", "output format: table|json|csv")
+	cmd.Flags().StringVar(&setPIIMode, "pii-mode", "", "PII redaction mode: off|customers|all")
+	cmd.Flags().BoolVar(&setPIIAllowRaw, "pii-allow-unredacted", false, "allow per-request --unredacted override")
 	return cmd
 }
 
@@ -89,6 +94,15 @@ func runConfigSet(cmd *cobra.Command, args []string) error {
 	if cmd.Flags().Changed("format") {
 		existing.Format = setFormat
 	}
+	if cmd.Flags().Changed("pii-mode") {
+		if !pii.IsValidMode(setPIIMode) {
+			return fmt.Errorf("invalid --pii-mode: %q (expected off|customers|all)", setPIIMode)
+		}
+		existing.PIIMode = setPIIMode
+	}
+	if cmd.Flags().Changed("pii-allow-unredacted") {
+		existing.PIIAllowUnredacted = setPIIAllowRaw
+	}
 
 	if err := config.Save(path, existing); err != nil {
 		return err
@@ -112,6 +126,8 @@ func runConfigGet(cmd *cobra.Command, args []string) error {
 		fmt.Fprintf(out, "client-secret: %s\n", c.ClientSecret)
 		fmt.Fprintf(out, "default-mailbox: %d\n", c.DefaultMailbox)
 		fmt.Fprintf(out, "format: %s\n", c.Format)
+		fmt.Fprintf(out, "pii-mode: %s\n", c.PIIMode)
+		fmt.Fprintf(out, "pii-allow-unredacted: %t\n", c.PIIAllowUnredacted)
 		return nil
 	}
 
@@ -124,6 +140,10 @@ func runConfigGet(cmd *cobra.Command, args []string) error {
 		fmt.Fprintf(out, "%d\n", c.DefaultMailbox)
 	case "format":
 		fmt.Fprintf(out, "%s\n", c.Format)
+	case "pii-mode":
+		fmt.Fprintf(out, "%s\n", c.PIIMode)
+	case "pii-allow-unredacted":
+		fmt.Fprintf(out, "%t\n", c.PIIAllowUnredacted)
 	default:
 		return fmt.Errorf("unknown config key: %s", args[0])
 	}

internal/cmd/config_test.go

@@ -22,10 +22,12 @@ func TestConfigSet(t *testing.T) {
 	t.Setenv("HS_CLIENT_ID", "")
 	t.Setenv("HS_CLIENT_SECRET", "")
 	t.Setenv("HS_FORMAT", "")
+	t.Setenv("HS_PII_MODE", "")
+	t.Setenv("HS_PII_ALLOW_UNREDACTED", "")
 
 	buf := new(bytes.Buffer)
 	rootCmd.SetOut(buf)
-	rootCmd.SetArgs([]string{"inbox", "config", "set", "--client-id", "myid", "--client-secret", "mysecret", "--default-mailbox", "42", "--format", "json"})
+	rootCmd.SetArgs([]string{"inbox", "config", "set", "--client-id", "myid", "--client-secret", "mysecret", "--default-mailbox", "42", "--format", "json", "--pii-mode", "customers", "--pii-allow-unredacted"})
 	require.NoError(t, rootCmd.Execute())
 
 	assert.Contains(t, buf.String(), "Config saved")
@@ -36,6 +38,8 @@ func TestConfigSet(t *testing.T) {
 	assert.Equal(t, "mysecret", loaded.ClientSecret)
 	assert.Equal(t, 42, loaded.DefaultMailbox)
 	assert.Equal(t, "json", loaded.Format)
+	assert.Equal(t, "customers", loaded.PIIMode)
+	assert.True(t, loaded.PIIAllowUnredacted)
 }
 
 func TestConfigSet_Partial(t *testing.T) {
@@ -49,12 +53,15 @@ func TestConfigSet_Partial(t *testing.T) {
 	t.Setenv("HS_CLIENT_ID", "")
 	t.Setenv("HS_CLIENT_SECRET", "")
 	t.Setenv("HS_FORMAT", "")
+	t.Setenv("HS_PII_MODE", "")
+	t.Setenv("HS_PII_ALLOW_UNREDACTED", "")
 
 	require.NoError(t, config.Save(cfgFile, &config.Config{
 		ClientID:       "original-id",
 		ClientSecret:   "original-secret",
 		DefaultMailbox: 10,
 		Format:         "table",
+		PIIMode:        "off",
 	}))
 
 	buf := new(bytes.Buffer)
@@ -68,6 +75,7 @@ func TestConfigSet_Partial(t *testing.T) {
 	assert.Equal(t, "original-secret", loaded.ClientSecret)
 	assert.Equal(t, 10, loaded.DefaultMailbox)
 	assert.Equal(t, "table", loaded.Format)
+	assert.Equal(t, "off", loaded.PIIMode)
 }
 
 func TestConfigSet_MutualFields(t *testing.T) {
@@ -81,6 +89,8 @@ func TestConfigSet_MutualFields(t *testing.T) {
 	t.Setenv("HS_CLIENT_ID", "")
 	t.Setenv("HS_CLIENT_SECRET", "")
 	t.Setenv("HS_FORMAT", "")
+	t.Setenv("HS_PII_MODE", "")
+	t.Setenv("HS_PII_ALLOW_UNREDACTED", "")
 
 	buf := new(bytes.Buffer)
 	rootCmd.SetOut(buf)
@@ -104,12 +114,16 @@ func TestConfigGet(t *testing.T) {
 	t.Setenv("HS_CLIENT_ID", "")
 	t.Setenv("HS_CLIENT_SECRET", "")
 	t.Setenv("HS_FORMAT", "")
+	t.Setenv("HS_PII_MODE", "")
+	t.Setenv("HS_PII_ALLOW_UNREDACTED", "")
 
 	require.NoError(t, config.Save(cfgFile, &config.Config{
-		ClientID:       "myid",
-		ClientSecret:   "mysecret",
-		DefaultMailbox: 42,
-		Format:         "json",
+		ClientID:           "myid",
+		ClientSecret:       "mysecret",
+		DefaultMailbox:     42,
+		Format:             "json",
+		PIIMode:            "all",
+		PIIAllowUnredacted: true,
 	}))
 
 	buf := new(bytes.Buffer)
@@ -122,6 +136,8 @@ func TestConfigGet(t *testing.T) {
 	assert.Contains(t, output, "client-secret: mysecret")
 	assert.Contains(t, output, "default-mailbox: 42")
 	assert.Contains(t, output, "format: json")
+	assert.Contains(t, output, "pii-mode: all")
+	assert.Contains(t, output, "pii-allow-unredacted: true")
 }
 
 func TestConfigGet_SingleKey(t *testing.T) {
@@ -135,6 +151,8 @@ func TestConfigGet_SingleKey(t *testing.T) {
 	t.Setenv("HS_CLIENT_ID", "")
 	t.Setenv("HS_CLIENT_SECRET", "")
 	t.Setenv("HS_FORMAT", "")
+	t.Setenv("HS_PII_MODE", "")
+	t.Setenv("HS_PII_ALLOW_UNREDACTED", "")
 
 	require.NoError(t, config.Save(cfgFile, &config.Config{
 		ClientID:       "myid",
@@ -164,3 +182,45 @@ func TestConfigPath(t *testing.T) {
 
 	assert.Equal(t, cfgFile+"\n", buf.String())
 }
+
+func TestConfigGet_SinglePIIModeKey(t *testing.T) {
+	saveRestore(t)
+	versionStr = "dev"
+
+	dir := t.TempDir()
+	cfgFile := filepath.Join(dir, "config.yaml")
+	cfgPath = cfgFile
+
+	t.Setenv("HS_CLIENT_ID", "")
+	t.Setenv("HS_CLIENT_SECRET", "")
+	t.Setenv("HS_FORMAT", "")
+	t.Setenv("HS_PII_MODE", "")
+	t.Setenv("HS_PII_ALLOW_UNREDACTED", "")
+
+	require.NoError(t, config.Save(cfgFile, &config.Config{
+		PIIMode: "customers",
+	}))
+
+	buf := new(bytes.Buffer)
+	rootCmd.SetOut(buf)
+	rootCmd.SetArgs([]string{"inbox", "config", "get", "pii-mode"})
+	require.NoError(t, rootCmd.Execute())
+
+	assert.Equal(t, "customers\n", buf.String())
+}
+
+func TestConfigSet_InvalidPIIMode(t *testing.T) {
+	saveRestore(t)
+	versionStr = "dev"
+
+	cfgPath = filepath.Join(t.TempDir(), "config.yaml")
+	t.Setenv("HS_PII_MODE", "")
+	t.Setenv("HS_PII_ALLOW_UNREDACTED", "")
+
+	buf := new(bytes.Buffer)
+	rootCmd.SetOut(buf)
+	rootCmd.SetArgs([]string{"inbox", "config", "set", "--pii-mode", "bad"})
+	err := rootCmd.Execute()
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "invalid --pii-mode")
+}