Sink auth failures (401/403) cause a restart loop instead of a terminal failure

[chunxi]

Symptom: A Turbo pipeline with a webhook sink whose endpoint returns 401 sits in "restarting" indefinitely. It never reports a failure the user can act on, and the checkpoint never advances.

Repro: Deploy any pipeline with a webhook sink pointing at an endpoint that returns 401 (wrong/rotated auth header, or a secret_name whose credentials expired).

Root cause — three parts

1. The 401 is correctly classified non-retriable, so nothing retries in-process.
   is_retriable_status is 408 / 429 / 5xx only — crates/streamling-core/src/operators/external_handlers.rs:296-300 (test asserts !is_retriable_status(UNAUTHORIZED) at :886). retry_if_retriable returns immediately (crates/streamling-core/src/retry.rs:137-140), the sink's write loop propagates it (crates/streamling-connectors/src/table_providers/http.rs:162), and the process exits.

2. The exit carries no terminal/fatal signal. Every Err maps to ExitCode::FAILURE (crates/streamling/src/main.rs:271-273). A permanent auth failure and a transient OOM produce the identical exit code, so the orchestrator can only restart. On restart the sink resumes from the last finalized checkpoint (docs/checkpointing-deep-dive.md:277) — the failing batch was never acked, so it replays the same rows and gets the same 401. Unbounded retry at the worst possible period (full process restart), with no application-level backoff.

3. The error is mislabeled as a platform error. It is built with StreamlingError::new(...) (external_handlers.rs:607), which sets internal: true (crates/streamling-common/src/error.rs:50-61). So the structured log emits error.internal = true (main.rs:251-258) for a customer credential problem. Per this repo's own convention (crates/streamling/src/lib.rs:281-285) that routes it to the platform team instead of surfacing it to the user — which is exactly why nobody gets told "your webhook auth is wrong."

Blast radius

Same pattern in every HTTP-ish sink path:

• Webhook sink and the HTTP handler transform (shared ExternalHandlerClient).
• ClickHouse sink: transient is is_server_error() || 429 (crates/streamling-connectors/src/table_providers/clickhouse.rs:2115-2116); the non-transient error uses streamling_err! → also internal: true (crates/streamling-common/src/error.rs:554-558).
• Kafka schema registry: retries only retryable SRCErrors, so a 401 there behaves the same.

Not affected: RPC-source 401s, which fail differently (JSON parse → FetchError::InvalidResponse → retried forever in-process by the sequential runner) — a separate issue, but same user-visible "stuck" outcome.

Proposed fix

1. In streamling (small, local): construct non-retriable 4xx errors with StreamlingError::user(...) / streamling_user_err! instead of new(...) / streamling_err!, in external_handlers.rs:607 and the ClickHouse equivalent. Makes error.internal = false truthful and routes the error to the customer. Update the assertions around external_handlers.rs:873-890 and :1516.
2. Across streamling + control plane: give the process a distinct exit code for user-facing terminal errors (e.g. 2) so the orchestrator can mark the pipeline FAILED with the user-facing message rather than restarting. Requires a matching control-plane change.

Item 1 alone doesn't stop the restart loop — it only makes the loop diagnosable. Item 2 is what actually fixes the limbo.

Open questions

Couldn't answer these from streamling or streamling-goldsky-plugins:

• What does the control plane currently do with a non-zero exit — is there any crashloop-backoff / give-up threshold, or does it restart indefinitely?
• Does anything consume error.internal from the log stream today, or is the exit code the only channel? (Nothing in streamling reads it back; it's emitted for logging only.)
• Is there prior art for a terminal-failure exit code, or would 2 be new?

Workaround for affected users

skip_on_error: true on the webhook sink lets the pipeline progress (external_handlers.rs:601-605) — but it drops the rows that got the 401, so it's only acceptable if the data is disposable. Rejected for output-producing handlers (external_handlers.rs:463).