Support Bundle and Incident Diagnostics

Support Bundle & Incident Diagnostics

Type: Operator Reference (Troubleshooting) Scope: nftban support, incident evidence collection, reading a bundle Since: v1.229.12 (incident evidence) · Verified against tag v1.229.12 @ da53598e Terminology: Glossary & Vocabulary

---

Purpose

nftban support collects a diagnostic bundle from the host. This page explains how to use it during an incident and how to read what comes back.

The bundle is read-only. It never mutates the host being diagnosed — a host in a failed state is evidence, and collection must not destroy it.

For the exact file-by-file contract, see the versioned operator document: Support bundle and incident evidence.

---

Collecting a bundle

nftban support --output /root               # bundle to a chosen directory
nftban support --email admin@example.com    # create the bundle AND email it
nftban support --quick                      # terminal diagnostics only, no file

On v1.229.11 and earlier, --output does not work — it exits 1 and produces no bundle at all. On those releases use plain nftban support and move the tarball from /tmp yourself. Fixed in v1.229.12.

Output is nftban-support-YYYYMMDD-HHMMSS.tar.gz in the output directory (default /tmp). nftban support-bundle is an accepted alias.

--email is transport, not authority. The bundle is written to disk before any mail is attempted. If mail fails, you still have the bundle — look in the output directory.

Why this is called out. Before v1.229.12 the flag failed outright (SUPPORT_OUTPUT_DIR: readonly variable, exit 1) and produced no bundle at all. The default path was unaffected, which is why it survived: routine use omits --output. An operator running the documented command mid-incident got nothing back.

Handling

Redaction removes known secret patterns, and is fail-closed — if the redactor is unavailable the affected stream is discarded rather than passed through raw. Review the bundle before sharing it outside your organization, because diagnostic data may still contain identifying or environment-specific information. No pattern-based redactor can guarantee that arbitrary sensitive material is absent. Never paste credentials anywhere; nothing in NFTBan asks for them.

---

Incident workflow

When an update, rebuild, or installer run goes wrong, work in this order.

1. establish whether firewall protection is currently active
2. collect `nftban support`
3. preserve the bundle off-host
4. inspect the installer / rebuild state
5. inspect the validation and rollback result
6. identify WHICH stage failed: render / apply / module re-apply / validation / rollback
7. only then consider repair or any other mutation

Step 1 — is protection active right now?

nft list tables                      # expect: table ip nftban, table ip6 nftban
systemctl is-active nftables nftband
nftban health

This is a different question from "did the run succeed", and it decides how much time you have.

Step 2–3 — collect and preserve.

nftban support --output /root
scp /root/nftban-support-*.tar.gz you@elsewhere:/path/

⛔ Do not run nftban-installer --repair first. Repair resumes from the phase recorded in the install state and re-runs it in full — for a FAILED_REBUILD state that is the entire Switch phase, including emergency SSH injection, ghost-table cleanup, and another complete firewall rebuild. If the original run's work actually succeeded, repair does substantial work to recover from nothing, and overwrites the evidence that would have shown that.

---

Reading the bundle

The four truth rules

The bundle is built so that absence of evidence is never rendered as evidence of absence:

UNKNOWN      != 0
UNAVAILABLE  != ABSENT
NOT_STARTED  != FAILED
DANGLING     != COMPLETE

A failed query is recorded as UNKNOWN with the error, never as a count. nft list table ip nftban 2>/dev/null | grep -c chain returns 0 for both an empty table and a failed nft, so rendering a tool failure as chains=0 would read as proof the firewall had been destroyed. The collectors refuse to do that.

A rebuild start with no matching end is reported as DANGLING, never paired with a later run's end line — pairing across runs would fabricate a duration.

Collection time is not incident time

Every evidence file carries a correlation header stating what it is bound to:

Kind of evidence	Example	Bound to
Live state	current chains, sets, service status	collected_at
Historical events	installer phases, rebuild durations, parser rejections	the log's own timestamps

Reading a rotated log next to a freshly observed live state as one story is the easiest way to reach a wrong conclusion from a technically correct bundle.

The incident/ directory

File	Answers
phase_timeline.txt	Real failure, or a false verdict from a deadline? Budget, phase markers, rebuild start/end/exit pairs.
chain_inventory.txt	Which chains are present, by identity — not just a count.
module_reapply.txt	Why module re-apply failed — per-module command, start/end, exit, and captured stdout/stderr.
parser_rejections.txt	How many feed/list elements the parser rejected and roughly what shape they were. Counted from installer.log only, split by a two-way heuristic (dash_range vs other), at most five sample values. An unreadable log yields total_rejections=UNKNOWN, never 0.
timeline_ruleset_lifecycle.txt	render → apply → validation → rollback → final live state.
timeline_installer_run.txt	phase → duration → exit → deadline/cancellation → terminal verdict.
nft_ruleset.json	The full live ruleset as nft -j list ruleset — parseable without scraping human text.
nft_sets.json	The live sets as nft -j list sets, same structured form.

Module re-apply evidence

Before this existed, a rebuild could record PRE protected (chains: 16) → POST degraded (chains: 6) with no recoverable cause: the module re-apply calls suppressed stderr, and their warnings never reached installer.log. The mechanism was reproducible; the trigger was not.

module_reapply.txt carries the newest three records from /var/log/nftban/rebuild-modules/, each with step, command, start, end, exit and result, plus bounded stdout/stderr excerpts. Content is redacted when written, and that redaction is fail-closed — if the redactor is unavailable the capture is withheld rather than emitted raw.

Read the absence carefully:

• directory missing → UNAVAILABLE. Rebuilds before this feature recorded nothing; that is not evidence the modules succeeded.
• directory present, records_present=0 → stated explicitly as not the same as "all modules succeeded".
• stdout=<empty> vs stdout=UNAVAILABLE → an empty stderr and a stderr that could not be captured are different facts, and are reported differently.

The two timelines are separate on purpose. They can disagree, and the disagreement is the diagnosis.

A healthy final live state alongside a terminal installer verdict does not mean the host is unprotected. Compare the two timelines before concluding anything.

Chain inventory: identity, not count

The base firewall contributes input, forward, and output per family. That structural fact — three base chains in each of ip and ip6 — is the claim to rely on. The resulting total of six is a consequence of it, not an invariant in its own right, and no regression test currently pins the number. Everything beyond the base chains is contributed by protection modules during module re-apply.

So a count alone (16 → 6) cannot say which protections went away; the identities can. Six survivors that are exactly the base chains means every module chain is absent.

Six chains is the base topology, not renderer corruption. Module chains are added by the module re-apply mechanism, which is a separate step with its own failure modes. Do not read a base-only topology as the base renderer having lost the module chains.

Installer phase attribution

The installer tests deadline expiry when entering a phase. An error naming phase X can therefore mean "the deadline had already expired before X started" — X may never have run. Cross-check the phase that was actually running against the rebuild start/end pairs before attributing blame. The bundle prints this warning unconditionally.

The global wall-clock budget is 300 seconds (cmd/nftban-installer/main.go). That constant is present in v1.229.11 too — but v1.229.11 does not log it, so on a current released host the budget is real yet invisible, and a bundle from that host reports it as UNKNOWN. Emitting installer global budget=300s deadline=... is unreleased v1.229.12 behavior. A firewall rebuild is exempt from the budget by policy; if the budget is exhausted and a policy-exempt operation has already returned successfully, the run continues under one fresh bounded budget, granted at most once per run. It is never "no deadline".

MANIFEST.txt

Records what the bundle contains and what it could not collect: every file with its size, an EMPTY list (collected but no content), and an UNAVAILABLE / FAILED list (sections that explicitly declared they could not collect). Without this, an absent file is ambiguous between "not collected", "collected empty", and "collection failed".

---

Rollback: what a successful rollback does and does not tell you

If rollback succeeds:

• the failed candidate state was rejected;
• rollback restored the prior protection state;
• this does not mean the initiating cause has been identified.

Treat a successful rollback as containment, not as diagnosis. The bundle collected before any repair is what lets you find the cause afterwards.

---

Known limitations

• A hung rebuild is not bounded. The rebuild subprocess runs on context.Background(), so the installer's deadline cannot terminate it, and nothing wraps the installer itself. NFTBan has no progress-aware supervision and therefore cannot distinguish a long rebuild from a hung one. The bundle can show a rebuild that started and never ended (DANGLING); it cannot tell you whether that rebuild is still making progress. Tracked as a GA backlog item.
• The bundle reports what the host recorded. If installer.log was rotated or is unreadable, the phase timeline says UNAVAILABLE rather than reconstructing events.

---

Related

• Troubleshooting & Selftest — smoke / selftest semantics
• CLI Commands Reference — nftban support flags
• Installation Guide — install outcomes and repair
• Known Limitations & Validation Scope
• Emergency recovery and rollback