Revision #2

Author: gsusanto-fastly

doc/GUIDE.md

@@ -2,13 +2,13 @@
 
 libfsm compiles regular expressions to deterministic finite state machines (FSMs) and generates executable code. FSM-based matching runs in **linear time O(n)** with **no backtracking**.
 
-> Regex engines like PCRE use backtracking to explore multiple possible match paths at **runtime**.
-> This means the same pattern can have different execution costs depending on the input.
->
-> libfsm instead resolves all match decisions at **compile time** by constructing a Deterministic Finite Automaton (DFA).
-> At runtime, matching is a single linear pass over the input with no alternative paths to explore.
->
-> As a result, libfsm avoids input-dependent slowdowns and is not susceptible to regular expression–based denial-of-service (ReDoS) attacks.
+Regex engines like PCRE use backtracking to explore multiple possible match paths at **runtime**.
+This means the same pattern can have different execution costs depending on the input.
+
+libfsm instead resolves all match decisions at **compile time** by constructing a Deterministic Finite Automaton (DFA).
+At runtime, matching is a single linear pass over the input with no alternative paths to explore.
+
+As a result, libfsm avoids input-dependent slowdowns and is not susceptible to regular expression–based denial-of-service (ReDoS) attacks.
 
 **libfsm is not a drop-in replacement for traditional regex engines.** It only supports patterns that can be compiled to FSMs.
 
@@ -21,6 +21,7 @@ libfsm compiles regular expressions to deterministic finite state machines (FSMs
 - [Writing Effective libfsm Patterns](#writing-effective-libfsm-patterns)
 - [Byte Search Optimization (Optional)](#byte-search-optimization-optional)
 - [Troubleshooting](#troubleshooting)
+- [Pattern Matches Empty String Unintentionally](#pattern-matches-empty-string-unintentionally)
 
 ## What libfsm Cannot Do
 
@@ -33,8 +34,6 @@ These PCRE features will not compile:
 * Conditional expressions (`(?(condition)then|else)`)
 * Recursion and subroutines (`(?R)`, `(?1)`)
 
----
-
 ## Quick Start
 
 Generate a matcher from a regex:
@@ -46,21 +45,14 @@ re -p -r pcre -l go -k str 'user\d+' > user_detector.go
 
 This produces a standalone matcher function.
 
----
-
 ## Supported Code Generation Targets
 
 libfsm provides stable, “first-class” code generation for:
+- High-level languages: C (via `-l vmc`), Go, Rust
+- LLVM IR
+- Native WebAssembly
 
-| Category             | Output                         |
-| -------------------- | ------------------------------ |
-| High-level languages | **C (via `-l vmc`), Go, Rust** |
-| Toolchains           | **LLVM IR**                    |
-| Virtualization       | **Native WebAssembly**         |
-
-> Adding code generation for new languages is straightforward and is defined in [src/libfsm/print/](../src/libfsm/print/).
-
----
+Adding code generation for new languages is straightforward and is defined in [src/libfsm/print/](../src/libfsm/print/).
 
 ## Workflow Overview
 
@@ -70,11 +62,11 @@ libfsm provides two main tools:
 
 A recommended workflow when using libfsm is:
 
-### 1. Validate the Regex
+1. Validate the Regex
 
 Test behavior using any PCRE-compatible tool (e.g., [pcregrep(1)](https://man7.org/linux/man-pages/man1/pcregrep.1.html) on the CLI or [https://regex101.com/](https://regex101.com/) in the browser).
 
-### 2. Verify libfsm Compatibility
+2. Verify libfsm Compatibility
 
 ```bash
 re -r pcre -l ast 'x*?'
@@ -87,13 +79,13 @@ rx -r pcre -l ast -d declined.txt 'x*?'
 
 If unsupported constructs exist, libfsm reports the failing location.
 
-### 3. Generate Code
+3. Generate Code
 
 ```bash
 re -p -r pcre -l rust -k str '^item-[A-Z]{3}\z' > item_detector.rs
 ```
 
-### 4. Multiple Patterns
+4. Multiple Patterns
 
 ```bash
 # re - patterns from command line:
@@ -109,32 +101,29 @@ Both tools:
 * Pattern ID is argument position for `re`, line number for `rx`
 * When encountering unsupported patterns: `rx` skips them to `-d` file and generates code with working patterns; `re` fails completely
 
----
-
 ### Flag Reference
 | Flag | Purpose                      | Common Options                             | Notes                                                            |
 | ---- | ---------------------------- | ------------------------------------------ | ---------------------------------------------------------------- |
 | `-r` | Regex dialect                | `pcre`, `literal`, `glob`, `native`, `sql` | `pcre` supports the widest set of features                       |
 | `-l` | Output language for printing | `go`, `rust`, `vmc`, `llvm`, `wasm`, `dot` | Use `vmc` for `C` code. Pipe `dot` into `idot` for visualization |
 | `-k` | Generated function I/O API   | `str`, `getc`, `pair`                      | `str` takes string, `pair` takes byte array, `getc` uses callback for streaming |
 | `-p` | Print mode                   | *(no value)*                               | Abbrv. of `-l fsm`. Print the constructed fsm, rather than executing it.        |
-| `-d` | Declined                     | filename                                   | Only applies to `rx` (batch mode)                                |
+| `-d` | Declined patterns            | filename                                   | Only applies to `rx` (batch mode)                                |
 
 This is not exhausted list. For full flag details, see [include/fsm/options.h](../include/fsm/options.h) and the [man pages](../man).
-The man pages can be built by running `bmake doc`, then view with `build/man/re.1/re.1`.
-
----
+The man pages can be built by running `bmake -r doc`, then view with `build/man/re.1/re.1`.
 
 ## Writing Effective libfsm Patterns
 
-### 1. Replace Broad Wildcards
+1. Replace Broad Wildcards
+
+Avoid `.*` and `.+` when possible. Wildcards match “anything,” which is often imprecise. And although they look compact, libfsm must enumerate every possible byte and continuation. This quickly leads to large DFAs.
 
-Avoid `.*` and `.+` when possible. Wildcards match “anything,” which is often imprecise and forces libfsm to build a large DFA.
+For example, a double-quoted string should not use `".*"` because the content cannot contain an unescaped quote. Using `.*` forces libfsm to consider all characters -- including both the presence and absence of the closing `"` at every step. This greatly increases the number of states.
 
-For example, a double-quoted string should not use `".*?"` because the content cannot contain an unescaped quote.
-Instead, restrict it to the actual valid characters `"[^"\r\n]*"`, which matches only what is allowd and will keep the DFA more compact.
+Instead, restrict it to the actual valid characters `"[^"\r\n]*"`, which matches only what is allowed and will keep the DFA more compact.
 
-Use negated character classes:
+Use negated character classes to match only the allowed content:
 
 | Avoid      | Better         |
 | ---------- | -------------- |
@@ -143,45 +132,46 @@ Use negated character classes:
 | `price=.+` | `price=[0-9]+` |
 | `var\s.+=` | `var\s[^=]+=`  |
 
-> This is often the cause of an “explosion” in the size of the generated FSM.
->
-> See [Compilation Takes Too Long](#compilation-takes-too-long) for more details.
+The overlap between `.*` or `.+` and strings that follow is often the cause of an “explosion” in the size of the generated FSM. So when compilation is slow or generated output is large, look for `.*` and `.+` first and replace them with a narrower character class.
 
----
-
-### 2. Anchor When Matching Full String
+2. Anchor When Matching Full String
 
 When the intention is to match an entire string, use anchors.
 Use `^` at the beginning and `\z` for the true end of the string.
 
 ```regex
 # Correct: matches only this exact hostname
+# Matches "web12.example.com"
+# Does not match "foo-web12.example.com-bar"
 ^web\d+\.example\.com\z 
 
 # Incorrect: would match inside a larger string
-web\d+\.example\.com        # also matches "foo-web12.example.com-bar"
+# Matches "web12.example.com"
+# Also matches "foo-web12.example.com-bar"
+web\d+\.example\.com
 ```
 
----
-
-### 3. Prefer `\z` Over `$` for End-of-String
+3. Prefer `\z` Over `$` for End-of-String
 
 `\z` always matches the end of the string.
 `$` will also match a trailing newline at the end of the string,
 so if you use this in combination with capturing groups, you may not be capturing what you expect.
-Also, `\z` is more efficient, so it is better to use it in places where `\n` cannot appear.
+Also, `\z` produces a smaller FSM, so it is better to use it in places where `\n` cannot appear.
 
 ```regex
-# Preferred
-/foo\z
-
-# Risky: $ may allow an extra newline
-/foo$
+# Preferred: matches only if the string ends with "bar"
+# Matches "/foo/bar"
+# Does NOT match "/foo/bar\n"
+/bar\z
+
+# Incorrect: allows a trailing newline,
+# which is usually unintended and adds unnecessary complexity
+# Matches "/foo/bar"
+# Also matches "/foo/bar\n"
+/bar$
 ```
 
----
-
-### 4. Escape Special Characters When Used As Literal
+4. Escape Special Characters When Used As Literal
 
 Many characters have special meaning in regex (for example `.`, `+`, `*`, `?`, `[`, `(`).
 If you mean to match them literally, escape them:
@@ -195,9 +185,7 @@ If you mean to match them literally, escape them:
 | `(test)`                   | `\(test\)`                  | `(` and `)` begin/end a group              |
 | Markdown link `[t](u)`     | `(\[[^]]*\]\([^)]*\))`      | Matches `[text](url)` without crossing `]` or `)` |
 
----
-
-### 5. Use Non-Capturing Groups
+5. Use Non-Capturing Groups
 
 Capture groups are _currently_ not supported (coming soon!).
 If you need grouping for alternation or precedence, use non-capturing syntax `(?:...)`:
@@ -210,8 +198,6 @@ If you need grouping for alternation or precedence, use non-capturing syntax `(?
 (private|no-store)
 ```
 
----
-
 ## Byte Search Optimization (Optional)
 
 Patterns that start with an **uncommon character** can be accelerated using an initial byte scan before running the FSM.
@@ -231,11 +217,7 @@ These prefixes (`#`, `@`, `[`, `{`, `'`, `"`) are rare in normal text, so a byte
 
 We found using `strings.IndexByte` before calling the generated matcher in Go code significantly improved performance when matching strings with a large (>5k) leading prefix.
 
----
-
-## Troubleshooting
-
-### Pattern Matches Empty String Unintentionally
+## Pattern Matches Empty String Unintentionally
 
 Pattern:
 
@@ -251,14 +233,3 @@ This is only an issue if that is not what you intend.
 
 * Require at least one match: `\s+`
 * Anchor context: `^\s+$` or alternatively, use `-Fb` flag
-
-### Compilation Takes Too Long
-
-This is often caused by unrestricted wildcards (`.*`, `.+`).
-Although they look compact, libfsm must enumerate every possible byte and every possible continuation, causing the state machine to grow quickly.
-
-For example, to match `var anything =`, a pattern such as `var\s.+=` looks simple, but `.+` forces libfsm to encode every possible byte
-and every possible continuation -- including both the presence and absence of `=`. This drastically increases the number of states.
-
-When compilation is slow, look for broad wildcards and replace them with more specific character classes (as shown [above](#writing-effective-libfsm-patterns)),
-such as: `var\s[^=]+=`.