#449: Needs parsed past end of section (#451)

* #449: Parsing Feature Needs from MD goes past next field. Added way to switch off parsing for segments.

Author: redcatbear

doc/changes/changes_4.2.0.md

@@ -1,4 +1,4 @@
-# OpenFastTrace 4.2.0, released ???
+# OpenFastTrace 4.2.0, released 2025-05-19
 
 Code name: Markdown code blocks
 
@@ -8,6 +8,8 @@ In this release we changed the behavior of the Markdown importer, so that if we
 
 We also added a whole section about understanding and fixing broken links between specification items to the user guide.
 
+The new token `oft:on|off` allows switching off OFT parsing for certain text passages in Markdown and RST documents.
+
 ## Features
 
 * #437: Upgrade build and test dependencies on top of 4.1.0
@@ -17,5 +19,6 @@ We also added a whole section about understanding and fixing broken links betwee
 
 * #427: Removed old `CHANGELOG.md` file and merged missing parts into release history.
 * #431: Documented "unwanted coverage" in user guide.
+* #449: Fix parsing past end of "needs" paragraph.
 * #440: Added Tag importer support for TOML files.
 * #442: Added support for javascript file extensions `.cjs`, `.mjs` and `.ejs`

doc/spec/design.md

@@ -220,6 +220,24 @@ Covers:
 
 Needs: impl, utest, itest
 
+### Line Parser for Lightweight Markup Import
+
+RST and Markdown share a common underlying parser that operates on a line-by-line basis.
+
+##### Disabling OFT Parsing for Parts of a Markup File
+`dsn~disabling-oft-parsing-for-parts-of-a-markup-file~1`
+
+When it encounters the token `oft:off`, the line parser stops extracting specification items until it
+
+* either encounters the token `oft:on`
+* or reaches the end of the current document.
+
+Covers:
+
+* `req~disabling-oft-parsing-for-parts-of-a-markup-file~1`
+
+Needs: impl, utest
+
 ## Tracing
 
 ### Tracing Needed Coverage
@@ -756,8 +774,10 @@ The Markdown Importer supports forwarding required coverage from one artifact ty
 
 The following example shows an architectural specification item that forwards the needed coverage directly to the detailed design and an integration test:
 
+<!-- oft:off -->
     arch --> dsn, itest : req~skip-this-requirement~1
- 
+<!-- oft:on -->
+
 Covers:
 
 * `req~artifact-type-forwarding-in-markdown~1`

doc/spec/system_requirements.md

@@ -101,6 +101,8 @@ The same benefits as for [Markdown](#markdown-import) apply:
 * is portable across platforms
 * easy to process with text manipulation tools
 
+Needs: req
+
 ### ReqM2 Import
 `feat~reqm2-import~1`
 
@@ -251,9 +253,43 @@ Needs: dsn
 
 ### Supported Formats
 
+#### Common Requirements for Lightweight Markup Import
+
+Typical OFT specification are written in a lightweight markup language like [Markdown](#markdown-import) or [ReStructured Text](#restructured-text-rst-import). Before we go into the specifics, this section discusses the common requirements.
+
+##### Disabling OFT Parsing for Parts of a Markup File
+`req~disabling-oft-parsing-for-parts-of-a-markup-file~1`
+
+OFT-enhanced markup allows excluding text blocks from OFT parsing with the syntax `oft:on|off`.
+
+Example for Markdown:
+
+    <!-- oft:off -->
+    This part of the document will not be parsed for OFT specification items.
+
+    Until the end marker or the end of the current document is reached
+    <!-- oft:on -->
+
+Example for RST:
+
+    .. oft:off
+    Not imported.
+    .. oft:on
+
+Rationale:
+
+This allows creating OFT examples that do not contribute to the code and avoid accidental recognition of specification items in text that is not supposed to contain them.
+
+Covers:
+
+* [feat~markdown-import~1](#markdown-import)
+* [feat~rst-import~1](#restructured-text-rst-import)
+
+Needs: dsn
+
 #### Markdown
 
-Markdown is a simple ASCII-based markup format that is designed to be human readable in the source. While it can be rendered into HTML, it is perfectly eye-friendly even before rendering.
+Markdown is a simple ASCII-based markup format that is designed to be human-readable in the source. While it can be rendered into HTML, it is perfectly eye-friendly even before rendering.
 
 Markdown focuses on content over formatting by giving the document structure like headlines, paragraphs and lists. The combination of being lightweight, human-readable and structure-oriented makes it a good fit for writing specifications as code.
 

doc/user_guide.md

@@ -261,8 +261,6 @@ Requirements should be accompanied by a rationale in all cases where the reason
     the details are up to the detailed design.
 
     Needs: dsn 
-    
-
 
 `Needs`, `Rationale` and `Comment` are OpenFastTrace keywords that tell OpenFastTrace how to process the following content. There are other keywords in the context of specification items written in Markdown described in the following sections.
 
@@ -298,6 +296,22 @@ Given the Feature `feat~rubber-ducky~1` exists and needs a `req`. A requirement
     Covers:
     - feat~rubber-ducky~1
 
+##### `Needs`
+
+The `Needs` keyword states which artifact types are needed to cover the current specification item. It is followed by a list of artifact types that are needed, each one written on a new line starting with a bullet character (`+`, `*`, or `-`) followed by the artifact type abbreviation. `Needs` comes in two flavors: as one-liner or as list. 
+
+**Variant a) one-line `needs`**
+
+    Needs: impl, utest, itest
+
+**Variant b) as List**
+
+    Needs:
+    - dsn
+    - uman
+
+Please note that you cannot mix the two styles in one specification item.
+
 ##### `Depends`
 
 The `Depends` keyword defines dependencies between specification items. It is followed by a list of items the current specification item depends on, each one written on a new line starting witch a bullet character (`+`, `*`, or `-`) followed by the referenced specification item id. At the moment this has no effect on the HTML or plaintext output, but only if the `-o aspec` option is used. This has no effect on the coverage of specification items.
@@ -332,6 +346,26 @@ is functionally equivalent to
 
 Tags are described in detail later in this document, see section [Distributing the Detailing Work](#distributing-the-detailing-work).
 
+### Excluding Parts of a Specification Document for OFT Parsing
+
+Sometimes you want specific sections or a whole document to be excluded from OFT parsing. One reason could be that it is a document that contains an OFT example, that should not contribute to the trace. Or, you could have data in a document and don't want to risk that something accidentally looks like an OFT artifact.
+
+To switch of scanning use the token `oft:on|off` in your document at the appropriate location.
+
+Markdown example:
+
+    <!-- oft:off -->
+    This part is ignored by OFT.
+    <!-- oft:on -->
+    Here OFT scans again.
+
+ReStructured text example:
+
+    .. oft:off
+    This part is ignored by OFT.
+    .. oft:on
+    Here OFT scans again.
+
 ### Delegating Requirement Coverage
 
 Consider a situation where you are responsible for the high-level software architecture of your project. You define the component breakdown, the interfaces and the interworking of the components. You get your requirements from a system requirement specification, but it turns out many of those incoming requirements are at a detail level that does not require design decisions on inter-component-level but rather affects the internals of a single component.

importer/lightweightmarkup/src/main/java/org/itsallcode/openfasttrace/importer/lightweightmarkup/statemachine/LineParserState.java

@@ -7,7 +7,7 @@
 public enum LineParserState
 {
     /**
-     * Parser started (at beginning of the file) or outside of a specification
+     * Parser started (at beginning of the file) or outside a specification
      * item
      */
     START,
@@ -23,8 +23,10 @@ public enum LineParserState
     RATIONALE,
     /** Inside a comment section */
     COMMENT,
-    /** Inside a section defining the required coverage */
-    NEEDS,
+    /** Inside a section defining the required coverage (inline form) */
+    NEEDS_LINE,
+    /** Required coverage (list form) */
+    NEEDS_LIST,
     /** Found a title */
     TITLE,
     /** Found tags */

importer/lightweightmarkup/src/main/java/org/itsallcode/openfasttrace/importer/lightweightmarkup/statemachine/LineParserStateMachine.java

@@ -2,6 +2,9 @@
 
 import java.util.*;
 import java.util.logging.Logger;
+import java.util.regex.Pattern;
+
+import static java.util.regex.Pattern.UNICODE_CHARACTER_CLASS;
 
 /**
  * This machine implements the core of a state based parser.
@@ -19,10 +22,15 @@
 public class LineParserStateMachine
 {
     private static final Logger LOG = Logger.getLogger(LineParserStateMachine.class.getName());
+    private static final Pattern PARSER_OFF_PATTERN = Pattern.compile("(?:^|\\W)oft:off(?:\\W|$)",
+            UNICODE_CHARACTER_CLASS);
+    private static final Pattern PARSER_ON_PATTERN = Pattern.compile("(?:^|\\W)oft:on(?:\\W|$)",
+            UNICODE_CHARACTER_CLASS);
 
     private LineParserState state = LineParserState.START;
     private String lastToken = "";
     private final Transition[] transitions;
+    private boolean enabled = true;
 
     /**
      * Create a new instance of the {@link LineParserStateMachine}
@@ -48,7 +56,29 @@ public LineParserStateMachine(final Transition[] transitions)
      *            patterns that span multiple lines like underlined titles in
      *            Markdown or RST.
      */
+    // [impl -> dsn~disabling-oft-parsing-for-parts-of-a-markup-file~1]
     public void step(final String line, final String nextLine)
+    {
+        if (enabled)
+        {
+            if (PARSER_OFF_PATTERN.matcher(line).find())
+            {
+                enabled = false;
+            }
+            else
+            {
+                stepEnabled(line, nextLine);
+            }
+        }
+        else
+        {
+            if (PARSER_ON_PATTERN.matcher(line).find()) {
+                enabled = true;
+            }
+        }
+    }
+
+    private void stepEnabled(final String line, final String nextLine)
     {
         boolean matched = false;
         for (final Transition entry : this.transitions)