Merge pull request #19 from jqnatividad/CSVWrangling-increased-accuracy

refactor: all accuracy benchmarks are now > 90%

Author: jqnatividad

README.md

@@ -164,11 +164,11 @@ The table below shows the dialect detection success ratio. Accuracy is measured
 
 | Data set | `csv-nose` | `CSVsniffer MADSE` | `CSVsniffer` | `CleverCSV` | `csv.Sniffer` | DuckDB `sniff_csv` |
 |:---------|:-----------|:-------------------|:-------------|:------------|:--------------|:-------------------|
-| POLLOCK  | **96.62%** | 95.27%             | 96.55%       | 95.17%      | 96.35%        | 84.14%             |
+| POLLOCK  | **97.30%** | 95.27%             | 96.55%       | 95.17%      | 96.35%        | 84.14%             |
 | W3C-CSVW[^2] | **99.55%** | 94.52%             | 95.39%       | 61.11%      | 97.69%        | 99.08%             |
-| CSV Wrangling | **87.15%** | 90.50%          | 89.94%       | 87.99%      | 84.26%        | 91.62%             |
-| CSV Wrangling CODEC | **86.62%** | 90.14%    | 90.14%       | 89.44%      | 84.18%        | 92.25%             |
-| CSV Wrangling MESSY | **84.92%** | 89.60%    | 89.60%       | 89.60%      | 83.06%        | 91.94%             |
+| CSV Wrangling | **92.74%** | 90.50%          | 89.94%       | 87.99%      | 84.26%        | 91.62%             |
+| CSV Wrangling CODEC | **91.55%** | 90.14%    | 90.14%       | 89.44%      | 84.18%        | 92.25%             |
+| CSV Wrangling MESSY | **90.48%** | 89.60%    | 89.60%       | 89.60%      | 83.06%        | 91.94%             |
 
 [^2]: csv-nose is optimized for the [W3C CSV on the Web Test Suite](https://w3c.github.io/csvw/tests/) - reaching 99.55% accuracy.
 
@@ -192,25 +192,25 @@ The F1 score is the harmonic mean of precision and recall, providing a balanced
 
 | Data set | `csv-nose` | `CSVsniffer MADSE` | `CSVsniffer` | `CleverCSV` | `csv.Sniffer` | DuckDB `sniff_csv` |
 |:---------|:-----------|:-------------------|:-------------|:------------|:--------------|:-------------------|
-| POLLOCK  | **0.966**  | 0.976              | 0.972        | 0.965       | 0.943         | 0.904              |
+| POLLOCK  | **0.973**  | 0.976              | 0.972        | 0.965       | 0.943         | 0.904              |
 | W3C-CSVW | **0.995**  | 0.967              | 0.967        | 0.748       | 0.730         | 0.986              |
-| CSV Wrangling | **0.872** | 0.950             | 0.945        | 0.935       | 0.724         | 0.956              |
-| CSV Wrangling CODEC | **0.866** | 0.948       | 0.948        | 0.944       | 0.728         | 0.959              |
-| CSV Wrangling MESSY | **0.849** | 0.943       | 0.943        | 0.943       | 0.705         | 0.956              |
+| CSV Wrangling | **0.927** | 0.950             | 0.945        | 0.935       | 0.724         | 0.956              |
+| CSV Wrangling CODEC | **0.916** | 0.948       | 0.948        | 0.944       | 0.728         | 0.959              |
+| CSV Wrangling MESSY | **0.905** | 0.943       | 0.943        | 0.943       | 0.705         | 0.956              |
 
 ### Component Accuracy
 
 csv-nose's delimiter and quote detection accuracy on each dataset:
 
 | Data set | Delimiter Accuracy | Quote Accuracy |
 |:---------|:-------------------|:---------------|
-| POLLOCK  | 96.62%             | 100.00%        |
+| POLLOCK  | 97.30%             | 100.00%        |
 | W3C-CSVW | 99.55%             | 100.00%        |
-| CSV Wrangling | 89.94%         | 96.65%         |
-| CSV Wrangling CODEC | 89.44%   | 96.48%         |
-| CSV Wrangling MESSY | 88.10%   | 96.03%         |
+| CSV Wrangling | 93.30%         | 99.44%         |
+| CSV Wrangling CODEC | 92.25%   | 99.30%         |
+| CSV Wrangling MESSY | 91.27%   | 99.21%         |
 
-> NOTE: See [PERFORMANCE.md](docs/PERFORMANCE.mdPERFORMANCE.md) for details on accuracy breakdowns and known limitations.
+> NOTE: See [PERFORMANCE.md](docs/PERFORMANCE.md) for details on accuracy breakdowns and known limitations.
 
 ### Benchmark Setup
 

src/benchmark.rs

@@ -261,7 +261,7 @@ fn parse_delimiter(name: &str) -> u8 {
         "space" => b' ',
         "vslash" | "pipe" => b'|',
         "colon" => b':',
-        "nsign" => 0xA7, // Section sign (§)
+        "nsign" => b'#', // Number sign (#)
         "slash" => b'/',
         _ => b',', // Default to comma
     }
@@ -435,6 +435,9 @@ mod tests {
         assert_eq!(parse_delimiter("space"), b' ');
         assert_eq!(parse_delimiter("vslash"), b'|');
         assert_eq!(parse_delimiter("colon"), b':');
+        // "nsign" is the CSVsniffer annotation name for number sign (#, 0x23).
+        // It must NOT map to 0xA7 (§, section sign), which is a different Unicode character.
+        assert_eq!(parse_delimiter("nsign"), b'#');
     }
 
     #[test]