Merge pull request #21 from ws-garcia/improved-delimiter-guesser

Improved delimiter guesser

Author: ws-garcia

.github/workflows/LABEL_OPENED_ISSUES_WORKFLOW.yml

@@ -13,5 +13,5 @@ jobs:
       - name: Label issues
         uses: andymckay/[email protected]
         with:
-          add-labels: "bug, unconfirmed"
+          add-labels: "unconfirmed"
           repo-token: ${{ secrets.GITHUB_TOKEN }}

README.md

@@ -6,20 +6,20 @@
 VBA CSV interface is the most complete, and open source, CSV/TSV VBA parser library nowadays. The library is RFC-4180 compliant and enables users to manipulate CSV content at the highest speed. All the modules were developed to accomplish the data exchange task with the greatest performance and to grant an easy use.
 
 ## Advantages
-* __Stable__. Fully Test Driven Developed (TDD) library, ([50/50 test passed](https://github.com/ws-garcia/VBA-CSV-interface/blob/master/testing/tests/results/)), that includes 500+ line of code for testing. See [VBA test library by Tim Hall](https://github.com/ws-garcia/vba-test).
+* __RFC-4180 specs compliant__.
+* __Stable__. Fully Test Driven Developed (TDD) library, ([60/60 test passed](https://github.com/ws-garcia/VBA-CSV-interface/blob/master/testing/tests/results/)), that includes 650+ line of code for testing. See [VBA test library by Tim Hall](https://github.com/ws-garcia/vba-test).
 * __Fast__. Writes and reads files at the highest speed.
 * __Memory-friendly__. CSV/[TSV](https://www.iana.org/assignments/media-types/text/tab-separated-values) files are processed using a custom stream technique, only 0.5MB are in memory at a time.
 * __Robust__. Parser and writer accept [Unix-style quotes escape sequences](https://www.loc.gov/preservation/digital/formats/fdd/fdd000323.shtml#notes). 
 * __Easy to use__. A few lines of code can do the work!
+* __Automatic delimiter guesser__. Don't worry if you forgot the file configuration. The interface has a solid strategy for guessing delimiters!
 * __Highly Configurable__. User can configure the parser to work with a wide range of CSV files.
 * __CSV data subsetting__. Split CSV data into a set of files with related data.
 * __Like SQL queries on CSV files__. Add your own logic to mimic SQL queries and filter data by criteria (=, <>, >=, <=, AND, OR).
-* __Automatic delimiter guesser__. Don't worry if you forgot the file configuration!
 * __Flexible__. Import only certain range of records from the given file, import fields (columns) by indexes or names, read records in sequential mode. 
 * __Dynamic Typing support__. Turn CSV data field to a desired VBA data type.
 * __Data sorting__. Sort CSV imported data using the hyper-fast(100k records per second) [Yaroslavskiy Dual-Pivot Quicksort](https://web.archive.org/web/20151002230717/http://iaroslavski.narod.ru/quicksort/DualPivotQuicksort.pdf) like Java.
 * __Microsoft Access compatible__. The library has a version for those who feel in comfort working through DAO databases, [download from here](https://github.com/ws-garcia/VBA-CSV-interface/raw/master/src/Access_version.zip).
-* __RFC-4180 specs compliant__.
 
 ## Getting started
 

docs/home/NON-Quoted-performance.png

@@ -162,7 +162,7 @@ conf.recordsDelimiter = vbCrLf
 
 The benchmark provided here is focused on the supposed most critical operation, this is the parse one for many authors.
 
-The class was tested against two solutions (the one from [@Senipah](https://github.com/Senipah/VBA-Better-Array) and the other from [@sdkn104](https://github.com/sdkn104/VBA-CSV)) using a laptop running `Win 10 Pro x64, Intel® Core™ i7-4500U CPU @1.80-2.40 GHz, 8 GB RAM, Excel 2019 x86`. We will call the import procedure over different files, increasing the file size, and the number of record per file, in each subsequent call. The CSV files are:
+The class was tested using a laptop running `Win 10 Pro x64, Intel® Core™ i7-4500U CPU @1.80-2.40 GHz, 8 GB RAM, Excel 2019 x86`. We will call the import procedure over different files, increasing the file size, and the number of record per file, in each subsequent call. The CSV files are:
 
 <table>
 <thead>
@@ -204,19 +204,9 @@ The images below shows the overall performance for the imports operations from t
 
 ![Quoted Benchmark](Quoted-performance.png)
 
-The benchmarks from the above charts are compared in the following chart:
+### Conclusion
 
-![Quoted Vs NONquoted Benchmark](Quoted-vs-NONquoted.png)
-
-Finally, the below chart shows the overheat for the Sorting and Dynamic Typing operations. These features are available on the VBA CSV interface since its version 3.
-
-![Overheat per functionality Benchmark](Overheat-per-functionality.png)
-
-### Conclusions
-
-- `ImportFromCSV` is the faster one import method.
-- The CSV syntax slow-down the performance. When the number of escaped fields are increased, the performance decrease, this is especially noticeable for the @sdkn104 solution.
-- The Dynamic Typing causes more overheat than the Sort operation. This can be explained by the great performance of the Yaroslavskiy sorting algorithm used.
+- The CSV syntax slow-down the performance. When the number of escaped fields are increased, the performance decrease.
 
 ## Licence
 Copyright (C) 2021  [W. García](https://github.com/ws-garcia/VBA-CSV-interface/).

docs/home/Overheat-per-functionality.png

@@ -30,23 +30,23 @@ In the table bellow all the rules of [RFC-4180](https://www.ietf.org/rfc/rfc4180
 </tr>
 <tr>
 <td style="text-align: left;"><em>There maybe an optional header line appearing as<br> the first line of the file with the same format<br> as normal record lines.  This header will contain<br> names corresponding to the fields in the file and<br> should contain the same number of fields as the<br> records in the rest of the file.</em></td>
-<td style="text-align: left;">In the same way. The presence or absence of the<br> header line should be indicated via the optional<br> "HeadersOmission" parameter.</td>
+<td style="text-align: left;">In the same way. The presence or absence of the<br> header line should be indicated via the option<br> <code>headersOmission</code>.</td>
 </tr>
 <tr>
 <td style="text-align: left;"><em>Within the header and each record, there may be<br> one or more fields, separated by commas.  Each<br> line should contain the same number of fields<br> throughout the file.  Spaces are considered part<br> of a field and should not be ignored.  The last<br> field in the record must not be followed by a<br> comma.</em></td>
 <td style="text-align: left;">The class accepts CSV files with different numbers<br> of fields per record. The spaces betwen the<br> fields separator char and a single field is ignored<br> only if that field is enclosed in double quotes.</td>
 </tr>
 <tr>
 <td style="text-align: left;"><em>Each field may or may not be enclosed in double<br> quotes (however some programs, such as Microsoft<br> Excel, do not use double quotes at all).  If<br> fields are not enclosed with double quotes, then<br> double quotes may not appear inside the fields</em></td>
-<td style="text-align: left;">In the same way. The class accepts also the<br> apostrophe char for indicate fields needing to<br> be escaped. It's important to notice that a<br> single CSV record may have fields enclosed and<br> not enclosed by the escape char.</td>
+<td style="text-align: left;">In the same way. The class accepts also the<br> apostrophe and tilde char for indicate fields needing to<br> be escaped. It's important to<br> notice that a single CSV record may have fields enclosed and<br> not enclosed by the escape char.</td>
 </tr>
 <tr>
 <td style="text-align: left;"><em>Fields containing line breaks (CRLF), double<br> quotes, and commas should be enclosed in double<br> quotes</em></td>
 <td style="text-align: left;">In the same way. Also accepts fields enclosed by<br> the apostrophe char.</td>
 </tr>
 <tr>
 <td style="text-align: left;"><em>If double-quotes are used to enclose fields, then<br> a double-quote appearing inside a field must be<br> escaped by preceding it with another double quote.</em></td>
-<td style="text-align: left;">Ignored rule. The class accepts the apostrophe<br> as escape char, and follow the specs claims<br> may cause conflict with some abbreviate US<br> slangs (e.g.: "<strong>isn't</strong>").</td>
+<td style="text-align: left;">In the same way. The class also accepts a Unix-style quote escape by preceding the quote with a<br> backslash (<strong>"\"</strong>").</td>
 </tr>
 </tbody>
 </table>

docs/home/Quoted-performance.png

@@ -13,17 +13,17 @@ VBA CSV interface is the most complete, and open source, CSV/TSV VBA parser libr
 {: .fs-6 .fw-300 }
 
 ## Advantages
-* __Stable__. Fully Test Driven Developed (TDD) library, ([50/50 test passed](https://github.com/ws-garcia/VBA-CSV-interface/blob/master/testing/tests/results/)), that includes 500+ line of code for testing. See [VBA test library by Tim Hall](https://github.com/ws-garcia/vba-test).
+* __RFC-4180 specs compliant__.
+* __Stable__. Fully Test Driven Developed (TDD) library, ([60/60 test passed](https://github.com/ws-garcia/VBA-CSV-interface/blob/master/testing/tests/results/)), that includes 650+ line of code for testing. See [VBA test library by Tim Hall](https://github.com/ws-garcia/vba-test).
 * __Fast__. Writes and reads files at the highest speed.
 * __Memory-friendly__. CSV/[TSV](https://www.iana.org/assignments/media-types/text/tab-separated-values) files are processed using a custom stream technique, only 0.5MB are in memory at a time.
-* __Robust__. Parser and writer accept [Unix-style quotes escape sequences](https://www.loc.gov/preservation/digital/formats/fdd/fdd000323.shtml#notes).
+* __Robust__. Parser and writer accept [Unix-style quotes escape sequences](https://www.loc.gov/preservation/digital/formats/fdd/fdd000323.shtml#notes). 
 * __Easy to use__. A few lines of code can do the work!
+* __Automatic delimiter guesser__. Don't worry if you forgot the file configuration. The interface has a solid strategy for guessing delimiters!
 * __Highly Configurable__. User can configure the parser to work with a wide range of CSV files.
 * __CSV data subsetting__. Split CSV data into a set of files with related data.
 * __Like SQL queries on CSV files__. Add your own logic to mimic SQL queries and filter data by criteria (=, <>, >=, <=, AND, OR).
-* __Automatic delimiter guesser__. Don't worry if you forgot the file configuration!
 * __Flexible__. Import only certain range of records from the given file, import fields (columns) by indexes or names, read records in sequential mode. 
 * __Dynamic Typing support__. Turn CSV data field to a desired VBA data type.
 * __Data sorting__. Sort CSV imported data using the hyper-fast(100k records per second) [Yaroslavskiy Dual-Pivot Quicksort](https://web.archive.org/web/20151002230717/http://iaroslavski.narod.ru/quicksort/DualPivotQuicksort.pdf) like Java.
-* __Microsoft Access compatible__. The library has a version for those who feel in comfort working through DAO databases, [download from here](https://github.com/ws-garcia/VBA-CSV-interface/raw/master/src/Access_version.zip).
-* __RFC-4180 specs compliant__.
+* __Microsoft Access compatible__. The library has a version for those who feel in comfort working through DAO databases, [download from here](https://github.com/ws-garcia/VBA-CSV-interface/raw/master/src/Access_version.zip).

docs/home/Quoted-vs-NONquoted.png

@@ -15,7 +15,7 @@ VBA is the version of Visual Basic shipped with Microsoft Office. At this time,
 
 At the same time, Microsoft aims the 32-bit version of Excel is [limited to 4GB of RAM](https://docs.microsoft.com/en-us/office/troubleshoot/excel/laa-capability-change), on x64 OS’s, for most recent versions and to 2GB for versions up to 2013. Again, VBA can’t avoid this limitation.
 
-By the fact CSV interface works, primarily, with strings and [VBA uses 10 bytes + string length](https://docs.microsoft.com/en-us/office/vba/language/reference/user-interface-help/data-type-summary) for store this data type, a huge working load can let application run out of memory. Then, it’s crucial to set some boundary over the file size users can work from the CSV interface library. 
+By the fact CSV interface works, primarily, with `Variant/Strings` and [VBA uses 10 bytes + string length](https://docs.microsoft.com/en-us/office/vba/language/reference/user-interface-help/data-type-summary) for store this data type, a huge working load can let application run out of memory. Then, it’s crucial to set some boundary over the file size users can work from the CSV interface library. 
 
 To achieve this, the first step is to study the performance of the VBA-CSV interface when parsing 5K records from the top and from the end of files with varying size, and then establish the recommended maximum amount of data that can be handled from the interface.