Add CSV file directive

[theletterf]

This adds a CSV file directive.

Fixes #1054

[github-actions]

🔍 Preview links for changed docs

• docs/syntax/csv-include.md
• docs/syntax/index.md

[Mpdreamz]

Nice!

Can we defer the parsing to when we actually need it?

This to keep the data on AST itself smaller.

We can leverage this csv library for low overhead reading the data out of the csv. Should help reduce allocations. E.g we could read the CSV inside the template (cshtml) and write the data to its output.

https://github.com/nietras/Sep

[theletterf]

@Mpdreamz I think this is it (as for using only Sep).

[Mpdreamz]

ReadCsv should be an IEnumerable so that we can read and discard data through exposing the reader from Sep directly.

That way if someone does use a 2gb CSV file it won't need to be loaded in memory all at once.

That said we should have some sane defaults and validation on the max size of the csv files wdy reckon @theletterf

[shainaraskas]

the csv I need to render is a somewhat biggie at ~250k so we should set a relatively large max

[theletterf]

@Mpdreamz @shainaraskas Changes made, perhaps we could test it with a humongous CSV?

New params:

max-rows: Maximum number of rows to display (default: 10,000)
max-columns: Maximum number of columns to display (default: 100)
max-size: Maximum file size to process (default: 10MB). Supports KB, MB, GB units.

[shainaraskas]

tested it out and the 3 params work as expected. tried with a big csv (20k rows) and it worked fine. I tested it with a 22 column, 2748 row spreadsheet and it was ok too.

[Mpdreamz]

@theletterf appologies should have been clearer on my end.

I think it's best to hard code these parameters and not make them configurable.

E.g these should be validations against abuse/unintended usages not open for configuration as options.

[Mpdreamz]

this should also be IEnumerable<string[]>, probably easier to make it a method.

That way the cshtml can lazily iterate over large csv files too.

Let me know if you want me to pick that up in a follow up @theletterf !

[KOTungseth]

Does this close #1054?

[theletterf]

@Mpdreamz @shainaraskas Done in 40c24f4:

• Directive renamed to csv-include (files, code, docs)
• Added lazy iteration for better perf
• Removed settings and kept hardcoded defaults

[shainaraskas]

20000 rows ✅
102 columns ✅
53mb ✅

my only note: what benefit does preview mode bring, given that we don't have a download option?

[theletterf]

@shainaraskas You're right! Preview mode removed. I also increased the row limit to 25k, since your test would have been truncated otherwise (I wonder though, do we really plan on showing such a long CSV file in the docs?)

[shainaraskas]

@theletterf I was just testing that the limits were working. my use case is a file under 3k lines.