Skip to content

Conversation

theletterf
Copy link
Contributor

@theletterf theletterf commented Aug 14, 2025

This adds a CSV file directive.

Fixes #1054

@theletterf theletterf requested review from a team as code owners August 14, 2025 15:31
@theletterf theletterf self-assigned this Aug 14, 2025
Copy link

github-actions bot commented Aug 14, 2025

🔍 Preview links for changed docs

Copy link
Member

@Mpdreamz Mpdreamz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

Can we defer the parsing to when we actually need it?

This to keep the data on AST itself smaller.

We can leverage this csv library for low overhead reading the data out of the csv. Should help reduce allocations. E.g we could read the CSV inside the template (cshtml) and write the data to its output.

https://github.com/nietras/Sep

@theletterf
Copy link
Contributor Author

@Mpdreamz I think this is it (as for using only Sep).

@theletterf theletterf requested a review from Mpdreamz August 14, 2025 16:51
Copy link
Member

@Mpdreamz Mpdreamz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ReadCsv should be an IEnumerable so that we can read and discard data through exposing the reader from Sep directly.

That way if someone does use a 2gb CSV file it won't need to be loaded in memory all at once.

That said we should have some sane defaults and validation on the max size of the csv files wdy reckon @theletterf

@shainaraskas
Copy link
Contributor

shainaraskas commented Aug 14, 2025

the csv I need to render is a somewhat biggie at ~250k so we should set a relatively large max

@theletterf
Copy link
Contributor Author

@Mpdreamz @shainaraskas Changes made, perhaps we could test it with a humongous CSV?

New params:

max-rows: Maximum number of rows to display (default: 10,000)
max-columns: Maximum number of columns to display (default: 100)
max-size: Maximum file size to process (default: 10MB). Supports KB, MB, GB units.

@shainaraskas
Copy link
Contributor

tested it out and the 3 params work as expected. tried with a big csv (20k rows) and it worked fine. I tested it with a 22 column, 2748 row spreadsheet and it was ok too.

@Mpdreamz
Copy link
Member

@theletterf appologies should have been clearer on my end.

I think it's best to hard code these parameters and not make them configurable.

E.g these should be validations against abuse/unintended usages not open for configuration as options.

@KOTungseth
Copy link
Contributor

KOTungseth commented Aug 18, 2025

Does this close #1054?

@theletterf
Copy link
Contributor Author

@Mpdreamz @shainaraskas Done in 40c24f4:

  • Directive renamed to csv-include (files, code, docs)
  • Added lazy iteration for better perf
  • Removed settings and kept hardcoded defaults

@shainaraskas
Copy link
Contributor

20000 rows ✅
102 columns ✅
53mb ✅

my only note: what benefit does preview mode bring, given that we don't have a download option?

@theletterf
Copy link
Contributor Author

@shainaraskas You're right! Preview mode removed. I also increased the row limit to 25k, since your test would have been truncated otherwise (I wonder though, do we really plan on showing such a long CSV file in the docs?)

@shainaraskas
Copy link
Contributor

@theletterf I was just testing that the limits were working. my use case is a file under 3k lines.

Copy link
Member

@Mpdreamz Mpdreamz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One last nit

@theletterf theletterf requested a review from Mpdreamz August 20, 2025 10:03
@Mpdreamz Mpdreamz changed the title Add CSV file directive Add CSV include directive Aug 20, 2025
Copy link
Member

@Mpdreamz Mpdreamz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! The max columns being a 100 seems odd to me since it will never render nicely I think. We should probably make it much lower e.g 10?

wdyt?

@theletterf
Copy link
Contributor Author

Yeah, it's a bit weird as a limit. Let me change that.

@theletterf theletterf requested a review from Mpdreamz August 20, 2025 10:34
@theletterf theletterf merged commit 741913c into main Aug 20, 2025
19 checks passed
@theletterf theletterf deleted the theletterf-add-csv-directivee branch August 20, 2025 10:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature Request]: Allow CSVs to be passed and rendered as tables
4 participants