[Performance] importing larger CSV files

[fruitl00p]

We're currently trying to import a somewhat large CSV (50 to 100MB with over 150 columns/fields of which we only need about 30 during rowhandling) and even though we chunk and queue it it really takes a long time. Not to actually parse / work the onRow method calls, but before and after the actual processing of a singel job usually takes minutes. (Even on a smaller CSV file of 2mb with smaller chunk sizes but same column counts)

We're currently chunking 1k rows, using batch memory cache. But since the jobs run sequentially this only seems to mitigate the memory issues.

My question is: is this normal for this lib? And if so, are there any plans to provide some sort of parallel job handling of sorts?

If I need to show code or setup, let me know.

Thnx

[patrickbrouwers]

@fruitl00p When dealing solely with CSV imports and wanting high performance, I'd recommend using csv helpers (fgetcsv) (or league/csv) directly. It drops all the overhead that PhpSpreadsheet adds to make it compatible with Excel formats.

And if so, are there any plans to provide some sort of parallel job handling of sorts

There're plans, however I don't have an ETA for it

[fruitl00p]

sorry for the late response, thanks. Are there docs available for using the CSV helpers (from the league/csv especially) and still utilizing this excellent package for quability / chunking et al?

[patrickbrouwers]

We do have plans on releasing a version of this package with league/csv on the background, but I don't have an ETA for the release of that one.

[James4645]

@patrickbrouwers For the record that sounds great. I too use this package for CSV files because of the chunking, upsets support, ect. It makes importing to or from a model effortless. I would love to see it keep the same interface but delegate to league/csv when working with a csv file and PhpSpreadsheet when working with an excel file automatically.

I noticed you have a 4.0 branch. Is this something you are targeting for 4.0?

[patrickbrouwers]

@James4645 It will be a separate package with the same approach. 4.0 release will be something else.