processLargeFile.withStreamingOptions()

[budschneehagen]

Hi. I can't get processLargeFile to use rowCacheSize. The entire sheet is being returned. Am I missing something? Even if I remove it, I get the whole sheet and not the 10 row default. Thank you.

options = {
bufferSize: 2048,
rowCacheSize: 25
}
variables.spreadsheet.processLargeFile(variables.filepath)
.withRowProcessor( processor )
.withStreamingOptions( options )
.withSheetName( "age_breed_group_map" )
.withFirstRowIsHeader()
.execute();

[cfsimplicity]

The entire sheet is being returned

Sorry but I'm not sure what you mean by that. ProcessLargeFile() doesn't return any data, it just runs your row processing function over each row as it's streamed from disk.

What does your row processing function look like?

If it takes each row and adds it to a variable, then obviously that will end up containing the entire sheet data, which case you're probably better off with readLargeFile().

[budschneehagen]

You're correct, I did misunderstand it. I thought it worked like maxrows. I already did something similar to your example. Thank you.

[cfsimplicity]

I wonder if you've misunderstood the purpose of rowCacheSize? It's basically just a tuning option which controls how many rows are read from disk at a time by the streaming reader. It has nothing to do with "returning" anything.

If you are wanting to read your sheet into smaller chunks that your application can then work with, then you could write your processor function to process the rows in batches. Here's an example:

spreadsheet = new spreadsheetCFML.SpreadSheet()
path = ExpandPath( "large.xlsx" )
result = {
  thisBatch: [],
  completedBatches: 0
}
processor = ( rowValues, rowNumber )=>{
  result.thisBatch.Append( rowValues )
  if( result.thisBatch.Len() == 25 ){
    // process this batch of 25 in some way...
    result.completedBatches++ // log how many batches we've processed
    result.thisBatch = [] // start a new batch
  }
}
spreadsheet.processLargeFile( path )
  .withRowProcessor( processor )
  .withFirstRowIsHeader()
  .execute()
WriteDump( result )