Best-practice read-parse-convert-write large file?

[yooakim]

I'm working on converting large files (usually 1.5 GB ) incoming TSV files, parse the records, do some conversions and then write out them in another format. records and classmaps works fine. CSVHelper is a joy to work with! I will write a console application using .NET 6)

Before I start I am wondering if there are any best-practices people here like to share for doing this type of work?

Should the process go record-by-record or should the process use GetRecords?

I have read a lot about streams, memory mapped files etc and I understand there are many options for doing this. So I am asking in the context of CSVHelper, what have been working and what hasn't worked so well with large files.

I really appreciate any pointers to good samples, suggestions or advice :-)

[JoshClose]

Streaming the records from one file to another will use the smallest amount of memory. You can use GetRecords as it yields results back; meaning it only works on a single row as you iterate. If you do any sort of projection like ToList, then the entire record set will be pulled into memory. You can loop the records without worry though.

Here is an example.

void Main()
{
    var config = new CsvConfiguration(CultureInfo.InvariantCulture)
    {
    };
    using var readStream = File.OpenRead("");
    using var reader = new StreamReader(readStream);
    using var csvReader = new CsvReader(reader, config);
    using var writeStream = File.Open("", FileMode.Create);
    using var writer = new StreamWriter(writeStream);
    using var csvWriter = new CsvWriter(writer, config);
    
    foreach (var foo in csvReader.GetRecords<Foo>())
    {
        // Convert from type Foo to type Bar.
        var bar = new Bar
        {
            Id = foo.Id,
            Name = foo.Name,
        };
        csvWriter.WriteRecord(bar);
    }
}

private class Foo
{
    public int Id { get; set; }
    public string Name { get; set; }
}

private class Bar
{
    public int Id { get; set; }
    public string Name { get; set; }
}

[yooakim]

Thanks for the clear example, you are very good at explaining the relevant stuff.

I did use GetRecord but as GetRecords yields it is even simpler, thanks.