Parallel parsing of Csv files

[nhirschey]

It should be more obvious how to parallelize Csv file parsing. This is important for big files and multicore PCs. Two possibilities:

1. Add some sort of Parallel option to CsvFile.Load() that triggers use of parallel map under the hood. For example a simple version of the function might be:

`

type test = CsvProvider<"test.csv">
let LazyParallelCsvLoadRows file =
    System.IO.File.ReadLines(file)
    |> Seq.skip 1
    |> PSeq.map(fun r -> test.ParseRows(r).[0])

let EagerParallelCsvLoadRows file =
    System.IO.File.ReadAllLines(file)
    |> Array.skip 1
    |> Array.Parallel.map(fun r -> test.ParseRows(r).[0])

`
2. Don't change any code, but add something like this as examples to the website documentation.

Thoughts?

[ovatsus]

For big files, the main issue is memory, not CPU. Switching from ReadLines to ReadAllLines would put the entire csv file in memory. Right now you can read a 8gb file do some transformation and write to another file without hitting memory issues, as only one row is kept at a time, and this would regress that. So I'm going to close this issue as by design

[nhirschey]

Thanks for the thoughts.

Also, I’m thinking of added functionality. Don’t want to change the current default.

For me, the issue using CsvProvider is CPU. I can’t be the only one. The lazy option above with readlines and PSeq gives us both what we want.

Loading a large multi-gig file into memory for analysis was taking me 8 minutes. The above two parallel versions both gave me 4x speedup on an 18 core PC. Not great scaling, but I’ll take 4x when I’m sitting there staring at the screen. I now use this code regularly.

At skills matters you asked for documentation pull requests. What about going with option 2 in my first note? Would you be open to me showing these parallel versions in a new documentation section ?

[ovatsus]

Sure, let's add examples then

[github-actions]

🤖 This is an automated response from Repo Assist.

Just to summarise what's been discussed here: parallel CSV parsing is very doable using the existing API, and the workaround @nhirschey described works well in practice.

The recommended parallel approach:

type TestCsv = CsvProvider<"test.csv">

// Eager parallel (returns array, all rows in memory):
let rows =
    System.IO.File.ReadAllLines("test.csv")
    |> Array.skip 1  // skip header
    |> Array.Parallel.map (fun line -> TestCsv.ParseRows(line).[0])

// Lazy parallel with FSharp.Collections.ParallelSeq:
// #r "nuget: FSharp.Collections.ParallelSeq"
open FSharp.Collections.ParallelSeq
let rows =
    System.IO.File.ReadLines("test.csv")
    |> Seq.skip 1
    |> PSeq.map (fun line -> TestCsv.ParseRows(line).[0])
    |> PSeq.toArray

Memory note: The Array.ReadAllLines approach loads the entire file into memory. For very large files, you may want to process chunks in parallel manually to avoid memory pressure.

Why it's not built in: Adding a parallel option to CsvFile.Load would require careful handling of line ordering, quoting across line boundaries (RFC 4180 allows quoted newlines in cells), and the lazy-vs-eager trade-off. These make it non-trivial to generalise. The ParseRows workaround sidesteps these issues by processing one line at a time and is already available.

A documentation PR adding these parallel examples would be a welcome contribution — the docs/ folder has the relevant CSV provider tutorial.

Generated by Repo Assist

To install this workflow, run gh aw add githubnext/agentics/workflows/repo-assist.md@b87234850bf9664d198f28a02df0f937d0447295. View source at https://github.com/githubnext/agentics/tree/b87234850bf9664d198f28a02df0f937d0447295/workflows/repo-assist.md.

[dsyme]

Good answer