@serhiy-storchaka

Thank you for review!

We need to consider not only the gain, but also the cost/benefit ratio. This change significantly complicates an already complex code.

I agree that it increases complexity.

I added an explanation before the switch block, and two macros:

https://github.com/python/cpython/pull/138214/files#diff-38fcce6bb475616052f5c9a0973eefd49489a4dff719f30e407534258e2a3ec3R1030-R1080

There are only two building blocks: jump with PyUnicode_FindChar, or parse each with PyUnicode_READ_CHAR. This might have been obscured by duplication before.

My thinking here goes that csv.reader is a global hot path, one of the most common usages of Python. Tuning this code has some real positive impact, and it's an explainable algorithmic improvement (without relying on low-level SIMD magic.)

I believe that conceptually it's simple: process the whole field at once.

The benefit is not unconditional either. In the IN_FIELD state, PyUnicode_FindChar() is called 4 times. This can actually slow down the code for long non-ASCII lines.

The ideal would be PyUnicode_strcspn(). :-)

I fully agree there are scenarios where it's worsening performance. My hunch is that CSV files without any fields (eg: the parser is spending significantly more time in states other than IN_FIELD and IN_QUOTED_FIELD) it'd worsen. How important are they?

What's the best way of sharing here? I already feel that all these compare_to --table --table-format md are overwhelming.

I'm not sure what is the best benchmarking strategy here. Unfortunately, all the combinations from the benchmark with --rigorous already take around ~30 minutes (despite NVMe, i9-12900K and 128G memory), so I cannot range randomly. I will think how to approach this.

I tried measuring a long non-ASCII line and still observed a significant benefit:

import csv
import io
import os
import pyperf
import random

NUM_ROWS = (1_000, 10_000)
NUM_COLS = (5, 10)
FIELD_LENGTH = (300, 1000)

CASES = [
    # (label, field_chars, delimiter, escapechar)
    (
        "nonascii_no_escape",
        "ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩαβγδεζηθικ",
        "λ",
        None,
    ),
    (
        "nonascii_escape",
        "ΑΒΓΔΕΖΗΘΙΚΛΜΝΞΟΠΡΣΤΥΦΧΨΩαβγδεζηθικ",
        "λ",
        "\\",
    ),
]


def generate_csv_data(rows, cols, field_len, ch, delim):
    # random.choice() so we're not cache-friendly
    field = "".join(random.choices(ch, k=field_len))
    row = delim.join([field] * cols)
    return os.linesep.join([row] * rows) + os.linesep


def benchmark_csv_reader(csv_data, delim, escapechar):
    rdr = csv.reader(
        io.StringIO(csv_data), delimiter=delim, escapechar=escapechar
    )
    for _ in rdr:
        pass


runner = pyperf.Runner()

for rows in NUM_ROWS:
    for cols in NUM_COLS:
        for field_len in FIELD_LENGTH:
            for label, ch, delim, esc in CASES:
                csv_data = generate_csv_data(rows, cols, field_len, ch, delim)
                runner.bench_func(
                    f"csv_reader({rows},{cols},{field_len})[{label}]",
                    benchmark_csv_reader,
                    csv_data,
                    delim,
                    esc,
                )

I updated the benchmark in the description to be more comprehensive.