gh-83273: Rewrite csv.Sniffer dialect detection using trial parsing

[serhiy-storchaka]

Summary

csv.Sniffer.sniff() now guesses the dialect by parsing the sample with every plausible combination of delimiter, quotechar and escapechar, using the actual CSV parser in strict mode, and choosing the combination which splits the sample into rows with the most consistent number of fields. The result is consistent with how csv.reader() will parse the sample — by construction.

The old heuristics are removed. They guessed the delimiter from characters adjacent to quotes (_guess_quote_and_delimiter) and from character frequency statistics (_guess_delimiter), so a delimiter inside a quoted field could win over the actual one, sniffing took quadratic time on ordinary quoted data (20+ seconds on a ~1 MB sample, which looked like a hang), and an escape character could not be detected at all.

A large sample is parsed incrementally: first only its beginning, then, after eliminating the combinations which are clearly worse than the leader, a several times larger part, and so on. Combinations with a quotechar are not parsed until the quote character occurs at the start of a field. The verdict that a delimiter does not delimit anything is not trusted to the first part of the sample, which covers its least representative beginning (titles, comments, headers).

User-visible changes

• csv.Sniffer can now detect escapechar='\\'.
• Explicitly requested delimiters are no longer restricted to ASCII.
• A delimiter inside a quoted field no longer wins over the actual delimiter, and sniffing no longer takes quadratic time on quoted samples.
• The sample can be cut off at an arbitrary point: in the middle of a row, of a quoted field or of an escaped sequence.
• Only '\r', '\n' and '\r\n' are treated as row separators, matching the reader; characters like '\x1c' can now be detected as a delimiter.
• Letters and digits are no longer guessed as a delimiter.
• A preamble (title or comment lines) before the data does not prevent detection if the data rows outnumber the preamble lines.
• A sample consisting of a single column of quoted fields now raises csv.Error instead of guessing a delimiter from the content of the fields, and Sniffer.has_header() no longer raises ValueError for such samples.
• doublequote is detected by comparing the two readings of the sample.

Results may differ from earlier Python versions on valid samples; this is documented with a versionchanged note and a What's New entry.

Performance

Release build, sniff() wall time:

sample	old	new
typical quoted CSV, 4 KB	0.1 ms	0.2 ms
quoted CSV, 64 KB	1.5 ms	2.3 ms
quoted CSV, 940 KB	> 20 s	30 ms
single column of quoted fields, 1.1 MB	minutes (gh-98820)	10 ms
random garbage, 1 MB	9 ms (meaningless answer)	~25 ms (Error)

The old regular expressions were quadratic on quoted data; the new code is linear in the sample size with a small factor.

Closed issues

• Closes Csv sniffer doesn't attempt to determine and set escape character. #83273: escapechar could not be detected.
• Closes csv.Sniffer().sniff() Hangs on Single-Column Data #98820: sniffing a single column of quoted fields took minutes.
• Closes csv.Sniffer regular expression has significant backtracing #109638: catastrophic backtracking on an artificial sample.
• Closes CSV sniffing detects wrong field delimiter #97611: a quoted tab won over the actual delimiter.
• Closes The CSV file delimiter found with csv.Sniffer() is interpreted incorrectly #119123: commas inside bracketed lists won over the actual delimiter.
• Closes CSV sniffing falsely detects space as a delimiter #88843: a space next to stray quotes was detected as the delimiter.
• Closes csv.Sniffer.snif doesn't set up the dialect properly for a csv created with dialect=csv.excel_tab and containing quote (") char #62029: tab-delimited data with quotes was detected as space-delimited.
• Closes In csv.py - guess_delimiter: add delimiters to list of eligible delimiters #111820: a non-ASCII delimiter in the delimiters argument was ignored.
• Closes csv.py _guess_quote_and_delimiter should be able to handle windows \r #103925: a quote before '\r\n' broke detection (the delimiter was detected as a letter).
• Closes Floating point rounding error in _guess_delimiter of csv module #111487: the floating-point threshold loop no longer exists.

Also addressed, at the triagers' discretion:

• csv.Sniffer.has_header() leaks ValueError instead of csv.Error for an inconsistent guessed dialect #153634: fixed at the root — sniff("''") now raises csv.Error, so has_header() cannot leak ValueError.
• csv.Sniffer uses a heuristic to determine doublequote that is often false-negative #109339: the doublequote false positive is fixed; the proposed default-to-True policy was deliberately not adopted — detection remains evidence-based.

Supersedes the stalled PRs gh-109639, gh-103926, gh-153635 and gh-27256, which patch the removed regular expressions. gh-109639 and gh-153635 remain valid as narrow backport candidates for the maintenance branches.

Notes for reviewers

• No backports: results change on valid inputs, so this is for 3.16 only. The hang class can be fixed on maintenance branches by the narrow regex patches mentioned above.
• escapechar detection is cleanly severable (one loop variant in candidate construction) if landing it separately is preferred; nine of the ten closed issues do not depend on it.
• Detection of quoting (Feature request: sniff csv quoting and lineterminator #129374) and lineterminator (csv.Sniffer does not detect lineterminator #75008) is deliberately out of scope; the new structure has a natural place for both (_detect_* methods on the winning combination).

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #33581348 | 📁 Comparing 7c9f890 against main (357c650)

  🔍 Preview build  

3 files changed

± library/csv.html
± whatsnew/3.16.html
± whatsnew/changelog.html