Proposal type
Other — preserve complete CSV records when enforcing the existing slug-sorting rule.
Affected scope
git-hooks/pre-commit.py, which sorts and stages the repository's CSV files before validation. This applies to all network and reference categories, including Algorand, Filecoin and Somnia. No column, category or data value is added.
Motivation / problem statement
At main commit e83cde059024fc5cfaf9881f5435c332905d54d4, sort_csv_by_slug() sorts the result of readlines() using split(','). A valid quoted multiline field is therefore split into independent sortable lines. Running the sorter can corrupt the CSV before downstream validation examines it, and the hook stages the rewritten file.
Minimal valid input:
slug,additionalNotes
zulu,"line one
continuation"
alpha,single
Actual output from the unmodified sorter:
slug,additionalNotes
alpha,single
continuation"
zulu,"line one
The parsed input has two data records; the output has an extra orphan record and a truncated quoted value. Separately, if a quoted field containing a comma precedes slug, splitting on the delimiter sorts on the wrong column.
This is a reproduced input-handling defect, not a claim that current production data has already been damaged. I found no multiline cells in the current checked-out CSV corpus.
Detailed proposal
Use Python's standard csv.reader to determine each record's fields and physical-line boundaries. Sort complete records by the parsed slug while preserving the original record text, quoting, embedded newlines and stable order for equal slugs. Keep the existing newline and parser compatibility behavior; no new schema or strict validation policy is introduced.
A focused implementation and four unittest test methods are ready. They cover LF/CRLF multiline fields, escaped quotes, quoted commas before the slug column, duplicate-key stability, a missing final newline, empty/header-only files, files without a slug column, and idempotence. On the original implementation the tests report three failing subcases; the fixed implementation passes. Comparing old and new sorter outputs on temporary copies of all 997 current repository CSVs produced zero differing output files. No dataset files were modified by this work.
Acceptance criteria
- Sorting preserves the parsed records and all cell values of a valid multiline CSV.
- The slug is found using CSV parsing, including when quoted commas precede it.
- Existing single-line repository data has identical sorter output.
- The regression tests pass offline using the Python standard library.
Reward eligibility
I am submitting this as a proposed database-integrity improvement under the program in Discussion #41. Please confirm whether this tooling proposal qualifies for the approved-DBIP reward; I am not assuming that ordinary network-data cell rewards apply, or that submission earns payment. A compatible receiving address will be provided only if an award is approved.
Authorship and validation
Prepared and tested by an AI coding assistant acting for this GitHub account. The account owner has not personally reviewed the code or verified the links. The reproduction and corpus comparison above were executed locally; the downloaded json-tools validation pipeline has not been run. Public project documentation and local source were used; no production service was probed or modified.
Proposal type
Other — preserve complete CSV records when enforcing the existing slug-sorting rule.
Affected scope
git-hooks/pre-commit.py, which sorts and stages the repository's CSV files before validation. This applies to all network and reference categories, including Algorand, Filecoin and Somnia. No column, category or data value is added.Motivation / problem statement
At
maincommite83cde059024fc5cfaf9881f5435c332905d54d4,sort_csv_by_slug()sorts the result ofreadlines()usingsplit(','). A valid quoted multiline field is therefore split into independent sortable lines. Running the sorter can corrupt the CSV before downstream validation examines it, and the hook stages the rewritten file.Minimal valid input:
Actual output from the unmodified sorter:
The parsed input has two data records; the output has an extra orphan record and a truncated quoted value. Separately, if a quoted field containing a comma precedes
slug, splitting on the delimiter sorts on the wrong column.This is a reproduced input-handling defect, not a claim that current production data has already been damaged. I found no multiline cells in the current checked-out CSV corpus.
Detailed proposal
Use Python's standard
csv.readerto determine each record's fields and physical-line boundaries. Sort complete records by the parsed slug while preserving the original record text, quoting, embedded newlines and stable order for equal slugs. Keep the existing newline and parser compatibility behavior; no new schema or strict validation policy is introduced.A focused implementation and four unittest test methods are ready. They cover LF/CRLF multiline fields, escaped quotes, quoted commas before the slug column, duplicate-key stability, a missing final newline, empty/header-only files, files without a slug column, and idempotence. On the original implementation the tests report three failing subcases; the fixed implementation passes. Comparing old and new sorter outputs on temporary copies of all 997 current repository CSVs produced zero differing output files. No dataset files were modified by this work.
Acceptance criteria
Reward eligibility
I am submitting this as a proposed database-integrity improvement under the program in Discussion #41. Please confirm whether this tooling proposal qualifies for the approved-DBIP reward; I am not assuming that ordinary network-data cell rewards apply, or that submission earns payment. A compatible receiving address will be provided only if an award is approved.
Authorship and validation
Prepared and tested by an AI coding assistant acting for this GitHub account. The account owner has not personally reviewed the code or verified the links. The reproduction and corpus comparison above were executed locally; the downloaded
json-toolsvalidation pipeline has not been run. Public project documentation and local source were used; no production service was probed or modified.