What if all validation records were in .jsonl files?!

[yarikoptic]

This is a complement to

• ENH: Make ./run_tests.sh to optionally store bids-validator outputs under derivatives/bids-validator #547
  inspired by
• Initial design notes bids-utils#4

and wonderful program https://www.visidata.org/ .

Inspired by bids-validator being able to produce json outputs, I did

• ENH: Machine-readable validate output with store/reload dandi/dandi-cli#1822

where dandi-cli already interfaces bids-validator (in addition to nwb inspector etc) to validate bids datasets.

The content here is the result of running

    for dd in */dataset_description.json; do d=$(dirname $dd); dod=$d/derivatives/dandi-validator/; mkdir -p $dod; duct --f-t 0 -p $dod/duct_ dandi validate -o $dod/output.jsonl $d; done

so it has also log files on what we spit out warnings etc captured by duct. Going back to wonders of visidata and jsonl (json lines). Since those logs are just dumps of validation records, I can just

cat */derivatives/dandi-validator/output.jsonl > dandi-validator-all.jsonl

and visualize that jsonl with all its tens of thousands of lines

❯ wc -l dandi-validator-all.jsonl
147586 dandi-validator-all.jsonl

in a second in visidata

and in another second figure out distribution across all issue codes:

in another subsecond concentrating on the one of direct interest to me

So, this PR is nothing to merge but rather potentially let people play with such representation of validation results using the validation results schema we have arrived to in dandi-cli (attn @candleindark)

[yarikoptic]

ok, it is in the history now. I guess with a little of jq we could extract and convert to jsonl regular bids-validator json logs too happen someone decides to practice