[ENG-6284] render tsv/csv

[aaxelb]

allow rendering search responses as lines of tab-separated or comma-separated values

main point:

• add simple_tsv and simple_csv renderers in trove.render
  • can be seen with query param acceptMediatype=text/tab-separated-values or acceptMediatype=text/csv
  • get default columns from static DEFAULT_TABULAR_SEARCH_COLUMN_PATHS in trove.vocab.osfmap
• allow "download" responses -- add withFileName=foo query param to get a response with Content-Disposition: attachment and a filename based on "foo"
• allow absurd page sizes

changes made along the way:

• introduce ProtoRendering as renderer output type, to better decouple rendering from view logic
  • include StreamableRendering for responses that might could be streamed, like csv/tsv (tho it's not currently handled any differently from SimpleRendering)
  • reshape BaseRenderer (and each existing renderer) to have a consistent call signature (and return ProtoRendering)
    • replace trove.render.get_renderer with trove.render.get_renderer_type -- instantiate the renderer with response data
• add trove.views._responder with common logic for building a django HttpResponse for a ProtoRendering
  • consistently handles withFileName/Content-Disposition
• move some osf-specific constants to trove.vocab.osfmap for easier reuse
• pull out some abstractable logic:
  • from existing trove.render.simple_json into trove.render._simple_trovesearch (for renderers that include only the list of search results)
  • from existing tests.trove.derive._base into tests.trove._input_output_tests (for tests following the same simple input/output pattern as deriver and renderer tests)
• add tests.trove.render to cover the new renderers simple_tsv and simple_csv, as well as the existing renderers jsonapi, simple_json, jsonld, and turtle
  • minimally update existing renderers to create consistent output

[coveralls]

coverage: 91.833% (+0.6%) from 91.22%
when pulling fb70351 on aaxelb:feat/render-tsv-csv
into 24bc70a on CenterForOpenScience:develop.

[mfraezz]

A couple nits or questions, but nothing blocking. Tests look sufficient, but behavior should still be confirmed manually on staging.

Pass complete

[mfraezz]

Minor: Is this maximum reasonable? Looks like it was previously 101.

Edit: I see the commit message called it "absurd," but I'm guessing it's also "justified for the sake of rendering files"?

[aaxelb]

yeah the need here is downloading all results in one response, but i hesitated to make that behavior automagic by mediatype... considered making withFileName obviate pagination whenever present, but overall i opted for consistent query param behavior, putting the onus on the client to string together all the params needed for the desired result (e.g. acceptMediatype=text/csv&page[size]=10000&withFileName=my-file-name for a full csv download with up to 10000 rows)

if 10000 at once turns out to be unreasonable in practice... a more complicated (but less costly all-at-once) alternative might be view logic that queries/renders smaller pages one at a time and streams the results

[aaxelb]

update: now streams, loading only one page (~100 rows) at a time, but streaming more than ~4000 items total still times out -- can further optimize or we can talk about increasing those timeouts for responses that are actively sending data...

[mfraezz]

We might not run into those same timeouts for ~4k items with production resourcing (or configuration -- unsure where you got that figure, but by default most nginx timeouts are between successive operations rather than the whole response), but I suspect it's fine for now and we can reevaluate if encountering that issue later.

[aaxelb]

am seeing timeouts at just after 30sec on production

looking at the .csv.part files, each one managed to stream a few pages before it dies... unsure what's stopping it (maybe making it less slow would help...)

[mfraezz]

LGTM