requirements to publish dcpy ingest code

[fvankrieken]

More focused on code/functionality here than docs

Main things from discussion on 2/19

• functionality to provide "template" via cli (or config -> custom template folder)
  • ideally, implement the latter and then we can move our own templates out of dcpy?
  • Might be good to separate our own "opinionated"/"operational" templates from what we use for the demo
  • Where do "demo" templates for the presentation live? In their own repo? In dcpy as documentation?
• easily runnable without configuration
  • default to just dumping data locally, s3 part is optional
  • a bit of customization/generalization of where data gets stored. Ideally this could happen either locally or in S3, not sure if there are other things we would support. Need to think about a more generalized api around the data store as well as part of this code
  • bit of a dependency of above - don't default to pushing to s3, and don't error out without s3 env vars set
• arcgis online sources
• simplify (or at least make clear) cli targets
• output formats? folks might want csv/shapefile but maybe this is more in nice-to-have (or wait to see if there's demand)

[fvankrieken]

More nice-to-haves

• extensions for custom connectors (and maybe make our own connectors private?)
• custom data checks (a bit blocked by first having data checks fully operational)
• generate templates from certain source types. i.e. given a socrata four-four, generate the template to ingest it, pulling url and description into the yml. You could argue maybe we could allow just running ingest given only a socrata four-four, but this would be a nice first step
• geocoding - I could imagine this being a really nice-to-have feature for some users. One day I'll get back to that pr...