develop more generic ingestion code

[jtauber]

consolidate the common elements between the various ingestion modules

This issue can track the generic work and individual issues can be created to update each module.