dev environments in DO spaces

[fvankrieken]

Seems maybe a bit overdue that it might be time to have dev environments for edm-recipes and edm-publishing. Maybe less important for publishing (at least on the builds side of things), but very much needed for recipes - we obviously have "dev" recipes as we add them, or when we make tweaks to one for our dev branch that causes nightly builds to fail.

EDIT - I think I would prefer separate buckets than hide this logic in new folders structures Could basically have something like this

edm-recipes
├- datasets
└- dev
   └- fvk-my-new-feature
       └- datasets
       └- raw_datasets

Other changes would then be

• in library call, have boolean flag to run in dev environment. Uses same "build_name" logic as our usual builds
• in recipes connector (for fetching an archived recipe) have some sort of env var to specify whether it's a dev build.
  • if not a dev build, run as normal
  • if so, first look for dev version of datasets, then fall back to "production" input datasets
• for build.yml, have a checkbox if this is a "dev" build. If so, look for dev versions first for all input datasets
• output to dev location similar to scheme above in recipes. It seems worthwhile that a "dev" build should not be able to be promoted in our usual environment, because it depends on a dev source dataset, meaning this build can't necessarily be repeated/verified/etc down the road

Outstanding questions/thoughts/etc

• should this be separated more fully? We could have prod/staging/dev buckets. Given our general workflows, I don't think we need a "staging" bucket, and for whatever reason it feels more burdensome to have individual dev buckets (or one dev bucket with a different substructure than our production bucket).
• should a dev dataset be able to be "promoted"/migrated to production? Or just re-run. We could imagine a case where we've archived a version that's no longer available, but for some reason we want that specific version for our production build of a product rather than the latest available

[fvankrieken]

Having individual dev buckets would be nice in that we're then only changing one thing for these utils (bucket), makes a nice simplicity. And we could just tear them down with our our weekly cleaning job. We don't need this often, so I don't think things would get that cluttered.

[sf-dcp]

• I'm leaning towards having a separate bucket because implementation seems to be minimal + it will replicate our tests where we use a fake bucket.
• I think re-running from production because using dev env is for to confirm the code works rather than to extract a specific data version. I also think this decision applies to only archiving datasets, not building. In the worst case, we could always manually copy files from dev to production.

Other thoughts:

• I don't think we need to add a dev option for build.yml because our build folder is for dev purposes by default, no?

@fvankrieken

[fvankrieken]

Yeah I agree very much with first two points.

For build - I think that

1. builds should be able to use archived source datasets from a dev bucket (dev version of edm-recipes), which would need this option in build.yml (logic being -> for all source datasets, look for dev version, use if found, fall back to production version). Since often we're changing source datasets (configs/preprocessing, like what Damon is doing in FacDB right now), we want to be able to run a complete pipeline with these dev source datasets.
2. builds should not be able to be promoted/etc if they've used a dev source dataset, for record-keeping purposes (using non-production source data) - hence why I think it might make sense just to dump them in their own bucket as well just to avoid any potential errors

[sf-dcp]

Oh makes sense! This sounds great :)