RFC: DE<>GIS Dataset Review Process + Internal Publishing Versions

[alexrichey]

Problem Statement

When DE has finished a build, we've often encountered some combination of the following problems:

• It's not clear to GIS (or Amanda) what build of a dataset should be reviewed.
• The status of a dataset in the publication lifecycle (being built, awaiting review, in review, approved) isn't clear.
• Files for datasets that have been reviewed are potentially (maybe even often) overwritten by subsequent builds.
• There's not clear lineage backwards from the published / distributed datasests (e.g. on Socrata / Bytes) back to their raw files.
• There's no review history for subsequent builds and reviews of a dataset.
  • e.g. say we're in our third QA cycle of PLUTO 24v1.1, where should we look to know why the second build failed QA?

Proposed Solution

For all of our products, we should add a subfolder under the version to indicate the draft publication version. The current state looks like this:

• current: edm-publishing / db-pluto / publish / 24v1.1 / dataset files
• proposed: edm-publishing / db-pluto / publish / 24v1.1 / draft publication version / dataset files

The draft publication version will be composed of an integer version, and a summary to describe the the build, similar to the summary line of a git commit. A list of builds versions could look like this:

• 1-initial-build
• 2-add-new-corrections
• 3-fix-zoning

I suggest an integer version instead of a timestamp because we don't really care when the draft was published, whereas the integer corresponds to something that we do care about. e.g. if we're in round three of PLUTO publishing, and you see that the last draft publication is 6-fix-the-issue then you immediately know something is wrong.

Draft Publication Github Issues

Our Publishing Github Action will create a Github Issue for every published build version. Decisions, discussions, etc should be documented on that issue. They should all be linked back to a parent Issue for a build of a dataset.

The Issue for the draft publication should use Github Labels to indicate the status. A list of statuses might be:

1. Ready
2. In QA
3. Passed
4. Failed

Perhaps we can auto-add all of GIS as an Assignee

Implementation Details (Technical)

• in DO: Migrate everything in the publish folder to this new scheme.
• in GHA: Modify our Publish action to accept a changes summary field, which will be used to generate the draft publication version. The integer part will be inferred from existing versions on DO.
• in dcpy: The draft publication version concept needs to be added to the edm publish connector. Publish functionality should refuse to overwrite existing data.

Implementation Details (Nontechnical)

• Within this new framework, everything under a products draft folder should be considered deletable.
• Need to work with GIS to change procedures.

Other Considerations

• I think it's nearly time (if not actually time) to do away with the latest folders. It's a convenient hack, but has the liability of being potentially out of sync with actual latest versions. As part of this, we could help GIS migrate off. We could either supply them python code to infer last build version, or add a REST endpoint to the QAQC app to redirect to the DO location.
• We could certainly use some utilities around tidying up our edm publications bucket. e.g. CLI tooling to flag or move "misplaced" items, e.g. edm-publishing / db-pluto / 23v2 which should be (presumably) under the draft folder.
• Regarding what we call the datasets in our publishing folders, is there a better term than draft publications? publication drafts? They really are just drafts of what we'll eventually publish.
• What repo should Draft Publication Issues live in?

Example Workflow for GIS (Copied from @sf-dcp's comment)

Suppose we're building PLUTO v24.1.

1. DE creates parent issue for a dataset build manually, lets call it PLUTO v24.1. DE kicks off a build.
2. DE reviews internally, and publishes via the Github action (no changes to process so far)
3. GHA generates new Issue PLUTO v24.1 1-initial build (child issue), tagged as Ready and GIS team is added to the issue
4. GIS updates the child issue's tag In review:
   Their QA review looks good. They put tag Passed for the child issue
   Their QA review asks for changes --> They put tag Failed for the child issue --> They note required changes in the child issue --> we repeat the process for generating consecutive child issues.

Whiteboarding

[sf-dcp]

Nice write-up! Thank you for doing it. Quick q: we can still use draft/ dir for DE internal testing, correct? And when we intend to build-build, we will be using publish/?

[sf-dcp]

I'm also a bit unclear about GH issues... It sounds like the workflow will be as follows:

• Create parent issue for a dataset build manually, lets call it PLUTO v24.1
• Automatically generated issue PLUTO v24.1 1-initial build (child issue) --> We add tag Ready --> GIS added to the issue --> GIS updates the child issue's tag In review:
  • Their QA review looks good. They put tag Passed for the child issue
  • Their QA review asks for changes --> They put tag Failed for the child issue --> They note required changes in the parent issue --> we repeat the process for generating consecutive child issues.

Does that sound right?

[alexrichey]

Nice write-up! Thank you for doing it. Quick q: we can still use draft/ dir for DE internal testing, correct? And when we intend to build-build, we will be using publish/?

Thank you, and yes. /draft would be for DE only.

[alexrichey]

I'm also a bit unclear about GH issues... It sounds like the workflow will be as follows:
...
Does that sound right?

@sf-dcp Yes - thank you, I was a little light on the details around the GIS workflow. Your distillation is very clear though - will copy that over to the main description above. That's exactly what I'd envisioned.

[alexrichey]

@sf-dcp Ah, the one difference I'd envisioned is that GIS would note required changes in the child issue, not the parent. Updated main text with your example flow.

[fvankrieken]

I really have basically no notes on this write-up. This all sounds great, and makes complete sense for how we generally interact with these once they're in this review state. My main note is that GIS support in terms of data access is a bit of a footnote, and that it really needs to be a key part of this, we can't have a result be that they have to struggle manually with a new folder structure.

Counterpoint would be to just keep "latest" for now. But seems like while we're touching all this stuff will be the best time to get work on a longer-term solution

[damonmcc]

@alexrichey
This all sounds great!

Regarding what we call the datasets in our publishing folders, is there a better term than draft publications? publication drafts? They really are just drafts of what we'll eventually publish.

I guess if these don't live in the DE-only draft folder, I'm hesitant to call them any type of draft

revisions? Also builds is the term we seem to naturally use when we're talking. so maybe draft builds, publication builds?

[damonmcc]

It seems like the verb "publish" here means "DE declares a certain build is ready for QA." So once a build passes QA, which verbs are next. I guess "package and "distribute"?

I really like "stage" as a verb for "DE declares a certain build is ready for QA", but wouldn't mind "publish" if we're happy with it only meaning that.