Product Lifecycle

Data Product Version Update Workflow

Parent Issue Creation:
The Data Engineering team (DE) creates a parent GitHub issue to track the entire version release lifecycle, from data loading to distribution.

Source Data Update and Build:
DE updates the source data as needed and runs a build. The build is archived in the build folder.

Initial QA and Draft Promotion:

DE conducts QA on the build and addresses (if applicable):

• All Github sub-issues under the parent issue, including:

  • Issues created by other teams (e.g., GIS, domain stakeholders) under the parent issue.
  • Issues communicated by email (DE will create a GitHub issue if notified this way).

• Any failed QA checks during the build — refer to the data check descriptions for context.

• The QAQC Streamlit page checks. Refer to the table blurbs for guidance.

Once everything is addressed, DE promotes the build to the draft folder in S3, initiating the QAQC process by domain experts.

QAQC workflow process with the GIS team

GIS team Open Data Update document

For data products reviewed by the GIS team, follow this workflow:

1. Draft Ready for Review:
   Once the draft is ready, DE adds a yellow "GIS" tag to the parent GitHub issue and creates a child GitHub issue for GIS QAQC. This issue is assigned to all GIS team members, named in the format: <Product Name> <Release Version> QA - Draft <Number> (e.g., "FacDB 24v2 QA - Draft 1").

2. GIS Team Review:
   The GIS team conducts the QA review, recording the results in the child issue. They close the issue after completing the review, regardless of the outcome.

3. Draft Promotion or Rebuild:

   • If the draft passes QA, DE promotes it to the publish folder.
   • If it fails, DE addresses the issues, creates a new build, and repeats the process, starting from step 1 with a new child issue for GIS QAQC.

⚠️ Note on PLUTO Minor:

The DE team doesn't create parent Github issues for PLUTO Minor to track the lifecycle. Instead, the GIS team builds PLUTO minor and promotes it to the drafts folder as a part of the Zoning Tax lots - 📁 DataLoading action. This action will automatically create an issue for PLUTO Minor QA which the GIS team can find under the "Issues" tab and mark it with the GIS tag.