Switch to new `flusurv` API endpoint by nmdefries · Pull Request #1278 · cmu-delphi/delphi-epidata

nmdefries · 2023-08-23T17:21:59Z

Summary:

Pull flusurv data from new CDC API endpoint. Ingest previously unlabelled age groupings. Ingest new race/ethnicity and sex breakdowns. Test flusurv.py functions. Add more comments, messaging, and assertions.

Closes #1247
Closes #242

Note:

Some of our CDC contacts are able to provide us with past versions of data from the API. We will want to patch those in. Many of the "new" strata are actually available back to 2009. We'll probably want to patch those in as well.
This also requires updates to the database table, the API server, and the API docs that will be made separately.

Prerequisites:

Unless it is a documentation hotfix it should be merged against the dev branch
Branch is up-to-date with the branch to be merged with, i.e. dev
Build is successful
Code is cleaned up and formatted

- rename input arg to `update` to avoid reassignment later - comment and reuse args_insert - spelling - comment magic constant used in output format - rename location-network/catchmentid map

Previously, age strata were numbered sequentially which allowed us to store rate values by position in a list. With the introduction of the new strata, this system is not robust enough to track all the different groups (e.g. ageids are no longer sequential and there are now race and sex groupings with separate numbering systems).

…piweek

src/acquisition/flusurv/new_grasp_location_result.json

melange396

nice work! i really appreciate the thorough commenting!

youll wanna pull in the changes from the dev branch, PR #1241 added tests for the flusurv endpoint.

src/acquisition/flusurv/flusurv.py

src/acquisition/flusurv/flusurv_update.py

nmdefries · 2025-10-15T21:40:23Z

Source data is here.

src/acquisition/flusurv/flusurv_update.py

src/acquisition/flusurv/api.py

src/acquisition/flusurv/reference/new_grasp_result.json

nmdefries · 2025-10-24T15:40:11Z

~~How long does this take to run for the last ~5 years for patching purposes? May want to go back farther, so can the history for demographic breakdowns~~
- ~~how long does that take (22 years)~~
- ~~what happens if try to insert new values for an existing obs/key? apparently keep newer value.~~
~~alert/warn if new demographic breakdowns are added [did this already?]~~
future TODO: generate SQL, other lists of output columns from constants.py so don't have so much duplication

integrations/server/test_flusurv.py

nmdefries · 2025-10-27T15:45:14Z

alert/warn if new demographic breakdowns are added

We warn about new demographic breakdowns here. Expected breakdowns are defined here.

nmdefries · 2025-10-27T17:30:47Z

How long does this take to run for the last ~22 years for patching purposes? Want to capture the history for demographic breakdowns

On my (old) computer this takes 54 s, so this shouldn't be a problem.

This source doesn't have version history, so when patching all the new data will be added under "today"'s issue date. I believe we have a connection at the source who can share their internal versioned data with us.

what happens if try to insert new values for an existing obs/key?

Based on the code, we apparently keep the newer value. However, the unique key is issue + epiweek + location. We aren't creating old issues, so all of the new patched data won't change the values for old issues.

…rity

…-endpoint

nmdefries · 2025-10-27T21:21:47Z

I wasnt paying attention and accidentally merged #1287 into this PR. I then reverted #1287 in #1426. This is a reminder that #1287 will eventually need to be re-merged into here (which is now probably best done by reverting the revert in #1426).

#1426 can't be reverted for some reason. I merged the ndefries/flusurv-new-columns branch in directly in f544328

sonarqubecloud · 2025-10-27T21:59:03Z

Quality Gate passed

Issues
8 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
8.9% Duplication on New Code

See analysis details on SonarQube Cloud

nmdefries added 11 commits September 14, 2023 17:47

initial switch to new API endpoint; doesn't account for format change

d99d2d0

separate fetch fn for whole api obj

70c3e16

get_current_issue to use existing json response

3580f3f

cleanup names and comments

9cb1da4

- rename input arg to `update` to avoid reassignment later - comment and reuse args_insert - spelling - comment magic constant used in output format - rename location-network/catchmentid map

define function to convert json obs to dict grouped by location and e…

cea25ab

…piweek

auto-map from valueids to ordinal and label-based group names

24dc088

add new strata to sql insert statement by name, not order

26eff97

pass seasonids around to use in requests for location-specific data

229a96c

include old and new example API responses

1e942a5

flusurv tests

f8a6706

nmdefries force-pushed the ndefries/flusurv-new-endpoint branch from 734dca9 to f8a6706 Compare September 15, 2023 15:50

Merge branch 'dev' into ndefries/flusurv-new-endpoint

d9581ac

nmdefries marked this pull request as ready for review September 15, 2023 16:25

nmdefries commented Sep 15, 2023

View reviewed changes

src/acquisition/flusurv/new_grasp_location_result.json Show resolved Hide resolved

nmdefries added 2 commits September 15, 2023 16:07

pass metadata around to reduce API calls

9ac2918

add season label as a descriptive column

aa3dd8c

nmdefries requested a review from brookslogan September 15, 2023 20:50

move example API responses to make it clear they are not for prod use

ca7e478

nmdefries removed the request for review from brookslogan September 15, 2023 21:51

nmdefries added 3 commits September 15, 2023 18:05

add new columns to flusurv endpoint spec

ed4dcf9

add new age, sex, race strata, and season to schema

73901c6

list new signals in documentation

108a000

nmdefries mentioned this pull request Sep 15, 2023

Update flusurv schema and docs with new age, sex, and race groups #1287

Merged

4 tasks

nmdefries requested a review from melange396 September 18, 2023 15:46

move new season column to end of table

16dc0c3

This was referenced Sep 18, 2023

flusurv request is missing columns cmu-delphi/epidatpy#25

Closed

flusurv request is missing columns cmu-delphi/epidatr#180

Closed

add migration script to add new fields to flusurv table

9a481de

melange396 requested changes Sep 26, 2023

View reviewed changes