Date formats when providing your own dummy data

[elsie-h]

I'm creating my own dummy data using this R script and am trying to run it using the following action:

  generate_study_population_vax:
    run: cohortextractor:latest generate_cohort --study-definition study_definition_vax --output-format feather
    dummy_data_file: analysis/lib/dummy_data_vax.feather
    needs: 
    - design
    outputs:
      highly_sensitive:
        cohort: output/input_vax.feather

However, when I include date variables, I'm getting the following error:

if query_args["date_format"] == "YYYY":
     KeyError: 'date_format'

I get the same error when using csv as the output format.

I assume I need to convert the date-time strings to a different class in R before writing to feather format (or csv)? I've tried as.POSIXct(.x), as.Date(.x), as.Date(as.character(.x) (as done here), and just as.character(.x), but all result in the above error.

[elsie-h]

Also worth noting: I got the above error when I used a date variable that was derived using categorised_as(), like this:

    elig_date=patients.categorised_as(
      {
       "2020-12-08": "jcvi_group = '01'",
       "2021-01-18": "DEFAULT"
       },
        return_expectations={
            "category": {"ratios": { "2020-12-08": 0.8, "2021-01-18": 0.2 }
            },
            "incidence": 1,
        },
    ), 

But I just tried swapping the above date variable to one derived like this:

    endoflife_0_date=patients.with_these_clinical_events(
        eol_codes,
        returning="date",
        date_format="YYYY-MM-DD",
        on_or_before="2020-12-12",
        find_last_match_in_period=True,
        return_expectations={
            "date": {"earliest": start_date, "latest": end_date},
            "rate": "exponential_increase",
            "incidence": 0.01
        },
    ),

And got the following (different) error:

Dummy data error: Invalid value `datetime.date(2020, 11, 11)` for endoflife_0_date in row 377

Where row 377 is the first non-missing date for that variable.

[iaindillingham]

Thanks for your question, @elsie-h.

The generate_cohort sub-command expects the elig_date column in the generated dummy data to match its entry in the study definition. Notice that the elig_date column in the study definition is the result of calling the patients.categorised_as function, and that this function accepts a dictionary with string keys and string values as its first argument: the keys are categories and the values are expressions of logic. It isn't possible to tell this function that the string keys are dates that correspond to a format, such as YYYY-MM-DD. Other functions, such as patients.with_these_clinical_events, accept a date_format argument. Not so, patients.categorised_as.

The elig_date column in the generated dummy data is a datetime* column. When checking that this column matches its entry in the study definition, the generate_cohort sub-command assumes that all datetime columns are associated with functions that accept date_format arguments. Here, that assumption is not correct: the elig_date column in the generated dummy data is a datetime column that is not associated with a function that accepts a date_format argument. Consequently, the generate_cohort sub-command raises an error 🙁. I will create a new issue; we should, as a minimum, raise a more informative error.

I'd suggest modifying the script that generates the dummy data to convert elig_date from a datetime to a string, using the format YYYY-MM-DD. I'm not an R programmer, unfortunately. Could @alexwalkerepi help?

I appreciate that this explanation isn't easy to understand; it's taken me a reasonable amount of time to understand the issue. If you'd like to discuss it further, then please feel free to message on Slack.

*The generate_cohort sub-command is written in Python, so I'll use types from Python; there will be equivalent types from R.

---

I had to modify project.yaml and the study definition to recreate the issue. Notes to self:

• Added an output to design
• Temporally removed dummy_data_file from generate_study_population_vax
• Temporally added a scripted action to execute analysis/dummy_data_vax.R; this introduced a circular dependency
• Restored dummy_data_file to generate_study_population_vax

[iaindillingham]

See also opensafely-core/cohort-extractor#686

[iaindillingham]

opensafely-core/cohort-extractor#686 is fixed by opensafely-core/cohort-extractor#687

[alexwalkerepi]

I'm also not much of an R ninja, so will have to pass the buck to someone like @wjchulme or @acagreen17?

[iaindillingham]

Thanks for updating the script that generates the dummy data file @elsie-h. I've used it to generate a new dummy data file, and the version of the study on the update-dummy-data branch now runs without error. To do the same:

opensafely upgrade # Upgrades the opensafely command
opensafely pull # Upgrades the cohortextractor command
opensafely run run_all -f

This assumes that analysis/lib/dummy_data_vax.feather exists. Because of the fix introduced by opensafely-core/cohort-extractor#687, we need not convert dates to strings.