limit on length of strings used in categorised_as()

[elsie-h]

Hopefully an easy fix, but there appears to be a limit on the length of strings used in the conditions for categorised_as().

I'm trying to create a variable that is conditional on region:

test=patients.categorised_as(
        {
            # "yorkshire": "region.startswith('Yorkshire')", # doesn't work
            "yorkshire": "region = 'Yorkshire and The Humber'",
            "other": "DEFAULT"
        }
    )

But this returns the error:

       File "/app/cohortextractor/expressions.py", line 103, in validate_string
         raise ValueError(f"String literals must be 16 characters or less: {value}")
     ValueError: String literals must be 16 characters or less: Yorkshire and The Humber

Full study definition here in the string-error-example branch.

[iaindillingham]

Hello @elsie-h, thanks for your question. The limit is arbitrary; it makes sense to have a limit, but the number of characters isn't significant. I'll update cohort-extractor and report back.

[iaindillingham]

Thanks for reporting this issue, @elsie-h. It's fixed in opensafely-core/cohort-extractor#691, which is now released. You will need to run opensafely pull to update cohort-extractor. Alas, your study still errors, but for a different reason:

ValueError: String literals can only contain alphanumeric characters, underscore and the comparators '<', '>', '=', and '~': Yorkshire and The Humber

I'd suggest replacing this:

    test=patients.categorised_as(
        {
            # "yorkshire": "region.startswith('Yorkshire')",
            "yorkshire": "region = 'Yorkshire and The Humber'",
            "other": "DEFAULT"
        }
    )

With this:

    test=patients.categorised_as(
        {
            # "yorkshire": "region.startswith('Yorkshire')",
            "yorkshire": "region = 'Yorkshire_and_The_Humber'",  # 👀 
            "other": "DEFAULT"
        }
    )

[elsie-h]

Hi @iaindillingham, thanks for sorting the issue relating to the string length limit!

However, I'm a bit confused about the suggestion to replace spaces with underscores. Suppose the region variable is defined as follows (I've pasted this from the OpenSAFELY documentation, but it was similarly defined in study_definition_vax.py):

region=patients.registered_practice_as_of(
    "2020-02-01",
    returning="nuts1_region_name",
    return_expectations={
        "rate": "universal",
        "category": {
            "ratios": {
                "North East": 0.1,
                "North West": 0.1,
                "Yorkshire and the Humber": 0.1,
                "East Midlands": 0.1,
                "West Midlands": 0.1,
                "East of England": 0.1,
                "London": 0.2,
                "South East": 0.2,
            },},},)

I believe that "nuts1_region_name" will return the same strings as have been included in return_expectations (I appreciate this is just for the dummy data, but I assume they have been specified as such because that's how they are in the real data). So then if all spaces are replaced by underscores, there would be no matches and the test column in the following example would default to "other" for all samples?

    test=patients.categorised_as(
        {
            # "yorkshire": "region.startswith('Yorkshire')",
            "yorkshire": "region = 'Yorkshire_and_The_Humber'",  # 👀 
            "other": "DEFAULT"
        }
    )

[iaindillingham]

Hello @elsie-h. Sorry for the confusion! To answer your last question first, yes, the only value in the test column would be other 🙁.

I think this is a case of one part of cohort-extractor not working especially well with another part of cohort-extractor. Stepping back slightly, I think you wish to transform values in one column to give values in another column; in this case, from long region names in the region column to short region names in the test column. For example, from Yorkshire and the Humber to yorkshire.

The long region names in the region column are returned by patients.registered_practice_as_of(..., returning="nuts1_region_name", ...). As you say, we'd expect the values given by the return_expectations argument to match the true values.

The short region names in the test column are returned by patients.categorised_as({...}), where the keys and values of the dictionary ({...}) map short region names to logic statements that are used to match (categorize) long region names. Unfortunately, cohort-extractor insists that the quoted text in these logic statements (e.g. 'Yorkshire and the Humber') contains only safe characters, which are alphanumeric characters, _, <, >, =, and ~. To cohort-extractor, a space is not a safe character.

I'll check whether treating a space as an unsafe character is an oversight; it may, unfortunately, be a necessary limitation. Could I ask how you wish to use the short region names? Could you accomplish the transformation in a downstream scripted action, for example?

[elsie-h]

@iaindillingham - thanks for getting back to me.

Sorry I should have clarified - the test variable is just a toy example. In reality, I wish to derive variables that assign a date to each patient based on the region in which their practice is registered, and the date at which they became eligible for vaccination. I.e., the RHS of an expression in categorised_as will look something like:

"elig_date = 2020-12-08 AND region_0 = 'Yorkshire and The Humber'"

Where elig_date and region_0 are previously specified variables.

The conditions are stored in a csv file, which is an output from a previous action and so not included in the repo currently, but I can add it if that would be useful (the file based on dummy data)? Then the date variables are created here in my study definition. These date variables are then used to define subsequent variables in the study definition. (Note: I can trim the date variables down to start_1_date and end_K_date - I don't really need start_1_date to start_K_date and end_1_date to end_K_date as I have now).

But I think (hope!) the example variable test above should be adequate for troubleshooting this issue?

[iaindillingham]

Hello @elsie-h, thanks for clarifying! Spaces are now permitted in string literals, such as 'Yorkshire and The Humber'. For more information, see: opensafely-core/cohort-extractor#692. You will need to run opensafely pull to update cohort-extractor.