Refactor service_code to allow multiple patterns

[clovis517]

1. Disallows inclusion of "CSTM-00" in an array containing other service codes
2. Avoids an issue with "const":"CSTM-00" being treated by the validator as a wild card in some environments

[Ankit1689]

"pattern": "^(0[1-9]|[0-9][1-9])$"
this pattern is failing for values 10,20,30.....90
so we can modify this to
"pattern": "^(0[1-9]|[1-9][0-9])$" ?

[Ankit1689]

or may be use this ^(?!00)\d{2}$

[clovis517]

"pattern": "^(0[1-9]|[0-9][1-9])$" this pattern is failing for values 10,20,30.....90 so we can modify this to "pattern": "^(0[1-9]|[1-9][0-9])$" ?

Correct. I initially forgot to change the second pattern but fixed that in a subsequent change. And yes, I think ^(?!00)\d{2}$ will do the same thing.

Note that the change from listing individual codes to the range allows codes not assigned by CMS which would have failed under schema 1.0, I don't recall seeing a discussion about that.

[clovis517]

Note that the change from listing individual codes to the range allows codes not assigned by CMS which would have failed under schema 1.0, I don't recall seeing a discussion about that.

To exclude the complete set of unassigned codes I think the pattern should be :

^(?!00)(?!2[8-9]|30)(?!3[5-9]|40)(?!4[3-8])(?!59)(?!6[3-4])(?!6[7-9]|70)(?!7[3-9]|80)(?!8[2-9])(?!9[0-8])\d{2}$

 ^(?!00)  → excludes 00
 (?!2[8-9]|30)  → excludes 28–30
 (?!3[5-9]|40)  → excludes 35–40
 (?!4[3-8])  → excludes 43–48
 (?!59)  → excludes 59
 (?!6[3-4])  → excludes 63–64
 (?!6[7-9]|70)  → excludes 67–70
 (?!7[3-9]|80)  → excludes 73–80
 (?!8[2-9])  → excludes 82–89
 (?!9[0-8])  → excludes 90–98
 \d{2}$  → matches any remaining two-digit code

[Ankit1689]

looks good, when are you going to merge it?

[shaselton-usds]

@clovis517 Thank you for this -- the regex is definitely more explicit to the available/allowable service codes. Good stuff.

Two things:

• Can you say a bit more about the "const" value being treated as a wild card in some environments? The current schema treats this as an either/or. The "oneOf" was supported in the json schema spec starting in draft 4.
• Would it be possible to change this PR to point to the draft branch vs master?

[clovis517]

@shaselton-usds yes, I will get it into the draft branch.

Regarding the oneOf issue. I say in some environments because I can't believe everyone is having the issue. oneOf is working properly in my modified schema. My suspicion (which I find hard to believe since I don't find reference to this anywhere) is that the RapidJSON Validator library is not parsing oneOf( {pattern}, {const} ) correctly. I verified one of the other two const entries is working correctly outside of oneOf. Maybe related - I am unable to get the EIN constraint to work. I presumed this was due to the missing ^...$ anchors in "\d{2}-?\d{7}" but adding the anchors didn't fix it, and I found it is even allowing short all-character values such as "ABC". Changing the const to a pattern didn't fix this so it seems to be an unrelated issue with the if/then being ignored.

The oneOf rabbit hole I went down:
All files fail with oneOf issue in the very first POS array. Confirmed the array contained valid POS codes.
Created a test file with a single valid POS code. Failed. Tried many other valid POS codes, all failed.
Tried "CSTM-00" and it passed.
I reviewed the Schema (below for reference) and did not see anything wrong with it, but noted that the patterns to the left and right of the OR pipe are not mutually exclusive.

        "service_code": {
          "type": "array",
          "items": {
            "type": "string",
            "oneOf": [
              {
                "pattern": "^([1-9][0-9]|[0-9][1-9])$"
              },
              {
                "const": "CSTM-00"
              }
            ]
          },
          "uniqueItems": true
        },

I re-wrote the pattern to remove the overlap: "^(0[1-9]|[1-9][0-9])$"
... and it did not change anything.
So it must be thinking that the POS code matched both the pattern and the const. I added a length constraint to the const as follows:

            ...
            "oneOf": [
              {
                "pattern": "^([1-9][0-9]|[0-9][1-9])$"
              },
              {
               "minLength": 7,
               "maxLength": 7,
                "const": "CSTM-00"
              }
            ]
          ...

and it fixed the oneOf problem:
"11" : Valid
"ZZ" : Invalid
"CSTM-00" : Valid
"ZZZZZZZ" : Valid!

So, the value following "const": appears to be ignored and instead matches any value passed to it.

Work arounds :

• Add |^CSTM-00$ to the Pattern. This worked however it does not prohibit reporting both CSTM-00 and POS codes in the same array. (nor does the oneOf with the length fix,).
• Changed to OneOf two Arrays: Either an Array of POS codes OR an array of "^CSTM-00$" to fix the problem of mixing values. I thought this might also fix the CONST problem, but it didn't.
• Expand the pattern to omit the unassigned POS values using the pattern in my last post.
• Investigate EIN issue; add anchors to the EIN pattern.

I would appreciate knowing if others are unable to get invalid EINs to fail.

[clovis517]

... many hours later - I have not been able to get the ! not ranges in the pattern to work, these turned the pattern into a wild card similar to the situation with const. I am going to do a complete uninstall and re-install of node JS and see if anything changes. In the meantime, the following does what it should : Only allows valid codes, does not allow dupes, does not allow CSTM-00 with other codes.

        "service_code": {
          "oneOf": [
            {
              "type": "array",
              "items": {
                "type": "string",
                "minLength": 2,
                "maxLength": 2,
                "pattern": "^(0[1-9]|1[0-9]|2[0-7]|3[1-4]|4[1-2]|4[9]|5[0-8]|6[0-2]|6[5-6]|7[1-2]|81|99)$"
              },
              "uniqueItems": true
            },
            {
              "type": "array",
              "items": {
                "type": "string",
                "minLength": 7,
                "maxLength": 7,
                "pattern": "^CSTM-00$"
              },
              "maxItems": 1
            }
          ]
        },

I need to re-test to see if I can remove the apparently unnecessary minLength maxLength stuff, which fixes the oneOf issue in some cases but I think it is just masking an underlying problem interpreting the pattern and/or const.

[smohammad01]

@clovis517 Thanks for your help here.
Do we have any plan/ETA to merge this in main branch? any workaround we should follow to avoid this error in meantime to meet our testing timelines?

[mmaligh]

@shaselton Can you pls let us know what's the ETA to merge the Validator update to fix the service code issue.Thanks!

[clovis517]

Small update :
I put the best correctly working pattern in my installation, which now functions correctly without redundant length constraints, in this pull request : d95ae15

@shaselton-usds if possible I would fix the missing anchors on the EIN pattern at the same time. 3164fbe
This doesn't fix the issue of the EIN if/then causing it to never trigger a validation failure in my environment but I confirmed the issue and fix using the schema web validator referenced here #879 (comment) - the URL provided pulls up both the schema and test MRF to demonstrate. I apologize for mistakenly bundling the EIN change with an Out of Network change rather than with this one, have not had time to figure out how to move it here.

I really don't like having to avoid using const and need to get the EIN if/then issue fixed so have been trying to figure out what is going on. I re-installed everything, also manually re-pulled and ran cmake on the rapidjson subfolder, and the problem did not go away. Next step will be to confirm I don't encounter the issue in a linux VM.

This might be a different manifestation of the service_code issue we are encountering: Tencent/rapidjson#2314

Maybe this issue is related is Windows use of cmake, and rapidjson being based on an antiquated version of cmake. I had to run cmake a parameter to override the version constraint. This and the links it points to relate to cmake version constraint in rapidjson: Tencent/rapidjson#2333

I have not been able to get the rapidjson google tests through cmake yet, similar version issues I think. Need to spend more time there. I expect some of the tests to fail given what's happening with MRF validator.

[clovis517]

I confirmed these issues are not a Windows-only problem. I did a fresh Ubuntu server install, installed latest versions of everything, and experience all of the same issues.

• oneOf / const issue, remediated by using use pattern instead of const
• If-Then not working so all EINs are accepted, including a single character such as "A". I have not found a fix for this.

If these issues are not universal I am guessing they relate to not sticking to the listed version numbers of the prerequisites:
Prerequisites
Node (version 16.x)
NPM (version 8.5.x)
Git (latest version recommended, tested using 2.27.0)
Docker (version 19.x)

[shaselton-usds]

@clovis517 Super appreciative with you continuing to look into this. I'll pull this down for some local testing -- if all looks good, I'll merge in early next week.

[clovis517]

Thanks @shaselton-usds . Right now I am trying to see if I can get the EIN validation to reject anything. I changed the const to a pattern, converted the IF to oneOf and added constraints to the numeric and text NPI entries (must be 10 digits with the first digit 1-4) but initial testing is still letting garbage through as if the constraint is not there. At least it isn't triggering the oneOf problem. If I am able to get the EIN validation to behave I will let you know.

update : I got EIN validation working, just running through several more tests before I post it

[clovis517]

@shaselton-usds I was able to work around all of the glitches I was encountering in schema handling. 🎆
#889

The following list of tests pass with the in network schema below.

inn.invalid.CSTM-00_CSTM-00.json inn.invalid.CSTM-02.json inn.invalid.ein_misplaced_hyphen.json inn.invalid.EIN_Short.json inn.invalid.ein_ssn.json inn.invalid.ein_too_long.json inn.invalid.NPI0x.json inn.invalid.pos.98.json inn.invalid.pos00.json inn.invalid.pos01_and_CSTM-00.json inn.invalid.tin_ein_without_name.json inn.invalid.tin_npi_too_short.json inn.valid.pos01.json inn.valid.pos01_99.json inn.valid.pos11.json inn.valid.pos_01_04_03.json inn.valid.tin_npi_without_name.json

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "https://github.com/CMSgov/price-transparency-guide/blob/master/schemas/in-network-rates/in-network-rates.json",
  "definitions": {
    "in_network": {
      "type": "object",
      "properties": {
        "negotiation_arrangement": {
          "enum": [
            "ffs",
            "bundle",
            "capitation"
          ]
        },
        "name": {
          "type": "string",
          "minLength": 1
        },
        "billing_code_type": {
          "$ref": "#/definitions/billing_code_types"
        },
        "severity_of_illness": {
          "type": "string",
          "minLength": 1
        },
        "billing_code_type_version": {
          "type": "string",
          "minLength": 1
        },
        "billing_code": {
          "type": "string",
          "minLength": 1
        },
        "description": {
          "type": "string",
          "minLength": 1
        },
        "negotiated_rates": {
          "type": "array",
          "items": {
            "$ref": "#/definitions/negotiated_rates"
          },
          "default": [],
          "minItems": 1
        },
        "covered_services": {
          "type": "array",
          "items": {
            "$ref": "#/definitions/contained_billing_code"
          },
          "default": [],
          "minItems": 1
        },
        "bundled_codes": {
          "type": "array",
          "items": {
            "$ref": "#/definitions/contained_billing_code"
          },
          "default": [],
          "minItems": 1
        }
      },
      "required": [
        "negotiation_arrangement",
        "name",
        "billing_code_type",
        "billing_code_type_version",
        "billing_code",
        "negotiated_rates",
        "description"
      ]
    },
    "contained_billing_code": {
      "type": "object",
      "properties": {
        "billing_code_type": {
          "$ref": "#/definitions/billing_code_types"
        },
        "billing_code_type_version": {
          "type": "string",
          "minLength": 1
        },
        "billing_code": {
          "type": "string",
          "minLength": 1
        },
        "description": {
          "type": "string",
          "minLength": 1
        }
      },
      "required": [
        "billing_code_type",
        "billing_code_type_version",
        "billing_code",
        "description"
      ]
    },
    "negotiated_rates": {
      "type": "object",
      "properties": {
        "negotiated_prices": {
          "type": "array",
          "items": {
            "$ref": "#/definitions/negotiated_price"
          },
          "uniqueItems": true,
          "default": [],
          "minItems": 1
        },
        "provider_references": {
          "type": "array",
          "items": {
            "type": "number"
          },
          "uniqueItems": true,
          "default": [],
          "minItems": 1
        }
      },
      "required": [
        "provider_references",
        "negotiated_prices"
      ]
    },
    "negotiated_price": {
      "type": "object",
      "properties": {
        "service_code": {
          "oneOf": [
            {
              "type": "array",
              "items": {
                "type": "string",
                "pattern": "^(0[1-9]|1[0-9]|2[0-7]|3[1-4]|4[1-2]|4[9]|5[0-8]|6[0-2]|6[5-6]|7[1-2]|81|99)$"
              },
              "uniqueItems": true
            },
            {
              "type": "array",
              "items": {
                "type": "string",
                "pattern": "^CSTM-00$"
              },
              "maxItems": 1
            }
          ]
        },
        "billing_class": {
          "enum": [
            "professional",
            "institutional",
            "both"
          ]
        },
        "setting": {
          "enum": [
            "inpatient",
            "outpatient",
            "both"
          ]
        },
        "negotiated_type": {
          "enum": [
            "negotiated",
            "derived",
            "fee schedule",
            "percentage",
            "per diem"
          ]
        },
        "billing_code_modifier": {
          "type": "array",
          "items": {
            "type": "string"
          },
          "uniqueItems": true,
          "default": [],
          "minItems": 1
        },
        "negotiated_rate": {
          "type": "number",
          "exclusiveMinimum": 0
        },
        "expiration_date": {
          "type": "string",
          "format": "date",
          "description": "This is a date format of YYYY-MM-DD",
          "minLength": 10,
          "maxLength": 10
        },
        "additional_information": {
          "type": "string",
          "description": "In situations in which alternative reimbursement arrangements can neither be expressed as a dollar amount nor as a percentage, this open text field can be used to provide information such as, a description of the formula, variables, methodology or other information necessary to understand the arrangement. The open text field may be utilized for reporting only if a plan or issuer cannot disclose its in-network rates as a dollar amount or a percentage.",
          "minLength": 1
        }
      },
      "required": [
        "negotiated_type",
        "billing_class",
        "negotiated_rate",
        "expiration_date",
        "setting"
      ],
      "if": {
        "properties": {
          "billing_class": {
            "const": "professional"
          }
        }
      },
      "then": {
        "required": [
          "service_code"
        ]
      }
    },
    "providers": {
      "type": "object",
      "properties": {
        "npi": {
          "type": "array",
          "items": {
            "type": "number",
            "minimum": 1000000000,
            "maximum": 9999999999
          },
          "uniqueItems": true,
          "default": [],
          "minItems": 1
        },
        "tin": {
          "type": "object",
          "properties": {
            "type": {
              "type": "string"
            },
            "value": {
              "type": "string",
              "minLength": 9,
              "maxLength": 10
            },
            "business_name": {
              "type": "string",
              "minLength": 1
            }
          },
          "oneOf": [
            {
              "properties": {
                "type": {
                  "pattern": "^ein$"
                },
                "value": {
                  "pattern": "^[0-9]{2}-?[0-9]{7}$"
                }
              },
              "required": ["type", "value", "business_name"]
            },
            {
              "properties": {
                "type": {
                  "pattern": "^npi$"
                },
                "value": {
                  "pattern": "^[1-9][0-9]{9}$"
                }
              },
              "required": ["type", "value"]
            }
          ]
        }
      },
      "required": [
        "npi",
        "tin"
      ]
    },
    "billing_code_types": {
      "enum": [
        "CPT",
        "HCPCS",
        "ICD",
        "MS-DRG",
        "R-DRG",
        "S-DRG",
        "APS-DRG",
        "AP-DRG",
        "APR-DRG",
        "APC",
        "NDC",
        "HIPPS",
        "LOCAL",
        "EAPG",
        "CDT",
        "RC",
        "CSTM-ALL"
      ]
    }
  },
  "type": "object",
  "properties": {
    "reporting_entity_name": {
      "type": "string",
      "minLength": 1
    },
    "reporting_entity_type": {
      "type": "string",
      "minLength": 1
    },
    "plan_name": {
      "type": "string",
      "minLength": 1
    },
    "issuer_name": {
      "type": "string",
      "minLength": 1
    },
    "plan_sponsor_name": {
      "type": "string",
      "minLength": 1
    },
    "plan_id_type": {
      "enum": [
        "ein",
        "hios"
      ]
    },
    "plan_id": {
      "type": "string",
      "minLength": 1
    },
    "plan_market_type": {
      "enum": [
        "group",
        "individual"
      ]
    },
    "last_updated_on": {
      "type": "string",
      "format": "date",
      "description": "This is a date format of YYYY-MM-DD",
      "minLength": 10,
      "maxLength": 10
    },
    "version": {
      "type": "string",
      "minLength": 1
    },
    "provider_references": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "provider_group_id": {
            "type": "integer"
          },
          "provider_groups": {
            "type": "array",
            "items": {
              "$ref": "#/definitions/providers"
            },
            "default": [],
            "minItems": 1
          },
          "network_name": {
            "type": "array",
            "items": {
              "type": "string",
              "minLength": 1
            }
          }
        },
        "required": [
          "provider_group_id",
          "provider_groups",
          "network_name"
        ]
      },
      "minLength": 1
    },
    "in_network": {
      "type": "array",
      "items": {
        "$ref": "#/definitions/in_network"
      },
      "default": [],
      "minItems": 1
    }
  },
  "if": {
    "properties": {
      "plan_id_type": {
        "enum": [
          "ein"
        ]
      }
    },
    "required": [
      "plan_id_type"
    ]
  },
  "then": {
    "required": [
      "plan_sponsor_name"
    ]
  },
  "required": [
    "reporting_entity_name",
    "reporting_entity_type",
    "last_updated_on",
    "in_network",
    "version"
  ],
  "dependencies": {
    "plan_name": [
      "plan_id_type",
      "plan_id",
      "plan_market_type",
      "issuer_name"
    ],
    "plan_id_type": [
      "plan_name",
      "plan_id",
      "plan_market_type",
      "issuer_name"
    ],
    "plan_id": [
      "plan_name",
      "plan_id_type",
      "plan_market_type",
      "issuer_name"
    ],
    "plan_market_type": [
      "plan_name",
      "plan_id_type",
      "plan_id",
      "issuer_name"
    ],
    "issuer_name": [
      "plan_name",
      "plan_id_type",
      "plan_market_type",
      "plan_id"
    ]
  }
}