Automated build failed with `FATAL: Maximum recursion depth reached`

[joverlee521]

Automated phylo workflow failed on Friday and Saturday with error from augur export v2

FATAL: Maximum recursion depth reached. You can set the env variable AUGUR_RECURSION_LIMIT to adjust this (current limit: 10000)

[joverlee521]

RecursionError is happening during validation of the produced Auspice JSON.

I attached to the AWS Batch build and ran the failed augur export v2 command with AUGUR_DEBUG=True

$ nextstrain build --aws-batch --attach 0b00c1b7-a120-436d-9617-5ad369981c2a .
$ nextstrain build . results/h5n1-cattle-outbreak/mp/default/auspice-dataset.json -s genome-focused/Snakefile --touch
$ nextstrain build --env AUGUR_DEBUG=True . results/h5n1-cattle-outbreak/mp/default/auspice-dataset.json -s genome-focused/Snakefile --force
Validating schema of 'results/h5n1-cattle-outbreak/mp/default/nt-muts.json'...
Validating schema of 'results/h5n1-cattle-outbreak/mp/default/aa-muts.json'...
Validating config file 'results/h5n1-cattle-outbreak/mp/default/auspice-config.json' against the JSON schema
Validating schema of 'results/h5n1-cattle-outbreak/mp/default/auspice-config.json'...
WARNING: Requested color-by field 'h5_label_clade' does not exist and will not be used as a coloring or exported.

WARNING: [config file] Trait 'genome_tree' is missing type information. We've guessed 'boolean'.

WARNING: These values for trait host were not specified in the colors file you provided:
	raw pet food, pet food.
	Auspice will create colors for them.

Validating produced JSON
Validating schema of 'results/h5n1-cattle-outbreak/mp/default/auspice-dataset.json'...
Traceback (most recent call last):
  File "/nextstrain/augur/augur/__init__.py", line 70, in run
    return args.__command__.run(args)
  File "/nextstrain/augur/augur/export_v2.py", line 1245, in run
    validate_data_json(args.output, args.validation_mode)
  File "/nextstrain/augur/augur/export_v2.py", line 498, in validate_data_json
    validate_v2(main_json=filename)
  File "/nextstrain/augur/augur/validate.py", line 221, in export_v2
    validate(main, main_schema, main_json)
  File "/nextstrain/augur/augur/validate.py", line 101, in validate_json
    errors = list(schema.iter_errors(jsonToValidate))
  File "/usr/local/lib/python3.10/site-packages/jsonschema/validators.py", line 384, in iter_errors
    for error in errors:
  File "/usr/local/lib/python3.10/site-packages/jsonschema/_keywords.py", line 296, in properties
    yield from validator.descend(
  File "/usr/local/lib/python3.10/site-packages/jsonschema/validators.py", line 432, in descend
    for error in errors:
  File "/usr/local/lib/python3.10/site-packages/jsonschema/_keywords.py", line 355, in oneOf
    errs = list(validator.descend(instance, subschema, schema_path=index))
  File "/usr/local/lib/python3.10/site-packages/jsonschema/validators.py", line 432, in descend
    for error in errors:
  File "/usr/local/lib/python3.10/site-packages/jsonschema/_keywords.py", line 275, in ref
    yield from validator._validate_reference(ref=ref, instance=instance)
  File "/usr/local/lib/python3.10/site-packages/jsonschema/validators.py", line 432, in descend
    for error in errors:
  File "/usr/local/lib/python3.10/site-packages/jsonschema/_keywords.py", line 296, in properties
    yield from validator.descend(
  File "/usr/local/lib/python3.10/site-packages/jsonschema/validators.py", line 432, in descend
    for error in errors:
  File "/usr/local/lib/python3.10/site-packages/jsonschema/_legacy_keywords.py", line 135, in items_draft6_draft7_draft201909
    yield from validator.descend(item, items, path=index)
  File "/usr/local/lib/python3.10/site-packages/jsonschema/validators.py", line 432, in descend
    for error in errors:
  File "/usr/local/lib/python3.10/site-packages/jsonschema/_keywords.py", line 275, in ref
    yield from validator._validate_reference(ref=ref, instance=instance)
  File "/usr/local/lib/python3.10/site-packages/jsonschema/validators.py", line 432, in descend
    for error in errors:
  File "/usr/local/lib/python3.10/site-packages/jsonschema/_keywords.py", line 296, in properties
    yield from validator.descend(
  File "/usr/local/lib/python3.10/site-packages/jsonschema/validators.py", line 432, in descend
    for error in errors:

[...last 6 lines keeps repeating...]

  File "/usr/local/lib/python3.10/site-packages/jsonschema/_keywords.py", line 296, in properties
    yield from validator.descend(
  File "/usr/local/lib/python3.10/site-packages/jsonschema/validators.py", line 432, in descend
    for error in errors:
  File "/usr/local/lib/python3.10/site-packages/jsonschema/_keywords.py", line 285, in type
    if not any(validator.is_type(instance, type) for type in types):
  File "/usr/local/lib/python3.10/site-packages/jsonschema/_keywords.py", line 285, in <genexpr>
    if not any(validator.is_type(instance, type) for type in types):
  File "/usr/local/lib/python3.10/site-packages/jsonschema/validators.py", line 455, in is_type
    return self.TYPE_CHECKER.is_type(instance, type)
  File "/usr/local/lib/python3.10/site-packages/jsonschema/_types.py", line 116, in is_type
    return fn(self, instance)
  File "/usr/local/lib/python3.10/site-packages/jsonschema/_types.py", line 45, in is_number
    return isinstance(instance, numbers.Number)
  File "/usr/local/lib/python3.10/abc.py", line 119, in __instancecheck__
    return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded in comparison
FATAL: Maximum recursion depth reached. You can set the env variable AUGUR_RECURSION_LIMIT to adjust this (current limit: 10000)

[joverlee521]

I was confused why we were hitting a recursion error when the mp tree didn't look that deep, but made more sense once I toggled to the time tree

I'm going to set AUGUR_RECURSION_LIMIT=20000 for the export rule for now to avoid this error, but maybe there's a better way to resolve this?