Figure out why analysis.py error did not cause AWS Batch job to fail and fix

[sandypreiss]

As part of the update to Python 3.11 and Pandas 2, we noticed that certain errors in analysis.py did not cause AWS jobs to fail, even though the execution of analysis.py failed.

We don't want silent failures, so we need to figure out why this error caused one and implement changes to avoid them in the future.

See example logs:

#
@timestamp
@message
1
2024-01-04T14:24:41.791Z
Completed 256.0 KiB/1.3 MiB (6.0 MiB/s) with 1 file(s) remaining Completed 512.0 KiB/1.3 MiB (11.6 MiB/s) with 1 file(s) remaining Completed 768.0 KiB/1.3 MiB (17.0 MiB/s) with 1 file(s) remaining Completed 1.0 MiB/1.3 MiB (22.1 MiB/s) with 1 file(s) remaining Completed 1.2 MiB/1.3 MiB (27.1 MiB/s) with 1 file(s) remaining Completed 1.3 MiB/1.3 MiB (9.4 MiB/s) with 1 file(s) remaining upload: ./output.csv to s3://crcsim-exp-crccp-sensitivity01/scenarios/fqhc5_implementation_lower_repeat_compliance/output_009.csv
2
2024-01-04T14:24:41.147Z
AttributeError: 'Series' object has no attribute 'iteritems'
3
2024-01-04T14:24:41.146Z
return object.__getattribute__(self, name)
4
2024-01-04T14:24:41.146Z
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
5
2024-01-04T14:24:41.145Z
for stage, value in onset_distrib.iteritems():
6
2024-01-04T14:24:41.145Z
^^^^^^^^^^^^^^^^^^^^^^^
7
2024-01-04T14:24:41.145Z
File "/usr/local/lib/python3.11/site-packages/pandas/core/generic.py", line 6204, in __getattr__
8
2024-01-04T14:24:41.144Z
component = fn(*varargs, **kwargs)
9
2024-01-04T14:24:41.144Z
^^^^^^^^^^^^^^^^^^^^^^
10
2024-01-04T14:24:41.144Z
File "/usr/local/lib/python3.11/site-packages/crcsim/analysis.py", line 1218, in run
11
2024-01-04T14:24:41.144Z
results = analysis.summarize()
12
2024-01-04T14:24:41.144Z
^^^^^^^^^^^^^^^^^^^^
13
2024-01-04T14:24:41.144Z
File "/usr/local/lib/python3.11/site-packages/crcsim/analysis.py", line 682, in summarize
14
2024-01-04T14:24:41.143Z
fire.Fire(run)
15
2024-01-04T14:24:41.143Z
File "/usr/local/lib/python3.11/site-packages/fire/core.py", line 141, in Fire
16
2024-01-04T14:24:41.143Z
component_trace = _Fire(component, args, parsed_flag_args, context, name)
17
2024-01-04T14:24:41.143Z
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
18
2024-01-04T14:24:41.143Z
File "/usr/local/lib/python3.11/site-packages/fire/core.py", line 475, in _Fire
19
2024-01-04T14:24:41.143Z
component, remaining_args = _CallAndUpdateTrace(
20
2024-01-04T14:24:41.143Z
^^^^^^^^^^^^^^^^^^^^
21
2024-01-04T14:24:41.143Z
File "/usr/local/lib/python3.11/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
22
2024-01-04T14:24:41.142Z
Traceback (most recent call last):
23
2024-01-04T14:24:41.142Z
File "/usr/local/bin/crc-analyze", line 8, in <module>
24
2024-01-04T14:24:41.142Z
sys.exit(main())
25
2024-01-04T14:24:41.142Z
^^^^^^
26
2024-01-04T14:24:41.142Z
File "/usr/local/lib/python3.11/site-packages/crcsim/analysis.py", line 1223, in main
27
2024-01-04T14:24:39.789Z
Completed 15.5 KiB/15.5 KiB (433.8 KiB/s) with 1 file(s) remaining download: s3://crcsim-exp-crccp-sensitivity01/scenarios/fqhc5_implementation_lower_repeat_compliance/params.json to ./params.json
28
2024-01-04T14:24:39.151Z
Creating output directory
crcsim/default/eb7605232c1c4f3fb1a84f0dc34fb9ed