TD Record Errors when Opt continues to have "running" status

[jaclark5]

Describe the bug
A TD record errors despite the associated Optimization records continuing to have a "running" status

Example

from collections import defaultdict
from qcportal import PortalClient

qc_client = PortalClient("https://api.qcarchive.molssi.org:443/", cache_dir=".")
ds = qc_client.get_dataset("torsiondrive", "OpenFF Lipid Torsion Drives v4.0")

record_dict = defaultdict(list)
for _, _, rec in ds.iterate_records():
    record_dict[rec.status.value].append(rec)

for status in ["running", "error", "waiting"]:
    if record_dict[status]:
        print(f"\n\n TD Status {status}")
        print("%"*50)
    for td_opt in record_dict[status]:
        print(f"\n\n    OPTIMIZATION SUCCESS ANALYSIS TD ID {td_opt.id}:")
        print("    ", "="*50)
        if status == "error":
            print("\n    ", td_opt.error, "\n")

        successful_angles = []
        problematic_angles = []
        problematic_records = []
        
        # len(td_opt.compute_history) == 1, no extra information
        print("    Problematic Angles:")
        for angle, rec_list in td_opt.optimizations.items():
            has_complete = any(rec.status.name == "complete" for rec in rec_list)
            
            if has_complete:
                successful_angles.append(angle)
            else:
                problematic_angles.append(angle)
                problematic_records.append(rec_list)
                status_summary = (f"    Complete: {sum(rec.status.name == 'complete' for rec in rec_list)}, "
                f"Error: {sum(rec.status.name == 'error' for rec in rec_list)}, "
                f"Waiting: {sum(rec.status.name == 'waiting' for rec in rec_list)}, "
                f"Running: {sum(rec.status.name == 'running' for rec in rec_list)}, ")
                print(f"    Angle {str(angle):>6s}: {len(rec_list)} records - {status_summary}")

        print(f"\n    Summary:")
        print(f"    Angles with at least one successful optimization: {len(successful_angles)}")
        print(f"    Angles with NO successful optimizations: {len(problematic_angles)}")

print(problematic_records[1][0].status)

 TD Status error
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


    OPTIMIZATION SUCCESS ANALYSIS TD ID 146649971:
     ==================================================

     {'error_type': 'service_iteration_error', 'error_message': 'Some task(s) did not complete successfully'} 

    Problematic Angles:
    Angle (105,): 1 records -     Complete: 0, Error: 0, Waiting: 0, Running: 1, 

    Summary:
    Angles with at least one successful optimization: 23
    Angles with NO successful optimizations: 1


    OPTIMIZATION SUCCESS ANALYSIS TD ID 146649959:
     ==================================================

     {'error_type': 'service_iteration_error', 'error_message': 'Some task(s) did not complete successfully'} 

    Problematic Angles:
    Angle (-120,): 1 records -     Complete: 0, Error: 0, Waiting: 0, Running: 1, 
    Angle (-105,): 1 records -     Complete: 0, Error: 0, Waiting: 0, Running: 1, 
    Angle (120,): 2 records -     Complete: 0, Error: 0, Waiting: 0, Running: 2, 

    Summary:
    Angles with at least one successful optimization: 21
    Angles with NO successful optimizations: 3

<RecordStatusEnum.running: 'running'>

[bennybp]

You are correct that this shouldn't happen. The way the queries are written should make this impossible, from what I can tell (outside of someone doing a record restart at a very precise time).

The record you mention here also appears consistent to me (although this was two weeks ago. Sorry, was teaching a bootcamp).

I ran a query directly on the database for any TD records with an error status but with any running or waiting optimizations, but couldn't find any.

Maybe this is another cache sync issue? If so, I guess I really need to prioritize figuring out why that keeps becoming inconsistent

[jaclark5]

As of now the output of this script is shown below. It seems that it is slow to update, but that it will eventually have the expected status, but sometimes not within the first 24 hours.

 TD Status error
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%


    OPTIMIZATION SUCCESS ANALYSIS TD ID 146649971:
     ==================================================

     {'error_type': 'service_iteration_error', 'error_message': 'Some task(s) did not complete successfully'} 

    Problematic Angles:
    Angle (105,): 1 records -     Complete: 0, Error: 1, Waiting: 0, Running: 0, 

    Summary:
    Angles with at least one successful optimization: 23
    Angles with NO successful optimizations: 1


    OPTIMIZATION SUCCESS ANALYSIS TD ID 146649959:
     ==================================================

     {'error_type': 'service_iteration_error', 'error_message': 'Some task(s) did not complete successfully'} 

    Problematic Angles:
    Angle (-105,): 1 records -     Complete: 0, Error: 1, Waiting: 0, Running: 0, 

    Summary:
    Angles with at least one successful optimization: 23
    Angles with NO successful optimizations: 1