Assistance with changing writing file to working directory rather than COMOUT

[PerryShafran-NOAA]

As instructed in PR #652 , I need to make updates so that a file is not written directly to COMOUT, but rather to the working directory and then copied to COMOUT.

The file is the completed_jobs.txt file. A job that completes is written out to this file. This file is needed so in case a job fails and needs restart, this file tells you which jobs were completed. I believe this file is read by the restart job to know which jobs already completed. This section was originally written by @MarcelCaron-NOAA.

The job is written directly to COMOUT, as written in this line in mesoscale_stats_grid2obs_create_job_script.py like this:

                  job_cmd_list_iterative.append(
                       "python -c "
                       + f"'import mesoscale_util; mesoscale_util.mark_job_completed("
                       + f"\"{os.path.join(RESTART_DIR, COMPLETED_JOBS_FILE)}\", "
                       + f"\"job{njob}\", job_type=\"{job_type}\")'"

where RESTART_DIR is part of the COMOUT directory.

Marcel had mentioned that the fix should be within the mark_job_completed routine in the mesoscale_util.py in the ush directory. That block of code looks like this:

def mark_job_completed(completed_jobs_file, job_name, job_type=""):
    with open(completed_jobs_file, 'a') as f:
        if job_type:
            f.write(job_type + "_" + job_name + "\n")
        else:
            f.write(job_name + "\n")

I am not sure how to fix this so that the file is written to the DATA directory. Additionally, this file would need to be copied to COMOUT each time there is a write to it, so that the updated file continually is seen in the COMOUT directory, as it is being done now when it's written directly to COMOUT.

I am requesting assistance to code all of this, as I am very unsure as to how to do it.

Thanks!

[malloryprow]

You can change the path passed to mesoscale_util.mark_job_completed to somewhere in DATA.

[malloryprow]

job_cmd_list_iterative is a list (set on line 367 in ush/mesoscale/mesoscale_stats_grid2obs_create_job_script.py); everything with job_cmd_list_iterative.append( is appending a new item to that list. The items in that list are then looped over and written to the job card on lines 912 and 913.

[PerryShafran-NOAA]

Ohhhhhh, I think it's starting to click. Instead of using cutil.run_shell_command, use the job_cmd_list.append to do this work.

[PerryShafran-NOAA]

Success! Everything looks great with the completed_jobs.txt file!!!

Thank you Mallory and Marcel!

[malloryprow]

Can we close this discussion?

[PerryShafran-NOAA]

Yes. I will need to do checks for the file's existence and also adding SENDCOM, but that is not in the scope of this issue and we can close it out. Thanks for your assistance.