Add restarts for PBS model run timeouts

[nefrathenrici]

Purpose

Adds restarts for PBS model runs specifically for model runs that are terminated by the PBS job scheduler.

To-do

Content

• The main code changes are in pbs_trap_block, which sets up a signal handler that catches SIGTERM and uses qrerun to requeue the job. This is added to the bash script generated by generate_pbs_script
• Specifies a tmpdir for derecho to prevent issues writing large checkpoint files in the user's home directory.
• This PR also has the changes from Add default status if PBS job status fails #217 , maybe they'd be easier to bundle together.

---

• I have read and checked the items on the review checklist.