Improved error handling in the PBSProProvider (#3853)

# Description

The `PBSProProvider` currently logs submit failures and returns `None`
instead of raising an exception as expected. This results in Parsl
repeatedly submitting jobs with little feedback to the user when
misconfigured. This PR addresses this issue by raising a
`SubmitException` which captures the stdout/err from the qsub command.
The relevant info trickles up via the `TooManyJobFailuresError` as the
examples below show:

Error raised when the `PBSProProvider` is misconfigured with a bad queue
name:

```
parsl.jobs.errors.TooManyJobFailuresError: Error 1:
	Failed to start block 0: Cannot launch job parsl.HighThroughputExecutor.block-0.1746723134.8832731: Submit command 'qsub  -q debug5 -A AuroraGPT /home/yadunand/parsl/parsl/providers/pbspro/runinfo/001/submit_scripts/parsl.HighThroughputExecutor.block-0.1746723134.8832731' failed; recode=None, stdout=, stderr=qsub: Unknown queue

Error 2:
	Failed to start block 1: Cannot launch job parsl.HighThroughputExecutor.block-1.1746723135.5695264: Submit command 'qsub  -q debug5 -A AuroraGPT /home/yadunand/parsl/parsl/providers/pbspro/runinfo/001/submit_scripts/parsl.HighThroughputExecutor.block-1.1746723135.5695264' failed; recode=None, stdout=, stderr=qsub: Unknown queue
```

Error raised when the account is incorrect:

```
parsl.jobs.errors.TooManyJobFailuresError: Error 1:
	Failed to start block 0: Cannot launch job parsl.HighThroughputExecutor.block-0.1746723277.4842422: Submit command 'qsub  -q debug -A NonExistentAccount /home/yadunand/parsl/parsl/providers/pbspro/runinfo/002/submit_scripts/parsl.HighThroughputExecutor.block-0.1746723277.4842422' failed; recode=None, stdout=, stderr=qsub: Request rejected.  Reason: not found: Project NonExistentAccount

Error 2:
	Failed to start block 1: Cannot launch job parsl.HighThroughputExecutor.block-1.1746723277.8245358: Submit command 'qsub  -q debug -A NonExistentAccount /home/yadunand/parsl/parsl/providers/pbspro/runinfo/002/submit_scripts/parsl.HighThroughputExecutor.block-1.1746723277.8245358' failed; recode=None, stdout=, stderr=qsub: Request rejected.  Reason: not found: Project NonExistentAccount
```

# Changed Behaviour

`PBSProProvider` when misconfigured will now raise an exception.

# Fixes

Fixes #3793 

## Type of change

Choose which options apply, and delete the ones which do not apply.

- Bug fix

Author: yadudoc

parsl/providers/pbspro/pbspro.py

@@ -5,6 +5,7 @@
 
 from parsl.jobs.states import JobState, JobStatus
 from parsl.launchers import SingleNodeLauncher
+from parsl.providers.errors import SubmitException
 from parsl.providers.pbspro.template import template_string
 from parsl.providers.torque.torque import TorqueProvider, translate_table
 
@@ -97,6 +98,14 @@ def _status(self):
 
         retcode, stdout, stderr = self.execute_wait("qstat -f -F json {0}".format(job_id_list))
 
+        # If qstat failed do not update job state
+        if retcode != 0:
+            logger.warning("qstat failed with retcode:%s STDOUT:%s STDERR:%s",
+                           retcode,
+                           stdout.strip(),
+                           stderr.strip())
+            return
+
         job_statuses = json.loads(stdout)
 
         if 'Jobs' in job_statuses:
@@ -198,10 +207,19 @@ def submit(self, command, tasks_per_node, job_name="parsl"):
                                               'job_stderr_path': job_stderr_path,
                                               }
         else:
-            message = "Command '{}' failed with return code {}".format(launch_cmd, retcode)
-            if (stdout is not None) and (stderr is not None):
-                message += "\nstderr:{}\nstdout{}".format(stderr.strip(), stdout.strip())
-            logger.error(message)
+            message = f"Submit command '{launch_cmd}' failed"
+            logger.error(
+                f"{message}\n"
+                f" Return code: {retcode}\n"
+                f" STDOUT: {stdout.strip()}\n"
+                f" STDERR: {stderr.strip()}"
+            )
+            raise SubmitException(
+                job_name=job_name,
+                message=message,
+                stdout=stdout,
+                stderr=stderr,
+            )
 
         return job_id