-
Notifications
You must be signed in to change notification settings - Fork 4
Description
The addition of file staging complicates the native id setting in local-type executors. Whereas before, we could Popen a process and immediately set the QUEUED state with the native ID, with file staging we cannot, since files must be staged before the Popen. And since STAGE_IN > QUEUED, this leads to an impossibility if the staging is to be done in the same process as the one PSI/J runs in. Even now, in psij-python we cheat a bit, since we trigger the QUEUED state after Popen is called.
One possible solution would be to have a job management process that handles both the staging and execution. It would be the PID of this process that would become the native ID of the job. This adds complexity, since we now have to communicate state transitions to the main PSI/J process from the job management process.
Another possible solution, which leads to a simpler implementation but may complicate use, is to only guarantee that the native ID is set when the job is in an ACTIVE state. This would be a specification change.