MultiBackendJobManager: avoid mixing openEO job status and internal status in same column

[soxofaan]

MultiBackendJobManager involves a status column which is used for

• official openEO status, e.g. "created", "queued", "running", "finished", ...
• internal housekeeping e.g. "not_started", "queued_for_start"

Mixing both has popped up quite often as a source of confusion during MultiBackendJobManager feature requests, bug fixes, and attempts to improve test coverage (especially when we started to do things in threads and the order of execution stopped being deterministic)

the official job status and internal status should be separated

[soxofaan]

example of last incident: #839 (comment)

in 706fafe I naively wanted to stop counting "created" jobs as "running", which sounded reasonable, but then a bunch of MultiBackendJobManager unit tests became flaky

[soxofaan]

cc @HansVRP

[HansVRP]

Indeed, let's put it higher on the agenda.

However I do believe that this should be internal cuisine and hidden from the user.
These days they build color-map tools based on the dataframe which look at the single status column.

So I would propose to tackle this in the database only if possible

[HansVRP]

 _column_requirements = _ColumnRequirements({
        # Official openEO backend status (from API calls)
        "backend_status": _ColumnProperties(dtype="str", default=None), -->"created", "queued", "running", "finished", "error", "canceled"
        
        # keep status with all non official states
        "status": _ColumnProperties(dtype="str", default="not_started"), --> "not_started", "created", "queued_for_start", "queued" "start_failed", "skipped", "running", finished, error, "downloading" 

Then in the code internally we can work and act upon the backend status where needed?