[Proposal] ORCA: remove the CJobQueue

[jiaqizho]

Proposers

jiaqizho

Proposal Status

Under Discussion

Abstract

About CJobQueue

As I understand it, CJobQueue is used to prevent do the same Cjob.

In CJob.h:L51 desc:

//		Job Queue:
//		Each job maintains a job queue CJob::m_pjq of other identical jobs that
//		are created while a given job is executing. For example, when exploring
//		a group, a group exploration job J1 would be executing. Concurrently,
//		another group exploration job J2 (for the same group) may be triggered
//		by another worker. The job J2 would be added in a pending state to the
//		job queue of J1. When J1 terminates, all jobs in its queue are notified
//		to pick up J1 results.

Also let me explain the logic of JobQueue in detail.

Before we start, let me introduce two type of CJob in CJobQueue

• Queued Cjob: the position in CJobQuque is >= 1
• Main CJob: the position in CJobQuque is 0

If the CJob being executed is the Queued Cjob

• Only the same Cjob Type(Exploration/Implementation) in the same Cgroup will be added to the CJobQuque
• Whatever the result of Cjob->FExecute() is , the Queued CJob will always return the false in CScheduler::FExecute.
• After CScheduler::FExecute, the Queued CJob will change to the EjrSuspended state, until the Main CJob release it.
• Once Main CJob call the NotifyCompleted, pop the Queued CJob and it will resume the parent. it's parent still push its StateMachine, because the Main CJob already do the same work in the CGroup

If the CJob being executed is the Main Cjob, it can be executed after StateMachine finished and call the NotifyCompleted logic.

Git history

I looked through the git history of CJobQuque. it's added in the first version of ORCA(76feb99 gpdb). In early versions of ORCA, concurrent execution existed until the commit Remove multi-threading code (#510)(61c7405). Before this PR, CScheduler can get the Queued Cjob, cause CScheduler will use the multi-threading to process the waitting list, and some of job may insert into the same CJobQueue.

Proposal

In current version of CBDB, the ORCA is single-thread mode. So the size of each CJobQuque is always 1, also in CScheduler won't get any Queued Cjob. I have verified this pointer in PR(#742, Although ORCA icw tests are not enabled in GITHUB CI, I have verified on my own machine that ICW-orca test does not trigger these assert(false)).

There are two reasons to remove CJobQueue:

• CJobQueue is dead logic of in current version ORCA.
• GP has proven that parallelization in CSchedule itself is a non-profitable endeavor. Maybe we should think of other acceleration processes.

Impact on cherry-pick GP

This content has not been changed in subsequent GP versions. left is CBDB(show 46 search results), right is GPDB(show 46 search results).

Motivation

pass

Implementation

pass

Rollout/Adoption Plan

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[my-ship-it]

Please paste pictures in English so other engineers could participant, thanks!

[jiaqizho]

updated

[my-ship-it]

As far as I know, it is possible that ORCA's design is based on the DXL structure. DXL supports DAG format, but the plan in PG is a tree structure, so the queue size is 1?