Does engine idle timeout work in HA mode? If so, how is it work

[HaoYang670]

I haven't tested but curious on how engine idle timeout could work in HA mode.

For instance, if an engine has sessions across multiple Kyuubi servers, how many SessionManagers are there? and how could the SessionManager know the number of open sessions in total?

I roughly go through the code of SessionManager but doesn't find how it achieves HA.

Thanks

[HaoYang670]

Hi @pan3793 Could you please help on it? Thanks a lot

[pan3793]

It does not matter whether HA is enabled or not, because "engine idle timeout check" happens on the engine side, not the server side.

Each Spark SQL engine (driver) holds a SparkSQLSessionManager, and it holds all active sessions (maybe come from one Kyuubi server, and maybe multiple Kyuubi servers).

[HaoYang670]

Oh, Thanks a lot. This really makes sense.

BTW, does the session idle check or operation idle check also happens on the engine side?

[pan3793]

techniquelly they could be done both, IIRC, Kyuubi implemented it on the server side.

[HaoYang670]

Thank you! It is really helpful. Thanks a lot

Other 2 questions are:

1. Is KyuubiSession and SparkSession has one-to-one mapping relation?
2. and same question for KyuubiOperation and SparkOperation, Thanks a lot

[pan3793]

Is KyuubiSession and SparkSession has one-to-one mapping relation?

Yes.

and same question for KyuubiOperation and SparkOperation.

Almost right, but there are some exceptions, for example, LaunchEngine is a server-side operator, it performs spark-submit, has no corresponding engine-side operator