feat: ClusterState does not cache session contexts

[milenkovicm]

... as session context was not cleaned up.

Which issue does this PR close?

Closes #1220.

Rationale for this change

• ClusterState was keeping HashMap of session_id -> SessionContext which was never cleaned up.
• ClusterState will never emit events about accessed or removed SessionContexts.
• There is a need to time out (and remove) contexts from scheduler, or do cluster wide cleanup when session context expires. There were two ways to do this:
  • bringing cache to ClusterState and emit events when cache expires or
  • externally listen for SessionContext events and react when session_id times out (keep cache out of ClusterState.
• Client SessionContext is authoritative not the context kept scheduler, it will simplify client scheduler interaction if initial create_context is skipped.
• ClusterState is stateless from connected client perspective

What changes are included in this PR?

In order to have expiring session contexts a LRU cache is needed, which I did not want to bring in. So this implementation proposes not to cache session contexts, rather to create them every time. This is trade off between correctness and keeping additional dependency in. It can be revisited later if needed.

Also, this change brings:

• Client sets session context id, no need for extra create_context call.
• get_context has been replaced with create_or_update_context
• SessionContext notifications are published (SessionAccessed, SessionRemoved)
• SessionAccessed, SessionRemoved added
• SessionUpdate removed
• change in ballista protocol to reflect create_or_update_context method change
• session_id is required to be set
• operation_id added to help with client-scheduler request mapping

Are there any user-facing changes?

• there are changes in ballista protocol

To be done

• add request id (uuid)

[Copilot]

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

[andygrove]

I don't currently have time to review large PRs like this unfortunately, but I skimmed through it and it looks reasonable.