User Experience and the CLI

[gilesknap]

User Experience Discussion

Intro

We need to work out what we want the user experience to be for beamlines with containerised IOCs.

We have two kinds of customer:-

• Controls engineers (and maybe other devs)
• Beamline Scientists

The whole point of ec is to make things simple for managing containerised IOCs.

HOWEVER:

• We have two ways of managing IOCs and two corresponding 'backends' for ec
  • kubectl and helm
  • argocd
• We do not yet have parity of function between them
• Some important functions may not be possible with the argocd backend

Missing Functionality

The ArgoCD backend for ec does not currently support deploy but we can implement that by writing to the repo. I am strongly in favour of implementing this as I feel it would be better to have programmatic editing of the deployment repo instead of manual. Having said this, we still need to support the manual approach - at least for now that is how DAQ want to do it.

Features that have no way of being implemented in ArgoCD are:-

• attach to IOC shell: absolutely essential feature
• deploy-local: The argoCD equivalent is tracking a branch but this requires that all tentative changes go in the git repo and must be squashed later. deploy-local is a really handy way to do quick iteration on an IOC.
• debugging: this feature is not yet implemented but I'd like to automate the process of switching an IOC to debug mode and connecting to it with VSCode remote dev. This essentially has the same requirements as deploy-local

Two back ends

discussed in #182

We could have both backends enabled simultaneously and choose which to use based on function.
(Marcell is not going to like this!)

There are a couple of problems with this idea:

• it requires two sets of credentials, one for ArgoCD and one for kubectl
  • although kubectl will be cached for long periods
• This is not enough to easily support deploy-local or debug-mode because you will need to turn off ArgoCD for the IOC in question and remember to turn it back on again

Default Setup on Beamline workstations

It would be good if using ec on a beamline workstation just worked. Simply setting up the default environment from environment.sh has a couple of issues:

• we have to decide between backends!
• sourcing environment.sh involves logging in to kubectl or ArgoCD - therefore has to be an interactive process

ec monitor service

One idea that can mitigate some of these issues is to have ec monitor running as a service in the cluster and serving its GUI over HTTP. Marcell has prototyped this and it looks really good.

This would mean there was a web page that anyone could go to (we could serve it to the beamline subnet onl.y I guess). This would allow anyone to:

• start and stop IOCs
• see what IOCs are running
• get IOC logs

The issue with this is it would be a single account running all of the calls to Argo/kubectl. We therefore loose any accountability (so just like the current state of play).

[gilesknap]

@coretl @GDYendell @marcelldls We should get together to discuss this. It's kind of a big hole in the container strategy right now.

[coretl]

@DiamondJoseph I'd be interested on your opinion too

[DiamondJoseph]

I'm WFH tomorrow, Monday and I believe this Friday but we can figure out a time otherwise.

From DAQ perspective, at least in Core [or the version in Joseph's head] where we're used to using both, Argo and Kubectl are different layers of the deployment stack- I don't find it too jarring to have my desired state defined in Argo and then still port-forward/exec into the shell of a container with kubectl or the kubernetes dashboard

[gilesknap]

OK - I think we are getting to a good place.

• the argocd support for ec now includes:
  • deploy
  • delete
  • both of these make changes to the main branch of the deployment repo

This means that we can pretty much just use argocd for normal work with IOCs. The remaining things we need are:-

• exec a shell in the container. ArgoCD GUI now supports this at DLS and I'm pretty sure that means we could add it to ec if we felt so inclined
• console: this is important but Tom has suggested that we expose the console on a port and just exec a process that connects it to stdio (i.e. telnet). Um - this is procserv right?

One thing that I'm still a little unhappy about is that ec has become an IOC only tool
Maybe that is not at issue - we will almost never touch epics-pvcs and epics-opis, and DAQ appear to prefer direct interaction with the deployment repo via git.

There is a slight disconnect over how controls and DAQ will use the deployment repo. But I'm still not sure it warrants a divorce.

[gilesknap]

I guess we still don't have:

• deploy local which is rather useful
• debugging

But perhaps when I have looked at DiamondLightSource/python-copier-template#234 (comment) we may have something that covers these use cases.

[marcelldls]

@gilesknap maybe there is an option for attach which is to exec a shell and then basically run kubectl attach from the container? Or a lower privilege option might be to start a screen session?

[gilesknap]

kubectl attach inside the shell is quite cool because it will get killed when you close the shell and avoid the we have no way to detach issue. Although - I wonder if we could do it with python and kubernetes library.

Only problem is we'd have to deploy kubectl inside the containers which is a bit klunky especially WRT version management.

screen is also an interesting idea ... I might have a play with that

[coretl]

• console: this is important but Tom has suggested that we expose the console on a port and just exec a process that connects it to stdio (i.e. telnet). Um - this is procserv right?

Ha! We come full circle... It is also only an apt-get away:
https://github.com/ralphlange/procServ?tab=readme-ov-file#linux-system-packages

I tried this locally and procServ -f -P 8000 /path/to/start.sh seems to do what we want. The only issue is that you can't make procServ die when the ioc dies, unless we have start.sh fork a process that sends CTRL+Q to procServ when the IOC dies:
https://epics.anl.gov/tech-talk/2018/msg00299.php

Having said that, then maybe procServ auto restart isn't too bad, as it lets us do restart of the main process without tearing down the container...

One thing that I'm still a little unhappy about is that ec has become an IOC only tool

I'm not sure what makes ec an IOC only tool? Although I didn't read this thread in detail... I'd be happy with some IOC specific commands in ec, but it would be nice if the majority would work with FastCS, blueapi and other copier template projects...

[gilesknap]

@coretl, @GDYendell and I do not like using procserv. It will undermine K8S control of the pod - K8S won't know about restarts and an idle pod will look like it is running OK. When you talk about forking processes and sending ctrl-Q that makes me hate it. I'm going to have a think about this and try to make it work without introducing any new code or concepts - I'm liking @marcelldls' kubectl attach inside the shell better than anything else right now.

Why ec is an IOC only tool is because we decided to separate out ec_services and services in the deployment repo's apps.yaml the distinction is that ec_services support a disable flag and that has to be supported in the helm chart that deploys them. Others could support that but at present only IOCs do.

disable allows us to stop and start IOCs without removing them from the cluster. That is nice because you can still see them in ArgoCD and in ec monitor so stopped IOCs are discoverable and available for starting. However this is probably not particularly kubethonic!!

[gilesknap]

I've been working on the environment modules for ec. So we will be able to do

• module load ec/i13-1
  etc.

This is oh so much nicer than the current workflow of cloning a repo to get the environment.sh.

I'm able to get the module to pull in k8s and argocd configuration and only request that you log in to either of these services when your token has expired. In fact I have supplied an entrypoint called ec-login and that currently logs you in to argocd but it COULD also log you in to kubectl if your token was expired.

If we go with that then we could relax the split between argocd and kubectl a little.

• always use the argocd API for all functions except:
• use kubectl for ec-attach

The argcd workflow is now very tidy as shown by: https://drive.google.com/file/d/1-hvIXT_QkfvGA7SqN_nZZ9bKsiiIXKRm/view. I therefore feel we have something viable for production.

EXCEPT: at some point we took the decision to always update the deployment repo when starting and stopping IOCs. This has a downside that everyone who is allowed to stop an IOC must have write access to that repo. Solutions to this inlcude:

1. give scientists write access to the deployment repo
2. only allow scientists to do restarts, not stop, start (restart is argocd API only)
3. use overrides instead of repo for starts and stops (I have concerns about how this interacts when redeploying a currently stopped IOC - I think it might get fiddly)
4. run ec as a web application using an account with the necessary permissions (this removes accountability)

I prefer 1

@GDYendell @coretl I'm interested in your thoughts on this.

[coretl]

I vote for 1

[gilesknap]

OK if we have a consensus on 1. the last thing to fix is attach.

See epics-containers/ibek#293
We now have a tidy way to do it:

• argocd has had it's exec capability turned on at DLS
• I believe this is accessible from the argocd cli
• therefore we can just exec a socat session inside the container and we'll get full multi-client console access

The deploy tools changes are also in.

[gilesknap]

UPDATE:

The tidy way has been moved to its own repo at https://github.com/epics-containers/stdio-socket.

BUT when testing in containers there were a few issues:

• stdio task is being terminated when the socket connects - this only affects the original process i.e. when you attach you get no stdin.
• Using console in argocd terminal works but when you click away and the console terminates so does the IOC itself

TODO

• I have been unable to see any exceptions in the tasks but Tom pointed out I should use BaseException in the catch. Try this to see if we can work out what is happening and catch any errors we don't want to kill the task.
• probably the best course of action is to not try and have stdin monitoring on the original process and just make sure that the stdio-socket wrapper only gets used in cluster. This would avoid stdin contention and extra stuff running in standalone containers.
• Tom has pointed out that switching to ProcServ would require code changes because ProcServ cannot be convinced to drop out of the process when it's child closes - we really want that in Kubernetes.

[gilesknap]

stdio-socket is now fully working.

We just need to add exec and attach to the argocd backend for ec now.