Providing demonstrations instances

[mbruns91]

Just opening this issue to have a place where we can condense all efforts of providing access to pyironflow for a broader audience.

At the moment we have

• a web interface (by @liamhuber)
  This a kind of "stipped down" version, where all embedding into the usual jupyter lab interface is removed.
• a docker image (by @mbruns91, not yet hosted or published anywhere)
  This could be used for providing access via the PMD infrastructure as well as binder environments.

We now should clarify

• Who is the user group for each solution?
• What kind of demonstration material do we want to fill into these environments?
• ???

Please add anything coming to your mind in this regard!

[liamhuber]

How do we modify the node environment? I.e. is this per-user, do we have a power-user role for different groups to have a respononsible "node manager" team member, or does the demonstration instance admin manage it? I can imagine it's different depending on the nature of the demo instance, but it bears thinking about.

[Tara-Lakshmipathy]

I think it would also be great if @liamhuber provides a more detailed intreoduction of the we portal to us. Right now it is unclear to me what the plan is for experimentalists to e.g., interface with their robotic equipment or upload datasets and nodes to the portal. And would these nodes and datasets be public? But how would nodes that interface with a particular equipment in a lab be interoperable? Would it be possible for experimental groups to set up their own web portals that have a data backend of their own shared server or something? So many questions... Or is the portal just an appetizer of sorts for a future product?

I have sort of similar questions about PMD-S given the diversity of the participant projects in PMD. But at least in my head, PMD-S is a bit easier to answer.

[Tara-Lakshmipathy]

How do we modify the node environment? I.e. is this per-user, do we have a power-user role for different groups to have a respononsible "node manager" team member, or does the demonstration instance admin manage it? I can imagine it's different depending on the nature of the demo instance, but it bears thinking about.

I see the word "demonstration" being used a lot. So it is just an appetizer.. but to what? Hmmm... the plot thickens.

[liamhuber]

I think it would also be great if @liamhuber provides a more detailed intreoduction of the we portal to us. Right now it is unclear to me what the plan is for experimentalists

I see the word "demonstration" being used a lot. So it is just an appetizer.. but to what? Hmmm... the plot thickens.

Yes, the point here was always for this to be a proof-of-concept demo serving two roles: First, for users and groups without code experience or even necessarily access to your controlled compute cloud to play around with the GUI experience. This allows them to give feedback to the GUI team (led by you @Tara-Lakshmipathy) about what they like in the UX, what capabilities they might be missing, etc. Second, for those who take a shining to it, it gives a code-free platform for them to start getting their hands dirty running lightweight production runs; e.g. to upload some small or medium sized data and do some analysis and maybe generate a plot or two.

The first is now available, anyone in your orbit just needs to sign up and can log on and play with the GUI as it is. It's relatively straightforward for me to push updated GUI infrastructure, so we can push updated UX to them more or less continuously to get a nice feedback loop if they're interested.

The second is as much a question for me as for you! I'm very happy to help push the ball forward here, but I don't even know these people, so somebody else needs to take the lead on talking with candidate groups and working with them to develop nodes that they would find useful.

But how would nodes that interface with a particular equipment in a lab be interoperable? Would it be possible for experimental groups to set up their own web portals that have a data backend of their own shared server or something?

Two-way interaction with lab equipment would probably be super tricky to manage over a remotely served instance of python. In principle it should be possible if the equipment is also exposed to the internet, and you can go over http signals or whatever, but I don't imagine anyone wants that. For automated control of equipment, I think you would want a local python server running pyironflow without talking over the internet at all.

[Tara-Lakshmipathy]

The current use case I'm looking into does interface with lab equipment, but I can try inquiring for some post processing workflows. Maybe we can try to have a node that downloads data (e.g., CSV files from the lab equipment, or TEM images or XRD stuff) from a repository, does some post-processing and a final node that uploads results back into the repository. I could see something like that working for experimentalists in general.

But BAM is little bit of a different case because of its super-tight security. The chosen platform for sharing data within BAM is openBIS, but this is not accessible from any network on the outside. So, BAM would restricted to data that can be uploaded to public platforms. On the other hand, if this becomes successful, I could see a similar portal set up for BAM that interfaces directly with our openBIS instance.

By the way, the SFB1394 also works with openBIS and has some experimentalists. So, probably @niklassiemer and @pkruzikova could get in touch with them to see if we can get something from them?

[Tara-Lakshmipathy]

I spoke yesterday with the experimentalists I'm collaborating with in BAM. There may be some very interesting post-processing workflows in the future including NN/Bayesian-optimization. But for now, the post-processing workflows that they use have very few parameters to play around with. The workflows would most likely consist of nodes that just read some files and do some predefined things and plot it. So, they won't be too interesting in terms of showing off the capabilities of the portal. But on the bright side, these workflows are apparently widely used, and would hence be useful for many other experimental groups out there. I'll look into these when I get back to Germany and try to make them as interesting as possible.

[liamhuber]

@Tara-Lakshmipathy, that is super cool! In analogy to what you're describing, I've updated the demo at pyironflow.com to use an array from np.save, do a fit to the (now noisy) data, and save the resulting figure to file. I made a relatively simple file browser widget and stuck it below the main GUI, so you can upload your own numpy files and download the resulting figure to your local machine. For downloading, the only catch is that you need to "refresh" the file browser after running the workflow, because I didn't yet find a way to robustly sync the browser widget with pyironflow widget activity.

I haven't made the files persistent yet, but we're ready for it as each user is getting their own portion of the file system to up/download data at. You can look at the source for the "array" or "show" nodes to see how this is managed, but basically the storage root is stored in an env parameter os.environ["PYIRONFLOW_DATA_STORAGE"]. I hope this demo is similar enough to the type of analysis you're talking about to be a good springboard. Just ping me whenever you have a particular set of application nodes ready and we can get them deployed on the website!

[Tara-Lakshmipathy]

Quick question: how much space does each user get? And maybe in the future, there can be a quota usage indicator somehwere.

And a quick observation: i forgot my password for the portal and reset it successfully. But after resetting it, the screen did not display a message that it was successful, instead the screen showed a new form to reset the password again. The new form however cannot be used to reset again (not that I want to reset again, but I just tried it out anyway) due to token expired error.

[liamhuber]

Right now the storage space is not permanent as I'm having trouble attaching storage without switching the application back to a US server. This isn't a fundamental problem, but is annoying for the present. Otherwise all users together should stay to <50GB of persistent storage or I'll need to work out how to get the application talking directly to AWS storage (or similar). Since the contract was for demo + early adopters to do some "lightweight" production runs, I think that's outside scope.

In the meantime, since users can anyhow freely up/download to the application I think we have enough capability to get rolling. I've internally limited each user's directory to 100 files, and each file upload to 1 MB, but I can scale those up/down (within reason for the file size) super easily. The application instances have 8GB RAM right now, but I can scale that up pretty easily too if there's a need.

Regarding the forgotten password: are you still locked out? Absolute worst case, just try opening a private browsing session and going to pyironflow.com and you should get a clean login.

I just tried resetting my own password, and reset+"back"+login worked ok and reset+new tab+login worked ok, but reset+reenter pyironflow.com in reset tab --> 403. Re-entering pyironflow.com cleanly in the reset tab a second time still worked though.

Getting Voila to play nicely with the authentication redirects was a massive pain in the butt, and so I try to clean up the state on each successful/failed login. It looks like it's possible to hit an edge case I don't handle via the reset password route, but anyhow just going to the website twice in a row should (and does for me) resolve it by successfully cycling through my intended cleanup cycle.

LMK if you're still stuck!

[Tara-Lakshmipathy]

No no, I'm not locked out. I just had to open a new tab and I was back at the login page and everything worked from there. It's just that there was no confirmation message that the password change was sucessful. Instead, it just gave me a new form to fill on the same page. But a new tab did the trick.

[liamhuber]

No no, I'm not locked out. I just had to open a new tab and I was back at the login page and everything worked from there. It's just that there was no confirmation message that the password change was sucessful. Instead, it just gave me a new form to fill on the same page. But a new tab did the trick.

Ok, good 😅 That is not ideal, but IMO it is tolerable.