Enable running of COSIMA Cookbook code outside of NCI

[taimoorsohail]

Following on from discussion in #657, this issue proposes to enable the COSIMA Cookbook code to run outside of NCI restrictions. Data could still be hosted on NCI, but the actual group/project permissions currently pose a hurdle to run the recipes interactively. cc @charles-turner-1 @navidcy

[adele-morrison]

This seems like a big task? Is the aim here for others outside Australia to be able to analyse COSIMA data more easily (i.e. beyond the recipes)? Or just to run the recipes? How much demand is there for this?

[taimoorsohail]

I made this issue in response to the discussion in the JOSE paper PR. While I suspect the demand currently probably doesn't justify the work, @charles-turner-1 mentioned he has some ideas to implement this. Thought it would be better to move that discussion out here so as not to clutter the paper discussion with unrelated matters.

[charles-turner-1]

Cheers for opening this one up @taimoorsohail!

To my mind, the 'run the cookbook anywhere' issue really boils down to the choice of two paradigms:

1. Bring the compute to the data. This is what we're doing now and it works great, provided a user has NCI access. My personal opinion is that this is a real issue when it comes to people getting their feet wet, despite being super effective for what I'm gonna very vaguely refer to as power users.
2. Bring the data to the compute. This is somewhat more aligned with the cloud way of thinking about things. This has a bunch of additional limitations that the first approach doesn't, namely it means we can make fewer assumptions about the nature of the compute that the user has available, but also that it requires data to be curated a bit more cleanly.

What I had in mind, was to try to mirror data from a bunch of the recipes into something like Pawseys Acacia object store, or NCI's Nirin. We can then set bucket permissions, which makes the data remotely available.

We wouldn't be able to just do a lift and shift here - it would require probably reformatting into Zarr or virtualisation, and then we'd also need to rechunk the data to make weaker assumptions about the amount of compute power. Nowadays, I think aiming for something that could run with 8gb of ram would probably be a pretty safe bet.

I wrote up some investigations into virtualisation here - https://charles-turner-1.github.io/personal-homepage/blog/virtualising-on-acacia - it's a bit wordy as it's effectively reference material for myself, but there should be a lot of extra relevant detail near the start.

I don't think we'd want to do this with every recipe - just a curated subset so people could get going on their own machines/colab or something.