WIP: [AEP-1] Roadmap for v1.0

[transientlunatic]

This is a draft plan of ideas which I want to implement prior to asimov reaching version 1.
My aim is to complete most (perhaps all) of the work on this by the end of 2026, with as many parts as possible ready to test on the Summer 2026 LVK observing run, and certaintly in advance of O5 commencing.

Reproducibility

One of the long-term goals of asimov has been to improve the reproducibility of LVK (and non-LVK) GW analyses. Doing this requires asimov to be equipped with the means to run analyses with no hand-tuning on non-LVK resources. This is largely achieved in the case of GWTC analyses, though some steps still need to be fully incorporated into asimov workflows.
In addition to increasing the amount of the LVK workflow which can be reproduced using asimov, it would be desirable to have a means of easily identifying analyses which are compatible with each other, or are suitable for a subsequent analysis.
To do this we would need to have a more robust and standardised format for specifying analysis settings; the current blueprint approach in asimov is an obvious starting point for this, but these should be standardised so that pipelines can be reliably configured with minimal changes (e.g. by requiring the the sampling frequency is always specified as likelihood>sampling rate). This would also allow tools to be developed which would identify errors in blueprints before they're run, and make maintaining a database of previous analyses plausible.

• Introduce a validator for asimov blueprints (Introduce a blueprint validator #7)
• Standardise asimov blueprints (e.g. via a YAML schema)
• Allow asimov to export and import completed analyses (which can then be used in "downstream" analyses run by asimov)
• Refactor the handling of prior settings in asimov (Refactor prior handling #32)
• Comprehensively document the numerous parameters supported in the blueprints
• Produce standardised testing tools for pipeline integrations (Run scheduler tests in github actions #11)
• Deeper integration with software environment managers
• Packaging of results with reproduction blueprints

Flexibility

Another important aim for Asimov is to make running parameter estimation on GW events (and injections) simple and reliable, ultimately reducing the time between deciding to perform an analysis and getting the results considerably. It already does this to some extent, but there is much room for improvement.

• Improve support for "downstream" analyses, e.g. cosmology and population analyses via recent event and project analysis interfaces
• Allow the simple creation of large numbers of analyses according to some plan (e.g. setting up IMRPhenomXPHM runs on all events from a given set of events, or creating injections across a range of distances and chirp masses, and then analysing each with an identical RIFT setup).
• Allow asimov to work with a broader family of schedulers (e.g. slurm), and run shorter jobs (e.g. those which take under a minute) independently of a scheduler (but outwith the main monitor loop). (Make scheduler interface more flexible #1)
• Increased flexibility in creating configuration files, esp. in YAML format where far less templating might be required than for older formats.

Performance

There are a number of on-going problems with performance in the asimov codebase which need to be addressed.
Asimov struggles to complete the "monitor loop" efficiently for very large projects (>~100 events), and the use of a YAML file to store the database of settings (the "ledger") is prone to corruption, especially if the asimov monitor command is run in multiple places simultaneously. Updates made to this file are also not transactional, so it is perfectly possible to apply an update to an event (e.g. adding an analysis), and this to be lost because an ongoing, simultaneous update overwrites it.

• Reimplement the ledger in a performant database
• Refactor the monitor loop to both improve modularity and performance
• Consider parallelising the monitor loop / rewriting parts in e.g. rust
• Improve logging, observability, and the collection of telemetry data

Quality of Life

Some aspects of using asimov are less easy than they should be.
Adding an analysis to an event requires the user to lookup the correct URL for a given blueprint, add that to the event, and potentially do this many times for each event. This adds a lot of friction to the process, and it's not always clear that the process has been successful. This category of issue is likely to have the largest number of required changes, but each of these may be quite small.

• Redesign the asimov webpages to provide more useful data to accompany analyses, and allow e.g. rejected runs to be hidden
• Improve CLI tools for interacting with standard blueprints (e.g. more easily setting sensible pipeline defaults etc)
• Create a "wizard"-like CLI for adding new events and analyses to reduce friction and the barrier-to-entry for creating analyses
• Create a web-interface to asimov

Project planning

To make the project more generally accessible (and to take advantage of infrastructure not readily available in LIGO gitlab) I plan to move all development of the main asimov codebase, and much of the ecosystem to Githhub, starting with the active development of v0.7.0. This has some related issues.

• Decide on the long-term home of the project (currently in my github userspace: should this go to an organisation? Daniel owns the etive.io domain which is an option)
• Consider the name of the project (asimov isn't available as an organisation name)

[ColmTalbot]

This looks like a solid plan, I think these changes will significantly improve quality of life for users.

Some initial thoughts from interactions I've had with asimov that aren't touched on here (I might add others later).

• On the bilby_pipe development side we've been discussing moving away from ini configuration files, likely to yaml. Since asimov is already strongly yaml-based it would be nice to support yaml configuration for pipelines.

Some thoughts on points here (feel free to ignore):

Standardise asimov blueprints (e.g. via a YAML schema)

I've used (and enjoyed) pydantic for writing pipeline code, just something to consider. I think it would pretty easily handle some of the other goals for reproducibility.

Reimplement the ledger in a performant database

👍

Redesign the asimov webpages to provide more useful data to accompany analyses, and allow e.g. rejected runs to be hidden

I think it's worth aiming higher here. Adding more interactivity to the asimov webpages would be extremely useful. I've played around with this a little (AI tools are pretty great at this)

[transientlunatic]

Thanks @ColmTalbot!
Pydantic is definitely one of the approaches I'd been considering (possibly from your previous suggestion).
The webpages definitely need an overhaul and I agree there's a lot of scope for doing more than just tidying up the noise.