Who uses slurm2sql? What are your use cases?

[rkdarst]

I started this to fill a need, but there are clearly some other users here so I'm wondering: what do you use this for? How important is it? What do you need?

Here are some of my principles:

• All values are the most basic (metric) units: bytes, seconds, seconds-since-epoch, etc.
• Try to use existing column names as much as possible.
• Try to support as many Slurm versions as possible, but if something becomes hard to support, don't worry too much about breaking compatibly. SchedMD support slurm for 18 months after release. Try to support at least those versions.
• Don't try to maintain database compatibility. It's expected that for all schema changes, you have to delete and re-import. But try to avoid this if not needed.

Most things are added based on our site's own needs, but I wonder if more flexibility is needed for use by others?

[megatron-uk]

I'm using slurm2sql to archive all of the slurm history from our legacy institutional HPC service which is due to be replaced within the next six months. This is a very valuable, rich set of data, and I absolutely do not want it to disappear along with the hardware.

For me, the ability to identify trends in historic data has already proven its worth by highlighting we are not a typical HPC shop, in that I mean an overwhelming majority of our users are not using MPI, are using less than two hosts, and are mostly single-digit core users.

Analysis of this type of data from sacct is indeed possible, but pretty laborious when you have 5+ years and 15+ million jobs to process.

Getting the data into a table where I can analyse, sort, group at will shows me all kinds of information which I just can't get from the normal slurm toolset.

Also, I can do this offline, without hammering the live Slurmdb instance!

[mrhawkin]

I also use it for similar reasons as @megatron-uk . I export out data from "our" account quota and the bring it into PowerBI to visualize trends over time so we have data to make decisions on investment for the future.

[megatron-uk]

One thing I should have added is that I modify the schema which is created to add start-year, start-month, end-year and end-month (and indices on those columns) of every job as that coincides with our reporting and 'billing' periods.

It's a bit of work initially, but once loaded and entries created for those fields it makes doing doing analyses over the given periods much simpler.