How can we measure progress?

[clizbe]

Our internal leadership wants to keep better tabs on our progress.
To me this is tricky considering our organic/iterative development - but I agree it would be nice to recognize early if we need to triage.

One way might be defining Key Performance Indicators (KPIs). These can optionally have a Threshold (requirement) and a Target (bonus).
Do you have ideas about what we could use for KPIs?
I thought maybe:

• Speed of building model of European Case
• Speed of solving European Case
• Number of features implemented?
• Size of problem solvable?
• Something to do with data handling?
• Something to do with pre-processing?
• Something to do with documentation?

To me KPIs are a nice way of focusing on quality over quantity. We can also make a list of core features and track as they're delivered, but I think including quality somewhere is important as well.

I thought maybe eSciencers would have ideas about this since you have more experience with programming projects.
@suvayu @abelsiqueira @lsoucasse @datejada @gnawin

[suvayu]

KPIs are a good idea, and those are all valid candidates. I think maybe we can treat them as categories, and within each category define KPIs under required & target, where we can choose between them based on how often that step needs to be done. So something along the lines of this maybe?

category	required	target
performance	benchmark model: Europe
	- model building < 1m
	- model solve < 1h
model features
problem size	(how to define this?)
data pipeline	- no data duplication	- explorable datasets
	- handle scenarios
	- handle solver options
	- customisable pre-processing
documentation

[clizbe]

That's a really good point that if we're doing it right, the size of the problem should be smaller than what other models are solving.
Maybe instead, the size of the grid and detail somehow... Like # energy carriers, length of solve (year), etc?

[clizbe]

As in, the size of the scenario, rather than how large it is when we go to solve it.
But maybe that's a hollow pursuit and we should just focus on the benchmark case.

[suvayu]

Maybe instead, the size of the grid and detail somehow... Like # energy carriers, length of solve (year), etc?

Yes, something like this. But there are some hidden caveats, for example time resolution of the profiles, I think for performance comparisons, we can pick the simplest case for these kinds of factors, and state them explicitly. That should cover it

As in, the size of the scenario, rather than how large it is when we go to solve it.

Not sure I follow. Do you mean to say include performance numbers for a multi-scenario run of the benchmark model? In that case I would say no. Since it doesn't add any uniquely new information.

[suvayu]

No data duplication - what does this mean?

Avoid the current "copy-paste" datasets to run a different scenario issue. This also could extend to how the datasets are handled for multiple scenarios inside Julia, e.g. shared arrays when the dataset doesn't change. And at the most extreme end of the wishlist is something like "Gather Update Solve Scatter".

[clizbe]

I think this is a good start! I'm going to swing it by management and if they're cool with it we'll solidify it. They might still want some kind of timeline, but I'm hoping just being able to SEE our progress at any point in time will be enough.

[clizbe]

Now we have a great planning! See the details of the Project.