Questions to consider as we shape the SCI

[atg-abhishek]

I think as a part of the scope of this SCI, we should try and at least come up with answers to the following questions so that the SCI becomes something that is meaningful and actually adopted in practice:

1. What are the properties of the entire software system that we should be measuring?
2. Which of these properties currently have/lack metrics for effective measurement?
3. As highlighted above, what consitutes effective measurement?
4. What are the current measurement methods that are being utilized?
5. What are the strengths / limitations of these methods?
6. Which circumstances are the ones where it is appropriate to use one methodology over another?
7. What are the different properties of the metrics that we desire?
8. What are the impacts of these properties and the chosen metrics on the evaluation process?
9. What data would be needed to enable these measurements to take place?
10. Which of those are already available and which ones need to be collected?

[jawache]

This is linked very closely to what we want to be sensitive to.

• So energy consumption is one property (so we become sensitive to energy efficiency improvements)

• Real-time marginal carbon intensity (so we become sensitive to carbon aware computing strategies)

• Hardware is another property and I think a great proxy for this in the cloud space is "cost" (so we become sensitive to hardware efficiency improvements, like arch changes that use less servers). Since I don't care what your server utilisation is, I care how much hardware you have reserved to work on your task, even the stuff that is idle and unused, because that means other people can't use that same hardware to do other things.

All of these should be across the full stack, so for an app it needs to include the cloud, networking and client-side.

Energy has some tooling but is modellable at least, we need to improve this space.

Hardware I think cost is a great carbon proxy measurement. This assumes there is a correlation between the cost of cloud services and the embodied carbon emissions of the cloud service. If there is a strong correlation between cost and carbon then it would also make things a lot more comparable. A service that costs X on one platform and 2X on another platform, we can expect the 2X costing one to have increased emissions.

[srini1978]

Are we also looking at SCI from a performance aspect ? By that, what I mean is: how do we provide recommendations around the inflection point where we know that gunning for higher software performance (additional hardware/testing cycles) correlates inversely with the amount of carbon released to achieve very little performance improvements. One of my thoughts is to provide guidance by dragging sustainability into the NFR realm as well . This way we could we could define a OKR saying " All requests to this web server should be served within 1 seconds 99% of the time" and if they want higher performance (say 0.5 seconds), then we define the sustainability cost of that performance improvement in $ of kgCo2 to make the impact more meaningful.

[jawache]

We should aim to separate "how we measure" from "recommendations to reduce" with SCI being just about "how we measure".

From experience, there will be many recommendations to reduce and we should aim to document them in the community WG. There will also be lots of arguments and disagreement, it's easier to agree on measurement than reduction.

The recommendations will need to be domain-specific (advice for web vs ml might be different) and perhaps even context-specific, two different mobile apps might have different recommendations given the context of use.

[atg-abhishek]

That's a great question @srini1978 - I agree with @jawache that given the remit of the SCI, it would be better to see how we can improve and standardize measurement so that people who want to make decisions and act (#19 ) on those measurements have the best possible information that allows them to make reductions as they see fit.

[vaughanknight]

1. What are the properties of the entire software system that we should be measuring?
I keep coming back to this one and how it would change how my team or customers would implement systems, how it would influence how they deliver software, and how it would inform others (or how they may inform others, customers etc).

The two categories I think that would influence implementation with these customers, that being carbon awareness enhancements, and efficiency enhancements. I'm not going to talk about efficiency for now as it is very complex.

(And it's because of this that I think a single SCI may be hard to calculate as soon as you are talking about a large scale distributed system... but this discussion for another time)

Carbon awareness enhancements feel a lot easier to measure and declare, and can have patterns that are implementation independent i.e. carbon aware orchestrators. A good example is batch jobs that run during green periods, but it's the scheduler that's carbon aware, not the batch job, and something that is likely in place for low pricing based batch processing. For big compute consumers, such as a bank, having to reimplement their underlying code may be impossible (old FORTRAN systems) but changing a scheduler to run that legacy/FORTRAN code is likely already in place.

The reason I see this being relevant standalone, is that this can also be something that's measured by industry regulators i.e. regulator mandates top 4 banks need to hit carbon awareness score of 42 by 2025. If that code has to be refactored from an efficiency perspective to hit a number/goal, that year goes to 2050 - note both are relevant, but one is likely to be easier to apply to legacy systems.

[edit] - we discussed today that even if you are doing energy_consumed * carbon_intensity + carbon as SCI, you still reduce that even if all you can adjust is carbon_intensity (via carbon awareness) ... I still think there is a place for the carbon_intensity just by itself, and gives the big corporates to show they are taking steps.

[jawache]

I totally agree carbon awareness can be the first stepping stone into this domain, least effort but also the least risk and I think it's the risk that has most people concerned.

The challenge is that the way we calculate carbon emissions right now with the GHG protocol doesn't take into account carbon awareness. So it's hard to get investment in this space, most investment going into efforts that reduce a number that's calculated through the GHG protocol. And most of what you can do with software is not captured in that protocol. So with the SCI, it has to be sensitive to carbon awareness, if an organisation adopts the SCI standard, then investing in carbon awareness becomes easier.

[atg-abhishek]

(And it's because of this that I think a single SCI may be hard to calculate as soon as you are talking about a large scale distributed system... but this discussion for another time)

@vaughanknight can you open a new issue to track and discuss this?

I really like this idea @jawache in the sense that there are then two levers for enacting change reducing the reasons that someone can say that they don't have avenues for improving their carbon stance.