Have we thought about how SRE principles will be affected? What are the meaningful metrics we can measure?

[greenhsu123]

Have we thought about how SRE principles will be affected? What are the meaningful metrics we can measure?

1. Change the frequency of scraping based on SLO targets?
2. Can (should) we measure the total carbon emissions for a given SLI?
3. Can (should) we do monitoring on demand?

[atg-abhishek]

Thank you for starting this discussion @greenhsu123 - here are some tentative thoughts on this:

1. I think the SLO needs to be harmonized with the "green" targets that the organization has so that we can achieve the functional requirements at the same time being able to meet environmental commitments as well.
2. Yes I think we should.
3. Monitoring on demand is an interesting idea, not sure how that would impact the other downstream SRE metrics that are reliant on monitoring being ever-present. Do you have any potential literature that you could share on this @greenhsu123 ?

[navveenb]

Good point, For SRE, we need to define the metrics that can be measured uniformly.
For Point 2 on SLI, it depends on the granularity and scope. If the SLI is on API latency (based on percentile), how do you calculate the carbon footprint. For instance API having 1 ms latency v/s 3 ms latency, does not imply 1 ms has lower carbon footprint. So specification should include scope, granularity and meaningful metrics to measure that can be reported uniformly. For instance, if we choose API , it might be too granular and we would need supporting infra, network, bytes transferred, region etc.. to come up with a score.

[greenhsu123]

1. We can think about introducing another green attribute to the 4 golden signals? so we are examining new features with an sustainable angle
2. agree with above that considering the total carbon emission per SLI can be v convoluted and too granular of a task. Instead of per SLI, how about we evaluate carbon emissions of ways of alerting on an SLO i.e BadMinute v.s BurnRate (more ways suggested https://sre.google/workbook/alerting-on-slos/ ). Or, all SLOs of a particular product, then again, this can be a v convoluted exercise

[atg-abhishek]

What are the 4 golden signals @greenhsu123 ?

[greenhsu123]

@atg-abhishek

The four golden signals are the attributes recommended by the SRE book to measure for monitoring user-facing systems. They are latency, traffic, errors, saturation.

With this new attribute, we can reject a new feature if it leads to the breaching of SLOs, continuously monitoring a carbon budget, and uphold promises to our customers that our system is still sustainable etc

[srini1978]

One of the options we can do while defining SRE of a system and where we want to bring in the concept of sustainability is if we start qualifying the SRE metric with our CI or C values so that we get uniformity in comparison.

For example , if we say that the latency SLI of an application is 99% of requests should be served within 5 msec over a 15 minute observation window. If we want to bring in the green quotient here, can we qualify this by adding "...and not emitting more than x kgCO2 of carbon emissions"

[navveenb]

Sharing some thoughts below

Let’s take a requirement, SRE for carbon footprint for a given cloud application.

Now the application can be shared or standalone. A shared application uses shared database , used by many application, so you need to drill down at the schema level, so there is level of complexity based on your application and infra setup).

One approach is to look at all these complexity, so it’s easier to implement the specification by the vendor

Other approach is have SRE requirements at a broader scope level for initial release - like environment (production, dev, QA).

The production environment consumes x carbon.
The production environment can be drilled down into supporting infrastructure (like database consumes x carbon, VMs, server less etc..) . As an SRE, you would need to identify where the bottleneck is.
This can cover all components/services and you can define SLO for each infra service/product, which adds to overall environment..
SLI can be defined based on agreed timeline (a day ?), Avg carbon consumption for database or overall production environment should not cross X..

If we need this data for a specific shared application, then we need to have something like tags, where we can tag each components and system can find out carbon footprint of application using the tags and have the defined SLO/SLI for the app.

[srini1978]

My 2 cents:
Typically SRE is considered for production environments only however that does not mean we should not calculate for lower environments but within the gamut of a software application and the SCI principles that we are trying to evolve, I am looking at production being the first "guinea pig" where we will try to define SRE in the context of carbon.

One additional thought is if we want to bring in the carbon co-efficient for SRE can we look at relative metrices like defining SRE as one where the "SCI of the application doesnt vary +-20% over "x" observation window " . My thought process is that this would give application developers and architects a window of opportunity to measure the values and also, as we have seen in the SCI principle we dont react to changes in absolute values like carbon emissions.

Of course the computation complexity is more difficult for shared applications but to me, shared applications with shared databases "by architecture" will consume less carbon than a single web application provisioned on a single DsV2 Virtual machine. We definitely have to have a way to document SRE for shared applications for sure but maybe in a subsequent iteration

[greenhsu123]

Probably worthwhile for us to come up with a couple of typical production scenarios across the board so that we can put together use cases for measuring SLIs,/SLOs, and eventually evaluating different alerting methods. Finding the right alerting methods for different type of signals can reduce not just the operational burden but also the load on systems used, and personally, I think it's one of the few things we can do here to make a difference to the overall emission level.

I agree with the above, having a reliable metric that allow us to continuously measure the carbon usage of an application seems like something we should introduce as part of the SCI spec, as currently, the focus of SCI if very much development focused

[srini1978]

@Green-Software

That is a great thought ! Some typical production scenarios for which we can define SCI can be:

1. Multi-tier web application on cloud / on-premise.
2. Desktop applications with backend services on cloud.
3. Long running processes.

In trying to define SCI for these applications, I think we would need to define an observation window for monitoring. So I guess from a SCI spec perspective, I think if we define the spec itself should be sufficient for these production scenarios.

[atg-abhishek]

@srini1978 would you be game to write up a short use-case in more detail against which we could test out the SCI? Something akin to this? Thanks!

[greenhsu123]

just some thoughs off the top of my head-
how about we change 1 area at once, e.g. chaning the udnerlying infra/hosting option...although this may turn out to be a very long winded exercise...

• a serverless java web service
• an ecs java web service
• an eks java web service
• any other hosting option

and agree with the observation window for the SCI, we can aim to provide a table similar to https://sre.google/sre-book/availability-table/ for different availability levels

[atg-abhishek]

This is great @navveenb - would love to have you expand on this and exchange during the WG call this week. Thanks!

[navveenb]

@atg-abhishek , sure we discuss this during the call and if we have consensus, can create an issue to reiterate further to cover it as part of spec.

[atg-abhishek]

All great ideas and would be good to discuss on the call today!

…

On Thu, Sep 23, 2021, 5:47 PM greenhsu123 ***@***.***> wrote: Probably worthwhile for us to come up with a couple of typical production scenarios across the board so that we can put together use cases for measuring SLIs,/SLOs, and eventually evaluating different alerting methods. Finding the right alerting methods for different type of signals can reduce not just the operational burden but also the load on systems used, and personally, I think it's one of the few things we can do here to make a difference to the overall emission level. I agree with the above, having a reliable metric that allow us to continuously measure the carbon usage of an application seems like something we should introduce as part of the SCI spec, as currently, the focus of SCI if very much development focused — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#72 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA6CZPDKK3CD4M2RMEQH3OLUDMLGJANCNFSM5DW37E7A> .

[jawache]

I do apologise I'm not well versed in SRE, is what's being discussed here along the lines of "production monitoring of live SCI scores for a running application" and what to do if the SCI changes or breaches some limit?

[navveenb]

From an SRE perspective, the standard measurement attributes like latency, error rates are well defined. Like typical API Latency is between 3-5 sec and you define the objectives - 95% of Average API latency should follow this, else raise an alert.

For, carbon footprint, the standard attribute for measurement needs to be defined.

Some of the discussion points are around –

1. The scope needs to be defined for measurement – you can’t have a narrow scope like APIs to start with (pls see discussion above)
2. We need to measure carbon footprint at various scope levels – let’s say for an environment, not just limited to production as development systems (like training models happen on dev machines or always-on dev VMs) needs to be investigated from reducing the carbon footprint.
3. For a shared application, the cloud providers (or any system) need to provide a provision to calculate the carbon footprint of resources used. This can be through tags or explicit descriptors. Usually these granular level of details are not provided by the cloud vendors.
4. If we come up with any measurement attribute, we want to come up with acceptable benchmarked threshold (like for API is 3-5 sec), frequency for capturing the measurement (i.e., hour/day) and rules.

The first step to address the above challenge, is coming up with SRE requirements that allows providers (systems/cloud providers/on-premises) etc. to report carbon footprint for the application (dedicated, shared) based on the environment.

For instance, Production environment (usually customers would have a separate VPC) = Carbon footprint for database + VMs + server-less etc.… for running the application.

This can be a good starting point for vendors to implement as part of specification. The individual resource level (database, VMs etc.) carbon specification can provide a starting point for any debug activity.

Once we have a base value, the deviation can be defined based on the application requirements. Each application is different and underlying infrastructure and services used would be different. You many not run on a free carbon region, as your region doesn’t support it and you can port to other region due to legal data regulations. The point is you may not be able to generalize. In future, as systems start reporting this, we can get some benchmarks based on no of users, environment usages etc.

Given the above considerations, in my view, the spec should address the SRE requirement for carbon footprint being calculated for dedicated/shared application , at-least based on a given environment for the first specification and it next iterations, we can look at the intersection of existing SLOs areas in detail, once we have benchmarks available.

[jawache]

Thanks that's clarified a few points. It sounds like the discussion is around adding a constraint to the calculation of the SCI rather than changing the equation of the SCI itself. So a constraint(s) along the lines of if your context is X then when you calculate the SCI you MUST do Y. Is my understanding correct?

[srini1978]

Sure Abhishek. Will do a first cut between today and tomorrow.

…

On Mon, Sep 27, 2021 at 6:52 PM Abhishek Gupta ***@***.***> wrote: @srini1978 <https://github.com/srini1978> would you be game to write up a short use-case in more detail against which we could test out the SCI? Something akin to this <#54>? Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#72 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACPKIRVSY6JVPPUTZXX7WS3UEBV27ANCNFSM5DW37E7A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[srini1978]

@atg-abhishek Please find the link to my first cut of an attempt to define impact of SCI on SRE principles - #92

[srini1978]

@navveenb Thanks for your feedback. Let me try to incorporate your feedback into my write up and I will publish again. I will retain the literature on intersection of existing SLO areas and we can have a discussion if we want to defer that granularity to a future release of the specification, maybe!

[navveenb]

@srini1978 Thanks, can you please share the link to the writeup.

[atg-abhishek]

@navveenb are you looking for this? #92

[srini1978]

@atg-abhishek @navveenb

I am collating more thoughts into the above write up and hopefully should be done by EOD tomorrow. Will drop a note for you guys to review

[srini1978]

@atg-abhishek @navveenb I am done with the updateshttps://github.com//discussions/92 . How do i request for review for this article?