How should we think about functional units?

[jawache]

Is a functional unit linked to a "persona", for example a functional unit for the end-user vs. a functional unit for a machine learning engineer/researcher.

This was a discussion topic in the workshops and I thought it was a good way to categorize functional units.

An end-user functional unit might be Per Prompt, since that aligns with both what an end-user understand, but also is a unit of scale that makes sense to them - an end user will be able to scale their usage perhaps just by limiting their prompts so giving them a metric they understand that makes sense.

For a researcher/ml-engineer it might be Per Training Cycle. Since for them what they understand and care about is reducing the carbon emissions for a training cycle.

There was also conversation about how for different functional units you might include different parts of the lifecycle, for instance for an end-user, per-prompt, you might only include the operational costs, e.g. inference and supporting infra of running their prompt and leave everything else out.

Maybe the SCI for AI is a range of functional units which in aggregate cover the whole lifecycle but each functional unit is not responsible for including the whole lifecycle?

[bokelley]

I don't know if it's helpful but I've been pondering the idea of a session, which is one user trying to achieve a task. For instance I ask Claude "how do I get this code to compile" and we go back and forth a few times. Or I go to Sora and generate images until I get one that I can use in my LinkedIn post.

A great model might answer these perfectly in one turn, or ask a great couple of questions to refine before they do the expensive task of generating code or images. An ok model might take a bunch of turns, or never generate what I want. So while a great model might use more carbon per prompt, it would use far less carbon overall. I think this is important because we want to encourage effective use of AI.

So my proposed functional unit is "cost per successful session" or "cost per successful task".

This works well for the corporate use case where JP Morgan or somebody updates their customer service chat bot to prompt better and answers 20% more questions without having to call somebody - in this case the denominator goes up by 20% even if carbon per prompt is the same.

[jawache]

This is interesting it reminds me of the use case that Amadeus gave me for their adoption of the SCI. They are a flight/travel search company. So searching for flights is their main user feature.

But rather than pick Carbon / Flight Search, they chose Carbon / Booking.

Since people search for flights multiple times before they finally book. The choice of Carbon / Booking meant that they were incentivized to actually help the user make fewer flight searches rather than only incentivized to make each flight search as efficient as possible.

I've thought similarly regarding LLMs, the challenge is "how do you decide an LLM session is complete?". One idea is to model it on the Google Analytics definition of a session on a webpage, they say after 30 mins of inactivity the session is complete (ref)

I have several very long running LLM sessions, it's useful as each chat holds a memory, but anecdotally I can say that 30 mins of inactivity means that I've finished asking that chat.. for now.

So perhaps "Carbon / Session" where Session is any continuous chat conversation until there is 30 mins of inactivity.

[jawache]

WG:

• @navveenb proposed perhaps this is at the UI/UX level, each chat session shows the carbon emissions of that chat.

[kluskaj]

Maybe to add to this discussion, there are functionnal units defined in the AFNOR general framework on Frugal AI:

1. AI system Functional Unit: "Providing the system for one year for x queries". This one-year period must correspond to
   a period of stabilized inference (constant inference or known growth). ":
   𝑎𝑥 + 𝑏 + 𝑐𝑦
   a = environmental cost of an inference
   x = number of queries over one year
   b = environmental cost of training and other fixed costs
   c = environmental cost of retraining
   y = number of retrainings over one year (may be zero)

It is interesting here to note that retraining is counted separately of initial trainings.

2. AI Service FU: "Providing the service for one year to all users".
   This FU is based on the previous AI System FU, and accounts for additional servers (e.g. for web hosting),
   network transfers, and end-user devices required to provide the service, including both AI and non-AI
   components.

[jawache]

Thanks @kluskaj, do you have a reference to this at all? It looks very similar to the Green AI Index functional units.

[kluskaj]

Sure @jawache, it is the AFNOR Spec 2314 you can find the spec here:
https://www.boutique.afnor.org/en-gb/standard/afnor-spec-2314//fa208976/421140