Adding ESG Metrics for Energy and CO2 Consumption

[elisasimoni]

As we analyze and optimize token usage in Large Language Models (LLMs), it's important to consider their environmental impact. I propose integrating ESG (Environmental, Social, and Governance) metrics into the Litellm tool to track CO2 emissions and energy consumption (kWh) per token for each LLM call made.

This would provide users with a clearer understanding of the environmental footprint of their LLM usage, making the process of developing and interacting with LLMs more sustainable. Here's how it would work:

Key Features:

• CO2 Emissions per Token: Each API call would track the amount of CO2 emissions produced by processing the request. Token per an index of cosumption
• Energy Consumption (kWh) per Token: For each token generated, we could estimate the energy used in kWh. This would account for the computation power required to process the request.
• Real-time ESG Feedback: Litellm could provide real-time feedback on the environmental impact of each request, including the energy used and CO2 emitted for every token processed.

By embedding ESG metrics into LiteLLM, we empower users to track, analyze, and reduce the environmental footprint of LLMs. This initiative will help foster responsible AI usage while aligning with corporate sustainability goals.

Sources:

• EPA
• Stanford AI Index Report
• Google Sustainability AI Report
• Carbon Footprint of AI Models

[krrishdholakia]

how would we know this information? @elisasimoni

[elisasimoni]

A couple of existing frameworks could help provide estimates:

1. Eco2AI (https://github.com/sb-ai-lab/Eco2AI)

Tracks CPU/GPU power consumption and estimates CO₂ emissions.
Could be adapted to estimate energy per API call, considering token count and model type.

2. Energy Usage Report (Paper: https://arxiv.org/abs/1911.08354)

Proposes a standardized way to report energy use for machine learning.
Could help generate benchmarks for LLM inference energy costs.

Temporary Solution:
While waiting for more precise benchmarks, LiteLLM could introduce optional configuration parameters for models, such as:

co2_per_token

energy_kwh_per_token

Users could manually input values based on available estimates or data from specific LLM providers. Some companies might already have this information, and allowing for user-defined values would provide flexibility while working towards standardized benchmarks

[flefevre]

Hi and thanks for the great work on LiteLLM!

I am aligned with your proposition and I would like to suggest with you a feature idea focused on sustainability: providing users with estimated CO₂ and energy impact per inference, along with an optional dashboard to track cumulative usage over time.

✨ Feature Proposal: Inference-level Environmental Impact Estimation
Motivation
As usage of LLMs grows, there's increasing interest in understanding their environmental footprint. Surfacing these metrics can:

• Help users understand the hidden cost of their queries.
• Encourage more conscious, efficient usage.
• Support sustainability goals within teams or organizations.

Suggested Functionality
⚡ Per-Inference Impact Estimation
Display an estimated CO₂/energy footprint per request, based on token count and model size. For example:

Estimated impact: 0.2g CO₂eq (≈ 10 sec of YouTube streaming)

This could leverage public tools like:

• https://huggingface.co/spaces/genai-impact/ecologits-calculator
• https://impactco2.fr/

Make it optional, toggled via user or admin settings.

📊 Usage Dashboard (User-level or Global)
An optional dashboard showing:

• Total tokens generated
• Estimated cumulative energy/CO₂ use
• Real-world equivalents (e.g., lightbulb hours, video streaming time)
• Filters: daily / weekly / monthly
• (Optional) Aggregated stats for all users (with opt-in consent if needed)

No tracking unless explicitly enabled.

Works well for both individual and team-based setups.

Why It Matters
This feature would:

• Align LiteLLM with the growing push for sustainable AI.
• Be a unique offering among open-source LLM frontends.
• Promote responsible AI use by default.

Happy to discuss further or help refine the idea if there's interest!

[flefevre]