feat(electricity): add ElectricityMaps integration (part 1, recent data)

[Shillaker]

Summary

Add integration with the Electricity Maps API to fetch latest grid carbon intensity.

The integration is only active when the ELECTRICITY_MAPS_API_KEY environment variable (or config setting) is provided. When this is true, get_electrical_impact_factor will pull data from Electricity Maps latest endpoint. When not active, it will fall back to the existing hardcoded factors.yml data.

Note: this is part 1 of the integration. As suggested by @da-ekchajzer in #117, we could also support historical impact factors if we accepted a time frame or start date in the usage request object. With this, we could then query Electricity Maps to get the historical impact data for the given period.

Issue #294 is also relevant, as through this work I've noticed that our hard-coded factors could do with being updated. We could write a script to pull the right numbers from Electricity Maps for the last year too.

Changes

• Add electricitymaps module to query the ElectricityMaps API, using the electricitymaps-sdk, caching responses for 10 minutes by default
• Add country utility to convert ISO country codes (used by BoaviztAPI) to alpha-2 codes (used by Electricity Maps)
• Add electricity_maps_api_key and electricity_maps_cache_expiry_seconds config settings
• Update get_electrical_impact_factor to use Electricity Maps data for gwp when an API key is set
• Remove old commented-out ElectricityMap prototype from factor_provider.py
• Add unit tests for the new service, country utility, config, and factor provider
• Add e2e tests that runs against the real Electricity Maps API (skipped when no key is set)
• Update config and electricity factors documentation

Testing

The hard-coded value for the electricity impact of France is 0.098 kgCO2eq/kWh, and at the time of testing (16:20 Sunday 2nd February), the value from Electricity Maps was 0.036 kgCO2eq/kWh.

I ran the demo request to the cloud instance endpoint both with and without the Electricity Maps integration to check that it changed just the GWP usage impact.

Here is the request using the existing hard-coded value, giving 204 kgCO2eq:

Now here is the same request running using my Electricity Maps API key, giving 75 kgCO2eq:

[jonperron]

Not really fan of a generic pass without getting at least a log here. Or if the value is already an iso2, maybe you can check for it first ?

[Shillaker]

This branch will get executed a lot, as the locations WOR and EEE will often get passed to this function (as they are used as defaults and examples), so I don't think we want to log every time. The ISO3 format is just three characters, and WOR and EEE also fit this format, so we can't just do a simple check like if len(usage_location) == 3. I also can't guarantee that WOR and EEE will be the only non-country codes we ever use, and so maintaining a separate list of these locations is not a particularly nice idea either (i.e. if len(usage_location) == 3 and usage_location not in ["WOR", "EEE"]).

I have added an is_iso3 function, and now call this instetad to avoid using a try-except-pass. However, the is_iso3 function does the same thing as the iso3_to_iso2 function, as pycountry does not provide any kind of checking functions (we just have to do the same lookup twice now).

[jonperron]

I understand what you mean. Do you think you will reuse these functions a lot ?

[Shillaker]

For now it will only be with the Electricity Maps integration, and we will add at least one other endpoint to get historical data. Why do you ask? Can you be more specific?

[jonperron]

If the functions were not to be reused, I would have use a walrus operator in this part of the code or do something smaller like

if config.electricity_maps_api_key and impact_type == "gwp":
    country = pycountry.countries.get(alpha_3=usage_location)
    if country is None:
        raise ValueError(f"Unknown ISO3 country code: '{usage_location}'")

    return fetch_carbon_intensity(
        config.electricity_maps_api_key,
        country.alpha_2,
    )

So it calls get only once. But if the function will be reused somewhere else, then my comment has no reason to be 👍

[Shillaker]

Yes you're right, I could have inlined the calls to pycountry rather than put them in a separate utility. I'm pretty sure it will be useful like that, but if after we do the historical data it's not, I'll inline it.

The other issue is that the Boavizta code regularly uses location codes that are not valid ISO3 codes, e.g. WOR and EEE to express global and European averages (see the EEE definition here). These are perfectly valid in the Boavizta code, but will not work with ElectricityMaps. Therefore the code snippet you've suggested would throw a ValueError regularly, as the function will be passed WOR and EEE.

It is kind of confusing, and perhaps it would be better to create a Location class and pass that around, rather than overload a single string field to represent countries and regions. This Location could then have functions like is_country, get_iso2_code etc. I did take a look at this but it's a non-trivial change.