Duplicating generation assets

[ekatef]

Version Checks (indicate both or one)

• I have confirmed this bug exists on the lastest release of powerplantmatching.

• I have confirmed this bug exists on the current master branch of powerplantmatching.

Issue Description

When using powerplantmatching for some non-European countries in PyPSA-Earth, we have observed some duplication, as mentioned in this old issue. For example, such duplicating appears when running build_powerplants rule of PyPSA-Earth 0.8.0 for Colombia. After some investigation, the reason is the way GEM datasets are being treated if taken in combination with other databases.

Reproducible Example

## basic code

import powerplantmatching as pm
import yaml
config = {
    'matching_sources': 
        [
            'GEO', 'GPD',
            'GBPT', 'GGPT', 'GCPT', 'GGTPT', 'GNPT', 'GSPT', 'GWPT', 'GHPT'
        ],
    'fully_included_sources': 
        [
            'GEO', 'GPD',
            'GBPT', 'GGPT', 'GCPT', 'GGTPT', 'GNPT', 'GSPT', 'GWPT', 'GHPT'
        ], 
    'target_countries': 'Colombia',
    'main_query': '(DateOut >= 2022 or DateOut != DateOut)'
}

ppl = (
    pm.powerplants(
        from_url=False,
        update=True,
        config_update=config,
    )
    .query('Fueltype not in ["Solar", "Wind"]')
)

ppl.to_csv("test_ppm_co_minimal.csv")

Expected Behavior

A modelling-ready dataset of power plants for Colombia. Instead, a dataset is produced with multiple duplicates which require manual cleaning-up ("test_ppm_co_minimal.csv"). E.g. Termosierra power plant present in two versions: as Termosierra Ccgt Colombia (426 MWe) and Termosierra (460 MWe).

This duplication can be removed if GEM-sources datasets ('GEO', 'GPD') are excluded or if only they are kept (the output files are test_ppm_co_minimal_without_gem.csv and test_ppm_co_minimal_gem_only.csv, respectively).

Installed Versions

Details

powerplantmatching 0.7.1 pyhd8ed1ab_0 conda-forge

[ekatef]

@cdgaete tagging you here as agreed. I know that you have been investigating integration of new datasets into ppm in details and interactions with GEM could be probably of interest for you.

[fneum]

The problem is if the matching fails, but the power plant is included in multiple datasets listed under fully_included_sources. Then, multiple unmatched versions of the same power plant may be included. Generally, the recommendation is to lower the number of datasets in fully_included_sources.