Fix raw OSM build and release process

[brynpickering]

Fixes issues found in a soft fork project.

Result of the two changes can be found in this zenodo record: https://zenodo.org/records/18161054

Changes proposed in this Pull Request

• Delete column filtering when preparing release. It was previously filtering out columns it would later want access to, after column renaming.
• Use invariant names for virtual buses. This means that when the OSM data updates, the virtual bus ID does not change (previously, the integer naming convention was based on the order of processing the raw data rows, which was not deterministic)

Checklist

• I tested my contribution locally and it works as intended.
• Code and workflow changes are sufficiently documented.
• Changed dependencies are added to pixi.toml (using pixi add <dependency-name>).
• Changes in configuration options are added in config/config.default.yaml.
• Changes in configuration options are documented in doc/configtables/*.csv.
• For new data sources or versions, these instructions <https://pypsa-eur.readthedocs.io/en/latest/data_sources.html>_ have been followed.
• A release note doc/release_notes.rst is added.

[bobbyxng]

Thanks @brynpickering for the PR and suggestion, I like the idea very much to make the virtual buses more persistent/invariant, at the time of implementation, numbering was the easier way to go. Using endpoints is also a good idea, I made a few comments, see the individual conversations. Here are some summarised thoughts:

• Suggestion to implement some basic sorting (ensuring that bus0 stays 0 and bus1 stays 1 and does not "flip" at some point)
• Suggestion to clean up the names (needs to be addressed in other parts of the workflow/script), so namings like this "DE_way/25046847-2-110_0-380" do not happen (110 and 380 are "contradicting", ideally only one of the two suffixes is needed --> also unique).
• Open question: A virtual bus always connects two lines. Both endpoints go into the pool of virtual bus candidates. Before, there was a simple method to drop duplicates based on identical geometry and voltage (and to prioritise "real" substations over virtuals in the naming). I think this was fine before, as there was no "ambition" of having persistent virtual bus names. Now, if we do this correctly, we need to make sure, that we have an invariant/deterministic way of determining the "name-giving" endpoint ("entering" or "exiting" line). One simple idea could be to take the longer of the two lines (similar to taking the largest substation, if they are merged together).

[brynpickering]

Thanks for the review @bobbyxng. I have been working with a fork that still uses pypsa <v1 hence the difference with the bug you've seen. PyPSA-Eur doesn't yet require pypsa v1 (the config sets >=0.35.0) so I guess a fix needs to be cross-compatible.

[brynpickering]

Open question: A virtual bus always connects two lines.

Except at stubs, where the virtual bus is the end of one line and that is all. But for the other cases you mention, I see what you mean.

One simple idea could be to take the longer of the two lines (similar to taking the largest substation, if they are merged together)

Seems reasonable. As with other concerns around the underlying data changing, I think it's fine if this isn't strictly persistent if the OSM data changes and the longest line changes.

[bobbyxng]

The updates look good! Almost ready to approve, there are some merge conflicts, that I cannot resolve due to missing rights to your fork. Thanks @brynpickering for improving the OSM grid building process!

[bobbyxng]

@brynpickering: @lkstrp fixed some bugs in the upstream wrt. to workflow testing (#1992). Could you please merge it into your branch/PR? Only when the checks are successful, I can approve.

Thanks!

Edit:// General comment: Missing push rights is due to this fork being managed by OET Github user group (which I am not part of). I don't know if there's a better way to handle this in the future, otherwise everything just takes longer and creates a lot of friction @fneum

[fneum]

Normally, maintainers of the repositories for which PR is made gain push rights on the branch of the PR, but it might be blocked from OET-org side.

[fneum]

Failing CI due to #1930, not this PR's fault.

[bobbyxng]

looks good, thanks @brynpickering!

[brynpickering]

Thanks @bobbyxng ! Any chance you could publish a new zenodo version to the existing record, based on this update (and updating the OSM content more generally)?

[bobbyxng]

Thanks @bobbyxng ! Any chance you could publish a new zenodo version to the existing record, based on this update (and updating the OSM content more generally)?

Yes, just did not have the time yet :) I wanted to make another PR on improving the workflow (comparison of individual builds) and update the zenodo record in this process. Will prioritise this next! @brynpickering