Add pythonisation for LinkNavigator

[dudarboh]

As the title says.
It would be nice to have this feature in python as well.
As, otherwise one needs to manually loop over link collections

import podio
podio.LinkNavigator
AttributeError: module 'podio' has no attribute 'LinkNavigator'

[jmcarcell]

There is a way:

In [1]: import ROOT

In [2]: ROOT.gInterpreter.LoadFile("podio/LinkNavigator.h")
Out[2]: 0

In [3]: from ROOT import podio

In [4]: podio.LinkNavigator
Out[4]: <cppyy.Template 'podio::LinkNavigator' object at 0x7f61c8183c50>

[dudarboh]

Thanks 👀

Just for reference putting here the full reproducible example maybe it will be useful for somebody, as it was not directly trivial for me and required also load of the LinkCollection.h

import edm4hep
import ROOT
ROOT.gInterpreter.LoadFile("podio/LinkNavigator.h")
ROOT.gInterpreter.LoadFile("podio/LinkCollection.h")


pfo = edm4hep.ReconstructedParticle()
mc = edm4hep.MCParticle()

link_col = ROOT.podio.LinkCollection[edm4hep.ReconstructedParticle, edm4hep.MCParticle]()

link = link_col.create()
link.setFrom(pfo)
link.setTo(mc)
link.setWeight(1.0)

nav = ROOT.podio.LinkNavigator[ROOT.podio.LinkCollection[edm4hep.ReconstructedParticle, edm4hep.MCParticle]]( link_col )
print(f"RecoParticle {pfo} is linked to: ")
for weight_object in nav.getLinked(pfo):
    print(f"MCParticle {weight_object.o} with weight {weight_object.weight}")

[tmadlener]

We should probably add the two necessary LoadFile bits into our existing python infrastructure. We can do it similarly to the datasource: https://github.com/AIDASoft/podio/blob/master/python/podio/data_source.py

Is the full specialization of the constructor here necessary?

nav = ROOT.podio.LinkNavigator[ROOT.podio.LinkCollection[edm4hep.ReconstructedParticle, edm4hep.MCParticle]]( link_col )

Technically everything should be deducible from the link collection as that has all the types, i.e.

nav = ROOT.podio.LinkNavigator(link_coll)

should also work, but I am not sure if cppyy is clever enough for that.

[dudarboh]

Yes, I need to specify the template arguments directly, otherwise, if I use nav = ROOT.podio.LinkNavigator(link_coll) with the same reproducible as you proposed I get:

Traceback (most recent call last):
  File "/home/foxwise/projects/experiments/key4hep-doc/sandbox/read_podio.py", line 19, in <module>
    nav = ROOT.podio.LinkNavigator( link_col )
  File "/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2025-12-21/x86_64-almalinux9-gcc14.2.0-opt/root/6.38.00-2jf5cm/lib/root/cppyy/_cpython_cppyy.py", line 158, in __call__
    return self.__getitem__(*args)
           ~~~~~~~~~~~~~~~~^^^^^^^
  File "/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2025-12-21/x86_64-almalinux9-gcc14.2.0-opt/root/6.38.00-2jf5cm/lib/root/cppyy/_cpython_cppyy.py", line 101, in __getitem__
    pyclass = _backend.MakeCppTemplateClass(*newargs)
SyntaxError: could not construct C++ name from provided template argument.

Actually there is yet another problem...
When reading the collection elements in a python loop from EDM4hep file returns Mutable objects.

reader = root_io.Reader("my_file.edm4hep.root")
for event in reader.get("events"):
    for pfo in event.get("PandoraPFOs"):
        print(pfo) # <cppyy.gbl.edm4hep.MutableReconstructedParticle object at 0x1b343b60>

And the above reproducible doesn't work with Mutable, because I cannot pass the Mutable as a template type for the LinkCollection, I get:

    link_col = ROOT.podio.LinkCollection[edm4hep.MutableReconstructedParticle, edm4hep.MCParticle]()
               ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2025-12-21/x86_64-almalinux9-gcc14.2.0-opt/root/6.38.00-2jf5cm/lib/root/cppyy/_cpython_cppyy.py", line 101, in __getitem__
    pyclass = _backend.MakeCppTemplateClass(*newargs)
TypeError: 'podio::LinkCollection<::edm4hep::MutableReconstructedParticle,::edm4hep::MCParticle>' is not a known C++ class

[tmadlener]

Alright, I see. Pity that cppyy doesn't see through that apparently (or at least not in our case).

You will have to define the LinkCollection using the immutable handle types, i.e. podio.LinkCollection[edm4hep.ReconstructedParticle, edm4hep.MCParticle]. You will otherwise run into these static_asserts:

podio/include/podio/detail/LinkCollectionImpl.h

Lines 40 to 43 in c91eb9c

	static_assert(std::is_same_v<FromT, detail::GetDefaultHandleType<FromT>>,
	"Links need to be instantiated with the default types!");
	static_assert(std::is_same_v<ToT, detail::GetDefaultHandleType<ToT>>,
	"Links need to be instantiated with the default types!");

The resulting LinkCollection and the LinkNavigator work with Mutable types regardless.

[m-fila]

Actually there is yet another problem... When reading the collection elements in a python loop from EDM4hep file returns Mutable objects.

I think we discussed it at some point, it's also inconsistent with accessing elements by index (which by pythonized to return immutable objects). It's because cppyy creates Python iterator from from non-const C++ iterator - pythonizing to use const C++ iterator seemed more complicated, because cppyy is using heuristics to identify some classes as "iterators" and apply internal pythonizations for them

[dudarboh]

@tmadlener,
well, for me it doesn't, that is why I started to play with Mutable.

Running this:

import edm4hep
import ROOT
ROOT.gInterpreter.LoadFile("podio/LinkNavigator.h")
ROOT.gInterpreter.LoadFile("podio/LinkCollection.h")

pfo = edm4hep.MutableReconstructedParticle()
mc = edm4hep.MCParticle()

link_col = ROOT.podio.LinkCollection[edm4hep.ReconstructedParticle, edm4hep.MCParticle]()

link = link_col.create()
link.setFrom(pfo)
link.setTo(mc)
link.setWeight(1.0)

nav = ROOT.podio.LinkNavigator[ROOT.podio.LinkCollection[edm4hep.ReconstructedParticle, edm4hep.MCParticle]]( link_col )
print(f"RecoParticle {pfo} is linked to: ")
for weight_object in nav.getLinked(pfo):
    print(f"MCParticle {weight_object.o} with weight {weight_object.weight}")

I get:

RecoParticle <cppyy.gbl.edm4hep.MutableReconstructedParticle object at 0xf9b8940> is linked to: 
Traceback (most recent call last):
  File "/home/foxwise/projects/experiments/key4hep-doc/sandbox/read_podio.py", line 23, in <module>
    for weight_object in nav.getLinked(pfo):
                         ~~~~~~~~~~~~~^^^^^
TypeError: Template method resolution failed:
  none of the 2 overloaded methods succeeded. Full details:
  vector<podio::detail::links::WeightedObject<edm4hep::ReconstructedParticle> > podio::LinkNavigator<podio::LinkCollection<edm4hep::ReconstructedParticle,edm4hep::MCParticle> >::getLinked(const podio::LinkNavigator<podio::LinkCollection<edm4hep::ReconstructedParticle,edm4hep::MCParticle> >::ToT& object, podio::detail::links::ReturnFromTag) =>
    TypeError: takes at least 2 arguments (1 given)
  vector<podio::detail::links::WeightedObject<edm4hep::MCParticle> > podio::LinkNavigator<podio::LinkCollection<edm4hep::ReconstructedParticle,edm4hep::MCParticle> >::getLinked(const podio::LinkNavigator<podio::LinkCollection<edm4hep::ReconstructedParticle,edm4hep::MCParticle> >::FromT& object, podio::detail::links::ReturnToTag) =>
    TypeError: takes at least 2 arguments (1 given)
  none of the 2 overloaded methods succeeded. Full details:
  vector<podio::detail::links::WeightedObject<edm4hep::ReconstructedParticle> > podio::LinkNavigator<podio::LinkCollection<edm4hep::ReconstructedParticle,edm4hep::MCParticle> >::getLinked(const podio::LinkNavigator<podio::LinkCollection<edm4hep::ReconstructedParticle,edm4hep::MCParticle> >::ToT& object) =>
    TypeError: could not convert argument 1
  vector<podio::detail::links::WeightedObject<edm4hep::MCParticle> > podio::LinkNavigator<podio::LinkCollection<edm4hep::ReconstructedParticle,edm4hep::MCParticle> >::getLinked(const podio::LinkNavigator<podio::LinkCollection<edm4hep::ReconstructedParticle,edm4hep::MCParticle> >::FromT& object) =>
    TypeError: could not convert argument 1
  vector<podio::detail::links::WeightedObject<edm4hep::MCParticle> > podio::LinkNavigator<podio::LinkCollection<edm4hep::ReconstructedParticle,edm4hep::MCParticle> >::getLinked(const podio::LinkNavigator<podio::LinkCollection<edm4hep::ReconstructedParticle,edm4hep::MCParticle> >::FromT& object) =>
    TypeError: could not convert argument 1
  Failed to instantiate "getLinked(::edm4hep::MutableReconstructedParticle*)"
  vector<podio::detail::links::WeightedObject<edm4hep::MCParticle> > podio::LinkNavigator<podio::LinkCollection<edm4hep::ReconstructedParticle,edm4hep::MCParticle> >::getLinked(const podio::LinkNavigator<podio::LinkCollection<edm4hep::ReconstructedParticle,edm4hep::MCParticle> >::FromT& object) =>
    TypeError: could not convert argument 1

[tmadlener]

Argh, then it's cppyy that apparently doesn't consider the conversion operator that we explicitly put in for going from Mutable to the non mutable version. In c++ the following compiles without issues:

auto linkColl = podio::LinkCollection<edm4hep::ReconstructedParticle, edm4hep::MCParticle>{};
auto nav = podio::LinkNavigator(linkColl);
auto reco = edm4hep::MutableReconstructedParticle{};
const auto linked = nav.getLinked(reco);

We might be able to fix the latter bit here with additional overloads in the LinkNavigator that explicitly take Mutable objects. For the instantiation of the LinkNavigator we would probably have to write some explicit python code that does some of the "heavy" lifting of instantiating the right template.