`podio-dump` output of MCParticle table can be improved

[dudarboh]

The current output of the MCParticles table from podio-dump -d is extremely hard to read compared to the formatting from dumpevent.

Here are things that can be improved:

• header alignment with the columns content
• column values formatting
• simulatorStatus should print readable simulation flags [sbvtcls] instead of the integer encoding
• id columns, and ids in "parents" and "daughters" columns should not contain an extra "a1cba250|+" string identical for all entries.
• parents and daughters should be part of the table as separate columns rather than creating independent new lines.
  Side to side comparison:

Output from podio-dump -d

MCParticles
          id:         PDG:generatorStatus:simulatorStatus:      charge:        time:        mass:                   vertex [ x, y, z]:                 endpoint [ x, y, z]:                 momentum [ x, y, z]:       momentumAtEndpoint [ x, y, z]:    helicity:
a1cba250|+0          +11           +3           +0 -1.000000e+00 +0.000000e+00 +0.000000e+00 +0.000000e+00 +0.000000e+00 +0.000000e+00  +0.000000e+00 +0.000000e+00 +0.000000e+00  +0.000000e+00 +0.000000e+00 +4.550000e+01  +0.000000e+00 +0.000000e+00 +0.000000e+00            +0 
      parents : 
      daughters : a1cba250|+2 a1cba250|+3 
a1cba250|+1          -11           +3           +0 +1.000000e+00 +0.000000e+00 +0.000000e+00 +0.000000e+00 +0.000000e+00 +0.000000e+00  +0.000000e+00 +0.000000e+00 +0.000000e+00  +0.000000e+00 +0.000000e+00 -4.550000e+01  +0.000000e+00 +0.000000e+00 +0.000000e+00            +0 
      parents : 
      daughters : a1cba250|+2 a1cba250|+3 
a1cba250|+2           -3           +2           +0 +3.333333e-01 +0.000000e+00 +6.743496e-07 +0.000000e+00 +0.000000e+00 +0.000000e+00  +0.000000e+00 +0.000000e+00 +0.000000e+00  +2.625073e+01 +3.538229e+01 -1.136850e+01  +0.000000e+00 +0.000000e+00 +0.000000e+00            +0 
      parents : a1cba250|+0 a1cba250|+1 
      daughters : a1cba250|+4 
a1cba250|+3           +3           +2           +0 -3.333333e-01 +0.000000e+00 +6.743496e-07 +0.000000e+00 +0.000000e+00 +0.000000e+00  +0.000000e+00 +0.000000e+00 +0.000000e+00  -2.625073e+01 -3.538229e+01 +1.136850e+01  +0.000000e+00 +0.000000e+00 +0.000000e+00            +0 
      parents : a1cba250|+0 a1cba250|+1 
      daughters : a1cba250|+4 
a1cba250|+4          +94           +2           +0 +0.000000e+00 +0.000000e+00 +9.100000e+01 +0.000000e+00 +0.000000e+00 +0.000000e+00  +0.000000e+00 +0.000000e+00 +0.000000e+00  +0.000000e+00 +0.000000e+00 +0.000000e+00  +0.000000e+00 +0.000000e+00 +0.000000e+00            +0 
      parents : a1cba250|+2 a1cba250|+3 
      daughters : a1cba250|+5 a1cba250|+6 

// to highlight a simStatus difference

a1cba250|+24         +225           +2   +134217728 +0.000000e+00 +0.000000e+00 +1.265306e+00 +0.000000e+00 +0.000000e+00 +0.000000e+00  +9.753921e-309 +1.414426e-308 -2.289286e-309  +4.309140e+00 +6.248727e+00 -1.011373e+00  +4.309140e+00 +6.248727e+00 -1.011373e+00            +0 
      parents : a1cba250|+18 
      daughters : a1cba250|+42 a1cba250|+43 

Output from dumpevent:

--------------- print out of MCParticle collection --------------- 

  flag:  0x0
  simulator status bits: [sbvtcls]  s: created in simulation b: backscatter v: vertex is not endpoint of parent t: decayed in tracker c: decayed in calorimeter l: has left detector s: stopped o: overlay


[   id   ]index|      PDG |    px,     py,        pz    | px_ep,   py_ep , pz_ep      | energy  |gen|[simstat ]| vertex x,     y   ,   z     | endpoint x,    y  ,   z     |    mass |  charge |            spin             | colorflow | [parents] - [daughters]

[00012557]    0|        11| 0.00e+00, 0.00e+00, 4.55e+01| 0.00e+00, 0.00e+00, 0.00e+00| 4.55e+01| 3 |[        ]| 0.00e+00, 0.00e+00, 0.00e+00| 0.00e+00, 0.00e+00, 0.00e+00| 0.00e+00|-1.00e+00| 0.00e+00, 0.00e+00, 0.00e+00|  (0, 0)   | [] - [2,3] 
[00012558]    1|       -11| 0.00e+00, 0.00e+00,-4.55e+01| 0.00e+00, 0.00e+00, 0.00e+00| 4.55e+01| 3 |[        ]| 0.00e+00, 0.00e+00, 0.00e+00| 0.00e+00, 0.00e+00, 0.00e+00| 0.00e+00| 1.00e+00| 0.00e+00, 0.00e+00, 0.00e+00|  (0, 0)   | [] - [2,3] 
[00012559]    2|        -1| 2.97e+01,-3.11e+01, 1.49e+01| 0.00e+00, 0.00e+00, 0.00e+00| 4.55e+01| 2 |[        ]| 0.00e+00, 0.00e+00, 0.00e+00| 0.00e+00, 0.00e+00, 0.00e+00| 0.00e+00| 3.33e-01| 0.00e+00, 0.00e+00, 0.00e+00|  (0, 1)   | [0,1] - [4] 
[00012560]    3|         1|-2.97e+01, 3.11e+01,-1.49e+01| 0.00e+00, 0.00e+00, 0.00e+00| 4.55e+01| 2 |[        ]| 0.00e+00, 0.00e+00, 0.00e+00| 0.00e+00, 0.00e+00, 0.00e+00| 0.00e+00|-3.33e-01| 0.00e+00, 0.00e+00, 0.00e+00|  (1, 0)   | [0,1] - [4] 
[00012561]    4|        94| 0.00e+00, 0.00e+00, 0.00e+00| 0.00e+00, 0.00e+00, 0.00e+00| 9.10e+01| 2 |[        ]| 0.00e+00, 0.00e+00, 0.00e+00| 0.00e+00, 0.00e+00, 0.00e+00| 9.10e+01| 0.00e+00| 0.00e+00, 0.00e+00, 0.00e+00|  (0, 0)   | [2,3] - [5,6] 
[00012562]    5|        -1| 2.89e+01,-3.02e+01, 1.45e+01| 0.00e+00, 0.00e+00, 0.00e+00| 4.57e+01| 2 |[        ]| 0.00e+00, 0.00e+00, 0.00e+00| 0.00e+00, 0.00e+00, 0.00e+00| 1.16e+01| 3.33e-01| 0.00e+00, 0.00e+00, 0.00e+00|  (0, 500)   | [4] - [7,8] 

// to highlight a simStatus difference

[00012635]   78|        11| 1.83e-03,-1.70e-03,-1.40e-03|-0.00e+00, 0.00e+00,-0.00e+00| 2.91e-03| 0 |[s vt  s ]| 4.71e+02,-6.27e+02,-2.30e+03| 4.71e+02,-6.28e+02,-2.30e+03| 5.11e-04|-1.00e+00| 0.00e+00, 0.00e+00, 0.00e+00|  (0, 0)   | [72] - [] 

[tmadlener]

There are a few things to consider here, most importantly dumpevent is specific to LCIO (and the types it defines), whereas podio-dump has to work for all EDMs that are generated via podio. podio also generates the operator<< for all of the generated classes (in contrast to LCIO where these are handwritten):

podio/python/templates/macros/implementations.jinja2

Lines 195 to 231 in c91eb9c

	{% macro ostream_operator(type, members, single_relations, multi_relations, get_syntax) %}
	std::ostream& operator<<(std::ostream& o, const {{ type }}& value) {
	if (!value.isAvailable()) {
	return o << "[not available]";
	}
	o << " id: " << value.id() << '\n';
	{% for member in members %}
	{% if member.is_array %}
	o << " {{ member.name }} : ";
	for (size_t i = 0; i < {{ member.array_size }}; ++i) {
	o << value.{{ member.getter_name(get_syntax) }}()[i] << "|";
	}
	o << '\n';
	{% else %}
	o << " {{ member.name }} : " << value.{{ member.getter_name(get_syntax) }}() << '\n';
	{% endif %}
	{% endfor %}
	{% for relation in single_relations %}
	o << " {{ relation.name }} : " << value.{{ relation.getter_name(get_syntax) }}().id() << '\n';
	{% endfor %}
	{% for relation in multi_relations %}
	o << " {{ relation.name }} : ";
	for (unsigned i = 0; i < value.{{ relation.name }}_size(); ++i) {
	{% if type == relation.bare_type %}
	o << value.{{ relation.getter_name(get_syntax) }}(i).id() << " ";
	{% else %}
	o << value.{{ relation.getter_name(get_syntax) }}(i) << " ";
	{% endif %}
	}
	o << '\n';
	{% endfor %}
	return o;
	}
	{%- endmacro %}

and

podio/python/templates/macros/collections.jinja2

Lines 100 to 143 in c91eb9c

	{% macro ostream_operator(class, members, single_relations, multi_relations, vector_members, get_syntax, settings) %}
	std::ostream& operator<<(std::ostream& o, const {{ class.bare_type }}Collection& v) {
	{% set col_width = 12 %}
	const auto old_flags = o.flags();
	o << "{{ 'id' | ostream_collection_header(col_width=col_width) }}:
	{%- for header in settings.header_contents -%}
	{{ header | ostream_collection_header(col_width=col_width) }}:
	{%- endfor -%}" << '\n';
	for (const auto&& el : v) {
	o << std::scientific << std::showpos << std::setw({{ col_width }}) << el.id() << " "
	{% for member in members %}
	{% if not member.is_array %}
	<< std::setw({{ col_width }}) << el.{{ member.getter_name(get_syntax) }}() << " "
	{% endif %}
	{% endfor %}
	<< std::endl;
	{% for relation in multi_relations %}
	o << " {{ relation.name }} : ";
	for (unsigned j = 0, N = el.{{ relation.name }}_size(); j < N; ++j) {
	o << el.{{ relation.getter_name(get_syntax) }}(j).id() << " ";
	}
	o << std::endl;
	{% endfor %}
	{% for relation in single_relations %}
	o << " {{ relation.name }} : ";
	o << el.{{ relation.getter_name(get_syntax) }}().id() << std::endl;
	{% endfor %}
	{% for member in vector_members %}
	o << " {{ member.name }} : ";
	for (unsigned j = 0, N = el.{{ member.name }}_size(); j < N; ++j) {
	o << el.{{ member.getter_name(get_syntax) }}(j) << " ";
	}
	o << std::endl;
	{% endfor %}
	}
	o.flags(old_flags);
	return o;
	}

This means that it's generally hard (to currently impossible) to simply change things for only certain types, specifically:

• It's generally hard to add specific interpretations of certain fields, e.g. simulatorStatus as a bit field and associating different short codes for different bit values,
• We cannot generally strip off the extra "a1cba250|+" part from relations. It might be the case that all of the parents / daughters live in the same collection for MCParticles, but for other types this might not be the case (e.g. ReconstructedParticles will point to e.g. tracks and clusters, and clusters might point to hits in different collections),
• The latter is also the reason why the relations are separate for now, because the table will get extremely wide once the number of related object becomes even slightly large

We could probably improve the overall printing of tables and the column alignment with a somewhat modest amount of work. We have a printTable already here that we use to print the non-detailed podio-dump tables. I think this could be re-used for other purposes as well, but it would probably require to implement a format based printing for the collections and objects first.

[jmcarcell]

Having an edm4hep-dump would not be hard but then we would have to maintain two tools, and the usage of all the EDM4hep or many would shift to use edm4hep-dump instead of podio-dump. I'm not sure if that's the best option.

[tmadlener]

That is a good point as well.

I think a possible way forward with this would be to

• switch to fmt-lib (or even better directly to std::format, maybe something that's compatible with both)
• We could then keep a default formatting (e.g. in the already existing operator<<)
• We define our own podio::CollectionFormatter (maybe also a podio::ObjectFormatter) with the following capabilities
  • By default use the default formatting
  • If there is a user defined custom_formatting method (with a defined signature) we can use ADL and the detection idiom in the custom formatter(s) that we define and call that iff it's present

A Gemini generated solution seems to work on a basic level. There might be a few details still to be hashed out from this implementation in one TU to one that works across library boundaries, but I think in general this approach could work.

With this it would then be possible for e.g. EDM4hep to override the formatting for MCParticleCollection in the EDM4hep source code and that would then be the one that is called when running podio-dump. All the types that do not have a custom formatting defined would fall back to the one that we define in any case from the generated code.