[ntuple] Change entry handling in RNTupleProcessor to properly handle missing values

[enirolf]

This PR adds a significant change to the way field values can be accessed from an RNTupleProcessor, and, under the hood, how the RNTupleProcessor handles the entries that manage these values. The motivation behind this change is that the current processor interface has two main shortcomings:

1. It is too flexible, allowing users to call GetPtr on the entry inside the event loop which is generally considered a bad practice.
2. It cannot properly handle values missing from entries as the result of an incomplete join or missing fields in subsequent chains.

To address these shortcomings, we introduce two new classes, the RNTupleProcessorEntry and the RNTupleProcessorOptionalPtr, which are described in more detail below. With the introduction of these classes, the use of the RNTupleModel is removed from the public interface entirely.

The changes in this PR are also in preparation of the use of the RNTupleProcessor as an RDataFrame data source. A minimal working example has been developed to validate that this new RNTupleProcessor interface is compatible with the abstract interface of RDataSource.

The RNTupleProcessorEntry

The RNTupleProcessorEntry is an internal class that largely mirrors the REntry interface, but with additional functionality for the RNTupleProcessor. Most notably, fields and their values present in the entry have a notion of validity. Only when a field is valid, its value can be read.

In addition, it keeps track of which fields are auxiliary if a join operation is involved. This is needed in order to properly resolve from which underlying RNTuple the field data should be read. Since the RNTupleProcessor is composable, this is also relevant for the chain-based processor.

The RNTupleProcessorOptionalPtr

The RNTupleProcessorOptionalPtr is what the user or upstream application will actually iteract with to read field values from a processor entry. It is obtained when registering a field in the processor through RegisterField(), which has to be done before and processor iteration starts (because, as stated before, afterwards the entry will be frozen).
The RNTupleProcessorOptionalPtr can be used to get a const reference or a pointer to the field's value. If the field is marked invalid in the entry at the time of access, the const reference access method will throw an exception, whereas the pointer access method returns a nullptr.

Code example

Old interface

auto model = RNTupleModel::Create();
auto fldX = model->MakeField<float>("x");

auto proc =
  RNTupleProcessor::CreateChain({{"ntuple", "ntuple1.root"}, {"ntuple", "ntuple1.root"}}, std::move(model));

for (const auto &entry : *proc) {
  // Either (no way to guarantee the validity of `fldX` in this entry):
  std::cout << "x = " << *fldX << std::endl;
  // Or, same issue as above, plus expensive:
  std::cout << "x = " << entry.GetPtr<float>("x") << std::endl;
}

New interface

auto proc =
  RNTupleProcessor::CreateChain({{"ntuple", "ntuple1.root"}, {"ntuple", "ntuple1.root"}});

// Returns RNTupleProcessorOptionalPtr
auto x = proc->RequestField<float>("x");

// Instead of the full entry, now only provide the entry number as iterator value
for (const auto &idx : *proc) {
  // Without this check, `*x` will throw for entries where it is invalid.
  if (x.HasValue())
    std::cout << "x = " << *x << std::endl;
}

[github-actions]

Test Results

    22 files      22 suites   3d 18h 48m 58s ⏱️
 3 701 tests  3 700 ✅ 0 💤 1 ❌
79 468 runs  79 466 ✅ 0 💤 2 ❌

For more details on these failures, see this check.

Results for commit 0520c5f.

♻️ This comment has been updated with latest results.

[hahnjo]

First code review - I have two overall design questions that I will post separately so their inline comments are not mixed with these.

[hahnjo]

My two questions:

1. Do we (want to) support conversion and schema evolution when reading fields via the RNTupleProcessor? (triggered by #19111 (comment))
2. At the moment, every RNTupleProcessor has its own RNTupleProcessorEntry that are "connected together" via shared pointers. However, the field validity is not shared which could lead to problems with nested joins where the outer one is valid, but the inner one does not find its value (see also https://github.com/root-project/root/pull/19111/files#r2194283077).

[pcanal]

My two questions:

1. Do we (want to) support conversion and schema evolution when reading fields via the `RNTupleProcessor`? (triggered by [#19111 (comment)](https://github.com/root-project/root/pull/19111#discussion_r2194261277))

In my opinion we have no choice but to (eventually) support schema evolution (otherwise the RNTupleProcessor would be unusable to read any non-current files).

[enirolf]

Thank you for you extensive review, @hahnjo! To answer your questions:

1. Do we (want to) support conversion and schema evolution when reading fields via the RNTupleProcessor? (triggered by #19111 (comment))

Yes, to echo what Philippe wrote, we have no choice (not just for schema evolution, but also to allow alternative field types through e.g., RDF). I admit that I didn't yet take this into account in this particular PR, but I also think what with the proposed interface change there's no blocker not to have this. I will address it and depending on how much it would blow up this PR even more, I'll either add it here or in a follow-up PR.

2. At the moment, every RNTupleProcessor has its own RNTupleProcessorEntry that are "connected together" via shared pointers. However, the field validity is not shared which could lead to problems with nested joins where the outer one is valid, but the inner one does not find its value (see also https://github.com/root-project/root/pull/19111/files#r2194283077).

That's a good point, and something that needs to indeed also be addressed more properly. For this one as well, I will investigate more and either add it to this PR or create a follow-up.

[vepadulano]

The work is already at a really advanced state and the direction is very positive, thanks! A few minor suggestions/considerations. Furthermore, I don't see the new API to check for missing values e.g. FieldIsValid` being used in the tests, would be good to have at least one simple test already in this PR. A more thorough testing campaign could be added later.

EDIT: Sorry, I was looking only for FieldIsValid, I see there are uses of HasValue in the tests 👍

[jblomer]

I think the tutorials look much cleaner with this change!

[vepadulano]

Thank you! This is already very mature and close to being ready. A few comments from my side before the final push.

[vepadulano]

Great work, thank you! Please try to simplify the commit history where it makes sense before merging.

[hahnjo]

My last round of comments has been addressed 😃

[jblomer]

Very nice!