Handles : several issues of these user facing objects

[sponce]

One of the main problems LHCb users have been facing when using DD4hep is the concept of Handles and its oddities compared to 'standard' C++ objects. Just to cite a few :

• passing handles as reference is a recipe for disaster while it's the recommended way for all other C++ objects. You should know you have to copy them.
• handles should not have members, although the code should be there, kind of breaks the object orientation concept
• handles are not inheriting from each other, even though the underlying objects do

This last point is the most problematic of all, and creates a lot of troubles. in particular it breaks the whole idea of virtuality at the base of many C++ constructs. Think of a simple vector<Handles>, hosting obviously Handles on different types all inheriting from the same base. It just does not work !

This also has another bad consequence when implementing the code of a pair MyObject/MyElement=Handle<MyObject> : you basically need to have all the code in MyObject and then replicate all signatures in the Handle. I known the theory is that the code should be in the Handle, but that actually quickly fails as soon as some functions are calling each other, since the object has no possibility to call a function on the Handle. Net result is that all our Handles are pure facade, not really needed and only creating troubles.

[MarkusFrankATcernch]

Handles are an intrinsic ingredient to DD4hep meant to support
various views onto relatively simple structures.
DD4hep internally prefers to expose data structures which are
as simple as possible while at the same time being as complicated as necessary.
These objects are then manipulated by Handles, smart pointer objects
which sometimes have specialized functionality depending on the contained data type.
It is seen as a clear advantage that these smart pointer objects do not impose
any restrictions on the underlying object except the accessibility
of the contained data.

The lifetime of DD4hep objects is clearly defined by the lifetime
of the dd4hep::Detector object (which in turn defines the lifetime of
ROOT's TGeoManager). Conditions are managed by the ConditionsManager
object under the steering of the embedding framework. There is also no memory model
imposed which would possibly lead to performance penalties.
Hence, handles can be applied to pointers to any data structures independent of their origin:

• allocated from the stack
• allocated from the heap
• allocated as single objects or in bulk by array constructors.

This freedom to attach handle based facades to virtually any data item allows
for optimized data views depending on the application needs, and special
views can be constructed for reconstruction, simulation, tracking, etc. applications.

In particular when designing DD4hep the aim was to support these
three types of data views:

• Convenience Views provide higher level abstractions and internally group complex
  calculations. Such views simplify the life of the end-users.
• Optimization Views allow end-users to extend the data of the common detector
  element and store pre-computed results, which would be expensive to obtain repeatedly.
• Compatibility Views help to ensure smooth periods of software redesign. During the
  life-time of the experiment often various software constructs are for functional reasons re-
  designed and re-engineered. Compatibility Views either adapt new data designs to existing
  application code or expose new behavior based on existing data.

This design was described in the first DD4hep publication:
M Frank et al 2014 J. Phys.: Conf. Ser. 513 022010.

However, having said that, there is absolutely no restriction which would inhibit toolkit
users from extending any of the internally used DD4hep classes such as
e.g. the dd4hep::DetElement data object dd4hep::detail::DetElementObject
with their own implementation supporting further enhanced functionality
both in terms of additional data and additional member functions
provided the user specialized class inherits from the DD4hep provided base.
Obviously such enhancements do not hold for the geometry classes provided
by ROOT because here the ROOT framework internally calls its object constructors.
Hence, though DD4hep internally uses templated handles to manipulate data objects,
DD4hep allows for all freedom to extend any of the internally used objects
and a priori no restrictions are imposed by the DD4hep framework besides
the mentioned inheritance.

The design approach of DD4hep to internally use handles to manipulate objects
which is also offered to clients clearly has difficulties to support the
Liskov substitution principle. This was a conscious decision when designing DD4hep.

Frameworks which see the substitution principle for the implementation of the
simple data structures mandatory for its functionality clearly are limited and cannot
benefit from the advantages handles offer, unless

• the depending framework use consistently the operator->() provided by all
  handles to directly access the underlying object which may very well be structured
  with inheritance as described above or
• the depending framework uses DD4hep and it is provided only internally
  and provide the access of data to top level user code only with pointers/references
  to the data structures in their full object oriented glory. Such an approach would also
  leverage the problems mentioned with vectors of handles, since these would then be
  replaced by vectors of pointers as in any freely defined C++ application.

To conclude:

• Handles are an intrinsic and necessary ingredient to the internal functioning of DD4hp.
• For most client the handles greatly simplify dealing with the detector description.
• Enhancements of the existing internal data structures using user provided inheritance
  is possible at nearly all levels. Restrictions are hence minimal.

I hope this explanation clarifies the situation.
For the above reasons I have serious problems to see here any restriction imposed
by DD4hep. DD4hep has clearly broken with the object oriented approach to
describe data together with functionality in one "class".
This was a conscious decision that so far was not regretted and shall not change.
This limitation though can easily be overcome when overloading and exposing
objects held by handles as described above.

[sponce]

Although I can understand the logic of what you're saying, it feels like you've chosen the wrong language to implement your handle concept !
Indeed, you're "consciously" colliding frontally with several C++ foundations, in particular the substitution principle, as you mention it, but more widely inheritance and object orientation. This makes the end user completely disoriented as most of the main principles and good practices of C++ are not applicable, if not dangerous in DD4hep world. The only thing that can come out of such a conflict is chaos and failures. And we've nicely tested it for you in LHCb :-(

Now, independently of the conceptual problem, I'd like practical answers to practical problems, so again :

• how do I prevent seg faults do to people passing handles by reference, thinking they are well behaving objects, and following what they are told by C++ guidelines ?
• how do I work around the need for a vector, with different types of Handle there, and virtuality needed ?
• how do I prevent my whole code to be in the real object, breaking completely the handle concept as you expose it ?
  These are the key problems generated by handles, and I'm sure you have answers as you seem to have thought about it in depth.

[sponce]

And last small question : how do I reopen this issue ? It seems that commenting did not reopen it ?