Representing objects and large arrays in the metadata

[pannalukacs]

Hi,

I'm working on the description of COMPSs workflows using the Provenance Run Crate profile. Based on the CWL parameter mapping examples, each method parameter is typically represented by a FormalParameter (for the parameter definition) and a corresponding PropertyValue (for the actual value).

I have two related questions:

1. How should complex Python objects be represented as PropertyValue instances?
   For example, if a parameter is a custom class instance or a nested data structure, is there a recommended way to represent this in the metadata?

2. How should very large arrays be handled in the metadata?
   Adding the entire array would probably be impractical, so should we include only a portion of it or do you recommend any other approach?
   Also, it would be interesting (though I’m not sure what is the standard way to do that) to include the array’s shape or size somehow too.

Thanks in advance!

[simleo]

There is no recommended way to represent an arbitrary Python (or other language) object. The profiles are agnostic with respect to the internals of the computational tools used for the execution, and parameters are the means through which tools and workflows communicate with the external environment. If two tasks exchange arbitrary objects directly in some specific programming language, they are part of the same process / tool from an external perspective. Otherwise, the objects should be serialized in some way, in which case they could be stored to files and represented as file parameters in the WRROC metadata. Very large arrays could also be serialized and passed as files, while array shape and size would probably need custom terms. It may be that the more general COMPSs model does not really fit in the current specification, though this use case never came up before.

[pannalukacs]

Thank you for the answer!
In COMPSs, our goal is to make the metadata more human-readable and user-friendly. So, we’ve decided to include all extra information (including array shape) in the description field of the PropertyValue, as also suggested by Schema.org.