Use Cases Questions

[jimmarino]

Hi,

I have a few questions regarding various use cases:

1. Support for large data

Are there plans to add streaming support in the runtime? On the client, It looks as if the connector materializes the entire entity into memory as opposed to streaming segments during processing. My understanding is the core IDS Information Model classes assume this. Or have I misunderstood this? For example, the contents of an IDS artifact response could be a pointer to a data stream that is processed at some later point downstream? Streaming support could be really challenging, particularly in the context of policy handling, but it's a common requirement in environments that process large amounts of data.

2. Integration with other data transfer technologies

Companies may already be using existing technologies that they would like to leverage in the context of data transfer. For example, a company may be using cloud object storage for machine learning data sets. Object storage systems often have APIs that are optimized for highly parallel data transfer. Is it possible for the connecter to leverage these technologies? For example, the connector implements the trust flow and initial data request; however, the actual data transfer is completed by out-of-process infrastructure that uses those storage technologies. Note that this would be different than the connector using a "data app" that it provisions as a Linux container. These other technologies would be infrastructure that is deployed outside the ambit of the connector (possibly for security reasons).

Thanks for your help!
Jim