ArrayDataset converts ndarray to numpy array

[leezu]

Description

            if isinstance(data, ndarray.NDArray) and len(data.shape) == 1:
                data = data.asnumpy()

This unexpected and undocumented. Is there any reason for this or can it be deleted?

Environment info (Required)

https://github.com/apache/incubator-mxnet/blob/52116d4d3205361f861141013eb0de40be003e3c/python/mxnet/gluon/data/dataset.py#L151

[frankfliu]

@mxnet-label-bot [Question, Python]

[NRauschmayr]

Yes there is a reason for it. If the array has only one dimension, then it won't be converted to an mx.nd.array and as such it won't be copied to the GPU. You can find a similar thread here: https://discuss.mxnet.io/t/what-happen-one-of-the-data-from-the-same-iterator-is-on-the-cpu-and-the-other-is-on-the-gpu-is-this-a-bug/961

[leezu]

Thanks for the context @NRauschmayr, I now understand how that line of code got there in the first place. However, as this is undocumented, it should be considered a bug. mx.gluon.data.ArrayDataset seems to be a general purpose tool for representing data. The case of single dimensional arrays can easily arise when representing sparse data in COO format.

I don't see why ArrayDataset should contain an obscure optimization to speed up metrics when the user wrongly tries loading data to GPU? Is there not some more appropriate place than the constructor of ArrayDataset?

[ThomasDelteil]

I think the way forward would be to document the feature, the work-around is fairly simple (.reshape(-1, 1) your array).