Keyerror shape on SimDB Server

[prasad-sawantdesai]

Hi,
This came up as I was testing branch `add-data-endpoint-for-simdb`` on SimDB test server. I am using same branch on SimDB client and server.

manifest file

simdb sim ingest /home/ITER/sawantp1/github/SimDB/imaspulse/manifest_1210250002.yaml
simdb sim push  7104c15f-55ee-11f1-9575-48df37d69a7

Client log

Uploading file //home/ITER/sawantp1/github/SimDB/imaspulse/master.h5    Complete
Uploading file //home/ITER/sawantp1/github/SimDB/imaspulse/core_sources.h5 Complete
Uploading file //home/ITER/sawantp1/github/SimDB/imaspulse/core_profiles.h5 Complete
Uploading file //home/ITER/sawantp1/github/SimDB/imaspulse/wall.h5      Complete
Uploading file //home/ITER/sawantp1/github/SimDB/imaspulse/pf_active.h5 Complete
Uploading file //home/ITER/sawantp1/github/SimDB/imaspulse/summary.h5   Complete
Uploading file //home/ITER/sawantp1/github/SimDB/imaspulse/nbi.h5       Complete
Uploading file //home/ITER/sawantp1/github/SimDB/imaspulse/distribution_sources.h5 Complete
Uploading file //home/ITER/sawantp1/github/SimDB/imaspulse/equilibrium.h5 Complete
Uploading simulation data ... Error: HTTP error 500 returned from endpoint http://10.154.84.240/scenarios/api/v1.2/simulations.

Server log

May 22 16:04:38 io-ls-simdb-t01.iter.org gunicorn[461910]:   File "/home/simdbt01/SimDB/src/simdb/remote/core/pydantic_utils.py", line 276, in wrapper
May 22 16:04:38 io-ls-simdb-t01.iter.org gunicorn[461910]:     validated = _validate_param(model_type, source)
May 22 16:04:38 io-ls-simdb-t01.iter.org gunicorn[461910]:                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 22 16:04:38 io-ls-simdb-t01.iter.org gunicorn[461910]:   File "/home/simdbt01/SimDB/src/simdb/remote/core/pydantic_utils.py", line 160, in _validate_param
May 22 16:04:38 io-ls-simdb-t01.iter.org gunicorn[461910]:     return model.model_validate_json(request_data)
May 22 16:04:38 io-ls-simdb-t01.iter.org gunicorn[461910]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 22 16:04:38 io-ls-simdb-t01.iter.org gunicorn[461910]:   File "/home/simdbt01/simdb_env/lib64/python3.11/site-packages/pydantic/main.py", line 782, in model_validate_json
May 22 16:04:38 io-ls-simdb-t01.iter.org gunicorn[461910]:     return cls.__pydantic_validator__.validate_json(
May 22 16:04:38 io-ls-simdb-t01.iter.org gunicorn[461910]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
May 22 16:04:38 io-ls-simdb-t01.iter.org gunicorn[461910]:   File "/home/simdbt01/SimDB/src/simdb/remote/models.py", line 137, in _deserialize_numpy
May 22 16:04:38 io-ls-simdb-t01.iter.org gunicorn[461910]:     return np.frombuffer(np_bytes, dtype=v["dtype"]).reshape(v["shape"])
May 22 16:04:38 io-ls-simdb-t01.iter.org gunicorn[461910]:                                                              ~^^^^^^^^^
May 22 16:04:38 io-ls-simdb-t01.iter.org gunicorn[461910]: KeyError: 'shape'

Currently patched this file to test further
/home/ITER/sawantp1/github/SimDB/src/simdb/remote/models.py

def _deserialize_numpy(v: Any) -> Any:
    if isinstance(v, np.ndarray):
        return v
    if isinstance(v, dict) and v.get("_type") == "numpy.ndarray":
        np_bytes = base64.b64decode(v["bytes"].encode())
        arr = np.frombuffer(np_bytes, dtype=v["dtype"])
        if "shape" in v:
            arr = arr.reshape(v["shape"])
        return arr
    raise ValueError(f"Cannot deserialize {v} to np.ndarray"