cannot get ensemble inferenced successfully

[breadbread1984]

the pipeline generated from tensorflow trained model cannot be inferenced successfully.

• merline docker: nvcr.io/nvidia/merlin/merlin-tensorflow:23.12

• the scripts I use:

  • training and pipeline generation script
  • testing script

• how tritonserver is launched

tritonserver --model-repository=pipeline_tf --http-port=8081 --grpc-port=8082 --metrics-port=8083 --model-control-mode=poll

• the error I met：

RuntimeError: Error: <class 'TypeError'> - data type 'list' not understood,
Traceback:
  File "/root/raid/common_models/recommend_system/merlin/movielens/pipeline_tf/0_transformworkflowtriton/1/model.py", line 117, in execute
    transformed = self.runner.run_workflow(input_tensors)
  File "/usr/local/lib/python3.10/dist-packages/merlin/systems/workflow/base.py", line 103, in run_workflow
    transformed = LocalExecutor().transform(transformable, self.workflow.graph)
  File "/usr/local/lib/python3.10/dist-packages/merlin/dag/executors.py", line 102, in transform
    transformed_data = self._execute_node(node, transformable, capture_dtypes, strict)
  File "/usr/local/lib/python3.10/dist-packages/merlin/dag/executors.py", line 116, in _execute_node
    upstream_outputs = self._run_upstream_transforms(
  File "/usr/local/lib/python3.10/dist-packages/merlin/dag/executors.py", line 130, in _run_upstream_transforms
    node_output = self._execute_node(
  File "/usr/local/lib/python3.10/dist-packages/merlin/dag/executors.py", line 116, in _execute_node
    upstream_outputs = self._run_upstream_transforms(
  File "/usr/local/lib/python3.10/dist-packages/merlin/dag/executors.py", line 130, in _run_upstream_transforms
    node_output = self._execute_node(
  File "/usr/local/lib/python3.10/dist-packages/merlin/dag/executors.py", line 122, in _execute_node
    transform_output = self._run_node_transform(node, transform_input, capture_dtypes, strict)
  File "/usr/local/lib/python3.10/dist-packages/merlin/dag/executors.py", line 250, in _run_node_transform
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/merlin/dag/executors.py", line 237, in _run_node_transform
    transformed_data = node.op.transform(selection, input_data)
  File "/usr/local/lib/python3.10/dist-packages/nvtabular/ops/join_external.py", line 159, in transform
    new_df = self._merge(df, self._ext)
  File "/usr/local/lib/python3.10/dist-packages/nvtabular/ops/join_external.py", line 120, in _ext
    return convert_data(self._ext_cache, cpu=self.cpu)
  File "/usr/local/lib/python3.10/dist-packages/merlin/core/dispatch.py", line 674, in convert_data
    _x = x if isinstance(x, pd.DataFrame) else x.to_pandas()
  File "pyarrow/array.pxi", line 830, in pyarrow.lib._PandasConvertible.to_pandas
  File "pyarrow/table.pxi", line 3908, in pyarrow.lib.Table._to_pandas
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/pandas_compat.py", line 812, in table_to_blockmanager
    ext_columns_dtypes = _get_extension_dtypes(
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/pandas_compat.py", line 865, in _get_extension_dtypes
    pandas_dtype = _pandas_api.pandas_dtype(dtype)
  File "pyarrow/pandas-shim.pxi", line 146, in pyarrow.lib._PandasAPIShim.pandas_dtype
  File "pyarrow/pandas-shim.pxi", line 149, in pyarrow.lib._PandasAPIShim.pandas_dtype
  File "/usr/local/lib/python3.10/dist-packages/pandas/core/dtypes/common.py", line 1781, in pandas_dtype
    npdtype = np.dtype(dtype)