error message in openFE

[boranhan]

There's an error messsage in openFE when running the spaceship-titanic

WARNING:autogluon_assistant.transformer.feature_transformers.base:FeatureTransformer OpenFETransformer failed to fit.
WARNING:autogluon_assistant.transformer.feature_transformers.base:Traceback (most recent call last):
  File "/media/deephome/autogluon-assistant/src/autogluon_assistant/transformer/feature_transformers/base.py", line 29, in fit
    self._fit_dataframes(
  File "/media/deephome/autogluon-assistant/src/autogluon_assistant/transformer/feature_transformers/openfe.py", line 340, in _fit_dataframes
    self.features = AssistantOpenFE().fit(
  File "/media/deephome/autogluon-assistant/src/autogluon_assistant/transformer/feature_transformers/openfe.py", line 92, in fit
    self.process_and_save_data()
  File "/opt/conda/envs/aga/lib/python3.10/site-packages/openfe/openfe.py", line 331, in process_and_save_data
    self.data.reset_index().to_feather(self.tmp_save_path)
  File "/opt/conda/envs/aga/lib/python3.10/site-packages/pandas/core/frame.py", line 2938, in to_feather
    to_feather(self, path, **kwargs)
  File "/opt/conda/envs/aga/lib/python3.10/site-packages/pandas/io/feather_format.py", line 66, in to_feather
    feather.write_feather(df, handles.handle, **kwargs)
  File "/opt/conda/envs/aga/lib/python3.10/site-packages/pyarrow/feather.py", line 156, in write_feather
    table = Table.from_pandas(df, preserve_index=preserve_index)
  File "pyarrow/table.pxi", line 4751, in pyarrow.lib.Table.from_pandas
  File "/opt/conda/envs/aga/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 625, in dataframe_to_arrays
    arrays = [convert_column(c, f)
  File "/opt/conda/envs/aga/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 625, in <listcomp>
    arrays = [convert_column(c, f)
  File "/opt/conda/envs/aga/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 612, in convert_column
    raise e
  File "/opt/conda/envs/aga/lib/python3.10/site-packages/pyarrow/pandas_compat.py", line 606, in convert_column
    result = pa.array(col, type=type_, from_pandas=True, safe=safe)
  File "pyarrow/array.pxi", line 360, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 87, in pyarrow.lib._ndarray_to_array
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: ("Could not convert 'unknown' with type str: tried to convert to boolean", 'Conversion failed for column 36821469 with type object')

[AnirudhDagar]

Thanks, I'm able to reproduce this issue too. @canerturkmen do you recall facing such an error before during the addition of the openfe transformer? I do not have the full context on this, not sure if this is an openfe issue or can be fixed on our end?

[canerturkmen]

Interesting, I don't remember anything like this but it looks like OpenFE is getting fed something that feather/arrow (which OpenFE uses to store the data) doesn't like.

[AnirudhDagar]

Thanks for the prompt response, good to know that this was not happening before. I can checkout an older version to debug further.

[boranhan]

It is somehow working now, despite I didn't change anything. If it occurs again, I'll reopen this issue.