Finetune Environment Configuration Failed

[UsanoCoCr]

I followed the README instructions to set up the environment on a fresh machine (without uv cache), but encountered an error when running scripts/train/psi0/finetune-simple-psi0.sh:

Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 80/80 [00:00<00:00, 16376[90/1840]
[rank1]: Traceback (most recent call last):
[rank1]: File "/data3/zixi/Psi0/scripts/train.py", line 364, in
[rank1]: train(config) # type: ignore
[rank1]: File "/data3/zixi/Psi0/scripts/train.py", line 191, in train
[rank1]: train_dataset, val_dataset = trainer.create_datasets()
[rank1]: File "/data3/zixi/Psi0/src/psi/trainers/finetune.py", line 226, in create_datasets
[rank1]: self.train_dataset = self.data_cfg(split="train", transform_kwargs=transform_kwargs)
[rank1]: File "/data3/zixi/Psi0/src/psi/config/data_lerobot.py", line 45, in call
[rank1]: train_dataset = LeRobotDatasetWrapper(self, split=split)
[rank1]: File "/data3/zixi/Psi0/src/psi/data/lerobot/lerobot_ext.py", line 39, in init
[rank1]: self.base_dataset = lerobot_dataset_class(
[rank1]: File "/data3/zixi/Psi0/.venv-psi/lib/python3.10/site-packages/lerobot/datasets/lerobot_dataset.py", line 480, in init
[rank1]: self.hf_dataset = self.load_hf_dataset()
[rank1]: File "/data3/zixi/Psi0/.venv-psi/lib/python3.10/site-packages/lerobot/datasets/lerobot_dataset.py", line 601, in load_hf_dataset
[rank1]: hf_dataset = load_dataset("parquet", data_dir=path, split="train")
[rank1]: File "/data3/zixi/Psi0/.venv-psi/lib/python3.10/site-packages/datasets/load.py", line 2062, in load_dataset
[rank1]: builder_instance = load_dataset_builder(
[rank1]: File "/data3/zixi/Psi0/.venv-psi/lib/python3.10/site-packages/datasets/load.py", line 1819, in load_dataset_builder
[rank1]: builder_instance: DatasetBuilder = builder_cls(
[rank1]: File "/data3/zixi/Psi0/.venv-psi/lib/python3.10/site-packages/datasets/builder.py", line 395, in init
[rank1]: self.info = DatasetInfo.from_directory(self._cache_dir)
[rank1]: File "/data3/zixi/Psi0/.venv-psi/lib/python3.10/site-packages/datasets/info.py", line 279, in from_directory
[rank1]: return cls.from_dict(dataset_info_dict)
[rank1]: File "/data3/zixi/Psi0/.venv-psi/lib/python3.10/site-packages/datasets/info.py", line 284, in from_dict
[rank1]: return cls(**{k: v for k, v in dataset_info_dict.items() if k in field_names})
[rank1]: File "", line 20, in init
[rank1]: File "/data3/zixi/Psi0/.venv-psi/lib/python3.10/site-packages/datasets/info.py", line 170, in post_init
[rank1]: self.features = Features.from_dict(self.features)
[rank1]: File "/data3/zixi/Psi0/.venv-psi/lib/python3.10/site-packages/datasets/features/features.py", line 1888, in from_dict
[rank1]: obj = generate_from_dict(dic)
[rank1]: File "/data3/zixi/Psi0/.venv-psi/lib/python3.10/site-packages/datasets/features/features.py", line 1468, in generate_from_dict
[rank1]: return {key: generate_from_dict(value) for key, value in obj.items()}
[rank1]: File "/data3/zixi/Psi0/.venv-psi/lib/python3.10/site-packages/datasets/features/features.py", line 1468, in
[rank1]: return {key: generate_from_dict(value) for key, value in obj.items()}
[rank1]: File "/data3/zixi/Psi0/.venv-psi/lib/python3.10/site-packages/datasets/features/features.py", line 1474, in generate_from_dict
[rank1]: raise ValueError(f"Feature type '{_type}' not found. Available feature types: {list(_FEATURE_TYPES.keys())}")
[rank1]: ValueError: Feature type 'List' not found. Available feature types: ['Value', 'ClassLabel', 'Translation', 'TranslationVariableLanguages', 'LargeList', 'Sequence', 'Array2D', 'Array3D', 'Array4D', 'Array5D', 'Audio', 'Image', 'Video', 'Pdf', 'VideoFrame']

This error seems to be related to the datasets version not being 4.0.0. However, when I upgrade to datasets==4.0.0, the lerobot library becomes incompatible. It appears that the lerobot version and data format used by Psi0 do not support directly upgrading datasets to 4.0.0.
Is there any way to avoid this error? Or has anyone encountered a similar issue during reproduction? Looking forward to your responses.

[songlin]

Hi, Sorry for the inconvenience. This is a well-known version compatibility issue between LeRobot and datasets. Unforturnately, the LeRobot version we use does not work with datasets==4.0.0.

So, $\Psi_0$ worked around this issue by fixing the lerobot version to lerobot @ git+https://github.com/songlin/lerobot.git@09929d8057b044b53aecaf5c6d7eb71f99e8beb9 and the datasets==3.6.0.

Or, you can try to directly apply a patch to the LeRobot.

# HOT-Fix for lerobot
cp src/lerobot_patch/common/datasets/lerobot_dataset.py \
  .venv-psi/lib/python3.10/site-packages/lerobot/common/datasets/lerobot_dataset.py

Best,
Songlin

[mileret]

Hi @songlin,
This is a really cool project, thanks for open-sourcing it!

I also follow the instructions in readme and encountered the same issue.

I found that I still cannot fix this issue although my lerobot version is lerobot @ git+https://github.com/songlin/lerobot.git@09929d8057b044b53aecaf5c6d7eb71f99e8beb9 and the datasets==3.6.0

I also found that the path .venv-psi/lib/python3.10/site-packages/lerobot/common is not exist, so i can't just apply the patch as you mentioned.

I'd love to know if there's a workaround for this. Any help or pointers would be much appreciated!

[Ausbxuse]

Hi @mileret, could you please provide the full commands you ran for setting up the env? Could you also try reinstalling the env following the first 5 steps from examples/quick_start.md, and start training again?

[UsanoCoCr]

Hi @songlin @Ausbxuse

All my commands are:
uv venv .venv-psi --python 3.10 source .venv-psi/bin/activate GIT_LFS_SKIP_SMUDGE=1 uv sync --all-groups --index-strategy unsafe-best-match --active

(I downloaded flash-attn following pyproject.toml without commenting out the flash-attn)

python -c "import psi;print(psi.__version__)" -> 0.0.0 python -c "from psi.data.lerobot.compat import LEROBOT_LAYOUT; print(LEROBOT_LAYOUT)" -> datasets

After downloading data and pretrained weights from Hugging Face, I ran:
scripts/train/psi0/finetune-real-psi0.sh $task, where I modified the paths for data and model.
I have repeatedly deleted the environment, cleared caches, and reinstalled, but still get the same error "ValueError: Feature type 'List' not found" after "Resolving data files". I am not sure if you can reproduce my error with the same installation steps? Or is there something wrong with mine? I look forward to your guidance.

[Ausbxuse]

I wasn't able to reproduce the error. Could you try following examples/quick_start.md up to step 6?

[mileret]

Hi @Ausbxuse
Thanks for you reply.

I tried instructions in examples/quick_start.md, but when I came to the second instruction git submodule update --init --recursive, it returns:

ERROR: Repository not found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:songlin/SIMPLE.git' into submodule path '/home/milleret/humanoid_vla/Psi0/third_party/SIMPLE' failed

It seems like this repo doesn't exist?

Besides, the full commands I ran for setting up the env are:

git clone git@github.com:physical-superintelligence-lab/Psi0.git 
cd Psi0

uv venv .venv-psi --python 3.10
source .venv-psi/bin/activate
GIT_LFS_SKIP_SMUDGE=1 uv sync --all-groups --index-strategy unsafe-best-match --active
uv pip install flash_attn==2.7.4.post1 --no-build-isolation


cp .env.sample .env
source .env
echo $PSI_HOME


export task=Pick_bottle_and_turn_and_pour_into_cup

hf download USC-PSI-Lab/psi-data \
  real/$task.zip \
  --local-dir=$PSI_HOME/data \
  --repo-type=dataset

unzip $PSI_HOME/data/real/$task.zip -d $PSI_HOME/data/real

CUDA_VISIBLE_DEVICES=0 scripts/train/psi0/finetune-real-psi0.sh $task

I also changed the paths and api keys in .env and in finetune-real-psi0.sh, and downloaded the necessary datas and model checkpoints from huggingface.

After all these steps, I still got the same issue as @UsanoCoCr mentioned.

Looking forward to your reply!

[UsanoCoCr]

I suspect this might be an issue with my remote server, so I set up the environment again on a fresh Linux Ubuntu 24.04 machine following the instructions in examples/quick_start.md. Since you haven't uploaded SIMPLE yet, I skipped steps 2 and 3 and removed the SIMPLE-related parts from flake.nix. All other steps were exactly the same as in examples/quick_start.md. However, I ended up with the same error:

ValueError: Feature type 'List' not found. Available feature types: ['Value', 'ClassLabel', 'Translation', 'TranslationVariableLanguages', 'LargeList', 'Sequence', 'Array2D', 'Array3D', 'Array4D', 'Array5D', 'Audio', 'Image', 'Video', 'Pdf', 'VideoFrame']

If you have successfully set it up, could you provide information about your machine setup or more specific guidance? I really like this work and hope to reproduce it successfully, but I'm currently encountering some issues. Looking forward to your response!