MEDOR

Official codebase for Mesh-based dynamics with occlusion reasoning for cloth manipulation

Installation

Clone the repo

git clone --recursive git@github.com:zxhuang97/medor.git
cd medor

Configure python environment

Mamba is highly recommended for configuring the python environment. It's a drop-in replacement for conda but much faster.

mamba env create -f release.yml

After softgym is installed, you may activate the environment by

. ./prepare_release.sh

Install Softgym

Step 1: Install docker and nvidia-container. Then pull the docker image by

docker pull xingyu/softgym

Step 2: Set the path to conda directory as CONDA_PATH, then enter the docker container.

export CONDA_PATH=/home/zixuanh/miniforge3
sudo docker run \
--runtime=nvidia  \
-v ${PWD}/softgym_medor:/workspace/softgym   \
-v ${CONDA_PATH}:${CONDA_PATH} \
-v /tmp/.X11-unix:/tmp/.X11-unix  \
--gpus all   \
-e DISPLAY=$DISPLAY  \
-e QT_X11_NO_MITSHM=1   \
-it xingyu/softgym:latest bash

Step 3: Inside the docker container, run the following commands to compile the softgym.

cd softgym
export CONDA_PATH=/home/zixuanh/miniforge3
export PATH=${CONDA_PATH}/bin:$PATH
. ./prepare_1.0.sh
export PATH=/usr/local/cuda/bin/:$PATH
LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
. ./compile_1.0.sh

Pretrained models

Please download the pretrained models and put them under data/release.

Mesh reconstruction model

We provide the pretrained models for Tshirt, Trousers, Skirt, and Dress:

• Tshirt
• Trousers
• Skirt
• Dress

Mesh dynamics model

We trained a mesh dynamics model on Trousers and found it generalizes well to other categories. We used this model for planning in all experiments.

• Trousers

Dataset

Currently, we only provide dataset for Tshirt. For training and testing, you need to full dataset. If you only want to run the demo, you can download the test set alone.

• Full dataset
• Test set

Mesh Reconstruction Demo

Download the pretrained model and put it under data/release.

data
└── release
    └── medor_Tshirt
dataset
└── Tshirt_dataset_release2

make_opt_gif will generate the gifs that visualize the optimization process and each gif will take around 3-4 mins.

. ./prepare_release.sh
python garmentnets/eval_pipeline.py \
--model_path data/release/medor_Tshirt/pipeline/ \
--tt_finetune --cloth_type Tshirt --max_test_num 5 \
--exp_name release_demo 

The results can be found in data/test/release_demo.

Planning with Mesh Dynamics Model

Download the simulator cached states and put the pickle files under softgym_medor/softgym/cached_initial_states. Download pre-processed cloth3d dataset and put it under dataset/cloth3d. The entire dataset is around ~30 GB. You may only download the categories you are interested in.

Make sure that the checkpoints of pretrained reconstruction model and dynamics model are downloaded and put under data/release.

python plan_gnn.py --cloth_type Tshirt --exp_name medor_Tshirt --num_trials 40

Results can be found in data/plan/medor_Tshirt.

Training

Train the canonicalization Networks

 python garmentnets/train_pointnet2.py \
  --exp_name tshirt_canon \
  --log_dir data/release/Tshirt_release  \
  --ds Tshirt_dataset_release2 \
  --cloth_type Tshirt

Train the mesh reconstruction pipeline

python garmentnets/train_pipeline.py \
--exp_name tshirt_pipeline \
--log_dir data/release/Tshirt_release \
--ds Tshirt_dataset_release2 \
--cloth_type Tshirt \
--canon_checkpoint data/release/Tshirt_release/tshirt_canon