Project Website: https://humanoid-ai.github.io/
This repository contains the updating implementation for the Humanoid Shadowing Transformer (HST) and the Humanoid Imitation Transformer (HIT), along with instructions for whole-body pose estimation and the associated hardware codebase.
Reinforcement learning in simulation is based on legged_gym and rsl_rl.
Install IsaacGym v4 first from the official source. Place the isaacgym fold inside the HST folder.
cd HST/rsl_rl && pip install -e .
cd HST/legged_gym && pip install -e .
To train HST:
python legged_gym/scripts/train.py --run_name 0001_test --headless --sim_device cuda:0 --rl_device cuda:0
To play a trained policy:
python legged_gym/scripts/play.py --run_name 0001_test --checkpoint -1 --headless --sim_device cuda:0 --rl_device cuda:0
Imitation learning in the real world is based on ACT repo and Mobile ALOHA repo.
conda create -n HIT python=3.8.10
conda activate HIT
pip install torchvision
pip install torch
pip install pyquaternion
pip install pyyaml
pip install rospkg
pip install pexpect
pip install mujoco==2.3.7
pip install dm_control==1.0.14
pip install opencv-python
pip install matplotlib
pip install einops
pip install packaging
pip install h5py
pip install ipython
pip install getkey
pip install wandb
pip install chardet
pip install h5py_cache
cd HIT/detr && pip install -e .
Collect your own data or download our dataset from here and place it in the HIT folder.
To set up a new terminal, run:
conda activate HIT
cd HIT
To train HIT:
# Fold Clothes task
python imitate_episodes_h1_train.py --task_name data_fold_clothes --ckpt_dir fold_clothes/ --policy_class HIT --chunk_size 50 --hidden_dim 512 --batch_size 48 --dim_feedforward 512 --lr 1e-5 --seed 0 --num_steps 100000 --eval_every 100000 --validate_every 1000 --save_every 10000 --no_encoder --backbone resnet18 --same_backbones --use_pos_embd_image 1 --use_pos_embd_action 1 --dec_layers 6 --gpu_id 0 --feature_loss_weight 0.005 --use_mask --data_aug --wandb
Hardware codebase is based on unitree_ros2.
install unitree_sdk
install unitree_ros2
conda create -n lowlevel python=3.8
conda activate lowlevel
install nvidia-jetpack
install torch==1.11.0 and torchvision==0.12.0:
please refer to the following links:
https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048
https://docs.nvidia.com/deeplearning/frameworks/install-pytorch-jetson-platform/index.html
Put your trained policy in the hardware-script/ckpt folder and rename it to policy.pt
conda activate lowlevel
cd hardware-script
python hardware_whole_body.py --task_name stand
For body pose estimation, please refer to WHAM. For hand pose estimation, please refer to HaMeR.