Pretrain Dataset Question:

[motor-x]

Hi,

When you used BC Z (Jang et al., 2021) as one of your pretrain dataset, did you realize that their action data is problematic？ For example:

The gripper action for the future 10 steps is all 0, and the state of gripper increased; However, as shown below, the gripper action for the future 10 steps is also all 0, the state of gripper decreased.

Did you ever fixed this? Could you please share the fixed data?

Thanks!

[csuastt]

In pre-training, we do not use actions. We use future states instead.

[motor-x]

Got it, thanks!

[motor-x]

Also, could you please let me know where you got the ALOHA dataset? It seems that the dataset is missing from the google drive of open-x dataset: https://console.cloud.google.com/storage/browser/gresearch/robotics?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))

Thanks!

[csuastt]

see https://drive.google.com/drive/folders/1FP5eakcxQrsHyiWBRDsMRvUfSxeykiDc

[motor-x]

Hi,

Just one further question, I noticed that you are also using RH20T dataset. Two questions about this:

• Where to find the state and action information? I noticed that they have:

  • join.npy
  • tcp_base.npy
    but, it seems that they are all state info, right?

• They have a number of camera views, may I ask which camera views did you use for pretrain?

Thanks!

[LBG21]

We use both joint.npy and tcp_base.npy for future state prediction. To leverage the multiple camera views in RH20T, we randomly select one camera view from all views available for each data sample during training.

[motor-x]

Thank you for the quick reply! Then, what about action? Are you using future joint and tcp_base as action?

Thanks!

[LBG21]

During pre-training, we do not use action but instead future states. In the case of RH20T, the model takes current joint pose and tcp_base as proprio, and predicts future joint pose and tcp_base.

[motor-x]

Got it, I appreciate your quick and detailed response, thank you!

[motor-x]

Sorry for bothering you again! In RH20T, for the following two configs:

Conf. | Robot Gripper |
Cfg 1 Flexiv Dahuan AG95
Cfg 2 Flexiv Dahuan AG95

The joint dim should be either 6 or 7, but their joint.npy file has 14 on it. May I ask how did you process this?

[LBG21]

During our pre-training processing, most (I'm not sure if it's all) of the joint.npy dimensions are 6 or 7, so we simply remove all data that is not 6 or 7 dimensions. You can refer to data/preprocess_scripts/rh20t.py for more information.