How to adjust camera angle to be a simulated camera on top of the robotic arm?

[alanwang8]

Hello,

How can I adjust the camera angle of the simulation for the robot arm such that the camera is attached to the robot (that is: I want my camera angle to be an angle that is a birds-eye view from the robotic arm itself and the camera view moves with the robotic arm as the arm moves)?

Thanks!

[xuanlinli17]

You can see how the google robot's camera is mounted on the robot's camera link and see the camera pose wrt robot link. You can set a fixed transformation between a robot link and the camera.

[alanwang8]

Ok, thanks for the comment! I'm also experimenting with using Simpler for OpenVLA experiments and was looking through the openvla policy pull request.

How can I run the OpenVLA model on Simpler to reproduce the Bridge dataset results on the pull request? It seems different from running the example.ipynb script in the main repository.

I previously tried running the instructions that another user had but got these errors:

~/simpler-env/SimplerEnv$ python tools/calc_metrics_evaluation_videos.py --task=bridge_put_on
***Bridge Put On Env results***
********** Results for put_spoon_on_tablecloth **********
/home/alanw2/miniconda3/envs/simpler/lib/python3.10/site-packages/numpy/core/fromnumeric.py:3464: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
/home/alanw2/miniconda3/envs/simpler/lib/python3.10/site-packages/numpy/core/_methods.py:192: RuntimeWarning: invalid value encountered in scalar divide
  ret = ret.dtype.type(ret / rcount)
WARNING: avg_sim_success is nan for ./results//rt_1_x_tf_trained_for_002272480_step/bridge_table_1_v1/arm_pd_ee_target_delta_pose_align2_gripper_pd_joint_pos/PutSpoonOnTableClothInScene-v0
WARNING: avg_sim_success is nan for ./results//octo-base/bridge_table_1_v1/arm_pd_ee_target_delta_pose_align2_gripper_pd_joint_pos/PutSpoonOnTableClothInScene-v0_octo_init_rng_0
WARNING: avg_sim_success is nan for ./results//octo-base/bridge_table_1_v1/arm_pd_ee_target_delta_pose_align2_gripper_pd_joint_pos/PutSpoonOnTableClothInScene-v0_octo_init_rng_2
WARNING: avg_sim_success is nan for ./results//octo-base/bridge_table_1_v1/arm_pd_ee_target_delta_pose_align2_gripper_pd_joint_pos/PutSpoonOnTableClothInScene-v0_octo_init_rng_4
WARNING: avg_sim_success is nan for ./results//octo-small/bridge_table_1_v1/arm_pd_ee_target_delta_pose_align2_gripper_pd_joint_pos/PutSpoonOnTableClothInScene-v0_octo_init_rng_0
WARNING: avg_sim_success is nan for ./results//octo-small/bridge_table_1_v1/arm_pd_ee_target_delta_pose_align2_gripper_pd_joint_pos/PutSpoonOnTableClothInScene-v0_octo_init_rng_2
WARNING: avg_sim_success is nan for ./results//octo-small/bridge_table_1_v1/arm_pd_ee_target_delta_pose_align2_gripper_pd_joint_pos/PutSpoonOnTableClothInScene-v0_octo_init_rng_4
WARNING: avg_sim_success is nan for ./results//openvla-7b/bridge_table_1_v1/arm_pd_ee_target_delta_pose_align2_gripper_pd_joint_pos/PutSpoonOnTableClothInScene-v0
sim visual matching partial success {'rt-1-x': nan, 'octo-base': nan, 'octo-small': nan, 'openvla-7b': nan}
real partial success {'rt-1-x': 0.042, 'octo-base': 0.5, 'octo-small': 0.542, 'openvla-7b': 0.1}
visual matching MMRV (partial success) 0.254
visual matching pearson correlation (partial success)  nan
avg kruskal (partial success):
       each checkpoint kruskal:
             KruskalResult(statistic=0.9999999999997733, pvalue=0.3173105078629661)
             KruskalResult(statistic=15.66666666666664, pvalue=7.554389326826452e-05)
             KruskalResult(statistic=17.45714285714282, pvalue=2.938580582975732e-05)
             KruskalResult(statistic=2.0434782608695747, pvalue=0.15285977016579425)
sim visual matching success {'rt-1-x': nan, 'octo-base': nan, 'octo-small': nan, 'openvla-7b': nan}
real success {'rt-1-x': 0.0, 'octo-base': 0.333, 'octo-small': 0.417, 'openvla-7b': 0.1}
visual matching MMRV 0.2125
visual matching pearson correlation nan
avg kruskal:
       each checkpoint kruskal:
             all same, 1.0
             KruskalResult(statistic=9.399999999999977, pvalue=0.002169854427130147)
             KruskalResult(statistic=12.368421052631579, pvalue=0.0004366567257915252)
             KruskalResult(statistic=2.0434782608695747, pvalue=0.15285977016579425)
********************



********** Results for put_carrot_on_plate **********
WARNING: avg_sim_success is nan for ./results//rt_1_x_tf_trained_for_002272480_step/bridge_table_1_v1/arm_pd_ee_target_delta_pose_align2_gripper_pd_joint_pos/PutCarrotOnPlateInScene-v0
WARNING: avg_sim_success is nan for ./results//octo-base/bridge_table_1_v1/arm_pd_ee_target_delta_pose_align2_gripper_pd_joint_pos/PutCarrotOnPlateInScene-v0_octo_init_rng_0
WARNING: avg_sim_success is nan for ./results//octo-base/bridge_table_1_v1/arm_pd_ee_target_delta_pose_align2_gripper_pd_joint_pos/PutCarrotOnPlateInScene-v0_octo_init_rng_2
WARNING: avg_sim_success is nan for ./results//octo-base/bridge_table_1_v1/arm_pd_ee_target_delta_pose_align2_gripper_pd_joint_pos/PutCarrotOnPlateInScene-v0_octo_init_rng_4
WARNING: avg_sim_success is nan for ./results//octo-small/bridge_table_1_v1/arm_pd_ee_target_delta_pose_align2_gripper_pd_joint_pos/PutCarrotOnPlateInScene-v0_octo_init_rng_0
WARNING: avg_sim_success is nan for ./results//octo-small/bridge_table_1_v1/arm_pd_ee_target_delta_pose_align2_gripper_pd_joint_pos/PutCarrotOnPlateInScene-v0_octo_init_rng_2
WARNING: avg_sim_success is nan for ./results//octo-small/bridge_table_1_v1/arm_pd_ee_target_delta_pose_align2_gripper_pd_joint_pos/PutCarrotOnPlateInScene-v0_octo_init_rng_4
WARNING: avg_sim_success is nan for ./results//openvla-7b/bridge_table_1_v1/arm_pd_ee_target_delta_pose_align2_gripper_pd_joint_pos/PutCarrotOnPlateInScene-v0
sim visual matching partial success {'rt-1-x': nan, 'octo-base': nan, 'octo-small': nan, 'openvla-7b': nan}
real partial success {'rt-1-x': 0.167, 'octo-base': 0.5, 'octo-small': 0.208, 'openvla-7b': 0.1}
visual matching MMRV (partial success) 0.14375
visual matching pearson correlation (partial success)  nan
avg kruskal (partial success):
       each checkpoint kruskal:
             KruskalResult(statistic=4.272727272727169, pvalue=0.0387287232951513)
             KruskalResult(statistic=15.66666666666664, pvalue=7.554389326826452e-05)
             KruskalResult(statistic=5.4651162790697425, pvalue=0.019399763184203167)
             KruskalResult(statistic=2.0434782608695747, pvalue=0.15285977016579425)
sim visual matching success {'rt-1-x': nan, 'octo-base': nan, 'octo-small': nan, 'openvla-7b': nan}
real success {'rt-1-x': 0.0, 'octo-base': 0.25, 'octo-small': 0.083}
Traceback (most recent call last):
  File "/home/alanw2/simpler-env/SimplerEnv/tools/calc_metrics_evaluation_videos.py", line 1058, in <module>
    calc_bridge_put_on_env_stats(args.log_dir_root)
  File "/home/alanw2/simpler-env/SimplerEnv/tools/calc_metrics_evaluation_videos.py", line 1012, in calc_bridge_put_on_env_stats
    mean_maximum_rank_violation(list(sim_visual_matching_success.values()), list(real_success.values())),
  File "/home/alanw2/simpler-env/SimplerEnv/simpler_env/utils/metrics.py", line 234, in mean_maximum_rank_violation
    assert perf_sim.shape == perf_real.shape
AssertionError```

[xuanlinli17]

This script you have run is for calculating metrics according to our paper, not for OpenVLA evaluation.

To evaluate OpenVLA, run the scripts under scripts/ in https://github.com/DelinQu/SimplerEnv-OpenVLA/tree/main