Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Question] Problems with Collisions in Pusher-v4 #950

Closed
HernandezEduin opened this issue Mar 5, 2024 · 47 comments · Fixed by #1019
Closed

[Question] Problems with Collisions in Pusher-v4 #950

HernandezEduin opened this issue Mar 5, 2024 · 47 comments · Fixed by #1019
Labels
question Further information is requested

Comments

@HernandezEduin
Copy link

HernandezEduin commented Mar 5, 2024 •
edited
Loading

Question

I'm testing out the RL training with cleanRL, but I noticed in the video provided below that the robotic arm goes through both the table and the object it is supposed to be pushing.

The code below also has a similar behaviour:

import gymnasium as gym
env = gym.make("Pusher-v4", render_mode="human")
observation, info = env.reset(seed=39)
for _ in range(1000):
   action = env.action_space.sample()  # this is where you would insert your policy
   observation, reward, terminated, truncated, info = env.step(action)

   if terminated or truncated:
      observation, info = env.reset()

env.close()

Any idea what is going on or what might be missing?

System info

Installed via pip version 0.29.1

Python 3.8.12
Gymnasium 0.29.1
Mujoco 3.1.2
rl-video-episode-9000.mp4
@HernandezEduin HernandezEduin added the question Further information is requested label Mar 5, 2024
@Kallinteris-Andreas
Copy link
Collaborator

  1. does this also happen with mujoco==2.3.7
  2. can you produce the video with a different angle

@HernandezEduin
Copy link
Author

  1. Yes, same issue with Mujoco==2.3.7.
  2. Is there a way to modify the angle with the Gymnasium record_video?

@Kallinteris-Andreas
Copy link
Collaborator

Yes, you need to change the azimuth elevation camera properties

import gymnasium as gym
import numpy as np

DEFAULT_CAMERA_CONFIG = {
    "trackbodyid": -1,
    #"distance": 4.0,
    "distance": 3.0,
    "azimuth": 135.0,
    "elevation": -22.5,
}

env = gym.make("Pusher-v5", default_camera_config=DEFAULT_CAMERA_CONFIG, render_mode="human")

@HernandezEduin
Copy link
Author

  1. The changing of camera option is not present for Gymnasium 0.29.1. If I upgrade to gymnasium==1.0.0a1, the option becomes available along with Pusher-v5, but I'm no longer able to run CleanRL due to compatibilities issues of this version of Gym and Stable-Baselines3.

@Kallinteris-Andreas
Copy link
Collaborator

You could try this:

import gymnasium

DEFAULT_CAMERA_CONFIG = {
    "trackbodyid": -1,
    #"distance": 4.0,
    "distance": 3.0,
    "azimuth": 135.0,
    "elevation": -22.5,
}

env = gymnasium.make("Pusher-v4", render_mode="human")
env.unwrapped.mujoco_renderer.default_cam_config = DEFAULT_CAMERA_CONFIG
env.unwrapped.mujoco_renderer._set_cam_config()

@HernandezEduin
Copy link
Author

Okay, it seems to be working after I changed the camera angle for some unknown reason.

rl-video-episode-512.mp4

@Kallinteris-Andreas
Copy link
Collaborator

Kallinteris-Andreas commented Mar 7, 2024 •
edited
Loading

Are you using the same policy?
double check: does this also happen for mujoco==3.1.2
double check: does this work without env.unwrapped.mujoco_renderer._set_cam_config()

@HernandezEduin
Copy link
Author

HernandezEduin commented Mar 7, 2024 •
edited
Loading

Actually, I have to run it without env.unwrapped.mujoco_renderer._set_cam_config(), otherwise it gives me an assertion error on self.viewer is not None. As for whether it works with Mujoco==3.1.2, no. It goes right through the object it seems that the version is causing some errors.

With Mujoco==2.3.7

rl-video-episode-8.mp4

With Mujoco==3.1.2

rl-video-episode-8.mp4

@Kallinteris-Andreas
Copy link
Collaborator

Kallinteris-Andreas commented Mar 7, 2024 •
edited
Loading

also try mujoco==3.1.3 if it does not fix the issue

please find which version of mujoco broke it and report it to the mujoco team (https://github.com/google-deepmind/mujoco) also (tag me in the issue)

Thanks

@pseudo-rnd-thoughts
Copy link
Member

@HernandezEduin also, if you have some example actions list for a fixed seed for the MuJoCo team to use, it would be helpful

@HernandezEduin
Copy link
Author

Okay, so I've tested it out with various versions of Mujoco and it the results are as follows:

mujoco==2.3.6 - working
mujoco==2.3.7 - working
mujoco==3.0.0 - not working
mujoco==3.0.1 - not working
mujoco==3.1.0 - not working
mujoco==3.1.1 - not working
mujoco==3.1.2 - not working
mujoco==3.1.3 - not working

The collisions stopped working with Ver. 3.0.0. I'll report it soon to the Mujoco team.

@HernandezEduin HernandezEduin mentioned this issue Mar 11, 2024
Closed
@Kallinteris-Andreas
Copy link
Collaborator

Kallinteris-Andreas commented Mar 11, 2024 •
edited
Loading

Continuing from the mujoco issue

Can you render with contact forces? @HernandezEduin

import gymnasium
import mujoco

env = gymnasium.make("Pusher-v4", render_mode="rgb_array")
env.reset()
env.render()
env.unwrapped.mujoco_renderer.viewer.vopt.flags[mujoco.mjtVisFlag.mjVIS_CONTACTFORCE] = True
env.unwrapped.mujoco_renderer.viewer.vopt.flags[mujoco.mjtVisFlag.mjVIS_CONTACTPOINT] = True

...  # run and render your episode here

Thanks!

@pseudo-rnd-thoughts
Copy link
Member

pseudo-rnd-thoughts commented Mar 12, 2024 •
edited
Loading

@Kallinteris-Andreas given that dm-mujoco don't believe that this is a bug on their end and this is actually just an issue with the environment model.

Does this problem continues in v5, if so we fix it in v5 before the full release of v1.0.0?

@Kallinteris-Andreas
Copy link
Collaborator

Kallinteris-Andreas commented Mar 12, 2024 •
edited
Loading

@pseudo-rnd-thoughts The problem is that version 5 of the environments has already been released in 1.0.0a1 (Users might already have started using it in their research ), This should be fixed in a potential version 6 In the meantime, we should block the instantiation of the pusher environment When the MMujoco version is 3.0 or higher

@pseudo-rnd-thoughts
Copy link
Member

I think we should put a restriction on pusher-v4 for mujoco < 3.0 at the very least as we know this is bugged.

For v5, given that 1.0a requires users to intentionally install it, pip install gymnasium will install 0.29 then I don't think it is an issue to update the model currently. But it would be great to do before alpha 2

Do you think that it would be easy to get a better model?

@Kallinteris-Andreas
Copy link
Collaborator

I agree with the suggested change for version 4.

Normally, i would not agree with you, about changing the version 5 pusher, but because the bug is so critical and it was released in an alpha version of gymnasium we should fix it.

@Kallinteris-Andreas
Copy link
Collaborator

Kallinteris-Andreas commented Mar 15, 2024 •
edited
Loading

@HernandezEduin can you also test this model file?

pusher_v5.xml

<mujoco model="arm3d">
  <compiler inertiafromgeom="true" angle="radian" coordinate="local"/>
  <option timestep="0.01" gravity="0 0 0" iterations="20" integrator="Euler" />

  <default>
    <joint armature='0.04' damping="1" limited="true"/>
    <geom friction=".8 .1 .1" density="300" margin="0.002" condim="1" contype="0" conaffinity="0"/>
  </default>

  <worldbody>
    <light diffuse=".5 .5 .5" pos="0 0 3" dir="0 0 -1"/>
    <geom name="table" type="plane" pos="0 0.5 -0.325" size="1 1 0.1" contype="1" conaffinity="1"/>

    <body name="r_shoulder_pan_link" pos="0 -0.6 0">
      <geom name="e1" type="sphere" rgba="0.6 0.6 0.6 1" pos="-0.06 0.05 0.2" size="0.05" />
      <geom name="e2" type="sphere" rgba="0.6 0.6 0.6 1" pos=" 0.06 0.05 0.2" size="0.05" />
      <geom name="e1p" type="sphere" rgba="0.1 0.1 0.1 1" pos="-0.06 0.09 0.2" size="0.03" />
      <geom name="e2p" type="sphere" rgba="0.1 0.1 0.1 1" pos=" 0.06 0.09 0.2" size="0.03" />
      <geom name="sp" type="capsule" fromto="0 0 -0.4 0 0 0.2" size="0.1" />
      <joint name="r_shoulder_pan_joint" type="hinge" pos="0 0 0" axis="0 0 1" range="-2.2854 1.714602" damping="1.0" />

      <body name="r_shoulder_lift_link" pos="0.1 0 0">
        <geom name="sl" type="capsule" fromto="0 -0.1 0 0 0.1 0" size="0.1" />
        <joint name="r_shoulder_lift_joint" type="hinge" pos="0 0 0" axis="0 1 0" range="-0.5236 1.3963" damping="1.0" />

        <body name="r_upper_arm_roll_link" pos="0 0 0">
          <geom name="uar" type="capsule" fromto="-0.1 0 0 0.1 0 0" size="0.02" />
          <joint name="r_upper_arm_roll_joint" type="hinge" pos="0 0 0" axis="1 0 0" range="-1.5 1.7" damping="0.1" />

          <body name="r_upper_arm_link" pos="0 0 0">
            <geom name="ua" type="capsule" fromto="0 0 0 0.4 0 0" size="0.06" />

            <body name="r_elbow_flex_link" pos="0.4 0 0">
              <geom name="ef" type="capsule" fromto="0 -0.02 0 0.0 0.02 0" size="0.06" />
              <joint name="r_elbow_flex_joint" type="hinge" pos="0 0 0" axis="0 1 0" range="-2.3213 0" damping="0.1" />

              <body name="r_forearm_roll_link" pos="0 0 0">
                <geom name="fr" type="capsule" fromto="-0.1 0 0 0.1 0 0" size="0.02" />
                <joint name="r_forearm_roll_joint" type="hinge" limited="true" pos="0 0 0" axis="1 0 0" damping=".1" range="-1.5 1.5"/>

                <body name="r_forearm_link" pos="0 0 0">
                  <geom name="fa" type="capsule" fromto="0 0 0 0.291 0 0" size="0.05" />

                  <body name="r_wrist_flex_link" pos="0.321 0 0">
                    <geom name="wf" type="capsule" fromto="0 -0.02 0 0 0.02 0" size="0.01" />
                    <joint name="r_wrist_flex_joint" type="hinge" pos="0 0 0" axis="0 1 0" range="-1.094 0" damping=".1" />

                    <body name="r_wrist_roll_link" pos="0 0 0">
                      <joint name="r_wrist_roll_joint" type="hinge" pos="0 0 0" limited="true" axis="1 0 0" damping="0.1" range="-1.5 1.5"/>
                      <body name="tips_arm" pos="0 0 0">
                        <geom name="tip_arml" type="sphere" pos="0.1 -0.1 0." size="0.01" />
                        <geom name="tip_armr" type="sphere" pos="0.1 0.1 0." size="0.01" />
                      </body>
                      <geom type="capsule" fromto="0 -0.1 0. 0.0 +0.1 0" size="0.02" contype="1" conaffinity="1" />
                      <geom type="capsule" fromto="0 -0.1 0. 0.1 -0.1 0" size="0.02" contype="1" conaffinity="1" />
                      <geom type="capsule" fromto="0 +0.1 0. 0.1 +0.1 0." size="0.02" contype="1" conaffinity="1" />
                    </body>
                  </body>
                </body>
              </body>
            </body>
          </body>
        </body>
      </body>
    </body>

    <!--<body name="object" pos="0.55 -0.3 -0.275" >-->
    <body name="object" pos="0.45 -0.05 -0.275" >
      <geom rgba="1 1 1 0" type="sphere" size="0.05 0.05 0.05" density="0.01" conaffinity="0"/>
      <geom rgba="1 1 1 1" type="cylinder" size="0.05 0.05 0.05" density="0.01" contype="1" conaffinity="0"/>
      <joint name="obj_slidey" type="slide" pos="0 0 0" axis="0 1 0" range="-10.3213 10.3" damping="0.5"/>
      <joint name="obj_slidex" type="slide" pos="0 0 0" axis="1 0 0" range="-10.3213 10.3" damping="0.5"/>
    </body>

    <body name="goal" pos="0.45 -0.05 -0.3230">
      <geom rgba="1 0 0 1" type="cylinder" size="0.08 0.001 0.1" density='0.00001' contype="0" conaffinity="0"/>
      <joint name="goal_slidey" type="slide" pos="0 0 0" axis="0 1 0" range="-10.3213 10.3" damping="0.5"/>
      <joint name="goal_slidex" type="slide" pos="0 0 0" axis="1 0 0" range="-10.3213 10.3" damping="0.5"/>
    </body>
  </worldbody>

  <actuator>
    <motor joint="r_shoulder_pan_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_shoulder_lift_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_upper_arm_roll_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_elbow_flex_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_forearm_roll_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_wrist_flex_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_wrist_roll_joint" ctrlrange="-2.0 2.0" ctrllimited="true"/>
  </actuator>
</mujoco>

@HernandezEduin
Copy link
Author

Sorry for the delay! This is the result for Mujoco == 3.1.3 using Gymnasium == 0.29.1.

mujoco.3.1.3.with.Contact.Forces.mp4

Continuing from the mujoco issue

Can you render with contact forces? @HernandezEduin

import gymnasium
import mujoco

env = gymnasium.make("Pusher-v4", render_mode="rgb_array")
env.reset()
env.render()
env.unwrapped.mujoco_renderer.viewer.vopt.flags[mujoco.mjtVisFlag.mjVIS_CONTACTFORCE] = True
env.unwrapped.mujoco_renderer.viewer.vopt.flags[mujoco.mjtVisFlag.mjVIS_CONTACTPOINT] = True

...  # run and render your episode here

Thanks!

@HernandezEduin
Copy link
Author

@HernandezEduin can you also test this model file?

pusher_v5.xml

<mujoco model="arm3d">
  <compiler inertiafromgeom="true" angle="radian" coordinate="local"/>
  <option timestep="0.01" gravity="0 0 0" iterations="20" integrator="Euler" />

  <default>
    <joint armature='0.04' damping="1" limited="true"/>
    <geom friction=".8 .1 .1" density="300" margin="0.002" condim="1" contype="0" conaffinity="0"/>
  </default>

  <worldbody>
    <light diffuse=".5 .5 .5" pos="0 0 3" dir="0 0 -1"/>
    <geom name="table" type="plane" pos="0 0.5 -0.325" size="1 1 0.1" contype="1" conaffinity="1"/>

    <body name="r_shoulder_pan_link" pos="0 -0.6 0">
      <geom name="e1" type="sphere" rgba="0.6 0.6 0.6 1" pos="-0.06 0.05 0.2" size="0.05" />
      <geom name="e2" type="sphere" rgba="0.6 0.6 0.6 1" pos=" 0.06 0.05 0.2" size="0.05" />
      <geom name="e1p" type="sphere" rgba="0.1 0.1 0.1 1" pos="-0.06 0.09 0.2" size="0.03" />
      <geom name="e2p" type="sphere" rgba="0.1 0.1 0.1 1" pos=" 0.06 0.09 0.2" size="0.03" />
      <geom name="sp" type="capsule" fromto="0 0 -0.4 0 0 0.2" size="0.1" />
      <joint name="r_shoulder_pan_joint" type="hinge" pos="0 0 0" axis="0 0 1" range="-2.2854 1.714602" damping="1.0" />

      <body name="r_shoulder_lift_link" pos="0.1 0 0">
        <geom name="sl" type="capsule" fromto="0 -0.1 0 0 0.1 0" size="0.1" />
        <joint name="r_shoulder_lift_joint" type="hinge" pos="0 0 0" axis="0 1 0" range="-0.5236 1.3963" damping="1.0" />

        <body name="r_upper_arm_roll_link" pos="0 0 0">
          <geom name="uar" type="capsule" fromto="-0.1 0 0 0.1 0 0" size="0.02" />
          <joint name="r_upper_arm_roll_joint" type="hinge" pos="0 0 0" axis="1 0 0" range="-1.5 1.7" damping="0.1" />

          <body name="r_upper_arm_link" pos="0 0 0">
            <geom name="ua" type="capsule" fromto="0 0 0 0.4 0 0" size="0.06" />

            <body name="r_elbow_flex_link" pos="0.4 0 0">
              <geom name="ef" type="capsule" fromto="0 -0.02 0 0.0 0.02 0" size="0.06" />
              <joint name="r_elbow_flex_joint" type="hinge" pos="0 0 0" axis="0 1 0" range="-2.3213 0" damping="0.1" />

              <body name="r_forearm_roll_link" pos="0 0 0">
                <geom name="fr" type="capsule" fromto="-0.1 0 0 0.1 0 0" size="0.02" />
                <joint name="r_forearm_roll_joint" type="hinge" limited="true" pos="0 0 0" axis="1 0 0" damping=".1" range="-1.5 1.5"/>

                <body name="r_forearm_link" pos="0 0 0">
                  <geom name="fa" type="capsule" fromto="0 0 0 0.291 0 0" size="0.05" />

                  <body name="r_wrist_flex_link" pos="0.321 0 0">
                    <geom name="wf" type="capsule" fromto="0 -0.02 0 0 0.02 0" size="0.01" />
                    <joint name="r_wrist_flex_joint" type="hinge" pos="0 0 0" axis="0 1 0" range="-1.094 0" damping=".1" />

                    <body name="r_wrist_roll_link" pos="0 0 0">
                      <joint name="r_wrist_roll_joint" type="hinge" pos="0 0 0" limited="true" axis="1 0 0" damping="0.1" range="-1.5 1.5"/>
                      <body name="tips_arm" pos="0 0 0">
                        <geom name="tip_arml" type="sphere" pos="0.1 -0.1 0." size="0.01" />
                        <geom name="tip_armr" type="sphere" pos="0.1 0.1 0." size="0.01" />
                      </body>
                      <geom type="capsule" fromto="0 -0.1 0. 0.0 +0.1 0" size="0.02" contype="1" conaffinity="1" />
                      <geom type="capsule" fromto="0 -0.1 0. 0.1 -0.1 0" size="0.02" contype="1" conaffinity="1" />
                      <geom type="capsule" fromto="0 +0.1 0. 0.1 +0.1 0." size="0.02" contype="1" conaffinity="1" />
                    </body>
                  </body>
                </body>
              </body>
            </body>
          </body>
        </body>
      </body>
    </body>

    <!--<body name="object" pos="0.55 -0.3 -0.275" >-->
    <body name="object" pos="0.45 -0.05 -0.275" >
      <geom rgba="1 1 1 0" type="sphere" size="0.05 0.05 0.05" density="0.01" conaffinity="0"/>
      <geom rgba="1 1 1 1" type="cylinder" size="0.05 0.05 0.05" density="0.01" contype="1" conaffinity="0"/>
      <joint name="obj_slidey" type="slide" pos="0 0 0" axis="0 1 0" range="-10.3213 10.3" damping="0.5"/>
      <joint name="obj_slidex" type="slide" pos="0 0 0" axis="1 0 0" range="-10.3213 10.3" damping="0.5"/>
    </body>

    <body name="goal" pos="0.45 -0.05 -0.3230">
      <geom rgba="1 0 0 1" type="cylinder" size="0.08 0.001 0.1" density='0.00001' contype="0" conaffinity="0"/>
      <joint name="goal_slidey" type="slide" pos="0 0 0" axis="0 1 0" range="-10.3213 10.3" damping="0.5"/>
      <joint name="goal_slidex" type="slide" pos="0 0 0" axis="1 0 0" range="-10.3213 10.3" damping="0.5"/>
    </body>
  </worldbody>

  <actuator>
    <motor joint="r_shoulder_pan_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_shoulder_lift_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_upper_arm_roll_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_elbow_flex_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_forearm_roll_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_wrist_flex_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_wrist_roll_joint" ctrlrange="-2.0 2.0" ctrllimited="true"/>
  </actuator>
</mujoco>

The updated xml file seems to have solved the issue!

mujoco.3.1.3.with.updated.pusher.xml.mp4

@HernandezEduin
Copy link
Author

HernandezEduin commented Mar 19, 2024 •
edited
Loading

Actually, it seems it still has the problem... It goes through the object occasionally...

mujoco.3.1.3.with.updated.pusher.problem.mp4

@Kallinteris-Andreas
Copy link
Collaborator

Kallinteris-Andreas commented Mar 20, 2024 •
edited
Loading

  1. the object appears to be a lot heavier and harder to push (from your first video), so this might need an extra revision
  2. it appears that the fingers of the pusher arm tips have no collision

@HernandezEduin
Copy link
Author

  1. The density of the object being pushed was 10^-5 and it is now 10^-2.

@Kallinteris-Andreas
Copy link
Collaborator

Here is a new revision of the pusher model that should at least fix the first problem,

also, please use width=1280, height=720 in make() arguments, so that the generated video is of a higher resolution

Thanks!

<!--
Pusher model for `Pusher-v5` RC2, based on openai/gym/Pusher
modified by @kallinteris-Andreas
   - disabled body.object shephere geometry
   - reduced body.object cylinder density 0.00001 -> 0.001
-->
<mujoco model="arm3d">
 <compiler inertiafromgeom="true" angle="radian" coordinate="local"/>
 <option timestep="0.01" gravity="0 0 0" iterations="20" integrator="Euler" />

 <default>
   <joint armature='0.04' damping="1" limited="true"/>
   <geom friction=".8 .1 .1" density="300" margin="0.002" condim="1" contype="0" conaffinity="0"/>
 </default>

 <worldbody>
   <light diffuse=".5 .5 .5" pos="0 0 3" dir="0 0 -1"/>
   <geom name="table" type="plane" pos="0 0.5 -0.325" size="1 1 0.1" contype="1" conaffinity="1"/>

   <body name="r_shoulder_pan_link" pos="0 -0.6 0">
     <geom name="e1" type="sphere" rgba="0.6 0.6 0.6 1" pos="-0.06 0.05 0.2" size="0.05" />
     <geom name="e2" type="sphere" rgba="0.6 0.6 0.6 1" pos=" 0.06 0.05 0.2" size="0.05" />
     <geom name="e1p" type="sphere" rgba="0.1 0.1 0.1 1" pos="-0.06 0.09 0.2" size="0.03" />
     <geom name="e2p" type="sphere" rgba="0.1 0.1 0.1 1" pos=" 0.06 0.09 0.2" size="0.03" />
     <geom name="sp" type="capsule" fromto="0 0 -0.4 0 0 0.2" size="0.1" />
     <joint name="r_shoulder_pan_joint" type="hinge" pos="0 0 0" axis="0 0 1" range="-2.2854 1.714602" damping="1.0" />

     <body name="r_shoulder_lift_link" pos="0.1 0 0">
       <geom name="sl" type="capsule" fromto="0 -0.1 0 0 0.1 0" size="0.1" />
       <joint name="r_shoulder_lift_joint" type="hinge" pos="0 0 0" axis="0 1 0" range="-0.5236 1.3963" damping="1.0" />

       <body name="r_upper_arm_roll_link" pos="0 0 0">
         <geom name="uar" type="capsule" fromto="-0.1 0 0 0.1 0 0" size="0.02" />
         <joint name="r_upper_arm_roll_joint" type="hinge" pos="0 0 0" axis="1 0 0" range="-1.5 1.7" damping="0.1" />

         <body name="r_upper_arm_link" pos="0 0 0">
           <geom name="ua" type="capsule" fromto="0 0 0 0.4 0 0" size="0.06" />

           <body name="r_elbow_flex_link" pos="0.4 0 0">
             <geom name="ef" type="capsule" fromto="0 -0.02 0 0.0 0.02 0" size="0.06" />
             <joint name="r_elbow_flex_joint" type="hinge" pos="0 0 0" axis="0 1 0" range="-2.3213 0" damping="0.1" />

             <body name="r_forearm_roll_link" pos="0 0 0">
               <geom name="fr" type="capsule" fromto="-0.1 0 0 0.1 0 0" size="0.02" />
               <joint name="r_forearm_roll_joint" type="hinge" limited="true" pos="0 0 0" axis="1 0 0" damping=".1" range="-1.5 1.5"/>

               <body name="r_forearm_link" pos="0 0 0">
                 <geom name="fa" type="capsule" fromto="0 0 0 0.291 0 0" size="0.05" />

                 <body name="r_wrist_flex_link" pos="0.321 0 0">
                   <geom name="wf" type="capsule" fromto="0 -0.02 0 0 0.02 0" size="0.01" />
                   <joint name="r_wrist_flex_joint" type="hinge" pos="0 0 0" axis="0 1 0" range="-1.094 0" damping=".1" />

                   <body name="r_wrist_roll_link" pos="0 0 0">
                     <joint name="r_wrist_roll_joint" type="hinge" pos="0 0 0" limited="true" axis="1 0 0" damping="0.1" range="-1.5 1.5"/>
                     <body name="tips_arm" pos="0 0 0">
                       <geom name="tip_arml" type="sphere" pos="0.1 -0.1 0." size="0.01" />
                       <geom name="tip_armr" type="sphere" pos="0.1 0.1 0." size="0.01" />
                     </body>
                     <geom type="capsule" fromto="0 -0.1 0. 0.0 +0.1 0" size="0.02" contype="1" conaffinity="1" />
                     <geom type="capsule" fromto="0 -0.1 0. 0.1 -0.1 0" size="0.02" contype="1" conaffinity="1" />
                     <geom type="capsule" fromto="0 +0.1 0. 0.1 +0.1 0." size="0.02" contype="1" conaffinity="1" />
                   </body>
                 </body>
               </body>
             </body>
           </body>
         </body>
       </body>
     </body>
   </body>

   <!--<body name="object" pos="0.55 -0.3 -0.275" >-->
   <body name="object" pos="0.45 -0.05 -0.275" >
     <!-- <geom rgba="1 1 1 0" type="sphere" size="0.05 0.05 0.05" density="0.01" conaffinity="0"/> -->
     <geom rgba="1 1 1 1" type="cylinder" size="0.05 0.05 0.05" density="0.001" contype="1" conaffinity="0"/>
     <joint name="obj_slidey" type="slide" pos="0 0 0" axis="0 1 0" range="-10.3213 10.3" damping="0.5"/>
     <joint name="obj_slidex" type="slide" pos="0 0 0" axis="1 0 0" range="-10.3213 10.3" damping="0.5"/>
   </body>

   <body name="goal" pos="0.45 -0.05 -0.3230">
     <geom rgba="1 0 0 1" type="cylinder" size="0.08 0.001 0.1" density='0.00001' contype="0" conaffinity="0"/>
     <joint name="goal_slidey" type="slide" pos="0 0 0" axis="0 1 0" range="-10.3213 10.3" damping="0.5"/>
     <joint name="goal_slidex" type="slide" pos="0 0 0" axis="1 0 0" range="-10.3213 10.3" damping="0.5"/>
   </body>
 </worldbody>

 <actuator>
   <motor joint="r_shoulder_pan_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
   <motor joint="r_shoulder_lift_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
   <motor joint="r_upper_arm_roll_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
   <motor joint="r_elbow_flex_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
   <motor joint="r_forearm_roll_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
   <motor joint="r_wrist_flex_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
   <motor joint="r_wrist_roll_joint" ctrlrange="-2.0 2.0" ctrllimited="true"/>
 </actuator>
</mujoco>

@HernandezEduin
Copy link
Author

HernandezEduin commented Mar 20, 2024 •
edited
Loading

rl-video-episode-0.mp4
rl-video-episode-1.mp4
rl-video-episode-8.mp4

The object seems to be pushed very slowly now.

@Kallinteris-Andreas
Copy link
Collaborator

does this problem happen for all the episodes?

@HernandezEduin
Copy link
Author

Yes

@Kallinteris-Andreas
Copy link
Collaborator

I re-enabled the sphere collision geometry

<!--
Pusher model for `Pusher-v5` RC3, based on openai/gym/Pusher
modified by @kallinteris-Andreas
	- reduced body.object cylinder density 0.00001 -> 0.001
	- reduced body.object sphere density 0.00001 -> 0.001
-->
<mujoco model="arm3d">
  <compiler inertiafromgeom="true" angle="radian" coordinate="local"/>
  <option timestep="0.01" gravity="0 0 0" iterations="20" integrator="Euler" />

  <default>
    <joint armature='0.04' damping="1" limited="true"/>
    <geom friction=".8 .1 .1" density="300" margin="0.002" condim="1" contype="0" conaffinity="0"/>
  </default>

  <worldbody>
    <light diffuse=".5 .5 .5" pos="0 0 3" dir="0 0 -1"/>
    <geom name="table" type="plane" pos="0 0.5 -0.325" size="1 1 0.1" contype="1" conaffinity="1"/>

    <body name="r_shoulder_pan_link" pos="0 -0.6 0">
      <geom name="e1" type="sphere" rgba="0.6 0.6 0.6 1" pos="-0.06 0.05 0.2" size="0.05" />
      <geom name="e2" type="sphere" rgba="0.6 0.6 0.6 1" pos=" 0.06 0.05 0.2" size="0.05" />
      <geom name="e1p" type="sphere" rgba="0.1 0.1 0.1 1" pos="-0.06 0.09 0.2" size="0.03" />
      <geom name="e2p" type="sphere" rgba="0.1 0.1 0.1 1" pos=" 0.06 0.09 0.2" size="0.03" />
      <geom name="sp" type="capsule" fromto="0 0 -0.4 0 0 0.2" size="0.1" />
      <joint name="r_shoulder_pan_joint" type="hinge" pos="0 0 0" axis="0 0 1" range="-2.2854 1.714602" damping="1.0" />

      <body name="r_shoulder_lift_link" pos="0.1 0 0">
        <geom name="sl" type="capsule" fromto="0 -0.1 0 0 0.1 0" size="0.1" />
        <joint name="r_shoulder_lift_joint" type="hinge" pos="0 0 0" axis="0 1 0" range="-0.5236 1.3963" damping="1.0" />

        <body name="r_upper_arm_roll_link" pos="0 0 0">
          <geom name="uar" type="capsule" fromto="-0.1 0 0 0.1 0 0" size="0.02" />
          <joint name="r_upper_arm_roll_joint" type="hinge" pos="0 0 0" axis="1 0 0" range="-1.5 1.7" damping="0.1" />

          <body name="r_upper_arm_link" pos="0 0 0">
            <geom name="ua" type="capsule" fromto="0 0 0 0.4 0 0" size="0.06" />

            <body name="r_elbow_flex_link" pos="0.4 0 0">
              <geom name="ef" type="capsule" fromto="0 -0.02 0 0.0 0.02 0" size="0.06" />
              <joint name="r_elbow_flex_joint" type="hinge" pos="0 0 0" axis="0 1 0" range="-2.3213 0" damping="0.1" />

              <body name="r_forearm_roll_link" pos="0 0 0">
                <geom name="fr" type="capsule" fromto="-0.1 0 0 0.1 0 0" size="0.02" />
                <joint name="r_forearm_roll_joint" type="hinge" limited="true" pos="0 0 0" axis="1 0 0" damping=".1" range="-1.5 1.5"/>

                <body name="r_forearm_link" pos="0 0 0">
                  <geom name="fa" type="capsule" fromto="0 0 0 0.291 0 0" size="0.05" />

                  <body name="r_wrist_flex_link" pos="0.321 0 0">
                    <geom name="wf" type="capsule" fromto="0 -0.02 0 0 0.02 0" size="0.01" />
                    <joint name="r_wrist_flex_joint" type="hinge" pos="0 0 0" axis="0 1 0" range="-1.094 0" damping=".1" />

                    <body name="r_wrist_roll_link" pos="0 0 0">
                      <joint name="r_wrist_roll_joint" type="hinge" pos="0 0 0" limited="true" axis="1 0 0" damping="0.1" range="-1.5 1.5"/>
                      <body name="tips_arm" pos="0 0 0">
                        <geom name="tip_arml" type="sphere" pos="0.1 -0.1 0." size="0.01" />
                        <geom name="tip_armr" type="sphere" pos="0.1 0.1 0." size="0.01" />
                      </body>
                      <geom type="capsule" fromto="0 -0.1 0. 0.0 +0.1 0" size="0.02" contype="1" conaffinity="1" />
                      <geom type="capsule" fromto="0 -0.1 0. 0.1 -0.1 0" size="0.02" contype="1" conaffinity="1" />
                      <geom type="capsule" fromto="0 +0.1 0. 0.1 +0.1 0." size="0.02" contype="1" conaffinity="1" />
                    </body>
                  </body>
                </body>
              </body>
            </body>
          </body>
        </body>
      </body>
    </body>

    <!--<body name="object" pos="0.55 -0.3 -0.275" >-->
    <body name="object" pos="0.45 -0.05 -0.275" >
      <geom rgba="1 1 1 0" type="sphere" size="0.05 0.05 0.05" density="0.001" conaffinity="0"/>
      <geom rgba="1 1 1 1" type="cylinder" size="0.05 0.05 0.05" density="0.001" contype="1" conaffinity="0"/>
      <joint name="obj_slidey" type="slide" pos="0 0 0" axis="0 1 0" range="-10.3213 10.3" damping="0.5"/>
      <joint name="obj_slidex" type="slide" pos="0 0 0" axis="1 0 0" range="-10.3213 10.3" damping="0.5"/>
    </body>

    <body name="goal" pos="0.45 -0.05 -0.3230">
      <geom rgba="1 0 0 1" type="cylinder" size="0.08 0.001 0.1" density='0.00001' contype="0" conaffinity="0"/>
      <joint name="goal_slidey" type="slide" pos="0 0 0" axis="0 1 0" range="-10.3213 10.3" damping="0.5"/>
      <joint name="goal_slidex" type="slide" pos="0 0 0" axis="1 0 0" range="-10.3213 10.3" damping="0.5"/>
    </body>
  </worldbody>

  <actuator>
    <motor joint="r_shoulder_pan_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_shoulder_lift_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_upper_arm_roll_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_elbow_flex_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_forearm_roll_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_wrist_flex_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_wrist_roll_joint" ctrlrange="-2.0 2.0" ctrllimited="true"/>
  </actuator>
</mujoco>

@HernandezEduin
Copy link
Author

rl-video-episode-0.mp4
rl-video-episode-1.mp4
rl-video-episode-8.mp4

@Kallinteris-Andreas
Copy link
Collaborator

Hmm, I have little idea of why this is actually happening :-)

double check you used the correct model file, (the results are identical to the previous case, but that can be explained by the fact that the bug is the same for both cases)

here is RC4 it is identical to RC1, but with the sphere object disabled, if that does not work I guess we can try decreasing the object's density to 0.005?

<!--
Pusher model for `Pusher-v5` RC4, based on openai/gym/Pusher
modified by @kallinteris-Andreas
	- reduced body.object cylinder density 0.00001 -> 0.01
	- disabled body.object sphere (density 0.00001 -> 0.001)
-->
<mujoco model="arm3d">
  <compiler inertiafromgeom="true" angle="radian" coordinate="local"/>
  <option timestep="0.01" gravity="0 0 0" iterations="20" integrator="Euler" />

  <default>
    <joint armature='0.04' damping="1" limited="true"/>
    <geom friction=".8 .1 .1" density="300" margin="0.002" condim="1" contype="0" conaffinity="0"/>
  </default>

  <worldbody>
    <light diffuse=".5 .5 .5" pos="0 0 3" dir="0 0 -1"/>
    <geom name="table" type="plane" pos="0 0.5 -0.325" size="1 1 0.1" contype="1" conaffinity="1"/>

    <body name="r_shoulder_pan_link" pos="0 -0.6 0">
      <geom name="e1" type="sphere" rgba="0.6 0.6 0.6 1" pos="-0.06 0.05 0.2" size="0.05" />
      <geom name="e2" type="sphere" rgba="0.6 0.6 0.6 1" pos=" 0.06 0.05 0.2" size="0.05" />
      <geom name="e1p" type="sphere" rgba="0.1 0.1 0.1 1" pos="-0.06 0.09 0.2" size="0.03" />
      <geom name="e2p" type="sphere" rgba="0.1 0.1 0.1 1" pos=" 0.06 0.09 0.2" size="0.03" />
      <geom name="sp" type="capsule" fromto="0 0 -0.4 0 0 0.2" size="0.1" />
      <joint name="r_shoulder_pan_joint" type="hinge" pos="0 0 0" axis="0 0 1" range="-2.2854 1.714602" damping="1.0" />

      <body name="r_shoulder_lift_link" pos="0.1 0 0">
        <geom name="sl" type="capsule" fromto="0 -0.1 0 0 0.1 0" size="0.1" />
        <joint name="r_shoulder_lift_joint" type="hinge" pos="0 0 0" axis="0 1 0" range="-0.5236 1.3963" damping="1.0" />

        <body name="r_upper_arm_roll_link" pos="0 0 0">
          <geom name="uar" type="capsule" fromto="-0.1 0 0 0.1 0 0" size="0.02" />
          <joint name="r_upper_arm_roll_joint" type="hinge" pos="0 0 0" axis="1 0 0" range="-1.5 1.7" damping="0.1" />

          <body name="r_upper_arm_link" pos="0 0 0">
            <geom name="ua" type="capsule" fromto="0 0 0 0.4 0 0" size="0.06" />

            <body name="r_elbow_flex_link" pos="0.4 0 0">
              <geom name="ef" type="capsule" fromto="0 -0.02 0 0.0 0.02 0" size="0.06" />
              <joint name="r_elbow_flex_joint" type="hinge" pos="0 0 0" axis="0 1 0" range="-2.3213 0" damping="0.1" />

              <body name="r_forearm_roll_link" pos="0 0 0">
                <geom name="fr" type="capsule" fromto="-0.1 0 0 0.1 0 0" size="0.02" />
                <joint name="r_forearm_roll_joint" type="hinge" limited="true" pos="0 0 0" axis="1 0 0" damping=".1" range="-1.5 1.5"/>

                <body name="r_forearm_link" pos="0 0 0">
                  <geom name="fa" type="capsule" fromto="0 0 0 0.291 0 0" size="0.05" />

                  <body name="r_wrist_flex_link" pos="0.321 0 0">
                    <geom name="wf" type="capsule" fromto="0 -0.02 0 0 0.02 0" size="0.01" />
                    <joint name="r_wrist_flex_joint" type="hinge" pos="0 0 0" axis="0 1 0" range="-1.094 0" damping=".1" />

                    <body name="r_wrist_roll_link" pos="0 0 0">
                      <joint name="r_wrist_roll_joint" type="hinge" pos="0 0 0" limited="true" axis="1 0 0" damping="0.1" range="-1.5 1.5"/>
                      <body name="tips_arm" pos="0 0 0">
                        <geom name="tip_arml" type="sphere" pos="0.1 -0.1 0." size="0.01" />
                        <geom name="tip_armr" type="sphere" pos="0.1 0.1 0." size="0.01" />
                      </body>
                      <geom type="capsule" fromto="0 -0.1 0. 0.0 +0.1 0" size="0.02" contype="1" conaffinity="1" />
                      <geom type="capsule" fromto="0 -0.1 0. 0.1 -0.1 0" size="0.02" contype="1" conaffinity="1" />
                      <geom type="capsule" fromto="0 +0.1 0. 0.1 +0.1 0." size="0.02" contype="1" conaffinity="1" />
                    </body>
                  </body>
                </body>
              </body>
            </body>
          </body>
        </body>
      </body>
    </body>

    <!--<body name="object" pos="0.55 -0.3 -0.275" >-->
    <body name="object" pos="0.45 -0.05 -0.275" >
      <!-- <geom rgba="1 1 1 0" type="sphere" size="0.05 0.05 0.05" density="0.001" conaffinity="0"/> -->
      <geom rgba="1 1 1 1" type="cylinder" size="0.05 0.05 0.05" density="0.01" contype="1" conaffinity="0"/>
      <joint name="obj_slidey" type="slide" pos="0 0 0" axis="0 1 0" range="-10.3213 10.3" damping="0.5"/>
      <joint name="obj_slidex" type="slide" pos="0 0 0" axis="1 0 0" range="-10.3213 10.3" damping="0.5"/>
    </body>

    <body name="goal" pos="0.45 -0.05 -0.3230">
      <geom rgba="1 0 0 1" type="cylinder" size="0.08 0.001 0.1" density='0.00001' contype="0" conaffinity="0"/>
      <joint name="goal_slidey" type="slide" pos="0 0 0" axis="0 1 0" range="-10.3213 10.3" damping="0.5"/>
      <joint name="goal_slidex" type="slide" pos="0 0 0" axis="1 0 0" range="-10.3213 10.3" damping="0.5"/>
    </body>
  </worldbody>

  <actuator>
    <motor joint="r_shoulder_pan_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_shoulder_lift_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_upper_arm_roll_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_elbow_flex_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_forearm_roll_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_wrist_flex_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_wrist_roll_joint" ctrlrange="-2.0 2.0" ctrllimited="true"/>
  </actuator>
</mujoco>

@HernandezEduin
Copy link
Author

rl-video-episode-1.mp4
rl-video-episode-8.mp4
rl-video-episode-0.mp4

I copied the previous xml, but instead of decreasing the density for the object, I increased it to 0.5.

@HernandezEduin
Copy link
Author

rl-video-episode-0.mp4
rl-video-episode-1.mp4
rl-video-episode-8.mp4

More videos of the same, but this time the arms tips colliding.

@pseudo-rnd-thoughts
Copy link
Member

@HernandezEduin I'm guessing you using the pretrained model to test with this. Once we have the final xml model complete, it would be helpful to retrain the agent for scratch on the new model and check that the agent will learn the correct model

@Kallinteris-Andreas
Copy link
Collaborator

"""
I copied the previous xml, but instead of decreasing the density for the object, I increased it to 0.5.
"""

I am not sure what xml file you are talking about, please use the "RCx" name (found at the top of the files) just post your RC

Thanks!

@pseudo-rnd-thoughts we can move to validation after we have finalized the model

@HernandezEduin
Copy link
Author

Sorry, I meant I used the RC4 and modified the object's cylinder's density to 0.5.

<geom rgba="1 1 1 1" type="cylinder" size="0.05 0.05 0.05" density="0.5" contype="1" conaffinity="0"/>

@Kallinteris-Andreas
Copy link
Collaborator

Kallinteris-Andreas commented Mar 22, 2024 •
edited
Loading

The only difference between RC4 and RC2 is the cylinder density
with RC4 having 0.01 and rc2 having 0.001
where with RC4 the object's mass corresponds to 7.85398163e-06 which should be easily moveable by the pusher arm if trained

So I believe we should move to validation
we will need to use the current "Pusher-v5" environment in the 1.0alpha1, but with the new xml_file (RC4).
For training, you can use any state-of-the-art algorithm you want (if you have a fast enough computer you can do multiple algorithms), but it must be a standard implementation from a library like SB3, cleanRL or others, 1mil steps over episodes should be enough.
If this results in the trained policy being able to "solve" the environment, then we can lock in the xml_file, if not we will have to test a new revision model version RC5.

Also after that we will have to define the reward_threshold parameter

@HernandezEduin do you want to do it?

@HernandezEduin
Copy link
Author

Sure, I've been using the CleanRL library implementation of the TD3 algorithm. I've just been reloading the weights of the model I had trained on the pusher-v3 on mujoco==2.3.7. I'll update my gymnasium to the current version and do the retraining then!

@Kallinteris-Andreas
Copy link
Collaborator

Ah wait, you've been using version 3 instead of version 4?

@HernandezEduin
Copy link
Author

*I meant pusher-v4, sorry

Btw, what version of Stable-Baseline3 is compatible with Gymnasium 1.0alpha1?

@Kallinteris-Andreas
Copy link
Collaborator

this one DLR-RM/stable-baselines3#1837

@HernandezEduin
Copy link
Author

So I've tried to train the pusher-v5 and v4 models using RC4 with various densities for the object, however, it doesn't seem to properly learn as it would with Gymnasium 0.29.1.

Maybe the quick fixes I'm doing on the TD3 file from the cleanrl and the adjustment to the record_video from SB3 aren't appropriate?

Gymnasium == 1.0.0.a1
stable_baselines3 == 2.3.0a4

@HernandezEduin
Copy link
Author

For reference, I modified the file stable_baselines3\common\vec_env\vec_video_recorder.py as follows
Instead of from gymnasium.wrappers.monitoring import video_recorder, I do from gymnasium.wrappers.rendering import RecordVideo as video_recorder

@Kallinteris-Andreas
Copy link
Collaborator

sb3 video record does not work, yet here is what I used:

    video_folder = "videos/"
    video_length = 1000
    VIDEO_NAME = f"{args.env}_{args.algo}_run_{args.run}"

    frame_list = []

    obs = model.get_env().reset()
    for _ in range(video_length + 1):
        action, _state = model.predict(obs, deterministic=True)
        obs, _, _, _ = model.get_env().step(action)
        frame_list.append(model.get_env().render())

    model.get_env().close()

    from moviepy.video.io.ImageSequenceClip import ImageSequenceClip
    import os
    clip = ImageSequenceClip(frame_list, fps=eval_env.metadata["render_fps"])
    #moviepy_logger = None if self.disable_logger else "bar"
    path = os.path.join(video_folder, f"{VIDEO_NAME}.mp4")
    clip.write_videofile(path, logger="bar")
    #clip.write_videofile(path, logger=moviepy_logger)
    exit()

This was referenced Apr 6, 2024
Open
Open
Merged
@Kallinteris-Andreas
Copy link
Collaborator 

<!--
Pusher model for `Pusher-v5` RC5, based on openai/gym/Pusher
modified by @kallinteris-Andreas
	- reduced body.object cylinder density 0.00001 -> 0.005
	- disabled body.object sphere (density 0.00001 -> 0.001)
-->
<mujoco model="arm3d">
  <compiler inertiafromgeom="true" angle="radian" coordinate="local"/>
  <option timestep="0.01" gravity="0 0 0" iterations="20" integrator="Euler" />

  <default>
    <joint armature='0.04' damping="1" limited="true"/>
    <geom friction=".8 .1 .1" density="300" margin="0.002" condim="1" contype="0" conaffinity="0"/>
  </default>

  <worldbody>
    <light diffuse=".5 .5 .5" pos="0 0 3" dir="0 0 -1"/>
    <geom name="table" type="plane" pos="0 0.5 -0.325" size="1 1 0.1" contype="1" conaffinity="1"/>

    <body name="r_shoulder_pan_link" pos="0 -0.6 0">
      <geom name="e1" type="sphere" rgba="0.6 0.6 0.6 1" pos="-0.06 0.05 0.2" size="0.05" />
      <geom name="e2" type="sphere" rgba="0.6 0.6 0.6 1" pos=" 0.06 0.05 0.2" size="0.05" />
      <geom name="e1p" type="sphere" rgba="0.1 0.1 0.1 1" pos="-0.06 0.09 0.2" size="0.03" />
      <geom name="e2p" type="sphere" rgba="0.1 0.1 0.1 1" pos=" 0.06 0.09 0.2" size="0.03" />
      <geom name="sp" type="capsule" fromto="0 0 -0.4 0 0 0.2" size="0.1" />
      <joint name="r_shoulder_pan_joint" type="hinge" pos="0 0 0" axis="0 0 1" range="-2.2854 1.714602" damping="1.0" />

      <body name="r_shoulder_lift_link" pos="0.1 0 0">
        <geom name="sl" type="capsule" fromto="0 -0.1 0 0 0.1 0" size="0.1" />
        <joint name="r_shoulder_lift_joint" type="hinge" pos="0 0 0" axis="0 1 0" range="-0.5236 1.3963" damping="1.0" />

        <body name="r_upper_arm_roll_link" pos="0 0 0">
          <geom name="uar" type="capsule" fromto="-0.1 0 0 0.1 0 0" size="0.02" />
          <joint name="r_upper_arm_roll_joint" type="hinge" pos="0 0 0" axis="1 0 0" range="-1.5 1.7" damping="0.1" />

          <body name="r_upper_arm_link" pos="0 0 0">
            <geom name="ua" type="capsule" fromto="0 0 0 0.4 0 0" size="0.06" />

            <body name="r_elbow_flex_link" pos="0.4 0 0">
              <geom name="ef" type="capsule" fromto="0 -0.02 0 0.0 0.02 0" size="0.06" />
              <joint name="r_elbow_flex_joint" type="hinge" pos="0 0 0" axis="0 1 0" range="-2.3213 0" damping="0.1" />

              <body name="r_forearm_roll_link" pos="0 0 0">
                <geom name="fr" type="capsule" fromto="-0.1 0 0 0.1 0 0" size="0.02" />
                <joint name="r_forearm_roll_joint" type="hinge" limited="true" pos="0 0 0" axis="1 0 0" damping=".1" range="-1.5 1.5"/>

                <body name="r_forearm_link" pos="0 0 0">
                  <geom name="fa" type="capsule" fromto="0 0 0 0.291 0 0" size="0.05" />

                  <body name="r_wrist_flex_link" pos="0.321 0 0">
                    <geom name="wf" type="capsule" fromto="0 -0.02 0 0 0.02 0" size="0.01" />
                    <joint name="r_wrist_flex_joint" type="hinge" pos="0 0 0" axis="0 1 0" range="-1.094 0" damping=".1" />

                    <body name="r_wrist_roll_link" pos="0 0 0">
                      <joint name="r_wrist_roll_joint" type="hinge" pos="0 0 0" limited="true" axis="1 0 0" damping="0.1" range="-1.5 1.5"/>
                      <body name="tips_arm" pos="0 0 0">
                        <geom name="tip_arml" type="sphere" pos="0.1 -0.1 0." size="0.01" />
                        <geom name="tip_armr" type="sphere" pos="0.1 0.1 0." size="0.01" />
                      </body>
                      <geom type="capsule" fromto="0 -0.1 0. 0.0 +0.1 0" size="0.02" contype="1" conaffinity="1" />
                      <geom type="capsule" fromto="0 -0.1 0. 0.1 -0.1 0" size="0.02" contype="1" conaffinity="1" />
                      <geom type="capsule" fromto="0 +0.1 0. 0.1 +0.1 0." size="0.02" contype="1" conaffinity="1" />
                    </body>
                  </body>
                </body>
              </body>
            </body>
          </body>
        </body>
      </body>
    </body>

    <!--<body name="object" pos="0.55 -0.3 -0.275" >-->
    <body name="object" pos="0.45 -0.05 -0.275" >
      <!-- <geom rgba="1 1 1 0" type="sphere" size="0.05 0.05 0.05" density="0.001" conaffinity="0"/> -->
      <geom rgba="1 1 1 1" type="cylinder" size="0.05 0.05 0.05" density="0.005" contype="1" conaffinity="0"/>
      <joint name="obj_slidey" type="slide" pos="0 0 0" axis="0 1 0" range="-10.3213 10.3" damping="0.5"/>
      <joint name="obj_slidex" type="slide" pos="0 0 0" axis="1 0 0" range="-10.3213 10.3" damping="0.5"/>
    </body>

    <body name="goal" pos="0.45 -0.05 -0.3230">
      <geom rgba="1 0 0 1" type="cylinder" size="0.08 0.001 0.1" density='0.00001' contype="0" conaffinity="0"/>
      <joint name="goal_slidey" type="slide" pos="0 0 0" axis="0 1 0" range="-10.3213 10.3" damping="0.5"/>
      <joint name="goal_slidex" type="slide" pos="0 0 0" axis="1 0 0" range="-10.3213 10.3" damping="0.5"/>
    </body>
  </worldbody>

  <actuator>
    <motor joint="r_shoulder_pan_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_shoulder_lift_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_upper_arm_roll_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_elbow_flex_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_forearm_roll_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_wrist_flex_joint" ctrlrange="-2.0 2.0" ctrllimited="true" />
    <motor joint="r_wrist_roll_joint" ctrlrange="-2.0 2.0" ctrllimited="true"/>
  </actuator>
</mujoco>

I trained pusher-v5rc5.xml
for 2 mil steps with SB3's SAC and we can see that the cylidrical object is light enough to be moved by the pusher arm, the objects's mass is 7.85398163e-06 kg, this should be good enough for release

next is 10 episodes per 10 polices trained

Pusher-v5rc5_SAC_run_0.mp4
Pusher-v5rc5_SAC_run_1.mp4
Pusher-v5rc5_SAC_run_2.mp4
Pusher-v5rc5_SAC_run_3.mp4
Pusher-v5rc5_SAC_run_4.mp4
Pusher-v5rc5_SAC_run_5.mp4
Pusher-v5rc5_SAC_run_6.mp4
Pusher-v5rc5_SAC_run_7.mp4
Pusher-v5rc5_SAC_run_8.mp4
Pusher-v5rc5_SAC_run_9.mp4

@pseudo-rnd-thoughts
Copy link
Member

@Kallinteris-Andreas wjat are the performance curves compared to pusher v4 and mujoco < 3

@Ejafa
Copy link

Ejafa commented Apr 10, 2024 •
edited
Loading

is it okay? I saw somehow it's working

<!-- <body name="object" pos="0.55 -0.3 -0.275" > -->
    <body name="object" pos="0.45 -0.05 -0.275" >
      <geom rgba="1 1 1 0" type="sphere" `size="1` 0.05 0.05" density="0.00001" conaffinity="0"/>
      <geom rgba="1 1 1 1" type="cylinder" size="0.05 0.05 0.05" density="0.00001" contype="1" conaffinity="0"/>
      <joint name="obj_slidey" type="slide" pos="0 0 0" axis="0 1 0" range="-10.3213 10.3" damping="0.5"/>
      <joint name="obj_slidex" type="slide" pos="0 0 0" axis="1 0 0" range="-10.3213 10.3" damping="0.5"/>
    </body>

    <body name="goal" pos="0.45 -0.05 -0.3230">
      <geom rgba="1 0 0 1" type="cylinder" size="0.08 0.001 0.1" density='0.00001' contype="0" conaffinity="0"/>
      <joint name="goal_slidey" type="slide" pos="0 0 0" axis="0 1 0" range="-10.3213 10.3" damping="0.5"/>
      <joint name="goal_slidex" type="slide" pos="0 0 0" axis="1 0 0" range="-10.3213 10.3" damping="0.5"/>
    </body>
  </worldbody>

@Kallinteris-Andreas
Copy link
Collaborator

t_ret
SB3 fork with support for gymnasium==1.0.0a1 was used
the environments are:

  • Pusher-v5 the existing implementation of Pusher in gymnasium==1.0.0a1 with mujoco==2.3.7
  • Pusher-v5-rc4 and -rc5 is the existing implementation of Pusher but with the new models

Analysis

the rc4 model is close enough to the original model, that it should be used as the for the Pusher-v5 release

@HernandezEduin are you also running tests?

@pseudo-rnd-thoughts
Copy link
Member

@Kallinteris-Andreas the results look good, I'm happy to merge this new model to replace the old v5 model

@Kallinteris-Andreas Kallinteris-Andreas mentioned this issue Apr 15, 2024
Merged
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update Pusher-v5 model to one that is compatible with mujoco>=3 Farama-Foundation/Gymnasium
4 participants
@pseudo-rnd-thoughts @Ejafa @Kallinteris-Andreas @HernandezEduin