Remove gym dependency

[ernestum]

Description

Removes the dependency to gym in favor of gymnasium.

Motivation and Context

Fixes #398

• I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (update in the documentation)

Checklist:

• I've read the CONTRIBUTION guide (required)
• I have updated the changelog accordingly (required).
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.
• I have reformatted the code using make format (required)
• I have checked the codestyle using make check-codestyle and make lint (required)
• I have ensured make pytest and make type both pass. (required)

Note: we are using a maximum length of 127 characters per line

[ernestum]

Note: on this branch (maybe on master as well) I still have the issue, that I can't push models that use a VecNormalizeWrapper to the hub. Somehow the wrapper sets the render_mode to None.

To reproduce:

python -m rl_zoo3.train --algo ppo --env MountainCarContinuous-v0 -f reproduce_bug 
python -m rl_zoo3.push_to_hub --algo ppo --env MountainCarContinuous-v0 -f reproduce_bug --organization <YOUR ORGA>

gives me the followign output:

[...]
Saving model to: hub/ppo-MountainCarContinuous-v0/ppo-MountainCarContinuous-v0
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/m/Documents/CHAI/rl-baselines3-zoo/rl_zoo3/push_to_hub.py", line 411, in <module>
    package_to_hub(
  File "/home/m/Documents/CHAI/rl-baselines3-zoo/rl_zoo3/push_to_hub.py", line 248, in package_to_hub
    _generate_replay(model, eval_env, video_length, is_deterministic, repo_local_path)
  File "/home/m/Documents/CHAI/rl-baselines3-zoo/venv/lib/python3.8/site-packages/huggingface_sb3/push_to_hub.py", line 133, in _generate_replay
    env = VecVideoRecorder(
  File "/home/m/Documents/CHAI/rl-baselines3-zoo/venv/lib/python3.8/site-packages/stable_baselines3/common/vec_env/vec_video_recorder.py", line 50, in __init__
    assert self.env.render_mode == "rgb_array", f"The render_mode must be 'rgb_array', not {self.env.render_mode}"
AssertionError: The render_mode must be 'rgb_array', not None
Exception ignored in: <function VecVideoRecorder.__del__ at 0x7f2283824e50>
Traceback (most recent call last):
  File "/home/m/Documents/CHAI/rl-baselines3-zoo/venv/lib/python3.8/site-packages/stable_baselines3/common/vec_env/vec_video_recorder.py", line 113, in __del__
    self.close_video_recorder()
  File "/home/m/Documents/CHAI/rl-baselines3-zoo/venv/lib/python3.8/site-packages/stable_baselines3/common/vec_env/vec_video_recorder.py", line 103, in close_video_recorder
    if self.recording:
  File "/home/m/Documents/CHAI/rl-baselines3-zoo/venv/lib/python3.8/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 392, in __getattr__
    return self.getattr_recursive(name)
  File "/home/m/Documents/CHAI/rl-baselines3-zoo/venv/lib/python3.8/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 415, in getattr_recursive
    attr = self.venv.getattr_recursive(name)
  File "/home/m/Documents/CHAI/rl-baselines3-zoo/venv/lib/python3.8/site-packages/stable_baselines3/common/vec_env/base_vec_env.py", line 417, in getattr_recursive
    attr = getattr(self.venv, name)
AttributeError: 'DummyVecEnv' object has no attribute 'recording'

[ernestum]

Right now the pipeline fails because mypy does not like that we assign None to packages that can't be imported in import_envs.py. Now the question for me is:

1. Why was this not detected before?
2. Why can't I reproduce this locally with mypy?

[araffin]

Note: on this branch (maybe on master as well) I still have the issue, that I can't push models that use a VecNormalizeWrapper to the hub.

Should be fixed with DLR-RM/stable-baselines3#1671 (I will give it a try soon)

mypy does not like that we assign None to packages that can't be imported in import_envs.py. Now the question for me is:

let me check, might be due to an update to mypy or other packages.

[araffin]

Note: on this branch (maybe on master as well) I still have the issue, that I can't push models that use a VecNormalizeWrapper to the hub. Somehow the wrapper sets the render_mode to None.

I cannot reproduce with SB3 master version, so I guess it is fixed =)

[araffin]

Right now the pipeline fails because mypy does not like that we assign None to packages that can't be imported in import_envs.py. Now the question for me is:

I also have no idea for those questions, but I fixed mypy check.

[araffin]

LGTM, thanks =)