RL Algorithms and Environments by PyTorch

Welcome to the RL Algorithms and Environments (RLAlgoEnv) repository! This project provides a collection of reinforcement learning (RL) algorithms implemented in PyTorch and some customized or packed-up environments.

Table of Contents

• Requirements
• Algorithms
• Environments

Requirements

• The codes are well tested on pytorch==2.0.1+cu117.
• Install all dependent packages:

  pip install -r requirements.txt

• The project is based the gymnasium>=0.29.1 package, to render the mujoco, robotics, etc. environments, you need to modify the site-packages\gymnasium\envs\mujoco\mujoco_rendering.py file: replace the solver_iter (at around line #593) to solver_niter.
• The running scripts can be run directly in PyCharm, or you may need to execute: export PYTHONPATH=<path to RLEnvsAlgos>:$PYTHONPATH.

Algorithms

The project implements RL algorithms in separate independent classes, easy to read and modify.

The implementation of these algorithms is primarily based on the CleanRL library, which is also an excellent resource that we recommend for reference.

Algorithm	Description	Auther & Year	Discrete Control	Continuous Control
DQN	An enhanced version of Deep Q-Networks algorithm.	Mnih et al., 2015	✔️	❌
CategoricalDQN	An extension of DQN with categorical distributional Q-learning.	Bellemare et al., 2017	✔️	❌
NoisyNet (DQN)	An extension of DQN with noisy networks for exploration.	Fortunato et al., 2019	✔️	❌
PPO	Proximal Policy Optimization.	Schulman et al., 2017	✔️	✔️ ️
RPO	An improved version of PPO.	Md Masudur Rahman and Yexiang Xue, 2023	✔️	✔️
RND	Random Network Distillation, extended from PPO.	Burda et al., 2018	✔️	✔️
DDPG	Deep Deterministic Policy Gradient algorithm.	Silver et al., 2014	❌	✔️
TD3	Twin Delayed DDPG, an improved version of DDPG.	Fujimoto et al., 2018	❌	✔️
SAC	Soft Actor-Critic.	Haarnoja et al., 2018	✔️	✔️

Algorithms to be Implemented

• Soft Q-Learning (SQL)
• Advantage Actor-Critic (A2C)
• Asynchronous Advantage Actor-Critic (A3C)
• and more...

Running Scripts

We provide a variety of running scripts for different algorithms and environments. You can find them here.

• DQN algorithm (can only work on discrete action spaces):
  • DQN in classic control
  • DQN in Atari games
  • DQN in MiniGrid environments
• NoisyNet-DQN algorithm:
  • NoisyNet-DQN in classic control
  • NoisyNet-DQN MiniGrid environments
• PPO algorithm:
  • PPO in classic control
  • PPO in Atari games
  • PPO in continuous control (e.g., Mujoco)
  • PPO in robotics control (e.g., gymnasium-robotics, MyFetchRobot)
• RPO algorithm:
  • RPO in continuous control
• RND algorithm:
  • RND in continuous control
  • RND in robotics control (e.g., gymnasium-robotics, MyFetchRobot)
  • RND in MiniGrid environments
  • RND in Atari environments
• DDPG algorithm (can only work on continuous action spaces):
  • DDPG in continuous control
• TD3 algorithm (can only work on continuous action spaces):
  • TD3 in continuous control
  • TD3 in robotics control (e.g., gymnasium-robotics, MyFetchRobot)
• SAC algorithm:
  • SAC in continuous control
  • SAC in Atari games
  • SAC in robotics control (e.g., gymnasium-robotics, MyFetchRobot)
  • SAC in MiniGrid environments

Environments

We pack up and customize a variety of environments for testing and benchmarking RL algorithms. All environments packages can be found in RLEnvs folder.

• gymnasium: the OpenAI gymnasium library.
• MyMiniGrid: based on the MiniGrid environment, customized some wrappers and self-designed environments.
• MyPandaRobot: based on the panda-gym environment, customized some self-designed environments.
• MyFetchRobot: based on the gymnasium-robotics library, customized the reward function to only give sparse and delayed rewards for four FetchRobot tasks.
• MyMujoco: based on the gymnasium-mujoco library, customized the reward function to only give sparse and delayed rewards.
• MyMiniWorld: based on the MiniWorld environment, customized some self-designed and sparse-reward environments.

Templates

We provide some templates to interact with the environments:

• Using gymnasium
• Using gymnasium-robotics
• Using MyMiniGrid
• Using MyPandaRobot
• Using MyFetchRobot
• Using MyMujoco

Contributing

We welcome contributions!

Actually, the codes are not thoroughly tested, so we sincerely invite you to help us update the repository. If you have improvements or bug fixes, please feel free to open an issue or a pull request. Thanks in advance for your help!