Gradient Boosting Reinforcement Learning (GBRL)

GBRL is a Python-based Gradient Boosting Trees (GBT) library, similar to popular packages such as XGBoost, CatBoost, but specifically designed and optimized for reinforcement learning (RL). GBRL is implemented in C++/CUDA aimed to seamlessly integrate within popular RL libraries.

Overview

GBRL adapts the power of Gradient Boosting Trees to the unique challenges of RL environments, including non-stationarity and the absence of predefined targets. The following diagram illustrates how GBRL uses gradient boosting trees in RL:

GBRL features a shared tree-based structure for policy and value functions, significantly reducing memory and computational overhead, enabling it to tackle complex, high-dimensional RL problems.

Key Features:

• GBT Tailored for RL: GBRL adapts the power of Gradient Boosting Trees to the unique challenges of RL environments, including non-stationarity and the absence of predefined targets.
• Optimized Actor-Critic Architecture: GBRL features a shared tree-based structure for policy and value functions. This significantly reduces memory and computational overhead, enabling it to tackle complex, high-dimensional RL problems.
• Hardware Acceleration: GBRL leverages CUDA for hardware-accelerated computation, ensuring efficiency and speed.
• Seamless Integration: GBRL is designed for easy integration with popular RL libraries. We implemented GBT-based actor-critic algorithm implementations (A2C, PPO, and AWR) in stable_baselines3 GBRL_SB3.

Performance

The following results, obtained using the GBRL_SB3 repository, demonstrate the performance of PPO with GBRL compared to neural-networks across various scenarios and environments:

Getting started

Prerequisites

• Python 3.9 or higher
• LLVM and OpenMP (macOS).

Installation

To install GBRL via pip, use the following command:

pip install gbrl

For further installation details and dependencies see the documentation.

Usage Example

For a detailed usage example, see tutorial.ipynb

Current Supported Features

Tree Fitting

• Greedy (Depth-wise) tree building - (CPU/GPU)
• Oblivious (Symmetric) tree building - (CPU/GPU)
• L2 split score - (CPU/GPU)
• Cosine split score - (CPU/GPU)
• Uniform based candidate generation - (CPU/GPU)
• Quantile based candidate generation - (CPU/GPU)
• Supervised learning fitting / Multi-iteration fitting - (CPU/GPU)
  • MultiRMSE loss (only)
• Categorical inputs
• Input feature weights - (CPU/GPU)

GBT Inference

• SGD optimizer - (CPU/GPU)
• ADAM optimizer - (CPU only)
• Control Variates (gradient variance reduction technique) - (CPU only)
• Shared Tree for policy and value function - (CPU/GPU)
• Linear and constant learning rate scheduler - (CPU/GPU only constant)
• Support for up to two different optimizers (e.g, policy/value) - **(CPU/GPU if both are SGD)
• SHAP value calculation

Documentation

For comprehensive documentation, visit the GBRL documentation.

Citation

@article{gbrl,
  title={Gradient Boosting Reinforcement Learning},
  author={Benjamin Fuhrer, Chen Tessler, Gal Dalal},
  year={2024},
  eprint={2407.08250},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2407.08250}, 
}

Licenses

Copyright © 2024, NVIDIA Corporation. All rights reserved.

This work is made available under the NVIDIA Source Code License-NC. Click here. to view a copy of this license.