🕊️ Migration PPOv2 -> PPO

[qgallouedec]

What does this PR do?

Follows #2016

Fixes # (issue)

It basically:

• Deletes PPO (including trainer, config, tests, examples, but not all, see my comment)
• Renames PPOv2 to PPO (in trainer, config, examples, tests, doc, etc.)
• Creates an alias PPOv2 for PPO

I've done my best, but it may break some code. To put things in perspective, PPO isn't as popular as it used to be, at the time of writing, the last PPO model created on the hub was two weeks ago.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

While working on this, I noticed that most of our documentation and notebooks are based on PPO and may be outdated, relying on older datasets, models, references, etc. I believe updating all of our examples and guides falls outside the scope of this PR.

I recommend the following:

• For now, keep the existing outdated documentation, guides, and notebooks as they are.
• Begin a separate effort to rewrite these materials in future PR(s), primarily starting from scratch.

[qgallouedec]

This one is just the renaming of ppov2_config. the diff view makes no sense

[qgallouedec]

This one is just the tests/test_ppov2_trainer.py renamed. The diff view makes no sense

[edbeeching]

LGTM apart from a comment about the examples and 1 change

[lewtun]

Thanks for the migration @qgallouedec ! Overall LGTM with some nits about the docs since we can take this opportunity to nuke a bunch of outdated stuff

[lewtun]

Perhaps it's a bug in the GitHub diff, but this looks repeated from above

[sygi]

Would it be possible to keep the "old" PPO code around until the new trainer achieves feature parity (in particular peft and arbitrary reward support) with the old one?

[Void-025]

I would also like to have the old PPO code kept around, I was previously using the old ppotrainer.step() function to provide my own reward values for each query/response pair, but there doesn't seem to be an equivalent in PPOv2, unless I'm missing some other way of doing this?

[qgallouedec]

You should use trl==0.11