[Doc] More depth in VMAS docs

[matteobettini]

No description provided.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/1802

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (6 Unrelated Failures)

As of commit 847cc13 with merge base b632be9 ():

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Continuous Benchmark (PR) / CPU Pytest benchmark (gh)
  Workflow failed! Resource not accessible by integration
• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  Workflow failed! Resource not accessible by integration
• Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh)
  test/test_libs.py::TestGym::test_vecenvs_env[CartPole-v1]
• Unit-tests on Linux / tests-stable-gpu (3.8, 11.8) / linux-job (gh)
  test/test_rb.py::TestStorages::test_storage_state_dict[torchsnapshot-False-memmap-memmap]
• Unit-tests on Windows / unittests-gpu / windows-job (gh)
  ##[error]The operation was canceled.

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Habitat Tests on Linux / tests (3.9, 11.6) / linux-job (gh)
  test/test_libs.py::TestHabitat::test_habitat_render[False-HabitatPick-v0]

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[vmoens]

Thanks so much for this! I left some comments

[vmoens]

just to make sure: even when rewards can be consumed there is no max_steps predefined right? All envs are not truncated?

[matteobettini]

Yes, max_steps is additional to the (eventually implemented) scenario "done" function

[vmoens]

what does it mean that an env can define a done funtion? Some envs are never done? Can I choose if they have a done function?

[matteobettini]

Yes, each scenario optionally implements "done" function. Some scenarios do not present that yes. Those scenarios would be eventually terminated only if you use max_steps.

You can choose to implement the scenario done function if you implement a scenario yourself, otherwise it is a property of the scenario

[vmoens]

got it
What is the done function in practice?
How does it relate to the "done" key?
I'm asking this to clarify if it's required to mention a function that we do not see appearing anywhere or if we can just say that env can be non-terminating unless max_steps is provided.

[matteobettini]

I am also pro to not mention it. I just did cause I did not know how much depth is required here.

we can definitely just say what you proposed. Seems better to me to.

We can say something like: some scnearios are terminating and some not. In all cases max_steps provides an additional termination condition

[vmoens]

I don't think users need to know the inner machinery but they should know precisely how changing a kwarg will impact what they get from the environment.

[matteobettini]

Changed

[vmoens]

I still think the current formulation is confusing, because even if we don't set the truncated key this is still technically a truncation. What about

Unlike gym's `TimeLimit` transform or torchrl's :class:`~torchrl.envs.transforms.StepCounter`, this argument will not set a `"truncated"` entry in the tensordict.

[vmoens]

LGTM