[WIP] Compute lp during loss execution

[vmoens]

Stack from ghstack (oldest at bottom):

• -> [WIP] Compute lp during loss execution #2688
• [Refactor] Use default device instead of CPU in losses #2687
• [Feature] Make PPO compatible with composite actions and log-probs #2665

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2688

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• Failures because of deprecated version of actions/download-artifact: v3

❌ 19 New Failures, 1 Unrelated Failure

As of commit 443330b with merge base ed656a1 ():

NEW FAILURES - The following jobs have failed:

• Build Windows Wheels / pytorch/rl / upload / wheel-py3_9-cuda11_8 (gh)
  Unable to download artifact(s): Artifact not found for name: pytorch_rl__3.9_cu118_
• Build Windows Wheels / pytorch/rl / upload / wheel-py3_9-cuda12_4 (gh)
  Unable to download artifact(s): Artifact not found for name: pytorch_rl__3.9_cu124_
• Build Windows Wheels / pytorch/rl / upload / wheel-py3_9-cuda12_6 (gh)
  Unable to download artifact(s): Artifact not found for name: pytorch_rl__3.9_cu126_
• Continuous Benchmark (PR) / CPU Pytest benchmark (gh)
  Process completed with exit code 1.
• Continuous Benchmark (PR) / GPU Pytest benchmark (gh)
  Process completed with exit code 1.
• Habitat Tests on Linux / tests (3.9, 12.1) / linux-job (gh)
  AttributeError: _ARRAY_API not found
• Lint / python-source-and-configs / linux-job (gh)
  torchrl/objectives/ppo.py:8:1: F401 'warnings' imported but unused
• RLHF Tests on Linux / unittests (3.9, 12.1) / linux-job (gh)
  AttributeError: 'ModuleList' object has no attribute 'dist_params_keys'
• SOTA Tests on Linux / tests (3.9, 12.1) / linux-job (gh)
  AttributeError: 'ModuleList' object has no attribute 'dist_params_keys'
• Unit-tests on Linux / tests-cpu (3.10) / linux-job (gh)
  test/test_cost.py::TestPPO::test_ppo_composite_dists
• Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh)
  test/test_cost.py::TestPPO::test_ppo_composite_dists
• Unit-tests on Linux / tests-cpu (3.12) / linux-job (gh)
  test/test_cost.py::TestPPO::test_ppo_composite_dists
• Unit-tests on Linux / tests-cpu (3.9) / linux-job (gh)
  test/test_cost.py::TestPPO::test_ppo_composite_dists
• Unit-tests on Linux / tests-cpu-oldget (3.12) / linux-job (gh)
  test/test_cost.py::TestPPO::test_ppo_composite_dists
• Unit-tests on Linux / tests-gpu (3.11, 12.1) / linux-job (gh)
  test/test_loggers.py::TestMLFlowLogger::test_log_video[steps1]
• Unit-tests on Linux / tests-olddeps (3.8, 11.6) / linux-job (gh)
  test/test_storage_map.py::TestMCTSForest::test_forest_parent[simple-False]
• Unit-tests on Linux / tests-optdeps (3.11, 12.1) / linux-job (gh)
  test/test_cost.py::TestPPO::test_ppo_composite_dists
• Unit-tests on Linux / tests-stable-gpu (3.10, 11.8) / linux-job (gh)
  test/test_cost.py::TestPPO::test_ppo_composite_dists
• Unit-tests on Windows / unittests-cpu / windows-job (gh)
  Process completed with exit code 1.

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Build Windows Wheels / pytorch/rl / upload / wheel-py3_9-cpu (gh) (trunk failure)

This comment was automatically generated by Dr. CI and updates every 15 minutes.