[Feature] TensorDictMap Query module

[vmoens]

Stack from ghstack (oldest at bottom):

• [Feature] MCTS policy #2359
• [Feature] MCTS Scoring functions #2358
• [Feature] TensorSpec.enumerate() #2354
• [Feature] MCTSForest #2307
• [Feature] TensorDictMap #2306
• -> [Feature] TensorDictMap Query module #2305
• [Feature] TensorDictMap hashing functions #2304

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/rl/2305

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 2 Cancelled Jobs, 15 Unrelated Failures

As of commit 71ceadb with merge base 35a1c5b ():

NEW FAILURES - The following jobs have failed:

• Habitat Tests on Linux / tests (3.9, 12.1) / linux-job (gh)
  RuntimeError: Command docker exec -t 797d557395e79d3f44bd2cbf03073a9db955cb825cbeb14209b0bfc216a6bbe0 /exec failed with exit code 139
• Wheels / build-wheel-windows (3.8, 3.8) (gh)
  Process completed with exit code 1.
• Wheels / build-wheel-windows (3.9, 3.9) (gh)
  Process completed with exit code 1.

CANCELLED JOBS - The following jobs were cancelled. Please retry:

• Wheels / build-wheel-windows (3.10, 3.10.3) (gh)
• Wheels / build-wheel-windows (3.12, 3.12) (gh)

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Build Windows Wheels / pytorch/rl (pytorch/rl, python packaging/wheel/relocate.py, test/smoke_test.py, torchrl) / upload / wheel-py3_9-cpu (gh) (similar failure)
  Unable to find any artifacts for the associated workflow
• Build Windows Wheels / pytorch/rl (pytorch/rl, python packaging/wheel/relocate.py, test/smoke_test.py, torchrl) / upload / wheel-py3_9-cuda11_8 (gh) (similar failure)
  Unable to find any artifacts for the associated workflow
• Wheels / build-wheel-windows (3.11, 3.11) (gh) (matched win rule in flaky-rules.json)
  ##[error]The operation was canceled.

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Build Windows Wheels / pytorch/rl (pytorch/rl, python packaging/wheel/relocate.py, test/smoke_test.py, torchrl) / upload / wheel-py3_9-cuda12_1 (gh) (trunk failure)
  Unable to find any artifacts for the associated workflow
• Build Windows Wheels / pytorch/rl (pytorch/rl, python packaging/wheel/relocate.py, test/smoke_test.py, torchrl) / upload / wheel-py3_9-cuda12_4 (gh) (trunk failure)
  Unable to find any artifacts for the associated workflow
• Unit-tests on Linux / tests-cpu (3.10) / linux-job (gh) (trunk failure)
  test/test_transforms.py::TestRenameTransform::test_rename_done_reward[False]
• Unit-tests on Linux / tests-cpu (3.11) / linux-job (gh) (trunk failure)
  test/test_transforms.py::TestRenameTransform::test_rename_done_reward[False]
• Unit-tests on Linux / tests-cpu (3.12) / linux-job (gh) (trunk failure)
  test/test_transforms.py::TestRenameTransform::test_rename_done_reward[False]
• Unit-tests on Linux / tests-cpu (3.8) / linux-job (gh) (trunk failure)
  test/test_transforms.py::TestRenameTransform::test_rename_done_reward[False]
• Unit-tests on Linux / tests-cpu (3.9) / linux-job (gh) (trunk failure)
  test/test_transforms.py::TestRenameTransform::test_rename_done_reward[False]
• Unit-tests on Linux / tests-gpu (3.11, 12.1) / linux-job (gh) (trunk failure)
  test/test_transforms.py::TestRenameTransform::test_rename_done_reward[False]
• Unit-tests on Linux / tests-olddeps (3.8, 11.6) / linux-job (gh) (trunk failure)
  test/test_transforms.py::TestRenameTransform::test_rename_done_reward[False]
• Unit-tests on Linux / tests-optdeps (3.10, 12.1) / linux-job (gh) (trunk failure)
  test/test_transforms.py::TestRenameTransform::test_rename_done_reward[False]
• Unit-tests on Linux / tests-stable-gpu (3.10, 11.8) / linux-job (gh) (trunk failure)
  test/test_transforms.py::TestRenameTransform::test_rename_done_reward[False]
• Unit-tests on Windows / unittests-cpu / windows-job (gh) (trunk failure)
  Process completed with exit code 1.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of CPU Benchmark Tests

Total Benchmarks: 91. Improved: $\large\color{#35bf28}3$. Worsened: $\large\color{#d91a1a}6$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_single	60.3390ms	58.8772ms	16.9845 Ops/s	17.1476 Ops/s	$\color{#d91a1a}-0.95\%$
test_sync	33.1117ms	31.4462ms	31.8003 Ops/s	26.1018 Ops/s	$\textbf{\color{#35bf28}+21.83\%}$
test_async	87.8023ms	30.4040ms	32.8904 Ops/s	33.2748 Ops/s	$\color{#d91a1a}-1.16\%$
test_simple	0.4767s	0.4077s	2.4526 Ops/s	2.4249 Ops/s	$\color{#35bf28}+1.14\%$
test_transformed	0.6258s	0.5651s	1.7697 Ops/s	1.7534 Ops/s	$\color{#35bf28}+0.93\%$
test_serial	1.3072s	1.2498s	0.8001 Ops/s	0.7864 Ops/s	$\color{#35bf28}+1.75\%$
test_parallel	1.1643s	1.0967s	0.9118 Ops/s	0.9113 Ops/s	$\color{#35bf28}+0.06\%$
test_step_mdp_speed[True-True-True-True-True]	0.2745ms	24.1854μs	41.3472 KOps/s	41.5845 KOps/s	$\color{#d91a1a}-0.57\%$
test_step_mdp_speed[True-True-True-True-False]	62.4960μs	14.2443μs	70.2034 KOps/s	70.7626 KOps/s	$\color{#d91a1a}-0.79\%$
test_step_mdp_speed[True-True-True-False-True]	40.9260μs	14.1668μs	70.5876 KOps/s	71.7021 KOps/s	$\color{#d91a1a}-1.55\%$
test_step_mdp_speed[True-True-True-False-False]	52.4680μs	8.1066μs	123.3570 KOps/s	121.7608 KOps/s	$\color{#35bf28}+1.31\%$
test_step_mdp_speed[True-True-False-True-True]	61.1650μs	25.8320μs	38.7117 KOps/s	39.0522 KOps/s	$\color{#d91a1a}-0.87\%$
test_step_mdp_speed[True-True-False-True-False]	40.2950μs	15.5569μs	64.2801 KOps/s	64.4445 KOps/s	$\color{#d91a1a}-0.26\%$
test_step_mdp_speed[True-True-False-False-True]	48.1800μs	15.6001μs	64.1020 KOps/s	64.9707 KOps/s	$\color{#d91a1a}-1.34\%$
test_step_mdp_speed[True-True-False-False-False]	35.6870μs	9.5891μs	104.2853 KOps/s	104.9154 KOps/s	$\color{#d91a1a}-0.60\%$
test_step_mdp_speed[True-False-True-True-True]	69.1090μs	27.1902μs	36.7779 KOps/s	36.6058 KOps/s	$\color{#35bf28}+0.47\%$
test_step_mdp_speed[True-False-True-True-False]	38.8330μs	17.0893μs	58.5162 KOps/s	59.0397 KOps/s	$\color{#d91a1a}-0.89\%$
test_step_mdp_speed[True-False-True-False-True]	50.8350μs	15.3384μs	65.1957 KOps/s	64.3123 KOps/s	$\color{#35bf28}+1.37\%$
test_step_mdp_speed[True-False-True-False-False]	32.3600μs	9.6158μs	103.9955 KOps/s	103.7676 KOps/s	$\color{#35bf28}+0.22\%$
test_step_mdp_speed[True-False-False-True-True]	70.5410μs	28.3290μs	35.2995 KOps/s	34.8286 KOps/s	$\color{#35bf28}+1.35\%$
test_step_mdp_speed[True-False-False-True-False]	49.5020μs	18.3262μs	54.5668 KOps/s	54.1123 KOps/s	$\color{#35bf28}+0.84\%$
test_step_mdp_speed[True-False-False-False-True]	56.2850μs	16.8429μs	59.3722 KOps/s	59.8452 KOps/s	$\color{#d91a1a}-0.79\%$
test_step_mdp_speed[True-False-False-False-False]	42.4790μs	10.9027μs	91.7204 KOps/s	91.6066 KOps/s	$\color{#35bf28}+0.12\%$
test_step_mdp_speed[False-True-True-True-True]	61.1050μs	27.4096μs	36.4835 KOps/s	36.5433 KOps/s	$\color{#d91a1a}-0.16\%$
test_step_mdp_speed[False-True-True-True-False]	53.1590μs	17.0427μs	58.6760 KOps/s	58.8548 KOps/s	$\color{#d91a1a}-0.30\%$
test_step_mdp_speed[False-True-True-False-True]	57.2370μs	17.8292μs	56.0877 KOps/s	56.0744 KOps/s	$\color{#35bf28}+0.02\%$
test_step_mdp_speed[False-True-True-False-False]	33.1220μs	10.8499μs	92.1664 KOps/s	92.2295 KOps/s	$\color{#d91a1a}-0.07\%$
test_step_mdp_speed[False-True-False-True-True]	78.7270μs	28.6236μs	34.9362 KOps/s	34.9822 KOps/s	$\color{#d91a1a}-0.13\%$
test_step_mdp_speed[False-True-False-True-False]	43.4510μs	18.3026μs	54.6370 KOps/s	54.3562 KOps/s	$\color{#35bf28}+0.52\%$
test_step_mdp_speed[False-True-False-False-True]	83.0740μs	19.2396μs	51.9763 KOps/s	52.0869 KOps/s	$\color{#d91a1a}-0.21\%$
test_step_mdp_speed[False-True-False-False-False]	35.9670μs	12.2043μs	81.9385 KOps/s	81.5411 KOps/s	$\color{#35bf28}+0.49\%$
test_step_mdp_speed[False-False-True-True-True]	76.8009ms	33.1598μs	30.1570 KOps/s	33.2194 KOps/s	$\textbf{\color{#d91a1a}-9.22\%}$
test_step_mdp_speed[False-False-True-True-False]	64.7900μs	19.5214μs	51.2257 KOps/s	50.2446 KOps/s	$\color{#35bf28}+1.95\%$
test_step_mdp_speed[False-False-True-False-True]	58.7490μs	19.2291μs	52.0046 KOps/s	52.4927 KOps/s	$\color{#d91a1a}-0.93\%$
test_step_mdp_speed[False-False-True-False-False]	48.1200μs	11.9912μs	83.3948 KOps/s	82.3527 KOps/s	$\color{#35bf28}+1.27\%$
test_step_mdp_speed[False-False-False-True-True]	73.3970μs	31.1973μs	32.0540 KOps/s	32.0798 KOps/s	$\color{#d91a1a}-0.08\%$
test_step_mdp_speed[False-False-False-True-False]	55.1630μs	20.9363μs	47.7639 KOps/s	47.5251 KOps/s	$\color{#35bf28}+0.50\%$
test_step_mdp_speed[False-False-False-False-True]	55.1130μs	20.0751μs	49.8130 KOps/s	49.7161 KOps/s	$\color{#35bf28}+0.19\%$
test_step_mdp_speed[False-False-False-False-False]	53.1590μs	13.4632μs	74.2763 KOps/s	74.9468 KOps/s	$\color{#d91a1a}-0.89\%$
test_values[generalized_advantage_estimate-True-True]	11.3018ms	9.2905ms	107.6365 Ops/s	105.7400 Ops/s	$\color{#35bf28}+1.79\%$
test_values[vec_generalized_advantage_estimate-True-True]	38.0646ms	35.9271ms	27.8341 Ops/s	30.2165 Ops/s	$\textbf{\color{#d91a1a}-7.88\%}$
test_values[td0_return_estimate-False-False]	0.2475ms	0.1661ms	6.0190 KOps/s	5.9111 KOps/s	$\color{#35bf28}+1.83\%$
test_values[td1_return_estimate-False-False]	26.5610ms	23.1713ms	43.1568 Ops/s	42.2428 Ops/s	$\color{#35bf28}+2.16\%$
test_values[vec_td1_return_estimate-False-False]	37.0795ms	36.0380ms	27.7485 Ops/s	30.1738 Ops/s	$\textbf{\color{#d91a1a}-8.04\%}$
test_values[td_lambda_return_estimate-True-False]	36.8981ms	33.4890ms	29.8606 Ops/s	29.3167 Ops/s	$\color{#35bf28}+1.86\%$
test_values[vec_td_lambda_return_estimate-True-False]	38.0535ms	36.0679ms	27.7255 Ops/s	30.3372 Ops/s	$\textbf{\color{#d91a1a}-8.61\%}$
test_gae_speed[generalized_advantage_estimate-False-1-512]	10.0249ms	8.1272ms	123.0439 Ops/s	120.9742 Ops/s	$\color{#35bf28}+1.71\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	2.4673ms	1.9706ms	507.4694 Ops/s	509.3678 Ops/s	$\color{#d91a1a}-0.37\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.6305ms	0.3559ms	2.8099 KOps/s	2.8105 KOps/s	$\color{#d91a1a}-0.02\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	47.3114ms	45.1790ms	22.1342 Ops/s	25.1755 Ops/s	$\textbf{\color{#d91a1a}-12.08\%}$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	4.1962ms	3.0323ms	329.7830 Ops/s	330.0103 Ops/s	$\color{#d91a1a}-0.07\%$
test_dqn_speed	1.7070ms	1.2856ms	777.8331 Ops/s	755.7053 Ops/s	$\color{#35bf28}+2.93\%$
test_ddpg_speed	2.9843ms	2.6909ms	371.6169 Ops/s	370.5336 Ops/s	$\color{#35bf28}+0.29\%$
test_sac_speed	9.2406ms	7.9107ms	126.4114 Ops/s	126.5855 Ops/s	$\color{#d91a1a}-0.14\%$
test_redq_speed	13.7972ms	12.5601ms	79.6172 Ops/s	79.9664 Ops/s	$\color{#d91a1a}-0.44\%$
test_redq_deprec_speed	14.5390ms	12.4988ms	80.0079 Ops/s	79.5954 Ops/s	$\color{#35bf28}+0.52\%$
test_td3_speed	8.5039ms	7.8213ms	127.8560 Ops/s	128.0770 Ops/s	$\color{#d91a1a}-0.17\%$
test_cql_speed	38.6844ms	36.2754ms	27.5669 Ops/s	28.3872 Ops/s	$\color{#d91a1a}-2.89\%$
test_a2c_speed	7.9892ms	7.2038ms	138.8164 Ops/s	138.4840 Ops/s	$\color{#35bf28}+0.24\%$
test_ppo_speed	8.6718ms	7.5014ms	133.3085 Ops/s	133.4895 Ops/s	$\color{#d91a1a}-0.14\%$
test_reinforce_speed	7.1409ms	6.3876ms	156.5531 Ops/s	156.6303 Ops/s	$\color{#d91a1a}-0.05\%$
test_iql_speed	33.4514ms	31.6729ms	31.5727 Ops/s	31.5259 Ops/s	$\color{#35bf28}+0.15\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	7.4256ms	4.7694ms	209.6693 Ops/s	211.9973 Ops/s	$\color{#d91a1a}-1.10\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.5622ms	0.4677ms	2.1383 KOps/s	2.0984 KOps/s	$\color{#35bf28}+1.90\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.6079ms	0.4449ms	2.2475 KOps/s	2.2271 KOps/s	$\color{#35bf28}+0.92\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	6.9323ms	4.7051ms	212.5345 Ops/s	215.6771 Ops/s	$\color{#d91a1a}-1.46\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.7574ms	0.4674ms	2.1393 KOps/s	1.8186 KOps/s	$\textbf{\color{#35bf28}+17.64\%}$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.7083ms	0.4443ms	2.2509 KOps/s	2.2501 KOps/s	$\color{#35bf28}+0.03\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000]	2.2063ms	1.6809ms	594.9209 Ops/s	598.1273 Ops/s	$\color{#d91a1a}-0.54\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000]	2.1534ms	1.5920ms	628.1406 Ops/s	630.3179 Ops/s	$\color{#d91a1a}-0.35\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	7.0713ms	4.9076ms	203.7656 Ops/s	207.0743 Ops/s	$\color{#d91a1a}-1.60\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	1.4601ms	0.6014ms	1.6627 KOps/s	1.6356 KOps/s	$\color{#35bf28}+1.66\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.9157ms	0.5772ms	1.7326 KOps/s	1.6910 KOps/s	$\color{#35bf28}+2.46\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	4.9678ms	4.7559ms	210.2659 Ops/s	211.8134 Ops/s	$\color{#d91a1a}-0.73\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	0.5497ms	0.4710ms	2.1233 KOps/s	2.1037 KOps/s	$\color{#35bf28}+0.93\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	7.0733ms	0.4562ms	2.1921 KOps/s	2.2185 KOps/s	$\color{#d91a1a}-1.19\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	5.0053ms	4.6571ms	214.7267 Ops/s	215.1169 Ops/s	$\color{#d91a1a}-0.18\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	1.2015ms	0.4649ms	2.1510 KOps/s	2.1463 KOps/s	$\color{#35bf28}+0.22\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.6457ms	0.4458ms	2.2430 KOps/s	2.2570 KOps/s	$\color{#d91a1a}-0.62\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	7.6016ms	4.8703ms	205.3254 Ops/s	203.1853 Ops/s	$\color{#35bf28}+1.05\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	0.8191ms	0.6027ms	1.6592 KOps/s	1.6421 KOps/s	$\color{#35bf28}+1.04\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.9067ms	0.5814ms	1.7200 KOps/s	1.7204 KOps/s	$\color{#d91a1a}-0.02\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.1258s	6.1724ms	162.0103 Ops/s	160.8139 Ops/s	$\color{#35bf28}+0.74\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	17.1105ms	13.0297ms	76.7479 Ops/s	79.1991 Ops/s	$\color{#d91a1a}-3.09\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	97.1946ms	3.0436ms	328.5558 Ops/s	877.2687 Ops/s	$\textbf{\color{#d91a1a}-62.55\%}$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.1005s	5.6316ms	177.5707 Ops/s	172.3732 Ops/s	$\color{#35bf28}+3.02\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	16.6682ms	12.9734ms	77.0808 Ops/s	79.0539 Ops/s	$\color{#d91a1a}-2.50\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	1.6050ms	1.0570ms	946.1047 Ops/s	905.6743 Ops/s	$\color{#35bf28}+4.46\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.1043s	5.8347ms	171.3879 Ops/s	123.5779 Ops/s	$\textbf{\color{#35bf28}+38.69\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	17.1684ms	13.1776ms	75.8865 Ops/s	78.1330 Ops/s	$\color{#d91a1a}-2.88\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	5.1562ms	1.3002ms	769.1188 Ops/s	770.3753 Ops/s	$\color{#d91a1a}-0.16\%$

[github-actions]

$\color{#D29922}\textsf{\Large⚠\kern{0.2cm}\normalsize Warning}$ Result of GPU Benchmark Tests

Total Benchmarks: 94. Improved: $\large\color{#35bf28}1$. Worsened: $\large\color{#d91a1a}4$.

Expand to view detailed results

Name	Max	Mean	Ops	Ops on Repo HEAD	Change
test_single	0.1071s	0.1066s	9.3767 Ops/s	9.5617 Ops/s	$\color{#d91a1a}-1.93\%$
test_sync	96.4532ms	93.5879ms	10.6851 Ops/s	10.6277 Ops/s	$\color{#35bf28}+0.54\%$
test_async	0.1777s	87.1170ms	11.4788 Ops/s	11.5328 Ops/s	$\color{#d91a1a}-0.47\%$
test_single_pixels	0.1186s	0.1181s	8.4678 Ops/s	8.6487 Ops/s	$\color{#d91a1a}-2.09\%$
test_sync_pixels	84.4146ms	77.0640ms	12.9762 Ops/s	13.0832 Ops/s	$\color{#d91a1a}-0.82\%$
test_async_pixels	0.1410s	71.4899ms	13.9880 Ops/s	14.1742 Ops/s	$\color{#d91a1a}-1.31\%$
test_simple	0.7629s	0.7624s	1.3117 Ops/s	1.3138 Ops/s	$\color{#d91a1a}-0.16\%$
test_transformed	1.0852s	1.0134s	0.9867 Ops/s	1.0165 Ops/s	$\color{#d91a1a}-2.93\%$
test_serial	2.2708s	2.2030s	0.4539 Ops/s	0.4649 Ops/s	$\color{#d91a1a}-2.35\%$
test_parallel	1.9578s	1.8898s	0.5292 Ops/s	0.5279 Ops/s	$\color{#35bf28}+0.24\%$
test_step_mdp_speed[True-True-True-True-True]	88.4520μs	36.1809μs	27.6389 KOps/s	27.9812 KOps/s	$\color{#d91a1a}-1.22\%$
test_step_mdp_speed[True-True-True-True-False]	41.5620μs	21.0048μs	47.6082 KOps/s	48.2128 KOps/s	$\color{#d91a1a}-1.25\%$
test_step_mdp_speed[True-True-True-False-True]	40.7610μs	20.4062μs	49.0048 KOps/s	50.4436 KOps/s	$\color{#d91a1a}-2.85\%$
test_step_mdp_speed[True-True-True-False-False]	36.8110μs	11.8874μs	84.1226 KOps/s	86.1193 KOps/s	$\color{#d91a1a}-2.32\%$
test_step_mdp_speed[True-True-False-True-True]	56.0910μs	38.4784μs	25.9886 KOps/s	26.4561 KOps/s	$\color{#d91a1a}-1.77\%$
test_step_mdp_speed[True-True-False-True-False]	42.7510μs	22.8505μs	43.7627 KOps/s	43.8856 KOps/s	$\color{#d91a1a}-0.28\%$
test_step_mdp_speed[True-True-False-False-True]	44.1310μs	22.4502μs	44.5429 KOps/s	45.4996 KOps/s	$\color{#d91a1a}-2.10\%$
test_step_mdp_speed[True-True-False-False-False]	46.5300μs	13.9964μs	71.4469 KOps/s	72.5343 KOps/s	$\color{#d91a1a}-1.50\%$
test_step_mdp_speed[True-False-True-True-True]	58.0010μs	39.6216μs	25.2387 KOps/s	25.1996 KOps/s	$\color{#35bf28}+0.16\%$
test_step_mdp_speed[True-False-True-True-False]	48.3900μs	25.1329μs	39.7885 KOps/s	40.4341 KOps/s	$\color{#d91a1a}-1.60\%$
test_step_mdp_speed[True-False-True-False-True]	42.0420μs	22.5517μs	44.3426 KOps/s	45.7616 KOps/s	$\color{#d91a1a}-3.10\%$
test_step_mdp_speed[True-False-True-False-False]	30.9110μs	14.0210μs	71.3217 KOps/s	72.5758 KOps/s	$\color{#d91a1a}-1.73\%$
test_step_mdp_speed[True-False-False-True-True]	62.1510μs	41.7485μs	23.9529 KOps/s	23.7050 KOps/s	$\color{#35bf28}+1.05\%$
test_step_mdp_speed[True-False-False-True-False]	0.1037ms	27.1505μs	36.8317 KOps/s	36.8002 KOps/s	$\color{#35bf28}+0.09\%$
test_step_mdp_speed[True-False-False-False-True]	49.3810μs	24.6780μs	40.5219 KOps/s	40.7350 KOps/s	$\color{#d91a1a}-0.52\%$
test_step_mdp_speed[True-False-False-False-False]	31.6210μs	16.3403μs	61.1984 KOps/s	63.5121 KOps/s	$\color{#d91a1a}-3.64\%$
test_step_mdp_speed[False-True-True-True-True]	59.8510μs	39.7565μs	25.1531 KOps/s	25.1312 KOps/s	$\color{#35bf28}+0.09\%$
test_step_mdp_speed[False-True-True-True-False]	52.7220μs	25.2114μs	39.6646 KOps/s	40.4660 KOps/s	$\color{#d91a1a}-1.98\%$
test_step_mdp_speed[False-True-True-False-True]	51.4210μs	26.7805μs	37.3406 KOps/s	38.1062 KOps/s	$\color{#d91a1a}-2.01\%$
test_step_mdp_speed[False-True-True-False-False]	38.7010μs	15.6714μs	63.8106 KOps/s	63.3442 KOps/s	$\color{#35bf28}+0.74\%$
test_step_mdp_speed[False-True-False-True-True]	64.8210μs	41.8101μs	23.9177 KOps/s	24.1666 KOps/s	$\color{#d91a1a}-1.03\%$
test_step_mdp_speed[False-True-False-True-False]	48.5810μs	27.0982μs	36.9028 KOps/s	37.2105 KOps/s	$\color{#d91a1a}-0.83\%$
test_step_mdp_speed[False-True-False-False-True]	45.3820μs	28.3708μs	35.2475 KOps/s	35.1085 KOps/s	$\color{#35bf28}+0.40\%$
test_step_mdp_speed[False-True-False-False-False]	33.5910μs	17.9257μs	55.7858 KOps/s	56.6592 KOps/s	$\color{#d91a1a}-1.54\%$
test_step_mdp_speed[False-False-True-True-True]	4.0772ms	44.8606μs	22.2913 KOps/s	22.4434 KOps/s	$\color{#d91a1a}-0.68\%$
test_step_mdp_speed[False-False-True-True-False]	55.6610μs	29.9094μs	33.4343 KOps/s	34.0010 KOps/s	$\color{#d91a1a}-1.67\%$
test_step_mdp_speed[False-False-True-False-True]	48.5010μs	28.8069μs	34.7140 KOps/s	34.8774 KOps/s	$\color{#d91a1a}-0.47\%$
test_step_mdp_speed[False-False-True-False-False]	33.4900μs	18.0164μs	55.5049 KOps/s	55.5189 KOps/s	$\color{#d91a1a}-0.03\%$
test_step_mdp_speed[False-False-False-True-True]	67.1420μs	46.0340μs	21.7231 KOps/s	21.7392 KOps/s	$\color{#d91a1a}-0.07\%$
test_step_mdp_speed[False-False-False-True-False]	49.4010μs	31.5076μs	31.7383 KOps/s	32.2425 KOps/s	$\color{#d91a1a}-1.56\%$
test_step_mdp_speed[False-False-False-False-True]	55.7310μs	30.2047μs	33.1074 KOps/s	32.8030 KOps/s	$\color{#35bf28}+0.93\%$
test_step_mdp_speed[False-False-False-False-False]	40.1910μs	19.8649μs	50.3400 KOps/s	50.1448 KOps/s	$\color{#35bf28}+0.39\%$
test_values[generalized_advantage_estimate-True-True]	25.1960ms	24.5312ms	40.7644 Ops/s	41.5326 Ops/s	$\color{#d91a1a}-1.85\%$
test_values[vec_generalized_advantage_estimate-True-True]	91.2298ms	2.7250ms	366.9720 Ops/s	372.2005 Ops/s	$\color{#d91a1a}-1.40\%$
test_values[td0_return_estimate-False-False]	93.4220μs	66.7968μs	14.9708 KOps/s	15.5316 KOps/s	$\color{#d91a1a}-3.61\%$
test_values[td1_return_estimate-False-False]	55.4362ms	55.1375ms	18.1365 Ops/s	18.7280 Ops/s	$\color{#d91a1a}-3.16\%$
test_values[vec_td1_return_estimate-False-False]	1.3797ms	1.0877ms	919.3320 Ops/s	928.0694 Ops/s	$\color{#d91a1a}-0.94\%$
test_values[td_lambda_return_estimate-True-False]	87.6923ms	87.2108ms	11.4665 Ops/s	11.8196 Ops/s	$\color{#d91a1a}-2.99\%$
test_values[vec_td_lambda_return_estimate-True-False]	1.4180ms	1.0858ms	920.9450 Ops/s	930.7556 Ops/s	$\color{#d91a1a}-1.05\%$
test_gae_speed[generalized_advantage_estimate-False-1-512]	24.5721ms	24.2162ms	41.2948 Ops/s	42.4348 Ops/s	$\color{#d91a1a}-2.69\%$
test_gae_speed[vec_generalized_advantage_estimate-True-1-512]	0.9755ms	0.7237ms	1.3819 KOps/s	1.4036 KOps/s	$\color{#d91a1a}-1.55\%$
test_gae_speed[vec_generalized_advantage_estimate-False-1-512]	0.7562ms	0.6726ms	1.4867 KOps/s	1.5104 KOps/s	$\color{#d91a1a}-1.57\%$
test_gae_speed[vec_generalized_advantage_estimate-True-32-512]	1.5301ms	1.4711ms	679.7564 Ops/s	684.4602 Ops/s	$\color{#d91a1a}-0.69\%$
test_gae_speed[vec_generalized_advantage_estimate-False-32-512]	0.7485ms	0.6884ms	1.4527 KOps/s	1.4794 KOps/s	$\color{#d91a1a}-1.81\%$
test_dqn_speed	7.3588ms	1.3551ms	737.9270 Ops/s	758.3242 Ops/s	$\color{#d91a1a}-2.69\%$
test_ddpg_speed	3.0097ms	2.7624ms	362.0000 Ops/s	371.2379 Ops/s	$\color{#d91a1a}-2.49\%$
test_sac_speed	8.2130ms	7.9621ms	125.5953 Ops/s	127.6585 Ops/s	$\color{#d91a1a}-1.62\%$
test_redq_speed	12.3632ms	10.1069ms	98.9423 Ops/s	99.9850 Ops/s	$\color{#d91a1a}-1.04\%$
test_redq_deprec_speed	11.1841ms	10.9304ms	91.4880 Ops/s	90.2813 Ops/s	$\color{#35bf28}+1.34\%$
test_td3_speed	7.9930ms	7.9327ms	126.0602 Ops/s	129.1294 Ops/s	$\color{#d91a1a}-2.38\%$
test_cql_speed	26.4267ms	25.2588ms	39.5902 Ops/s	40.2291 Ops/s	$\color{#d91a1a}-1.59\%$
test_a2c_speed	6.2464ms	5.5631ms	179.7572 Ops/s	181.4860 Ops/s	$\color{#d91a1a}-0.95\%$
test_ppo_speed	6.1600ms	5.9191ms	168.9446 Ops/s	170.5873 Ops/s	$\color{#d91a1a}-0.96\%$
test_reinforce_speed	4.6704ms	4.4137ms	226.5663 Ops/s	221.2425 Ops/s	$\color{#35bf28}+2.41\%$
test_iql_speed	19.2470ms	18.9242ms	52.8425 Ops/s	51.6947 Ops/s	$\color{#35bf28}+2.22\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	6.8040ms	6.7254ms	148.6894 Ops/s	153.4119 Ops/s	$\color{#d91a1a}-3.08\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	1.9624ms	0.5236ms	1.9097 KOps/s	1.9568 KOps/s	$\color{#d91a1a}-2.40\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.6517ms	0.4984ms	2.0063 KOps/s	2.0305 KOps/s	$\color{#d91a1a}-1.19\%$
test_rb_sample[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	6.8933ms	6.6243ms	150.9594 Ops/s	156.5894 Ops/s	$\color{#d91a1a}-3.60\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	1.1816ms	0.5093ms	1.9637 KOps/s	1.9578 KOps/s	$\color{#35bf28}+0.30\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.6788ms	0.4921ms	2.0321 KOps/s	2.0526 KOps/s	$\color{#d91a1a}-1.00\%$
test_rb_sample[TensorDictReplayBuffer-LazyMemmapStorage-sampler6-10000]	2.1540ms	1.9794ms	505.2015 Ops/s	517.9004 Ops/s	$\color{#d91a1a}-2.45\%$
test_rb_sample[TensorDictReplayBuffer-LazyTensorStorage-sampler7-10000]	2.0445ms	1.8940ms	527.9914 Ops/s	544.1847 Ops/s	$\color{#d91a1a}-2.98\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	6.8838ms	6.8011ms	147.0351 Ops/s	150.3203 Ops/s	$\color{#d91a1a}-2.19\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	1.0840ms	0.6636ms	1.5070 KOps/s	1.5159 KOps/s	$\color{#d91a1a}-0.59\%$
test_rb_sample[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	0.8450ms	0.6533ms	1.5308 KOps/s	1.5541 KOps/s	$\color{#d91a1a}-1.50\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-RandomSampler-4000]	6.8327ms	6.7014ms	149.2220 Ops/s	153.4463 Ops/s	$\color{#d91a1a}-2.75\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-10000]	1.0194ms	0.5167ms	1.9355 KOps/s	1.9556 KOps/s	$\color{#d91a1a}-1.03\%$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-10000]	0.6791ms	0.4968ms	2.0131 KOps/s	2.0228 KOps/s	$\color{#d91a1a}-0.48\%$
test_rb_iterate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-4000]	6.8352ms	6.6009ms	151.4947 Ops/s	154.5941 Ops/s	$\color{#d91a1a}-2.00\%$
test_rb_iterate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-10000]	0.1240s	0.6784ms	1.4740 KOps/s	1.9803 KOps/s	$\textbf{\color{#d91a1a}-25.57\%}$
test_rb_iterate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-10000]	0.6917ms	0.4897ms	2.0419 KOps/s	2.0485 KOps/s	$\color{#d91a1a}-0.32\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-ListStorage-None-4000]	6.9705ms	6.8396ms	146.2081 Ops/s	151.1932 Ops/s	$\color{#d91a1a}-3.30\%$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-10000]	0.8821ms	0.6694ms	1.4939 KOps/s	1.2081 KOps/s	$\textbf{\color{#35bf28}+23.65\%}$
test_rb_iterate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-10000]	4.5890ms	0.6536ms	1.5300 KOps/s	1.5721 KOps/s	$\color{#d91a1a}-2.68\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-RandomSampler-400]	0.1295s	7.6142ms	131.3343 Ops/s	130.5077 Ops/s	$\color{#35bf28}+0.63\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-RandomSampler-400]	18.8942ms	16.4929ms	60.6320 Ops/s	63.5751 Ops/s	$\color{#d91a1a}-4.63\%$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-RandomSampler-400]	2.2653ms	1.2545ms	797.1137 Ops/s	763.8964 Ops/s	$\color{#35bf28}+4.35\%$
test_rb_populate[TensorDictReplayBuffer-ListStorage-SamplerWithoutReplacement-400]	0.1249s	7.5492ms	132.4636 Ops/s	133.2743 Ops/s	$\color{#d91a1a}-0.61\%$
test_rb_populate[TensorDictReplayBuffer-LazyMemmapStorage-SamplerWithoutReplacement-400]	0.1375s	18.7652ms	53.2901 Ops/s	63.6500 Ops/s	$\textbf{\color{#d91a1a}-16.28\%}$
test_rb_populate[TensorDictReplayBuffer-LazyTensorStorage-SamplerWithoutReplacement-400]	7.0212ms	1.3431ms	744.5286 Ops/s	797.9400 Ops/s	$\textbf{\color{#d91a1a}-6.69\%}$
test_rb_populate[TensorDictPrioritizedReplayBuffer-ListStorage-None-400]	0.1246s	7.7569ms	128.9174 Ops/s	128.5718 Ops/s	$\color{#35bf28}+0.27\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyMemmapStorage-None-400]	18.9507ms	16.5766ms	60.3259 Ops/s	63.3611 Ops/s	$\color{#d91a1a}-4.79\%$
test_rb_populate[TensorDictPrioritizedReplayBuffer-LazyTensorStorage-None-400]	7.2493ms	1.5184ms	658.5982 Ops/s	711.3550 Ops/s	$\textbf{\color{#d91a1a}-7.42\%}$