Skip to content

[Quality] Fewer recompiles with tensordict (#1015) #1897

[Quality] Fewer recompiles with tensordict (#1015)

[Quality] Fewer recompiles with tensordict (#1015) #1897

Triggered via push October 3, 2024 11:56
Status Failure
Total duration 10m 42s
Artifacts

benchmarks.yml

on: push
CPU Pytest benchmark
7m 51s
CPU Pytest benchmark
GPU Pytest benchmark
10m 23s
GPU Pytest benchmark
Fit to window
Zoom out
Zoom in

Annotations

3 errors and 12 warnings
GPU Pytest benchmark
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
GPU Pytest benchmark
WARNING: The directory '/github/home/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
GPU Pytest benchmark
# :warning: **Performance Alert** :warning: Possible performance regression was detected for benchmark **'GPU Benchmark Results'**. Benchmark result of this commit is worse than the previous benchmark result exceeding threshold `2`. | Benchmark suite | Current: 0cada70abd9a1331d7dcb66fcc821f6f2dfc32d1 | Previous: 38c54a1d0d782391976176cbdc751b2e500ee1bf | Ratio | |-|-|-|-| | `benchmarks/compile/tensordict_nn_test.py::test_mod_wrap[eager]` | `1279.946671645182` iter/sec (`stddev: 0.000010439399700566533`) | `4105.870613519885` iter/sec (`stddev: 0.00000587916809497675`) | `3.21` | | `benchmarks/compile/tensordict_nn_test.py::test_mod_wrap[compile]` | `1196.5375065872304` iter/sec (`stddev: 0.000012056446044823838`) | `3356.2752661378336` iter/sec (`stddev: 0.000018220149148620563`) | `2.80` | | `benchmarks/compile/tensordict_nn_test.py::test_mod_wrap_and_backward[eager]` | `241.78211707870747` iter/sec (`stddev: 0.000025218769756407773`) | `695.7221648967183` iter/sec (`stddev: 0.000016045632807803608`) | `2.88` | | `benchmarks/compile/tensordict_nn_test.py::test_mod_wrap_and_backward[compile]` | `245.66080254735687` iter/sec (`stddev: 0.00008215693476750638`) | `705.4696949490648` iter/sec (`stddev: 0.000020935005519053254`) | `2.87` | | `benchmarks/compile/tensordict_nn_test.py::test_seq_wrap[eager]` | `1083.561093345206` iter/sec (`stddev: 0.000012814472018065899`) | `2572.630147286971` iter/sec (`stddev: 0.000007224515064949263`) | `2.37` | | `benchmarks/compile/tensordict_nn_test.py::test_seq_wrap[compile]` | `1188.1668299478908` iter/sec (`stddev: 0.000010405929537850503`) | `3167.4311295287985` iter/sec (`stddev: 0.000006647567500485782`) | `2.67` | | `benchmarks/compile/tensordict_nn_test.py::test_func_call_runtime[False-eager]` | `424.49807005752433` iter/sec (`stddev: 0.000019962805815589012`) | `1361.9761923488245` iter/sec (`stddev: 0.000010441248501911535`) | `3.21` | | `benchmarks/compile/tensordict_nn_test.py::test_func_call_runtime[False-compile]` | `421.902047750821` iter/sec (`stddev: 0.00001949274725133026`) | `1276.873388994848` iter/sec (`stddev: 0.000010556363127480173`) | `3.03` | | `benchmarks/compile/tensordict_nn_test.py::test_func_call_runtime[True-eager]` | `398.10252555733433` iter/sec (`stddev: 0.00002200395862296179`) | `1105.980477431626` iter/sec (`stddev: 0.000011584526513610845`) | `2.78` | | `benchmarks/compile/tensordict_nn_test.py::test_func_call_runtime[True-compile]` | `420.50762474788235` iter/sec (`stddev: 0.000021385339763071816`) | `1250.245121454213` iter/sec (`stddev: 0.000009802022875934027`) | `2.97` | | `benchmarks/compile/tensordict_nn_test.py::test_func_call_cm_runtime[False-eager]` | `425.6486055792215` iter/sec (`stddev: 0.00001744289360124352`) | `1315.6345445081918` iter/sec (`stddev: 0.0000359541284514054`) | `3.09` | | `benchmarks/compile/tensordict_nn_test.py::test_func_call_cm_runtime[False-compile]` | `419.98937022022216` iter/sec (`stddev: 0.000017972885624632058`) | `1277.713630343951` iter/sec (`stddev: 0.000009926399125788087`) | `3.04` | | `benchmarks/compile/tensordict_nn_test.py::test_func_call_cm_runtime[True-eager]` | `380.5498610242864` iter/sec (`stddev: 0.000019203010044430432`) | `1002.2804914248471` iter/sec (`stddev: 0.000012365589734137923`) | `2.63` | | `benchmarks/compile/tensordict_nn_test.py::test_func_call_cm_runtime[True-compile]` | `411.87065119777293` iter/sec (`stddev: 0.000021606255739942755`) | `1205.1618540424668` iter/sec (`stddev: 0.000010205314733240485`) | `2.93` | | `benchmarks/compile/tensordict_nn_test.py::test_vmap_func_call_cm_runtime[compile]` | `411.6317392285656` iter/sec (`stddev: 0.000017520867388628477`) | `1182.0291642394807` iter/sec (`stddev: 0.00001093769141520421`) | `2.87` | | `benchmarks/nn/functional_benchmarks_test.py::test_exec_functorch` | `1009.0932806732706` iter/sec (`stddev: 0.00001026221657952303`) | `4844.906806958219` iter/sec (`stddev: 0.000004551148690586294`) | `4.80` | | `benchmarks/nn/functional_benchmarks_test.py::test_exec_functional_call` | `1005.4487428946755` iter/sec (`stddev: 0.00001078937
CPU Pytest benchmark
The following actions use a deprecated Node.js version and will be forced to run on node20: actions/checkout@v3, actions/setup-python@v4. For more info: https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/
GPU Pytest benchmark
Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20: actions/checkout@v3, actions/setup-python@v4. For more information see: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.
GPU Pytest benchmark
Performance alert! Previous value was 4105.870613519885 and current value is 1279.946671645182. It is 3.207845064546632x worse than previous exceeding a ratio threshold 2
GPU Pytest benchmark
Performance alert! Previous value was 3356.2752661378336 and current value is 1196.5375065872304. It is 2.804989603468943x worse than previous exceeding a ratio threshold 2
GPU Pytest benchmark
Performance alert! Previous value was 695.7221648967183 and current value is 241.78211707870747. It is 2.8774756930027188x worse than previous exceeding a ratio threshold 2
GPU Pytest benchmark
Performance alert! Previous value was 705.4696949490648 and current value is 245.66080254735687. It is 2.871722666513185x worse than previous exceeding a ratio threshold 2
GPU Pytest benchmark
Performance alert! Previous value was 2572.630147286971 and current value is 1083.561093345206. It is 2.374236361094012x worse than previous exceeding a ratio threshold 2
GPU Pytest benchmark
Performance alert! Previous value was 3167.4311295287985 and current value is 1188.1668299478908. It is 2.665813461286166x worse than previous exceeding a ratio threshold 2
GPU Pytest benchmark
Performance alert! Previous value was 1361.9761923488245 and current value is 424.49807005752433. It is 3.208439068201797x worse than previous exceeding a ratio threshold 2
GPU Pytest benchmark
Performance alert! Previous value was 1276.873388994848 and current value is 421.902047750821. It is 3.026468811426534x worse than previous exceeding a ratio threshold 2
GPU Pytest benchmark
Performance alert! Previous value was 1105.980477431626 and current value is 398.10252555733433. It is 2.7781297691675753x worse than previous exceeding a ratio threshold 2
GPU Pytest benchmark
Performance alert! Previous value was 1250.245121454213 and current value is 420.50762474788235. It is 2.973180622358047x worse than previous exceeding a ratio threshold 2