[Quality] Fewer recompiles with tensordict (#1015) #1897
benchmarks.yml
on: push
CPU Pytest benchmark
7m 51s
GPU Pytest benchmark
10m 23s
Annotations
3 errors and 12 warnings
GPU Pytest benchmark
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
|
GPU Pytest benchmark
WARNING: The directory '/github/home/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
|
GPU Pytest benchmark
# :warning: **Performance Alert** :warning:
Possible performance regression was detected for benchmark **'GPU Benchmark Results'**.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold `2`.
| Benchmark suite | Current: 0cada70abd9a1331d7dcb66fcc821f6f2dfc32d1 | Previous: 38c54a1d0d782391976176cbdc751b2e500ee1bf | Ratio |
|-|-|-|-|
| `benchmarks/compile/tensordict_nn_test.py::test_mod_wrap[eager]` | `1279.946671645182` iter/sec (`stddev: 0.000010439399700566533`) | `4105.870613519885` iter/sec (`stddev: 0.00000587916809497675`) | `3.21` |
| `benchmarks/compile/tensordict_nn_test.py::test_mod_wrap[compile]` | `1196.5375065872304` iter/sec (`stddev: 0.000012056446044823838`) | `3356.2752661378336` iter/sec (`stddev: 0.000018220149148620563`) | `2.80` |
| `benchmarks/compile/tensordict_nn_test.py::test_mod_wrap_and_backward[eager]` | `241.78211707870747` iter/sec (`stddev: 0.000025218769756407773`) | `695.7221648967183` iter/sec (`stddev: 0.000016045632807803608`) | `2.88` |
| `benchmarks/compile/tensordict_nn_test.py::test_mod_wrap_and_backward[compile]` | `245.66080254735687` iter/sec (`stddev: 0.00008215693476750638`) | `705.4696949490648` iter/sec (`stddev: 0.000020935005519053254`) | `2.87` |
| `benchmarks/compile/tensordict_nn_test.py::test_seq_wrap[eager]` | `1083.561093345206` iter/sec (`stddev: 0.000012814472018065899`) | `2572.630147286971` iter/sec (`stddev: 0.000007224515064949263`) | `2.37` |
| `benchmarks/compile/tensordict_nn_test.py::test_seq_wrap[compile]` | `1188.1668299478908` iter/sec (`stddev: 0.000010405929537850503`) | `3167.4311295287985` iter/sec (`stddev: 0.000006647567500485782`) | `2.67` |
| `benchmarks/compile/tensordict_nn_test.py::test_func_call_runtime[False-eager]` | `424.49807005752433` iter/sec (`stddev: 0.000019962805815589012`) | `1361.9761923488245` iter/sec (`stddev: 0.000010441248501911535`) | `3.21` |
| `benchmarks/compile/tensordict_nn_test.py::test_func_call_runtime[False-compile]` | `421.902047750821` iter/sec (`stddev: 0.00001949274725133026`) | `1276.873388994848` iter/sec (`stddev: 0.000010556363127480173`) | `3.03` |
| `benchmarks/compile/tensordict_nn_test.py::test_func_call_runtime[True-eager]` | `398.10252555733433` iter/sec (`stddev: 0.00002200395862296179`) | `1105.980477431626` iter/sec (`stddev: 0.000011584526513610845`) | `2.78` |
| `benchmarks/compile/tensordict_nn_test.py::test_func_call_runtime[True-compile]` | `420.50762474788235` iter/sec (`stddev: 0.000021385339763071816`) | `1250.245121454213` iter/sec (`stddev: 0.000009802022875934027`) | `2.97` |
| `benchmarks/compile/tensordict_nn_test.py::test_func_call_cm_runtime[False-eager]` | `425.6486055792215` iter/sec (`stddev: 0.00001744289360124352`) | `1315.6345445081918` iter/sec (`stddev: 0.0000359541284514054`) | `3.09` |
| `benchmarks/compile/tensordict_nn_test.py::test_func_call_cm_runtime[False-compile]` | `419.98937022022216` iter/sec (`stddev: 0.000017972885624632058`) | `1277.713630343951` iter/sec (`stddev: 0.000009926399125788087`) | `3.04` |
| `benchmarks/compile/tensordict_nn_test.py::test_func_call_cm_runtime[True-eager]` | `380.5498610242864` iter/sec (`stddev: 0.000019203010044430432`) | `1002.2804914248471` iter/sec (`stddev: 0.000012365589734137923`) | `2.63` |
| `benchmarks/compile/tensordict_nn_test.py::test_func_call_cm_runtime[True-compile]` | `411.87065119777293` iter/sec (`stddev: 0.000021606255739942755`) | `1205.1618540424668` iter/sec (`stddev: 0.000010205314733240485`) | `2.93` |
| `benchmarks/compile/tensordict_nn_test.py::test_vmap_func_call_cm_runtime[compile]` | `411.6317392285656` iter/sec (`stddev: 0.000017520867388628477`) | `1182.0291642394807` iter/sec (`stddev: 0.00001093769141520421`) | `2.87` |
| `benchmarks/nn/functional_benchmarks_test.py::test_exec_functorch` | `1009.0932806732706` iter/sec (`stddev: 0.00001026221657952303`) | `4844.906806958219` iter/sec (`stddev: 0.000004551148690586294`) | `4.80` |
| `benchmarks/nn/functional_benchmarks_test.py::test_exec_functional_call` | `1005.4487428946755` iter/sec (`stddev: 0.00001078937
|
CPU Pytest benchmark
The following actions use a deprecated Node.js version and will be forced to run on node20: actions/checkout@v3, actions/setup-python@v4. For more info: https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/
|
GPU Pytest benchmark
Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20: actions/checkout@v3, actions/setup-python@v4. For more information see: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.
|
GPU Pytest benchmark
Performance alert! Previous value was 4105.870613519885 and current value is 1279.946671645182. It is 3.207845064546632x worse than previous exceeding a ratio threshold 2
|
GPU Pytest benchmark
Performance alert! Previous value was 3356.2752661378336 and current value is 1196.5375065872304. It is 2.804989603468943x worse than previous exceeding a ratio threshold 2
|
GPU Pytest benchmark
Performance alert! Previous value was 695.7221648967183 and current value is 241.78211707870747. It is 2.8774756930027188x worse than previous exceeding a ratio threshold 2
|
GPU Pytest benchmark
Performance alert! Previous value was 705.4696949490648 and current value is 245.66080254735687. It is 2.871722666513185x worse than previous exceeding a ratio threshold 2
|
GPU Pytest benchmark
Performance alert! Previous value was 2572.630147286971 and current value is 1083.561093345206. It is 2.374236361094012x worse than previous exceeding a ratio threshold 2
|
GPU Pytest benchmark
Performance alert! Previous value was 3167.4311295287985 and current value is 1188.1668299478908. It is 2.665813461286166x worse than previous exceeding a ratio threshold 2
|
GPU Pytest benchmark
Performance alert! Previous value was 1361.9761923488245 and current value is 424.49807005752433. It is 3.208439068201797x worse than previous exceeding a ratio threshold 2
|
GPU Pytest benchmark
Performance alert! Previous value was 1276.873388994848 and current value is 421.902047750821. It is 3.026468811426534x worse than previous exceeding a ratio threshold 2
|
GPU Pytest benchmark
Performance alert! Previous value was 1105.980477431626 and current value is 398.10252555733433. It is 2.7781297691675753x worse than previous exceeding a ratio threshold 2
|
GPU Pytest benchmark
Performance alert! Previous value was 1250.245121454213 and current value is 420.50762474788235. It is 2.973180622358047x worse than previous exceeding a ratio threshold 2
|