Reduce even more GPU allocations #745

Job	Run time
CMake Matrix Test. (Linux Min GCC, ubuntu-22.04, gcc-12, g++-12, 3.9, 3.19.x, ON, OFF, static)	12m 31s
CMake Matrix Test. (Linux Min GCC, ubuntu-22.04, gcc-12, g++-12, 3.9, 3.19.x, ON, OFF, shared)	8m 55s
CMake Matrix Test. (Linux Min Clang, ubuntu-22.04, clang-12, clang++-12, 3.9, 3.19.x, ON, OFF, st...	5m 29s
CMake Matrix Test. (Linux Min Clang, ubuntu-22.04, clang-12, clang++-12, 3.9, 3.19.x, ON, OFF, sh...	15m 37s
CMake Matrix Test. (MacOS Min, macos-13, clang, clang++, 3.9, 3.19.x, ON, OFF, static)	13m 8s
CMake Matrix Test. (MacOS Min, macos-13, clang, clang++, 3.9, 3.19.x, ON, OFF, shared)	8m 59s
CMake Matrix Test. (Linux Max GCC, ubuntu-24.04, gcc-14, g++-14, 3.12, 3.22.x, ON, OFF, static)	23m 45s
CMake Matrix Test. (Linux Max GCC, ubuntu-24.04, gcc-14, g++-14, 3.12, 3.22.x, ON, OFF, shared)	16m 13s
CMake Matrix Test. (Linux SIMD, ubuntu-24.04, gcc-14, g++-14, 3.12, 3.22.x, OFF, ON, static)	20m 20s
CMake Matrix Test. (Linux SIMD, ubuntu-24.04, gcc-14, g++-14, 3.12, 3.22.x, OFF, ON, shared)	14m 42s
CMake Matrix Test. (Linux Max Clang, ubuntu-24.04, clang-18, clang++-18, 3.12, 3.22.x, ON, OFF, s...	10m 58s
CMake Matrix Test. (Linux Max Clang, ubuntu-24.04, clang-18, clang++-18, 3.12, 3.22.x, ON, OFF, s...	15m 29s
CMake Matrix Test. (MacOS Max, macos-14, clang, clang++, 3.12, 3.22.x, ON, OFF, static)	10m 5s
CMake Matrix Test. (MacOS Max, macos-14, clang, clang++, 3.12, 3.22.x, ON, OFF, shared)	8m 9s
	3h 4m 20s

Provide feedback