Reduce even more GPU allocations #751

Job	Run time
CMake Matrix Test. (Linux Min GCC, ubuntu-22.04, gcc-12, g++-12, 3.9, 3.19.x, ON, OFF, static)	15m 10s
CMake Matrix Test. (Linux Min GCC, ubuntu-22.04, gcc-12, g++-12, 3.9, 3.19.x, ON, OFF, shared)	11m 29s
CMake Matrix Test. (Linux Min Clang, ubuntu-22.04, clang-12, clang++-12, 3.9, 3.19.x, ON, OFF, st...	22m 20s
CMake Matrix Test. (Linux Min Clang, ubuntu-22.04, clang-12, clang++-12, 3.9, 3.19.x, ON, OFF, sh...	21m 55s
CMake Matrix Test. (MacOS Min, macos-13, clang, clang++, 3.9, 3.19.x, ON, OFF, static)	14m 48s
CMake Matrix Test. (MacOS Min, macos-13, clang, clang++, 3.9, 3.19.x, ON, OFF, shared)	15m 20s
CMake Matrix Test. (Linux Max GCC, ubuntu-24.04, gcc-14, g++-14, 3.12, 3.22.x, ON, OFF, static)	34m 58s
CMake Matrix Test. (Linux Max GCC, ubuntu-24.04, gcc-14, g++-14, 3.12, 3.22.x, ON, OFF, shared)	39m 39s
CMake Matrix Test. (Linux SIMD, ubuntu-24.04, gcc-14, g++-14, 3.12, 3.22.x, OFF, ON, static)	19m 20s
CMake Matrix Test. (Linux SIMD, ubuntu-24.04, gcc-14, g++-14, 3.12, 3.22.x, OFF, ON, shared)	38m 5s
CMake Matrix Test. (Linux Max Clang, ubuntu-24.04, clang-18, clang++-18, 3.12, 3.22.x, ON, OFF, s...	25m 29s
CMake Matrix Test. (Linux Max Clang, ubuntu-24.04, clang-18, clang++-18, 3.12, 3.22.x, ON, OFF, s...	37m 6s
CMake Matrix Test. (MacOS Max, macos-14, clang, clang++, 3.12, 3.22.x, ON, OFF, static)	12m 11s
CMake Matrix Test. (MacOS Max, macos-14, clang, clang++, 3.12, 3.22.x, ON, OFF, shared)	7m 54s
	5h 15m 44s

Provide feedback