Reduce even more GPU allocations #747

Job	Run time
CMake Matrix Test. (Linux Min GCC, ubuntu-22.04, gcc-12, g++-12, 3.9, 3.19.x, ON, OFF, static)	15m 47s
CMake Matrix Test. (Linux Min GCC, ubuntu-22.04, gcc-12, g++-12, 3.9, 3.19.x, ON, OFF, shared)	11m 35s
CMake Matrix Test. (Linux Min Clang, ubuntu-22.04, clang-12, clang++-12, 3.9, 3.19.x, ON, OFF, st...	10m 48s
CMake Matrix Test. (Linux Min Clang, ubuntu-22.04, clang-12, clang++-12, 3.9, 3.19.x, ON, OFF, sh...	10m 35s
CMake Matrix Test. (MacOS Min, macos-13, clang, clang++, 3.9, 3.19.x, ON, OFF, static)	14m 57s
CMake Matrix Test. (MacOS Min, macos-13, clang, clang++, 3.9, 3.19.x, ON, OFF, shared)	12m 30s
CMake Matrix Test. (Linux Max GCC, ubuntu-24.04, gcc-14, g++-14, 3.12, 3.22.x, ON, OFF, static)	28m 44s
CMake Matrix Test. (Linux Max GCC, ubuntu-24.04, gcc-14, g++-14, 3.12, 3.22.x, ON, OFF, shared)	19m 16s
CMake Matrix Test. (Linux SIMD, ubuntu-24.04, gcc-14, g++-14, 3.12, 3.22.x, OFF, ON, static)	26m 30s
CMake Matrix Test. (Linux SIMD, ubuntu-24.04, gcc-14, g++-14, 3.12, 3.22.x, OFF, ON, shared)	18m 10s
CMake Matrix Test. (Linux Max Clang, ubuntu-24.04, clang-18, clang++-18, 3.12, 3.22.x, ON, OFF, s...	14m 50s
CMake Matrix Test. (Linux Max Clang, ubuntu-24.04, clang-18, clang++-18, 3.12, 3.22.x, ON, OFF, s...	12m 16s
CMake Matrix Test. (MacOS Max, macos-14, clang, clang++, 3.12, 3.22.x, ON, OFF, static)	9m 31s
CMake Matrix Test. (MacOS Max, macos-14, clang, clang++, 3.12, 3.22.x, ON, OFF, shared)	9m 40s
	3h 35m 9s

Provide feedback