Reduce even more GPU allocations #756

Job	Run time
CMake Matrix Test. (Linux Min GCC, ubuntu-22.04, gcc-12, g++-12, 3.9, 3.19.x, ON, OFF, static)	17m 32s
CMake Matrix Test. (Linux Min GCC, ubuntu-22.04, gcc-12, g++-12, 3.9, 3.19.x, ON, OFF, shared)	14m 36s
CMake Matrix Test. (Linux Min Clang, ubuntu-22.04, clang-12, clang++-12, 3.9, 3.19.x, ON, OFF, st...	8m 56s
CMake Matrix Test. (Linux Min Clang, ubuntu-22.04, clang-12, clang++-12, 3.9, 3.19.x, ON, OFF, sh...	9m 24s
CMake Matrix Test. (MacOS Min, macos-13, clang, clang++, 3.9, 3.19.x, ON, OFF, static)	28m 13s
CMake Matrix Test. (MacOS Min, macos-13, clang, clang++, 3.9, 3.19.x, ON, OFF, shared)	10m 49s
CMake Matrix Test. (Linux Max GCC, ubuntu-24.04, gcc-14, g++-14, 3.12, 3.22.x, ON, OFF, static)	17m 47s
CMake Matrix Test. (Linux Max GCC, ubuntu-24.04, gcc-14, g++-14, 3.12, 3.22.x, ON, OFF, shared)	12m 59s
CMake Matrix Test. (Linux SIMD, ubuntu-24.04, gcc-14, g++-14, 3.12, 3.22.x, OFF, ON, static)	16m 37s
CMake Matrix Test. (Linux SIMD, ubuntu-24.04, gcc-14, g++-14, 3.12, 3.22.x, OFF, ON, shared)	12m 7s
CMake Matrix Test. (Linux Max Clang, ubuntu-24.04, clang-18, clang++-18, 3.12, 3.22.x, ON, OFF, s...	17m 54s
CMake Matrix Test. (Linux Max Clang, ubuntu-24.04, clang-18, clang++-18, 3.12, 3.22.x, ON, OFF, s...	13m 59s
CMake Matrix Test. (MacOS Max, macos-14, clang, clang++, 3.12, 3.22.x, ON, OFF, static)	10m 18s
CMake Matrix Test. (MacOS Max, macos-14, clang, clang++, 3.12, 3.22.x, ON, OFF, shared)	7m 43s
	3h 18m 54s

Provide feedback