WIP: Add boundary comm inner loop specializations for performance and allow setting vector length #5579

Job	Run time
CI-Rocm-HIP (hipcc, Release, hip, serial)	33m 53s
CI-Cuda-Host (g++, Release, cuda, serial)	41m 38s
CI-Rocm-HIP (hipcc, Release, hip, mpi)	34m 28s
CI-Cuda-Host (g++, Release, cuda, mpi)	42m 15s
CI-Rocm-HIP (hipcc, Debug, hip, serial)	55m 51s
CI-Rocm-HIP (hipcc, Debug, hip, mpi)	55m 28s
CI-Cuda-Host (g++, Release, host, serial)	10m 53s
CI-Cuda-Host (g++, Release, host, mpi)	11m 6s
CI-Cuda-Host (g++, Debug, cuda, serial)	1h 15m 1s
CI-Cuda-Host (g++, Debug, cuda, mpi)	1h 15m 0s
CI-Cuda-Host (g++, Debug, host, serial)	10m 21s
CI-Cuda-Host (g++, Debug, host, mpi)	10m 50s
CI-Cuda-Host (clang++-15, Release, cuda, serial)	29m 0s
CI-Cuda-Host (clang++-15, Release, cuda, mpi)	27m 34s
CI-Cuda-Host (clang++-15, Release, host, serial)	9m 48s
CI-Cuda-Host (clang++-15, Release, host, mpi)	10m 27s
CI-Cuda-Host (clang++-15, Debug, host, serial)	7m 40s
CI-Cuda-Host (clang++-15, Debug, host, mpi)	7m 31s
	9h 8m 44s

Provide feedback