Improve launch latency #2456
Comments
maleadt
added
performance
|
FWIW, I'm seeing more reasonable timings: 30us between copy and launch, of which around 10us taken by Still a lot of latency, and the timings surprise me, because the functionality in question has been thoroughly micro-optimized: julia> cache = CUDA.compiler_cache(context());
julia> src = GPUCompiler.methodinstance(typeof(identity), Tuple{Nothing});
julia> cfg = CUDA.compiler_config(device());
julia> @benchmark GPUCompiler.cached_compilation(cache, src, cfg, CUDA.compile, CUDA.link)
BenchmarkTools.Trial: 10000 samples with 788 evaluations.
Range (min … max): 159.770 ns … 322.471 ns ┊ GC (min … max): 0.00% … 0.00%
Time (median): 183.145 ns ┊ GC (median): 0.00%
Time (mean ± σ): 182.211 ns ± 10.717 ns ┊ GC (mean ± σ): 0.00% ± 0.00%
▂█▇▆▄
▂▅▆▃▂▂▂▂▂▂▁▂▂▂▁▂▂▃█████▇▄▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▁▁▂▁▁▁▂▂▂▂▂▁▁▁▁▂▂▂▂ ▃
160 ns Histogram: frequency by time 229 ns <
Memory estimate: 64 bytes, allocs estimate: 2. |
|
Running the benchmark a couple of times (the profiler can make short measurements like this weird) lowers the time of 
Are you sure the function kernel!(a)
i = threadIdx().x + blockDim().x*(blockIdx().x-1)
a[i] = CUDA.cos(a[i]) + i^0.6 + CUDA.tan(a[i])
return
end
NVTX.@annotate function benchmark(a)
threads = 256
blocks = cld(length(a), threads)
a_d = CuArray(a)
@cuda threads=threads blocks=blocks kernel!(a_d)
Array(a_d)
end
function main()
n = 2^12
a = rand(n)
benchmark(a)
benchmark(a)
benchmark(a)
return
end |
|
Here's a much more detailed trace: So it looks more like "death by a thousand papercuts" rather than one operation being slow:
Bottom line, the latency is much better on my system, but I agree things can be improved. I also want to emphasize once more that NSight makes these look much worse. Where each iteration takes around 50us in the profile trace, in a non-profiled session an iteration only takes 30us. |
maleadt
changed the title
|
I ran these benchmarks again and you're right in that the latency does go down when the profiler is run a few times. I also noticed you were focusing on the CUDA API line in Nsight systems. I usually look at the CUDA HW line in the timeline and maybe that was throwing me off too. Thanks for looking into this! |
For a fairly simple minimal working example shown below, I see a considerably large delay between the HtoD memcopy and the kernel launch.
The times reported by Nsight systems are:
On further inspection of
cufunction()insrc/compiler/execution.jlusing NVTX annotations, it seems like the calls tomethodinstanceandcached_compilationtake more time than expected.Reference discussion in the Julia discourse here.
MWE
Expected behavior
I expect the delay between memcopy and kernel launch to be around two orders of magnitude lower.
Project.toml
Manifest.toml
Version info
Details on Julia:
Details on CUDA:
Additional context
The Nsight systems profiling output is shown below:
The same MWE was also run on a P100 GPU and similar trends were seen although each block took shorter times.
For other larger cases, a large delay between kernel completion and DtoH memcopy was also noticed. However, this may be unrelated.
The text was updated successfully, but these errors were encountered: