Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory Usage: Add memory transferred between Kokkos Mem Spaces #272

Open
wants to merge 14 commits into
base: develop
Choose a base branch
from

Conversation

vlkale
Copy link
Contributor

@vlkale vlkale commented Sep 19, 2024

This PR adds memory transferred between Kokkos Mem Spaces to the Kokkos Tools memory-usage tool library. This is done based on a request in Kokkos Tools Github Issue #50.

This PR adds functionality to the tool so that the size of data transferred in the deep_copy in a Kokkos application program is accumulated during the execution of a Kokkos program. The deep_copy accumulation is done per Kokkos Memory Space dst->src pair (e.g., OpenMP on host CUDA on device). Note that the "dst" and "src" Kokkos memory space being the same means that there was a deep_copy in the same memory space.

When Kokkos is finalized in a Kokkos application program, the finalize callback prints out the deep_copy accumulations per memory space to the file.

@vlkale vlkale self-assigned this Sep 19, 2024
@vlkale
Copy link
Contributor Author

vlkale commented Sep 19, 2024

Here is output from the stream.cuda in the Kokkos-core benchmarks directory run on Perlmutter on 1 GPU. The number of bytes in the HighWater at the last point in time shown in the first table is equal to the data transferred between host and device.

-----@nid----:~/kks-clean/benchmarks/stream> cat nid001213-1948341-Cuda.memspace_usage
# Space Cuda
# Time(s)  Size(MB)   HighWater(MB)   HighWater-Process(MB)
0.018076 7629.4 7629.4 142.7
0.047229 15258.8 15258.8 143.5
0.071899 22888.2 22888.2 144.0
12.252089 15258.8 22888.2 23032.6
12.313410 7629.4 22888.2 23032.6
12.331277 0.0 22888.2 23032.6
# Data transferred between Kokkos Memory Spaces --- 
# DstSpace     SrcSpace    Data-Transferred(MB)
Cuda Cuda 0.0 
# DstSpace     SrcSpace    Data-Transferred(MB)
Cuda Host 22888.2 
# DstSpace     SrcSpace    Data-Transferred(MB)
Host Cuda 22888.2 
# DstSpace     SrcSpace    Data-Transferred(MB)
Host Host 0.0 

------@nid----:~/kks-clean/benchmarks/stream> cat nid001213-1948341-
nid001213-1948341-Cuda.memspace_usage  nid001213-1948341-Host.memspace_usage  
vkale3@nid001213:~/kks-clean/benchmarks/stream> cat nid001213-1948341-Host.memspace_usage 
# Space Host
# Time(s)  Size(MB)   HighWater(MB)   HighWater-Process(MB)
0.077138 7629.4 7629.4 144.0
0.768111 15258.8 15258.8 7773.6
1.455819 22888.2 22888.2 15403.0
12.218183 15258.8 22888.2 23032.6
12.229001 7629.4 22888.2 23032.6
12.240329 0.0 22888.2 23032.6
# Data transferred between Kokkos Memory Spaces --- 
# DstSpace     SrcSpace    Data-Transferred(MB)
Cuda Cuda 0.0 
# DstSpace     SrcSpace    Data-Transferred(MB)
Cuda Host 22888.2 
# DstSpace     SrcSpace    Data-Transferred(MB)
Host Cuda 22888.2 
# DstSpace     SrcSpace    Data-Transferred(MB)
Host Host 0.0 

@vlkale vlkale marked this pull request as ready for review September 19, 2024 23:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant