From 25708a4a440be94af06ecc18ff5e26bf2cd81c56 Mon Sep 17 00:00:00 2001 From: Abhishek Bagusetty Date: Wed, 6 Sep 2023 09:43:20 -0500 Subject: [PATCH 1/5] [SYCL] updates to oneapi compiler documentation --- .../compiling-and-linking/oneapi-compiler.md | 30 +++++++++---------- .../programming-models/sycl-polaris.md | 17 +++++++++-- 2 files changed, 30 insertions(+), 17 deletions(-) diff --git a/docs/polaris/compiling-and-linking/oneapi-compiler.md b/docs/polaris/compiling-and-linking/oneapi-compiler.md index 6da243420..151855e7f 100644 --- a/docs/polaris/compiling-and-linking/oneapi-compiler.md +++ b/docs/polaris/compiling-and-linking/oneapi-compiler.md @@ -1,36 +1,36 @@ -# OneAPI Compilers and Support +# oneAPI Compilers and Support -The Intel OneAPI compiler and Codeplay plugins for Nvidia GPUs are available on Polaris. +The Intel oneAPI compiler and Codeplay plugins for Nvidia GPUs are available on Polaris. The oneAPI compilers are not enabled under the Cray Programming Environment system but can be used separately. Two oneAPI variants are provided, the first being a "release" version based on Intel's officially released oneAPI toolkit. [Intel Release Notes](https://www.intel.com/content/www/us/en/developer/articles/release-notes/intel-oneapi-toolkit-release-notes.html) !!! note - The 2023.1 release of oneAPI Toolkit does not support oneMKL or oneDPL on Nvidia. + The 2023.2.0 release of oneAPI Toolkit does not support oneMKL or oneDPL on Nvidia. The other variant being a build of main from the open source. This variant will be more up-to-date at the risk of bugs and breakages based on code that has not undergone a full release cycle. The documentation is located on the [SYCL](../programming-models/sycl-polaris.md) page. ## Compile and Link -OneAPI uses the clang (or icx/icpx wrapper) for compiling and linking for the Nvidia A100 SM80 architecture. +oneAPI uses the clang (or icx/icpx wrapper) for compiling and linking for the Nvidia A100 SM80 architecture. ``` module load PrgEnv-nvhpc module use /soft/compilers/oneapi/release/modulefiles module load compiler -icpx -std=c++17 -fsycl -fsycl-targets=nvptx64-nvidia-cuda -Xsycl-target-backend --cuda-gpu-arch=sm_80 test.cpp +icpx -std=c++17 -fsycl -fsycl-targets=nvptx64-nvidia-cuda -Xsycl-target-backend --cuda-gpu-arch=sm_80 test.cpp ``` ``` harms@polaris-login-04:~/working/polaris/oneapi> icpx --version -Intel(R) oneAPI DPC++/C++ Compiler 2023.1.0 (2023.1.0.20230320) +Intel(R) oneAPI DPC++/C++ Compiler 2023.2.0 (2023.2.0.20230721) Target: x86_64-unknown-linux-gnu Thread model: posix -InstalledDir: /soft/compilers/oneapi/release/2023.1/compiler/2023.1.0/linux/bin-llvm -Configuration file: /soft/compilers/oneapi/release/2023.1/compiler/2023.1.0/linux/bin-llvm/../bin/icpx.cfg +InstalledDir: /soft/compilers/oneapi/release/2023.2/compiler/2023.2.1/linux/bin-llvm +Configuration file: /soft/compilers/oneapi/release/2023.2/compiler/2023.2.1/linux/bin-llvm/../bin/icpx.cfg ``` ## Running @@ -48,13 +48,13 @@ Expected output of sycl-ls and which platforms are available. ``` harms@x3004c0s7b0n0:~> which sycl-ls -/soft/compilers/oneapi/release/2023.1/compiler/2023.1.0/linux/bin/sycl-ls +/soft/compilers/oneapi/release/2023.2/compiler/2023.2.1/linux/bin/sycl-ls harms@x3004c0s7b0n0:~> sycl-ls -[opencl:acc:0] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device 1.2 [2023.15.3.0.20_160000] -[opencl:cpu:1] Intel(R) OpenCL, AMD EPYC 7543P 32-Core Processor 3.0 [2023.15.3.0.20_160000] -[ext_oneapi_cuda:gpu:0] NVIDIA CUDA BACKEND, NVIDIA A100-SXM4-40GB 0.0 [CUDA 11.4] -[ext_oneapi_cuda:gpu:1] NVIDIA CUDA BACKEND, NVIDIA A100-SXM4-40GB 0.0 [CUDA 11.4] -[ext_oneapi_cuda:gpu:2] NVIDIA CUDA BACKEND, NVIDIA A100-SXM4-40GB 0.0 [CUDA 11.4] -[ext_oneapi_cuda:gpu:3] NVIDIA CUDA BACKEND, NVIDIA A100-SXM4-40GB 0.0 [CUDA 11.4] +[opencl:acc:0] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device 1.2 [2023.16.7.0.21_160000] +[opencl:cpu:1] Intel(R) OpenCL, AMD EPYC 7543P 32-Core Processor 3.0 [2023.16.7.0.21_160000] +[ext_oneapi_cuda:gpu:0] NVIDIA CUDA BACKEND, NVIDIA A100-SXM4-40GB 8.8 [CUDA 11.4] +[ext_oneapi_cuda:gpu:1] NVIDIA CUDA BACKEND, NVIDIA A100-SXM4-40GB 8.8 [CUDA 11.4] +[ext_oneapi_cuda:gpu:2] NVIDIA CUDA BACKEND, NVIDIA A100-SXM4-40GB 8.8 [CUDA 11.4] +[ext_oneapi_cuda:gpu:3] NVIDIA CUDA BACKEND, NVIDIA A100-SXM4-40GB 8.8 [CUDA 11.4] ``` diff --git a/docs/polaris/programming-models/sycl-polaris.md b/docs/polaris/programming-models/sycl-polaris.md index 8053d9f00..585510dab 100644 --- a/docs/polaris/programming-models/sycl-polaris.md +++ b/docs/polaris/programming-models/sycl-polaris.md @@ -10,14 +10,27 @@ module load oneapi ``` -:warning: This module (compilers, libraries) gets built periodically from the latest open-source rather than releases. As such, these compilers will get new features and updates quickly that may break on occasion. +!!! note + +This module (compilers, libraries) gets built periodically from the latest open-source rather than releases. As such, these compilers will get new features and updates quickly that may break on occasion. Please submit issues at the respective github repositories for the compilers and libraries. + +## Components +- These are the list of components associated with this module +|-------------------|---------------------------------------------------------| +| User Application | Third-Party Library | +|-------------------|---------------------------------------------------------| +| Compilers | [DPC++](https://docs.nvidia.com/cuda/cublas/index.html) | +| oneMKL Interfaces | [oneMKL](https://github.com/oneapi-src/oneMKL) | +| oneDPL | [oneDPL](https://github.com/oneapi-src/onedpl) | +| SYCLomatic/DPCT | [dpct](https://github.com/oneapi-src/syclomatic) | +|-------------------|---------------------------------------------------------| ## Dependencies - SYCL programming model is supported through `oneapi` compilers that were built from source-code - Loading this module switches the default programming environment to GNU and with the following dependencies - PrgEnv-gnu - cudatoolkit-standalone -- Environment Variable set: `SYCL_DEVICE_SELECTOR=ext_oneapi_cuda:gpu` +- Environment variable is set when loading the module: `ONEAPI_DEVICE_SELECTOR=ext_oneapi_cuda:gpu` ## Example (memory intilization) From b5be8b96067c370b64f119842f82eb743a5cf5ca Mon Sep 17 00:00:00 2001 From: Abhishek Bagusetty Date: Wed, 6 Sep 2023 10:37:32 -0500 Subject: [PATCH 2/5] trying to format tables --- docs/polaris/programming-models/sycl-polaris.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/polaris/programming-models/sycl-polaris.md b/docs/polaris/programming-models/sycl-polaris.md index 585510dab..dc2be341a 100644 --- a/docs/polaris/programming-models/sycl-polaris.md +++ b/docs/polaris/programming-models/sycl-polaris.md @@ -16,6 +16,7 @@ This module (compilers, libraries) gets built periodically from the latest open- ## Components - These are the list of components associated with this module + |-------------------|---------------------------------------------------------| | User Application | Third-Party Library | |-------------------|---------------------------------------------------------| @@ -152,6 +153,7 @@ For further details regarding the arguments passed to `mpiexec` command shown ab [oneMKL Interfaces](https://github.com/oneapi-src/oneMKL) is an open-source implementation of the oneMKL Data Parallel C++ (DPC++) interface according to the [oneMKL specification](https://spec.oneapi.io/versions/latest/elements/oneMKL/source/index.html). It works with multiple devices (backends) using device-specific libraries underneath. oneMKL is part of oneAPI. Various backend supported are shown below. More Information [here](https://github.com/oneapi-src/oneMKL#supported-configurations). + | User Application | Third-Party Library | |------------------|--------------------------------------------------------------| | | [cuBLAS](https://docs.nvidia.com/cuda/cublas/index.html) | @@ -285,6 +287,7 @@ int main() { ``` Compile and Run + The user would need to provide paths the math-libraris as shown below. Also please provide AOCL library for CPU GEMM by `module load aocl`. Environment variables `MKLROOT` is defined with `oneapi` module & `AOCL_ROOT` is defined with `aocl` module. Note: Please pay attention to the linker options for AOCL & oneMKL libraries. From ef0a6782b9d969f3dee992c20d86f81de9838757 Mon Sep 17 00:00:00 2001 From: Abhishek Bagusetty Date: Wed, 6 Sep 2023 10:45:12 -0500 Subject: [PATCH 3/5] formating try-2 --- docs/polaris/programming-models/sycl-polaris.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/docs/polaris/programming-models/sycl-polaris.md b/docs/polaris/programming-models/sycl-polaris.md index dc2be341a..73da6c23b 100644 --- a/docs/polaris/programming-models/sycl-polaris.md +++ b/docs/polaris/programming-models/sycl-polaris.md @@ -12,19 +12,17 @@ module load oneapi !!! note -This module (compilers, libraries) gets built periodically from the latest open-source rather than releases. As such, these compilers will get new features and updates quickly that may break on occasion. Please submit issues at the respective github repositories for the compilers and libraries. + This module (compilers, libraries) gets built periodically from the latest open-source rather than releases. For more details on the release version of compiler, please find the details [here](../compiling-and-linking/oneapi-compiler.md). As such, these compilers will get new features and updates quickly that may break on occasion. Please submit any issues at the respective github repositories for the compilers and libraries. ## Components - These are the list of components associated with this module -|-------------------|---------------------------------------------------------| -| User Application | Third-Party Library | +| User Application | Component | |-------------------|---------------------------------------------------------| | Compilers | [DPC++](https://docs.nvidia.com/cuda/cublas/index.html) | | oneMKL Interfaces | [oneMKL](https://github.com/oneapi-src/oneMKL) | | oneDPL | [oneDPL](https://github.com/oneapi-src/onedpl) | | SYCLomatic/DPCT | [dpct](https://github.com/oneapi-src/syclomatic) | -|-------------------|---------------------------------------------------------| ## Dependencies - SYCL programming model is supported through `oneapi` compilers that were built from source-code @@ -160,7 +158,6 @@ oneMKL is part of oneAPI. Various backend supported are shown below. More Inform | oneMKL interface | [cuSOLVER](https://docs.nvidia.com/cuda/cusolver/index.html) | | | [cuRAND](https://docs.nvidia.com/cuda/curand/index.html) | - ## Example (using onemkl::gemm) The following snippet shows how to compile and run a SYCL code with oneMKL library. For instance, a GPU-based GEMM is performed using `mkl::gemm` API and the results are compared to a CPU-based GEMM performed using the traditional blas (e.g., AOCL-BLIS) library. ```c++ From 58b3c12386666a04ee7fe866d1f08933c1433c61 Mon Sep 17 00:00:00 2001 From: Abhishek Bagusetty Date: Wed, 6 Sep 2023 10:48:36 -0500 Subject: [PATCH 4/5] fix the note --- docs/polaris/programming-models/sycl-polaris.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/polaris/programming-models/sycl-polaris.md b/docs/polaris/programming-models/sycl-polaris.md index 73da6c23b..3b7e2b664 100644 --- a/docs/polaris/programming-models/sycl-polaris.md +++ b/docs/polaris/programming-models/sycl-polaris.md @@ -12,7 +12,7 @@ module load oneapi !!! note - This module (compilers, libraries) gets built periodically from the latest open-source rather than releases. For more details on the release version of compiler, please find the details [here](../compiling-and-linking/oneapi-compiler.md). As such, these compilers will get new features and updates quickly that may break on occasion. Please submit any issues at the respective github repositories for the compilers and libraries. +This module (compilers, libraries) gets built periodically from the latest open-source rather than releases. For more details on the release version of compiler, please find the details [here](../compiling-and-linking/oneapi-compiler.md). As such, these compilers will get new features and updates quickly that may break on occasion. Please submit any issues at the respective github repositories for the compilers and libraries. ## Components - These are the list of components associated with this module From b3f5894b1bba7fa770d1c8f1be2e954a70ca0d19 Mon Sep 17 00:00:00 2001 From: Abhishek Bagusetty Date: Wed, 6 Sep 2023 13:14:19 -0500 Subject: [PATCH 5/5] update module naming --- docs/polaris/compiling-and-linking/oneapi-compiler.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/docs/polaris/compiling-and-linking/oneapi-compiler.md b/docs/polaris/compiling-and-linking/oneapi-compiler.md index 151855e7f..10785c9f5 100644 --- a/docs/polaris/compiling-and-linking/oneapi-compiler.md +++ b/docs/polaris/compiling-and-linking/oneapi-compiler.md @@ -7,10 +7,10 @@ Two oneAPI variants are provided, the first being a "release" version based on I !!! note - The 2023.2.0 release of oneAPI Toolkit does not support oneMKL or oneDPL on Nvidia. + The 2023.2.0 release of oneAPI Toolkit does not yet support oneMKL or oneDPL on Nvidia devices. -The other variant being a build of main from the open source. This variant will be more up-to-date at the risk of bugs and breakages based on code that has not undergone a full release cycle. -The documentation is located on the [SYCL](../programming-models/sycl-polaris.md) page. +The other variant being a build from the open-source. This variant will be more up-to-date at the risk of bugs and breakages based on code that has not undergone a full release cycle. +The documentation is located on the [SYCL](../programming-models/sycl-polaris.md) page. Most notable differences being, `icx/icpx` are the names of C/C++ compilers respectively when using the release version of the module where as `clang/clang++` are for open-source variant. ## Compile and Link @@ -18,9 +18,7 @@ oneAPI uses the clang (or icx/icpx wrapper) for compiling and linking for the Nv ``` -module load PrgEnv-nvhpc -module use /soft/compilers/oneapi/release/modulefiles -module load compiler +module load oneapi/release icpx -std=c++17 -fsycl -fsycl-targets=nvptx64-nvidia-cuda -Xsycl-target-backend --cuda-gpu-arch=sm_80 test.cpp ```