Trilinos: OneAPI MKL support (for SYCL backend) #12941
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
brian-kelley
added
type: enhancement
|
Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request. |
|
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: Trilinos_PR_gcc-8.3.0
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-serial
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-debug
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_clang-11.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_python3
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_cuda-11.4.2-uvm-off
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_intel-2021.3
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_cuda-11.4.20-uvm
Jenkins Parameters
Using Repos:
Pull Request Author: brian-kelley |
|
Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED Pull Request Auto Testing has PASSED (click to expand)Build InformationTest Name: Trilinos_PR_gcc-8.3.0
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-serial
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-debug
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_clang-11.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_python3
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_cuda-11.4.2-uvm-off
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_intel-2021.3
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_cuda-11.4.20-uvm
Jenkins Parameters
|
|
Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging |
|
All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur... |
jhux2
reviewed
Apr 19, 2024
|
Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging |
|
All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur... |
bartlettroscoe
previously approved these changes
Apr 19, 2024
There was a problem hiding this comment.
Just mainly commenting on the TriBITS TPL upgrade of FindTPLMKL.cmake ...
... Tribits TPL expert (@bartlettroscoe ?) on the way the MKL cmake package is loaded (it's ugly but simple, and seems to work...)
This does not look that ugly to me. Other than some hackery around having to set CMAKE_CXX_COMPILER to icpx, this is basically just calling:
find_package(MKL)
tribits_extpkg_create_imported_all_libs_target_and_config_file( MKL
INNER_FIND_PACKAGE_NAME MKL
IMPORTED_TARGETS_FOR_ALL_LIBS MKL::MKL MKL::MKL_DPCPP)
which is the about simplest modern TriBITS FindTPL<tplName>.cmake file you can image as per:
The rest of the code in this FindTPLMKL.cmake file is just asserting the correct state after calling find_package(MKL) or using the old implementation calling TRIBITS_TPL_FIND_INCLUDE_DIRS_AND_LIBRARIES() when Kokkos_ENABLE_SYCL is not enabled.
My only concern is how this impacts the BLAS and LAPACK TPLs since MKL::MKL could contain libraries for BLAS and LAPACK. Therefore, what happens if Trilinos uses non-MKL for BLAS and/or LAPACK but calls find_package(MKL) and links in MKL::MKL like this?
I wish we could refactor FindTPLBLAS.cmake and FindTPLLAPACK.cmake to also use find_package(MKL) when the MKL when it is available (or when Kokkos_ENABLE_SYCL=ON) . But it seems this may not be so easy:
- https://gitlab.kitware.com/cmake/cmake/-/issues/22831
- https://community.intel.com/t5/Intel-oneAPI-Math-Kernel-Library/MKLConfig-cmake-is-great-but-should-add-further-granularity-for/m-p/1375464
The CMake package ecosystem is in such an unfortunate state :-(
bartlettroscoe
dismissed
their stale review
I did not mean to approve this PR. I was just trying to comment on it.
|
All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur... |
2 similar comments
|
All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur... |
|
All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur... |
|
Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request. |
|
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: Trilinos_PR_gcc-8.3.0
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-serial
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-debug
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_clang-11.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_python3
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_cuda-11.4.2-uvm-off
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_intel-2021.3
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_cuda-11.4.20-uvm
Jenkins Parameters
Using Repos:
Pull Request Author: brian-kelley |
MKL does provide Intel's implementations of blas and lapack routines. Right? |
|
Status Flag 'Pull Request AutoTester' - Error: Jenkins Jobs - A user has pushed a change to the PR before testing completed. NEW EVENT 'committed', ID C_kwDOAsJyMdoAKDQyMWI4MzRjZWU2YzQzYmJlMmMwMDExNDVkZjlhZmVlNDEwYmNjMzU... The Jenkins Jobs will be shutdown; Testing of this PR must occur again. |
|
Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED Note: Testing will normally be attempted again in approx. 2 Hrs 30 Mins. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run. Pull Request Auto Testing has FAILED (click to expand)Build InformationTest Name: Trilinos_PR_gcc-8.3.0
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-serial
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-debug
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_clang-11.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_python3
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_cuda-11.4.2-uvm-off
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_intel-2021.3
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_cuda-11.4.20-uvm
Jenkins Parameters
|
|
Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection is Not Necessary for this Pull Request. |
|
Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects: Pull Request Auto Testing STARTING (click to expand)Build InformationTest Name: Trilinos_PR_gcc-8.3.0
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-serial
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-debug
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_clang-11.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_python3
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_cuda-11.4.2-uvm-off
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_intel-2021.3
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_cuda-11.4.20-uvm
Jenkins Parameters
Using Repos:
Pull Request Author: brian-kelley |
|
@jhux2 That's correct, if MKL is enabled then we should use MKL for all BLAS/LAPACK functions and not try to link in OpenBLAS or something. The changes I added today fix this. @bartlettroscoe I figured out how to do the CMake side of things correctly - for each of the two paths for FindTPLMKL, I added the logic to also set up the BLAS and LAPACK targets (if they are enabled respectively). Now the problem is that Teuchos BLAS/LAPACK wrappers assume that all the functions take dimensions/strides as |
brian-kelley
added
the
AT: WIP
There was a problem hiding this comment.
@brian-kelley, it is excellent that the BLAS and LAPACK TPLs are made to use MKL when MKL is enabled. I am must questing if the BLAS-specific and LAPACK-specific logic should not be moved to FindTPLBLAS.cmake and FindTPLLAPACK.cmake, respectively (i.e. under if (TPL_ENABLE_MKL))?
|
Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED Pull Request Auto Testing has PASSED (click to expand)Build InformationTest Name: Trilinos_PR_gcc-8.3.0
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-serial
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_gcc-8.3.0-debug
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_clang-11.0.1
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_python3
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_cuda-11.4.2-uvm-off
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_intel-2021.3
Jenkins Parameters
Build InformationTest Name: Trilinos_PR_cuda-11.4.20-uvm
Jenkins Parameters
|
|
Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging |
|
All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur... |
It's int normally, or MKL_INT if MKL is enabled (MKL_INT may be int or long)
when MKL is providing the libraries
|
I think this is basically done, except for all the Teuchos LAPACK wrappers needing to deal with the conversion between int and TeuchosNumerics_Int. This is more painful than BLAS because now there are arrays of int (like |
|
Yesterday @lucbv gave a great suggestion - for Trilinos builds, just link in |
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@trilinos/tribits
@trilinos/kokkos-kernels
@trilinos/tpetra
I made this a draft because I would like to see feedback from at least @lucbv on the KK side and a Tribits TPL expert (@bartlettroscoe ?) on the way the MKL cmake package is loaded (it's ugly but simple, and seems to work...)
Motivation
Before,
TPL_ENABLE_MKL=ONfor SYCL/OneAPI builds wouldn't enable MKL at all.libmkl_rt.sowould be the only library linked in, but this only pulls in the host libraries (no SYCL). Because of this KokkosKernels explicitly disabled all MKL tpl wrappers when building as a Trilinos package. This enables MKL in that situation on both CPU and GPU so that KokkosKernels can use it (and re-enables the KK wrappers).There are some KokkosKernels changes here to get rid of the "SYCL override" variable that explicitly disables TPLs when building as a Trilinos package. This would need to get added into a separate KokkosKernels PR.
Stakeholder Feedback
Testing
On sunspot: built and ran KokkosKernels+tests as a Trilinos package, with MKL enabled. Some tests fail, but they're the same set as when building KokkosKernels standalone. Most Tpetra tests pass as well (not all passed before either).