Skip to content

[rpp] Add standardized test filters#8790

Open
LakshmiKumar23 wants to merge 1 commit into
developfrom
lk/test_filtering
Open

[rpp] Add standardized test filters#8790
LakshmiKumar23 wants to merge 1 commit into
developfrom
lk/test_filtering

Conversation

@LakshmiKumar23

Copy link
Copy Markdown
Contributor

Motivation

As per https://github.com/ROCm/TheRock/blob/main/docs/development/test_filtering.md, we want to add test filter options to RPP.

Technical Details

This PR adds these options to rpp ctests, with the existing test base. More tests can be added as required.

Label Tests Target Time
quick rpp_sanity_test_brightness_host_f32, rpp_sanity_test_brightness_hip_f32 < 5 min
standard quick tests + rpp_qa_tests_tensor_misc_host_all, rpp_qa_tests_tensor_misc_hip_all < 30 min
comprehensive All rpp_qa_tests_tensor_*_all (image, voxel, audio, misc for both HOST and HIP) < 2 hours
full Everything including qa_mode_0 and test_type_1 (performance) tests hours+

Test Plan

Test ctests with these labels:

ctest -L quick # Sanity checks only
ctest -L standard # PR validation
ctest -L comprehensive # Nightly tests
ctest -L full # Full test suite

Test Result

Example of quick test results:

lakshmi@kapu:~/work/rpp-test$ ctest -VV -L quick
UpdateCTestConfiguration  from :/home/lakshmi/work/rpp-test/DartConfiguration.tcl
Parse Config file:/home/lakshmi/work/rpp-test/DartConfiguration.tcl
UpdateCTestConfiguration  from :/home/lakshmi/work/rpp-test/DartConfiguration.tcl
Parse Config file:/home/lakshmi/work/rpp-test/DartConfiguration.tcl
Test project /home/lakshmi/work/rpp-test
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
    Start 1: rpp_sanity_test_brightness_host_f32

1: Test command: /usr/local/bin/ctest "--build-and-test" "/home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/share/rpp/test/HOST" "/home/lakshmi/work/rpp-test/HOST" "--build-generator" "Unix Makefiles" "--test-command" "Tensor_image_host" "/home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/share/rpp/test/TEST_IMAGES/three_images_mixed_src1" "/home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/share/rpp/test/TEST_IMAGES/three_images_mixed_src1" "/home/lakshmi/work/rpp-test/HOST/output" "2" "0" "0" "0" "1" "0" "0" "1" "0" "0" "1" "0" "0" "0" "0" "/home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/share/rpp/test"
1: Working Directory: /home/lakshmi/work/rpp-test
1: Test timeout computed to be: 1500
1: Internal cmake changing into directory: /home/lakshmi/work/rpp-test/HOST
1: ======== CMake output     ======
1: HIP: Using hipcc from relative path: /home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/bin/hipcc
1: -- test_suite/HOST: rpp found with find_package(rpp QUIET)
1: --   rpp_INCLUDE_DIR -- /home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/include
1: --   rpp_LIBRARY -- /home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/lib/librpp.so
1: --   rpp_BACKEND_TYPE -- HIP
1: --   rpp_AUDIO_AUGMENTATIONS_SUPPORT -- OFF
1: -- test_suite/HOST is supported to build and run with RPP HIP backend installation
1: -- Python3 found at /usr/bin/python3.11
1: -- test_suite/HOST All Tensor HOST tests set to build with OpenMP
1: -- test_suite/HOST Tensor-Image HOST tests set to build with OpenCV (decoder_type 1 and image dumps enabled)
1: -- Tensor_image_host -- Link Libraries: see target_link_libraries
1: -- test_suite/HOST Tensor-Voxel HOST tests set to build with libniftiio
1: -- test_suite/HOST Tensor-Voxel HOST tests set to build with OpenCV (slice JPG dumps enabled)
1: -- Warning: RPP_AUDIO_SUPPORT not found in RPP installation. It must be installed to build test_suite/HOST Tensor-Audio tests successfully!
1: Configuring done
1: Generating done
1: Build files have been written to: /home/lakshmi/work/rpp-test/HOST
1: ======== End CMake output ======
1: Change Dir: /home/lakshmi/work/rpp-test/HOST
1:
1: Run Clean Command:/usr/bin/gmake -f Makefile clean
1:
1: Run Build Command(s):/usr/bin/gmake -f Makefile && [ 16%] Building CXX object CMakeFiles/Tensor_image_host.dir/Tensor_image_host.cpp.o
1: [ 33%] Linking CXX executable Tensor_image_host
1: [ 33%] Built target Tensor_image_host
1: [ 50%] Building CXX object CMakeFiles/Tensor_voxel_host.dir/Tensor_voxel_host.cpp.o
1: [ 66%] Linking CXX executable Tensor_voxel_host
1: [ 66%] Built target Tensor_voxel_host
1: [ 83%] Building CXX object CMakeFiles/Tensor_misc_host.dir/Tensor_misc_host.cpp.o
1: [100%] Linking CXX executable Tensor_misc_host
1: [100%] Built target Tensor_misc_host
1:
1: Running test command: "/home/lakshmi/work/rpp-test/HOST/Tensor_image_host" "/home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/share/rpp/test/TEST_IMAGES/three_images_mixed_src1" "/home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/share/rpp/test/TEST_IMAGES/three_images_mixed_src1" "/home/lakshmi/work/rpp-test/HOST/output" "2" "0" "0" "0" "1" "0" "0" "1" "0" "0" "1" "0" "0" "0" "0" "/home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/share/rpp/test"
1:
1: Inputs for this test case are:
1: src1 = /home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/share/rpp/test/TEST_IMAGES/three_images_mixed_src1
1: src2 = /home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/share/rpp/test/TEST_IMAGES/three_images_mixed_src1
1: dst = /home/lakshmi/work/rpp-test/HOST/output
1: u8 / f16 / f32 / u8->f16 / u8->f32 / i8 / u8->i8 (0/1/2/3/4/5/6) = 2
1: outputFormatToggle (pkd->pkd = 0 / pkd->pln = 1) = 0
1: case number (0:101) = 0
1: number of times to run = 1
1: test type - (0 = unit tests / 1 = performance tests) = 0
1: layout type - (0 = PKD3 / 1 = PLN3 / 2 = PLN1) = 0
1: qa mode - 0/1 = 0
1: decoder type - (0 = packed .rgb / 1 = OpenCV) = 0
1: batch size = 1
1: Running brightness_f32_Tensor_HOST_PKD3_toPKD3 1 times (each time with a batch size of 1 images) and computing mean statistics...
1:
1: Execution Timings for Iteration 1:
1: CPU Backend Clock Time: 0.046 ms/batch
1: CPU Backend Wall Time: 0.0500679 ms/batch
1:
1: Execution Timings for Iteration 2:
1: CPU Backend Clock Time: 0.06 ms/batch
1: CPU Backend Wall Time: 0.0619888 ms/batch
1:
1: Execution Timings for Iteration 3:
1: CPU Backend Clock Time: 0.062 ms/batch
1: CPU Backend Wall Time: 0.0629425 ms/batch
1:
1/2 Test #1: rpp_sanity_test_brightness_host_f32 ...   Passed    6.14 sec
test 2
    Start 2: rpp_sanity_test_brightness_hip_f32

2: Test command: /usr/local/bin/ctest "--build-and-test" "/home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/share/rpp/test/HIP" "/home/lakshmi/work/rpp-test/HIP" "--build-generator" "Unix Makefiles" "--test-command" "Tensor_image_hip" "/home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/share/rpp/test/TEST_IMAGES/three_images_mixed_src1" "/home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/share/rpp/test/TEST_IMAGES/three_images_mixed_src1" "/home/lakshmi/work/rpp-test/HIP/output" "2" "0" "0" "0" "1" "0" "0" "1" "0" "0" "1" "0" "0" "0" "0" "/home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/share/rpp/test"
2: Working Directory: /home/lakshmi/work/rpp-test
2: Test timeout computed to be: 1500
2: Internal cmake changing into directory: /home/lakshmi/work/rpp-test/HIP
2: ======== CMake output     ======
2: HIP: Using hipcc from relative path: /home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/bin/hipcc
2: -- test_suite/HIP: rpp found with find_package(rpp QUIET)
2: --   rpp_INCLUDE_DIR -- /home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/include
2: --   rpp_LIBRARY -- /home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/lib/librpp.so
2: --   rpp_BACKEND_TYPE -- HIP
2: --   rpp_AUDIO_AUGMENTATIONS_SUPPORT -- OFF
2: -- test_suite/HIP is supported to build and run with RPP HIP backend installation
2: HIP: Using hipcc from relative path: /home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/bin/hipcc
2: -- HIP found at HIP_INCLUDE_DIRS - /home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/include and HIP_LIBRARIES - hip::host;hip::device
2: -- Python3 found at /usr/bin/python3.11
2: -- test_suite/HIP All Tensor HIP tests set to build with OpenMP
2: -- test_suite/HIP Tensor-Image HIP tests set to build with OpenCV (decoder_type 1 and image dumps enabled)
2: -- Tensor_image_hip -- Link Libraries: see target_link_libraries
2: -- test_suite/HIP Tensor-Voxel HIP tests set to build with libniftiio
2: -- test_suite/HIP Tensor-Voxel HIP tests set to build with OpenCV (slice JPG dumps enabled)
2: -- Tensor_voxel_hip -- Link Libraries: see target_link_libraries
2: -- Warning: RPP_AUDIO_SUPPORT not found in RPP installation. It must be installed to build test_suite/HIP Tensor-Audio tests successfully!
2: -- Tensor_misc_hip -- Link Libraries: rpp::rpp;hip::host;hip::device;stdc++fs;OpenMP::OpenMP_CXX
2: Configuring done
2: Generating done
2: Build files have been written to: /home/lakshmi/work/rpp-test/HIP
2: ======== End CMake output ======
2: Change Dir: /home/lakshmi/work/rpp-test/HIP
2:
2: Run Clean Command:/usr/bin/gmake -f Makefile clean
2:
2: Run Build Command(s):/usr/bin/gmake -f Makefile && [ 16%] Building CXX object CMakeFiles/Tensor_image_hip.dir/Tensor_image_hip.cpp.o
2: [ 33%] Linking CXX executable Tensor_image_hip
2: [ 33%] Built target Tensor_image_hip
2: [ 50%] Building CXX object CMakeFiles/Tensor_voxel_hip.dir/Tensor_voxel_hip.cpp.o
2: [ 66%] Linking CXX executable Tensor_voxel_hip
2: [ 66%] Built target Tensor_voxel_hip
2: [ 83%] Building CXX object CMakeFiles/Tensor_misc_hip.dir/Tensor_misc_hip.cpp.o
2: [100%] Linking CXX executable Tensor_misc_hip
2: [100%] Built target Tensor_misc_hip
2:
2: Running test command: "/home/lakshmi/work/rpp-test/HIP/Tensor_image_hip" "/home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/share/rpp/test/TEST_IMAGES/three_images_mixed_src1" "/home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/share/rpp/test/TEST_IMAGES/three_images_mixed_src1" "/home/lakshmi/work/rpp-test/HIP/output" "2" "0" "0" "0" "1" "0" "0" "1" "0" "0" "1" "0" "0" "0" "0" "/home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/share/rpp/test"
2:
2: Inputs for this test case are:
2: src1 = /home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/share/rpp/test/TEST_IMAGES/three_images_mixed_src1
2: src2 = /home/lakshmi/rock_0408/lib/python3.10/site-packages/_rocm_sdk_devel/share/rpp/test/TEST_IMAGES/three_images_mixed_src1
2: dst = /home/lakshmi/work/rpp-test/HIP/output
2: u8 / f16 / f32 / u8->f16 / u8->f32 / i8 / u8->i8 (0/1/2/3/4/5/6) = 2
2: outputFormatToggle (pkd->pkd = 0 / pkd->pln = 1) = 0
2: case number (0:101) = 0
2: number of times to run = 1
2: test type - (0 = unit tests / 1 = performance tests) = 0
2: layout type - (0 = PKD3/ 1 = PLN3/ 2 = PLN1) = 0
2: qa mode - 0/1 = 0
2: decoder type - (0 = packed .rgb / 1 = OpenCV) = 0
2: batch size = 1
2: Running brightness_f32_Tensor_HIP_PKD3_toPKD3 1 times (each time with a batch size of 1 images) and computing mean statistics...
2:
2: Execution Timings for Iteration 1:
2: GPU Backend Wall Time: 3.73983 ms/batch
2:
2: Execution Timings for Iteration 2:
2: GPU Backend Wall Time: 0.0448227 ms/batch
2:
2: Execution Timings for Iteration 3:
2: GPU Backend Wall Time: 0.0429153 ms/batch
2/2 Test #2: rpp_sanity_test_brightness_hip_f32 ....   Passed   13.19 sec

100% tests passed, 0 tests failed out of 2

Label Time Summary:
all              =  19.33 sec*proc (2 tests)
comprehensive    =  19.33 sec*proc (2 tests)
full             =  19.33 sec*proc (2 tests)
nightly          =  19.33 sec*proc (2 tests)
pr               =  19.33 sec*proc (2 tests)
pre-commit       =  19.33 sec*proc (2 tests)
quick            =  19.33 sec*proc (2 tests)
smoke            =  19.33 sec*proc (2 tests)
standard         =  19.33 sec*proc (2 tests)
weekly           =  19.33 sec*proc (2 tests)

Total Test time (real) =  19.33 sec

Submission Checklist

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant