Skip to content

Mark mandatory conformance tests for HW plugins #26015

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 36 commits into from
Closed

Mark mandatory conformance tests for HW plugins #26015

Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
85e813a
skip query model test while mandatory
Aug 12, 2024
7581a27
Revert "skip query model test while mandatory"
Aug 14, 2024
1137f38
Merge branch 'master' into haiqi/fix_query_model
Aug 21, 2024
dd84594
Merge branch 'master' into haiqi/fix_query_model
Aug 22, 2024
0aa7547
add query model macro
Aug 23, 2024
35dffc9
change mandatory to optional
Aug 23, 2024
6d58a4a
fix comment
Aug 23, 2024
88ff413
Update to MARK_MANDATORY_API_FOR_HW_DEVICE
peterchen-intel Aug 25, 2024
1eb33d8
change macro name
Aug 26, 2024
820a47f
fix empty string failure
Aug 27, 2024
7a5f9ab
Merge branch 'master' into haiqi/fix_query_model
Aug 29, 2024
2c6be6e
remove sw_plugin_in_target_device
Aug 29, 2024
d0cedf6
remove OVClassModelTestP in test suite
Sep 3, 2024
3da49e3
fix OVCheckChangePropComplieModleGetPropTests_DEVICE_ID
Sep 9, 2024
746f47c
remove sw_plugin_in_target_device in OVCheckChangePropComplieModleGet…
Sep 9, 2024
26aae19
MARK_MANDATORY_API_FOR_HW_DEVICE support NULL
peterchen-intel Sep 9, 2024
33ee9a0
Revert "MARK_MANDATORY_API_FOR_HW_DEVICE support NULL"
Sep 11, 2024
47f064a
remove sw_plugin_in_target_device
Sep 11, 2024
b6ee44f
remove sw_plugin_target_device in life_time.cpp
Sep 11, 2024
67809f9
add header
Sep 11, 2024
6716b56
Merge branch 'master' into haiqi/fix_query_model
Sep 11, 2024
e2f8ab8
add function point for MARK_MANDATORY_API_FOR_HW_DEVICE
Sep 11, 2024
d4b3a38
add function pointer for MARK_MANDATORY_API_FOR_HW_DEVICE
Sep 11, 2024
ad7a975
move pointer
Sep 11, 2024
d0eb4ca
fix hetero error
Sep 11, 2024
7e40e6a
fix code style
Sep 11, 2024
ec2b2e7
skip GetMetricAndPrintNoThrow_AVAILABLE_DEVICES for hetero
Sep 13, 2024
ca0d462
revert
Sep 13, 2024
5c7ae6d
remove pointer
Sep 13, 2024
19d7d1c
Merge branch 'master' into haiqi/fix_query_model
Sep 13, 2024
77c21f0
Merge branch 'master' into haiqi/fix_query_model
Sep 14, 2024
eb36a2e
Merge branch 'master' into haiqi/fix_query_model
Sep 16, 2024
e608d90
Merge branch 'master' into haiqi/fix_query_model
peterchen-intel Sep 18, 2024
54fc7c9
Merge branch 'master' into haiqi/fix_query_model
peterchen-intel Sep 22, 2024
f02dfa4
revert OVClassModelTestP
Sep 24, 2024
38f6524
fix code style
Sep 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion ...ns/hetero/tests/functional/shared_tests_instances/behavior/ov_plugin/properties_tests.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

#include "behavior/ov_plugin/properties_tests.hpp"

#include "base/ov_behavior_test_utils.hpp"
#include "behavior/compiled_model/properties.hpp"
#include "openvino/runtime/properties.hpp"

Expand Down Expand Up @@ -32,9 +33,11 @@ INSTANTIATE_TEST_SUITE_P(smoke_Hetero_BehaviorTests,
::testing::ValuesIn(hetero_properties)),
OVPropertiesTests::getTestCaseName);

static std::string (*NULL_getTestCaseName)(const testing::TestParamInfo<std::string>& info) = NULL;
INSTANTIATE_TEST_SUITE_P(smoke_HeteroOVGetMetricPropsTest,
OVGetMetricPropsTest,
::testing::Values(ov::test::utils::DEVICE_HETERO));
::testing::Values(ov::test::utils::DEVICE_HETERO),
MARK_MANDATORY_API_FOR_HW_DEVICE(NULL_getTestCaseName));

INSTANTIATE_TEST_SUITE_P(
smoke_HeteroOVCheckGetSupportedROMetricsPropsTests,
Expand Down
12 changes: 6 additions & 6 deletions .../plugin/conformance/test_runner/api_conformance_runner/src/ov_plugin/core_integration.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@
#include "behavior/ov_plugin/query_model.hpp"
#include "openvino/runtime/core.hpp"
#include "ov_api_conformance_helpers.hpp"
#include "behavior/ov_plugin/properties_tests.hpp"
#include "base/ov_behavior_test_utils.hpp"

using namespace ov::test::behavior;
using namespace ov::test::conformance;
Expand All @@ -15,17 +17,15 @@ namespace {
// OV Class Common tests with <pluginName, deviceName params>
//

INSTANTIATE_TEST_SUITE_P(ov_plugin_mandatory,
OVClassModelTestP,
::testing::Values(ov::test::utils::target_device));

INSTANTIATE_TEST_SUITE_P(ov_plugin,
OVClassModelOptionalTestP,
::testing::Values(ov::test::utils::target_device));

// OV Class Query network

INSTANTIATE_TEST_SUITE_P(ov_plugin_mandatory,
static std::string (*NULL_getTestCaseName)(const testing::TestParamInfo<std::string>& info) = NULL;
INSTANTIATE_TEST_SUITE_P(ov_plugin,
OVClassQueryModelTest,
::testing::Values(ov::test::utils::target_device));
::testing::Values(ov::test::utils::target_device),
MARK_MANDATORY_API_FOR_HW_DEVICE(NULL_getTestCaseName));
Copy link
Contributor

@iefode iefode Oct 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please explain the technical details of the changes. I why you create NULLTestCaseName for sw plugin... Is there any other approaches? I asked you because it is not so clear...

Copy link
Contributor

@peterchen-intel peterchen-intel Oct 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason is to support 2 types test cases with unify MACRO:
Some API test cases have implemented getTestCaseName()
For example OVHoldersTestOnImportedNetwork::getTestCaseName
Some are not. For example OVGetMetricPropsTest

Another option, define 2 MACROs:
MARK_MANDATORY_API_FOR_HW_DEVICE_0 without parameter.
MARK_MANDATORY_API_FOR_HW_DEVICE_1 with parameter

Copy link
Contributor

@iefode iefode Oct 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to rename as WITH_PARAM and WITHOUT_PARAM

In general, LGTM

} // namespace
8 changes: 5 additions & 3 deletions ...ctional/plugin/conformance/test_runner/api_conformance_runner/src/ov_plugin/life_time.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@

#include "behavior/ov_plugin/life_time.hpp"
#include "ov_api_conformance_helpers.hpp"
#include "base/ov_behavior_test_utils.hpp"

using namespace ov::test::behavior;
using namespace ov::test::conformance;
Expand All @@ -14,7 +15,8 @@ INSTANTIATE_TEST_SUITE_P(ov_plugin_mandatory, OVHoldersTest,
::testing::Values(ov::test::utils::target_device),
OVHoldersTest::getTestCaseName);

INSTANTIATE_TEST_SUITE_P(ov_plugin_mandatory, OVHoldersTestOnImportedNetwork,
::testing::Values(ov::test::utils::target_device),
OVHoldersTestOnImportedNetwork::getTestCaseName);
INSTANTIATE_TEST_SUITE_P(ov_plugin,
OVHoldersTestOnImportedNetwork,
::testing::Values(ov::test::utils::target_device),
MARK_MANDATORY_API_FOR_HW_DEVICE(OVHoldersTestOnImportedNetwork::getTestCaseName));
} // namespace
15 changes: 9 additions & 6 deletions ...tional/plugin/conformance/test_runner/api_conformance_runner/src/ov_plugin/properties.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,17 +105,20 @@ INSTANTIATE_TEST_SUITE_P(ov_plugin, OVCheckMetricsPropsTests_ModelDependceProps,
// OV Class GetMetric
//

INSTANTIATE_TEST_SUITE_P(
ov_plugin_mandatory, OVGetMetricPropsTest,
::testing::Values(ov::test::utils::target_device));
static std::string (*NULL_getTestCaseName)(const testing::TestParamInfo<std::string>& info) = NULL;
INSTANTIATE_TEST_SUITE_P(ov_plugin,
OVGetMetricPropsTest,
::testing::Values(ov::test::utils::target_device),
MARK_MANDATORY_API_FOR_HW_DEVICE(NULL_getTestCaseName));

INSTANTIATE_TEST_SUITE_P(
ov_plugin, OVGetMetricPropsOptionalTest,
::testing::Values(ov::test::utils::target_device));

INSTANTIATE_TEST_SUITE_P(
ov_plugin_mandatory, OVGetAvailableDevicesPropsTest,
::testing::Values(ov::test::utils::target_device));
INSTANTIATE_TEST_SUITE_P(ov_plugin,
OVGetAvailableDevicesPropsTest,
::testing::Values(ov::test::utils::target_device),
MARK_MANDATORY_API_FOR_HW_DEVICE(NULL_getTestCaseName));

//
// OV Class GetConfig
Expand Down
14 changes: 11 additions & 3 deletions src/tests/functional/plugin/shared/include/base/ov_behavior_test_utils.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,9 +30,17 @@
#include "common_test_utils/subgraph_builders/concat_with_params.hpp"
#include "common_test_utils/subgraph_builders/split_concat.hpp"

#define MARK_MANDATORY_PROPERTY_FOR_HW_DEVICE(GET_TEST_NAME) \
[](const testing::TestParamInfo<PropertiesParams>& info) { \
std::string name = GET_TEST_NAME(info); \
#define MARK_MANDATORY_PROPERTY_FOR_HW_DEVICE(GET_TEST_NAME) \
[](const testing::TestParamInfo<PropertiesParams>& info) { \
std::string name = GET_TEST_NAME(info); \
return (sw_plugin_in_target_device(ov::test::utils::target_device) ? "" : "mandatory_") + name; \
}

#define MARK_MANDATORY_API_FOR_HW_DEVICE(GET_TEST_NAME) \
[](const testing::TestParamInfo<std::string>& info) { \
std::string name = ""; \
if (NULL != GET_TEST_NAME) \
name += GET_TEST_NAME(info); \
return (sw_plugin_in_target_device(ov::test::utils::target_device) ? "" : "mandatory_") + name; \
}

Expand Down
3 changes: 0 additions & 3 deletions src/tests/functional/plugin/shared/include/behavior/ov_plugin/query_model.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,9 +76,6 @@ TEST_P(OVClassQueryModelTest, QueryModelWithMatMul) {
}

TEST_P(OVClassQueryModelTest, QueryModelHETEROWithDeviceIDNoThrow) {
if (sw_plugin_in_target_device(target_device)) {
return;
}
ov::Core ie = ov::test::utils::create_core();

auto deviceIDs = ie.get_property(target_device, ov::available_devices);
Expand Down
6 changes: 0 additions & 6 deletions src/tests/functional/plugin/shared/src/behavior/ov_plugin/life_time.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,9 +148,6 @@ void OVHoldersTestOnImportedNetwork::TearDown() {
}

TEST_P(OVHoldersTestOnImportedNetwork, LoadedTensor) {
if (sw_plugin_in_target_device(target_device)) {
return;
}
ov::Core core = ov::test::utils::create_core();
std::stringstream stream;
{
Expand All @@ -163,9 +160,6 @@ TEST_P(OVHoldersTestOnImportedNetwork, LoadedTensor) {
}

TEST_P(OVHoldersTestOnImportedNetwork, CreateRequestWithCoreRemoved) {
if (sw_plugin_in_target_device(target_device)) {
return;
}
ov::Core core = ov::test::utils::create_core();
std::stringstream stream;
{
Expand Down
12 changes: 0 additions & 12 deletions src/tests/functional/plugin/shared/src/behavior/ov_plugin/properties_tests.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -588,9 +588,6 @@ TEST_P(OVCheckMetricsPropsTests_ModelDependceProps, ChangeCorrectDevicePropertie
TEST_P(OVClassSetDefaultDeviceIDPropTest, SetDefaultDeviceIDNoThrow) {
ov::Core ie = ov::test::utils::create_core();
// sw plugins are not requested to support `ov::available_devices` and ` ov::device::id` property
if (sw_plugin_in_target_device(target_device)) {
return;
}
auto deviceIDs = ie.get_property(target_device, ov::available_devices);
if (std::find(deviceIDs.begin(), deviceIDs.end(), deviceID) == deviceIDs.end()) {
GTEST_FAIL();
Expand All @@ -613,9 +610,6 @@ TEST_P(OVSpecificDeviceSetConfigTest, GetConfigSpecificDeviceNoThrow) {
deviceID = target_device.substr(pos + 1, target_device.size());
}
// sw plugins are not requested to support `ov::available_devices`, `ov::device::id` and `ov::num_streams` property
if (sw_plugin_in_target_device(target_device)) {
return;
}
auto deviceIDs = ie.get_property(clear_target_device, ov::available_devices);
if (std::find(deviceIDs.begin(), deviceIDs.end(), deviceID) == deviceIDs.end()) {
GTEST_FAIL() << "No DeviceID" << std::endl;
Expand Down Expand Up @@ -672,9 +666,6 @@ TEST_P(OVSpecificDeviceGetConfigTest, GetConfigSpecificDeviceNoThrow) {
}

TEST_P(OVGetAvailableDevicesPropsTest, GetAvailableDevicesNoThrow) {
if (sw_plugin_in_target_device(target_device)) {
return;
}
ov::Core ie = ov::test::utils::create_core();
std::vector<std::string> devices;

Expand Down Expand Up @@ -727,9 +718,6 @@ TEST_P(OVClassSetDevicePriorityConfigPropsTest, SetConfigAndCheckGetConfigNoThro
}

TEST_P(OVGetMetricPropsTest, GetMetricAndPrintNoThrow_AVAILABLE_DEVICES) {
if (sw_plugin_in_target_device(target_device)) {
return;
}
ov::Core ie = ov::test::utils::create_core();
std::vector<std::string> device_ids;

Expand Down