Neutron whitebox tests default to single thread

When new test is added in whitebox-neutron-tempest-plugin [1],
it is executed using inclusive regex in multi-thread workflow step,
this may cause failures in CI jobs even if test is stable, that is
due to many tests added to neutron whitebox need to restart services,
capture traffic without interference, etc.

This MR changes inclusive regex to be in single-thread workflow step,
*otherwise new tests will break CI jobs running them*.

The tests skipped in single-thread explicitly instead of in multi-thread
according to latest results from neutron component job, any test class
that had cases in both single/multi was specified by method name.

The exclude/include in whitebox_neutron_tempest_jobs.yaml is made based
on same job, so results should be similar, validated with testproject.

Related ticket: OSPRH-10217

[1] https://opendev.org/x/whitebox-neutron-tempest-plugin

Author: m-blue-m

zuul.d/whitebox_neutron_tempest_jobs.yaml

@@ -91,71 +91,84 @@
           tempestRun:
             concurrency: 6
             includeList: |
-              whitebox_neutron_tempest_plugin.*
+              # neutron whitebox multi-thread tests
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_basic.NetworkBasicTest
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_broadcast.BroadcastTestIPv4Common
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_broadcast.BroadcastTestVlanTransparency
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_extra_dhcp_opts.ExtraDhcpOptionsTest
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_extra_dhcp_opts.OvnExtraDhcpOptionsTest
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_internal_dns.InternalDNSTestOvn
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_multicast.MulticastTestIPv4Common.test_flooding_when_special_groups
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_multicast.MulticastTestIPv4Common.test_igmp_snooping_same_network_and_unsubscribe
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_multicast.MulticastTestIPv4Ovn
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_multicast.MulticastTestVlanTransparency
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_portsecurity.PortSecurityTest
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.QosTestCommon.test_bw_limit_east_west
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.QosTestCommon.test_bw_limit_tenant_network
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.QosTestCommon.test_default_qos_policy
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.QosTestCommon.test_dscp_marking_east_west
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.QosTestCommon.test_dscp_marking_tenant_network
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.QosTestCommon.test_qos_after_cold_migration
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.QosTestCommon.test_qos_after_live_migration
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.QosTestDscpInheritanceOvn
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.QosTestOvn
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.QosTestSriovBwLimitTest
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.QosTestSriovMinBwPlacementEnforcementTest
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_security_groups.NetworkDefaultSecGroupTest
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_vlan_transparency.MtuVlanTransparencyTest
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_vlan_transparency.MultiPortVlanTransparencyTest
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_vlan_transparency.MultiVlanTransparencyTest
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_vrrp.VrrpTest
             excludeList: |
-              neutron_tempest_plugin.scenario.test_mtu.NetworkWritableMtuTest
               test_multicast.*ext*
               test_multicast.*restart
-              whitebox_neutron_tempest_plugin.*south_north
-              whitebox_neutron_tempest_plugin.tests.scenario.*test_mtu
               ^neutron_.*plugin..*scenario.test_.*macvtap
-              ^neutron_tempest_plugin.fwaas.*
-              ^whitebox_neutron_tempest_plugin.*many_vms
-              ^whitebox_neutron_tempest_plugin.*networker_reboot
-              ^whitebox_neutron_tempest_plugin.*ovn_controller_restart
-              ^whitebox_neutron_tempest_plugin.*reboot_node
-              ^whitebox_neutron_tempest_plugin.*test_previously_used_ip
-              ^whitebox_neutron_tempest_plugin.tests.scenario.test_internal_dns.InternalDNSInterruptions.*
-              ^whitebox_neutron_tempest_plugin.tests.scenario.test_l3ha_ovn.*
-              ^whitebox_neutron_tempest_plugin.tests.scenario.test_metadata_rate_limiting
-              ^whitebox_neutron_tempest_plugin.tests.scenario.test_ovn_dbs.OvnDbsMonitoringTest.*
-              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.*external
-              ^whitebox_neutron_tempest_plugin.tests.scenario.test_security_group_logging
-              ^whitebox_neutron_tempest_plugin.tests.scenario.test_router_flavors
-              ^whitebox_neutron_tempest_plugin.tests.scenario.test_dvr_ovn
-              ^whitebox_neutron_tempest_plugin.tests.scenario.test_ovn_fdb
-              ^whitebox_neutron_tempest_plugin.tests.scenario.test_api_server.*
             # https://review.opendev.org/892839
             # - neutron_tempest_plugin.scenario.test_mtu.NetworkWritableMtuTest
             # It's in Blacklist before, FWaaS tests are not executed in any of our setups so there is no need to keep them whitelisted
             #   ^neutron_tempest_plugin.fwaas.*
-            # More Flaky tests mblue is working on fixing it
-            # - whitebox_neutron_tempest_plugin.tests.scenario.test_router_flavors
-            # - whitebox_neutron_tempest_plugin.tests.scenario.test_dvr_ovn
         - stepName: single-thread-testing
           tempestRun:
             concurrency: 1
             includeList: |
-              whitebox_neutron_tempest_plugin.*south_north
-              ^whitebox_neutron_tempest_plugin.tests.scenario.test_security_group_logging
-              test_multicast.ext
-              test_multicast.*restart
-              whitebox_neutron_tempest_plugin.*south_north
-              whitebox_neutron_tempest_plugin.tests.scenario.*test_mtu
+              ^whitebox_neutron_tempest_plugin.*
               ^neutron_.*plugin..*scenario.test_.*macvtap
-              ^whitebox_neutron_tempest_plugin.many_vms
-              ^whitebox_neutron_tempest_plugin.ovn_controller_restart
-              ^whitebox_neutron_tempest_plugin.test_previously_used_ip
-              ^whitebox_neutron_tempest_plugin.tests.scenario.test_internal_dns.InternalDNSInterruptions.
-              ^whitebox_neutron_tempest_plugin.tests.scenario.test_l3ha_ovn.
-              ^whitebox_neutron_tempest_plugin.tests.scenario.test_metadata_rate_limiting
-              ^whitebox_neutron_tempest_plugin.tests.scenario.test_ovn_dbs.OvnDbsMonitoringTest.
-              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.*external
-              ^whitebox_neutron_tempest_plugin.tests.scenario.test_router_flavors
-              ^whitebox_neutron_tempest_plugin.tests.scenario.test_vlan_transparency.ProviderNetworkVlanTransparencyTest
-              ^whitebox_neutron_tempest_plugin.tests.scenario.test_router_flavors
-              ^whitebox_neutron_tempest_plugin.tests.scenario.test_dvr_ovn
-              ^whitebox_neutron_tempest_plugin.tests.scenario.test_ovn_fdb
-              ^whitebox_neutron_tempest_plugin.tests.scenario.test_api_server.*
-            # NOTE(mblue): Exclude list has test failures which need further debugging
-            # remove when bug OSPRH-7998 resolved
-            # - whitebox_neutron_tempest_plugin.tests.scenario.test_security_group_logging.*test_only_dropped_traffic_logged
-            # remove when issue OSPCIX-457 resolved
-            # - whitebox_neutron_tempest_plugin.tests.scenario.test_security_group_logging.*test_dropped_and_accepted_traffic_logged
+            # NOTE(mblue): If test skipped - please add related ticket to remove skip when issue resolved
             excludeList: |
-              whitebox_neutron_tempest_plugin.tests.scenario.test_metadata_rate_limiting
-              whitebox_neutron_tempest_plugin.tests.scenario.test_mtu.GatewayMtuTestIcmp.test_northbound_pmtud_icmp
-              whitebox_neutron_tempest_plugin.tests.scenario.test_security_group_logging.*test_only_dropped_traffic_logged
-              whitebox_neutron_tempest_plugin.tests.scenario.test_security_group_logging.StatefulSecGroupLoggingTest.test_only_accepted_traffic_logged
-              whitebox_neutron_tempest_plugin.tests.scenario.test_security_group_logging.StatelessSecGroupLoggingTest.test_only_accepted_traffic_logged
-              whitebox_neutron_tempest_plugin.tests.scenario.test_security_group_logging.*test_dropped_and_accepted_traffic_logged
+              # TODO(ralonsoh): enable "test_ovn_fdb" tests once [1] is merged and released.
+              # [1]https://review.opendev.org/c/x/whitebox-neutron-tempest-plugin/+/929410
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_ovn_fdb.*
+              # remove when bug OSPRH-9569 resolved
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_metadata_rate_limiting
+              # remove when bug OSPRH-7998 resolved
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_security_group_logging.*test_only_dropped_traffic_logged
+              # remove when issue OSPCIX-457 resolved
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_security_group_logging.*test_.*accepted_traffic_logged
+              # neutron whitebox multi-thread tests (executed on different workflow step)
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_basic.NetworkBasicTest
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_broadcast.BroadcastTestIPv4Common
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_broadcast.BroadcastTestVlanTransparency
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_extra_dhcp_opts.ExtraDhcpOptionsTest
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_extra_dhcp_opts.OvnExtraDhcpOptionsTest
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_internal_dns.InternalDNSTestOvn
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_multicast.MulticastTestIPv4Common.test_flooding_when_special_groups
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_multicast.MulticastTestIPv4Common.test_igmp_snooping_same_network_and_unsubscribe
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_multicast.MulticastTestIPv4Ovn
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_multicast.MulticastTestVlanTransparency
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_portsecurity.PortSecurityTest
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.QosTestCommon.test_bw_limit_east_west
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.QosTestCommon.test_bw_limit_tenant_network
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.QosTestCommon.test_default_qos_policy
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.QosTestCommon.test_dscp_marking_east_west
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.QosTestCommon.test_dscp_marking_tenant_network
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.QosTestCommon.test_qos_after_cold_migration
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.QosTestCommon.test_qos_after_live_migration
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.QosTestDscpInheritanceOvn
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.QosTestOvn
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.QosTestSriovBwLimitTest
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_qos.QosTestSriovMinBwPlacementEnforcementTest
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_security_groups.NetworkDefaultSecGroupTest
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_vlan_transparency.MtuVlanTransparencyTest
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_vlan_transparency.MultiPortVlanTransparencyTest
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_vlan_transparency.MultiVlanTransparencyTest
+              ^whitebox_neutron_tempest_plugin.tests.scenario.test_vrrp.VrrpTest