Bluetooth: controller: llcp: failing EBQ and Harmony tests

[kruithofa]

Updated on 27-7: removed all tests that pass as of 27-7

• LL/CON/CEN/BV-55-C Mandatory Minimum PDU Length, LE Coded
• LL/CON/PER/BV-57-C
• LL/DDI/SCN/BV-46-C
• LL/CON/INI/BI-27-C

[erbr-ot]

Failures (26) from fresh run of LL/CON/CEN and LL/CON/PER tests on Ellisys Bluetooth Qualifier | Release | 20212.8080.8084.20312 with tip of Zephyr with refactored LLCP:
(the ones marked with 'fixed' we have fixes for)

LL/CON/CEN/BI-02-C.Central T_Terminate Timer
LL/CON/CEN/BI-14-C.Invalid LLID
LL/CON/CEN/BV-08-C.Central Sending Termination
fixed LL/CON/CEN/BV-126-C.Data Length Update - Preserve Parameters After a PHY Change - LE 2M PHY
LL/CON/CEN/BV-27-C.Initiating Connection Parameter Request - different procedure collision - channel map update
fixed LL/CON/CEN/BV-31-C.Accepting Connection Parameter Request - preferred anchor points only
fixed LL/CON/CEN/BV-41-C.Initiating PHY Update Procedure
fixed LL/CON/CEN/BV-43-C.Responding to PHY Update Procedure
LL/CON/CEN/BV-47-C.Handling Protocol Collision - Different Procedure - Channel Map
LL/CON/CEN/BV-48-C.Handling Protocol Collision - Different Procedure - Connection Parameters
LL/CON/CEN/BV-49-C.Initiating PHY Update Procedure - Packet Time Restrictions
LL/CON/CEN/BV-50-C.Responding to PHY Update Procedure - Packet Time Restrictions
LL/CON/PER/BI-02-C.Peripheral T_Terminate Timer
fixed LL/CON/PER/BI-04-C.Rejecting Connection Change
fixed LL/CON/PER/BI-09-C.Responding to PHY Update Procedure - Instant In Past
LL/CON/PER/BI-17-C.Invalid LLID
fixed LL/CON/PER/BI-18-C.PHY Update Procedure, Invalid or Unsupported PHY, Central Initiated, LE Coded PHY Not Supported
fixed LL/CON/PER/BI-20-C.PHY Update Procedure, Invalid or Unsupported PHY, Central Initiated, Multiple PHYs Specified
fixed LL/CON/PER/BI-21-C.PHY Update Procedure, Invalid or Unsupported PHY, Central Initiated, RFU Bits Specified
fixed LL/CON/PER/BV-129-C.Data Length Update - Preserve Parameters After a PHY Change - LE 2M PHY
LL/CON/PER/BV-158-C.Listening for Packet With Window Widening, ACL
fixed LL/CON/PER/BV-30-C.Accepting Connection Parameter Request - preferred anchor points only
LL/CON/PER/BV-46-C.Handling Protocol Collision - Different Procedure - Channel Map
LL/CON/PER/BV-47-C.Handling Protocol Collision - Different Procedure - Connection Parameters
LL/CON/PER/BV-48-C.Handling Protocol Collision - Different Procedure - Connection Update
fixed LL/CON/PER/BV-85-C.Initiating Connection Parameter Request - Unsupported Without Feature Exchange

[erbr-ot]

Failures from fresh run of LL/CON/CEN and LL/CON/PER tests on Ellisys Bluetooth Qualifier | Release | 20212.8080.8084.20312 with tip of Zephyr with refactored LLCP:

• LL/CON/CEN/BI-02-C.Central T_Terminate Timer
• LL/CON/CEN/BI-14-C.Invalid LLID - NOT relateted to LLCP
• LL/CON/CEN/BV-08-C.Central Sending Termination
• LL/CON/CEN/BV-27-C.Initiating Connection Parameter Request - different procedure collision - channel map update
• LL/CON/CEN/BV-47-C.Handling Protocol Collision - Different Procedure - Channel Map
• LL/CON/CEN/BV-48-C.Handling Protocol Collision - Different Procedure - Connection Parameters
• LL/CON/CEN/BV-49-C.Initiating PHY Update Procedure - Packet Time Restrictions
• LL/CON/CEN/BV-50-C.Responding to PHY Update Procedure - Packet Time Restrictions
• LL/CON/PER/BI-02-C.Peripheral T_Terminate Timer
• LL/CON/PER/BI-17-C.Invalid LLID - NOT relateted to LLCP
• LL/CON/PER/BV-46-C.Handling Protocol Collision - Different Procedure - Channel Map
• LL/CON/PER/BV-47-C.Handling Protocol Collision - Different Procedure - Connection Parameters
• LL/CON/PER/BV-48-C.Handling Protocol Collision - Different Procedure - Connection Update

[sjanc]

Harmony 2021.20 (TCRL 2021-2), IUT nRF52833-DK (540 tests enabled in total, including DF)

• HCI/CCO/BI-51-C
• HCI/CCO/BI-53-C
• HCI/CCO/BI-54-C
• HCI/CCO/BI-56-C
• HCI/DDI/BI-05-C
• HCI/DDI/BI-06-C
• HCI/DDI/BI-07-C
• HCI/DDI/BI-11-C
• HCI/DDI/BI-13-C
• HCI/DDI/BI-14-C
• HCI/DDI/BV-07-C
• HCI/GEV/BV-01-C
• LL/CON/CEN/BI-14-C
• LL/CON/CEN/BV-13-C
• LL/CON/CEN/BV-27-C
• LL/CON/CEN/BV-31-C
• LL/CON/CEN/BV-48-C
• LL/CON/INI/BV-26-C
• LL/CON/INI/BV-28-C
• LL/CON/PER/BI-04-C
• LL/CON/PER/BI-09-C
• LL/CON/PER/BV-13-C (PASS on second run)
• LL/CON/PER/BV-30-C
• LL/CON/PER/BV-46-C
• LL/CON/PER/BV-47-C
• LL/CON/PER/BV-48-C
• LL/CON/PER/BV-85-C
• LL/DDI/ADV/BV-29-C
• LL/DDI/ADV/BV-30-C
• LL/DDI/ADV/BV-33-C
• LL/DDI/ADV/BV-35-C
• LL/DDI/ADV/BV-45-C
• LL/DDI/ADV/BV-48-C
• LL/DDI/ADV/BV-52-C
• LL/DDI/ADV/BV-54-C
• LL/DDI/SCN/BV-28-C (PASS on second run)
• LL/DDI/SCN/BV-62-C
• LL/DDI/SCN/BV-72-C
• LL/DDI/SCN/BV-73-C
• LL/ENC/ADV/BI-02-C
• LL/TIM/PER/BV-02-C
• LL/TIM/PER/BV-03-C
• LL/TIM/PER/BV-04-C
• LL/TIM/PER/BV-05-C
• LL/TIM/PER/BV-06-C
• LL/TIM/PER/BV-07-C
• LL/CON/CEN/BV-56-C
• LL/CON/CEN/BV-59-C
• LL/CON/CEN/BV-63-C
• LL/CON/CEN/BV-64-C (PASS on second run)
• LL/CON/CEN/BV-67-C
• LL/CON/CEN/BV-69-C
• LL/CON/CEN/BV-71-C
• LL/CON/CEN/BV-72-C
• LL/CON/CEN/BV-119-C
• LL/CON/CEN/BV-120-C
• LL/CON/CEN/BV-122-C
• LL/CON/CEN/BV-123-C
• LL/CON/CEN/BV-124-C
• LL/CON/CEN/BV-125-C
• LL/CON/CEN/BV-150-C
• LL/CON/CEN/BV-151-C
• LL/CON/PER/BV-60-C
• LL/CON/PER/BV-63-C
• LL/CON/PER/BV-67-C
• LL/CON/PER/BV-71-C
• LL/CON/PER/BV-73-C
• LL/CON/PER/BV-75-C
• LL/CON/PER/BV-76-C
• LL/CON/PER/BV-122-C
• LL/CON/PER/BV-123-C
• LL/CON/PER/BV-125-C
• LL/CON/PER/BV-126-C
• LL/CON/PER/BV-127-C
• LL/CON/PER/BV-128-C
• LL/CON/PER/BV-150-C
• LL/CON/PER/BV-151-C
• LL/DDI/ADV/BV-36-C
• LL/DDI/ADV/BV-37-C
• LL/DDI/ADV/BV-57-C
• LL/DDI/ADV/BV-58-C
• LL/DDI/ADV/BV-59-C
• LL/DDI/ADV/BV-60-C

[erbr-ot]

Just did run of these 134 LL/CON/CEN and LL/CON/PER tests with only two failures:
This is without Coded PHY support, with PARAM_CHECK disabled, without PR#45798, with my draft PR#45829 for PHY Update and with a minor fix to avoid sending stuff after terminate_ind

• LL/CON/CEN/BI-02-C
• LL/CON/CEN/BI-04-C
• LL/CON/CEN/BI-05-C
• LL/CON/CEN/BI-06-C
• LL/CON/CEN/BI-07-C
• LL/CON/CEN/BI-08-C
• LL/CON/CEN/BI-11-C
• LL/CON/CEN/BI-12-C
• LL/CON/CEN/BI-13-C
• LL/CON/CEN/BI-14-C
• LL/CON/CEN/BV-02-C
• LL/CON/CEN/BV-03-C
• LL/CON/CEN/BV-04-C
• LL/CON/CEN/BV-05-C
• LL/CON/CEN/BV-07-C
• LL/CON/CEN/BV-08-C
• LL/CON/CEN/BV-09-C
• LL/CON/CEN/BV-10-C
• LL/CON/CEN/BV-117-C
• LL/CON/CEN/BV-126-C
• LL/CON/CEN/BV-128-C
• LL/CON/CEN/BV-13-C
• LL/CON/CEN/BV-14-C
• LL/CON/CEN/BV-146-C
• LL/CON/CEN/BV-147-C
• LL/CON/CEN/BV-15-C
• LL/CON/CEN/BV-16-C
• LL/CON/CEN/BV-17-C
• LL/CON/CEN/BV-18-C
• LL/CON/CEN/BV-19-C
• LL/CON/CEN/BV-20-C
• LL/CON/CEN/BV-21-C
• LL/CON/CEN/BV-22-C
• LL/CON/CEN/BV-23-C
• LL/CON/CEN/BV-24-C
• LL/CON/CEN/BV-25-C
• LL/CON/CEN/BV-26-C
• LL/CON/CEN/BV-27-C
• LL/CON/CEN/BV-28-C
• LL/CON/CEN/BV-29-C
• LL/CON/CEN/BV-30-C
• LL/CON/CEN/BV-32-C
• LL/CON/CEN/BV-33-C
• LL/CON/CEN/BV-34-C
• LL/CON/CEN/BV-35-C
• LL/CON/CEN/BV-41-C
• LL/CON/CEN/BV-43-C
• LL/CON/CEN/BV-45-C
• LL/CON/CEN/BV-46-C
• LL/CON/CEN/BV-47-C
• LL/CON/CEN/BV-48-C
• LL/CON/CEN/BV-49-C
• LL/CON/CEN/BV-50-C
• LL/CON/CEN/BV-51-C
• LL/CON/CEN/BV-73-C
• LL/CON/CEN/BV-74-C
• LL/CON/CEN/BV-75-C
• LL/CON/CEN/BV-76-C
• LL/CON/CEN/BV-77-C
• LL/CON/CEN/BV-73-C
• LL/CON/CEN/BV-81-C
• LL/CON/CEN/BV-82-C
• LL/CON/PER/BI-01-C
• LL/CON/PER/BI-02-C Fails
• LL/CON/PER/BI-04-C
• LL/CON/PER/BI-05-C
• LL/CON/PER/BI-07-C
• LL/CON/PER/BI-08-C
• LL/CON/PER/BI-09-C
• LL/CON/PER/BI-10-C
• LL/CON/PER/BI-11-C
• LL/CON/PER/BI-14-C
• LL/CON/PER/BI-15-C
• LL/CON/PER/BI-16-C
• LL/CON/PER/BI-17-C Fails
• LL/CON/PER/BI-18-C
• LL/CON/PER/BI-20-C
• LL/CON/PER/BI-21-C
• LL/CON/PER/BV-02-C
• LL/CON/PER/BV-04-C
• LL/CON/PER/BV-05-C
• LL/CON/PER/BV-06-C
• LL/CON/PER/BV-10-C
• LL/CON/PER/BV-109-C
• LL/CON/PER/BV-11-C
• LL/CON/PER/BV-110-C
• LL/CON/PER/BV-12-C
• LL/CON/PER/BV-129-C
• LL/CON/PER/BV-13-C
• LL/CON/PER/BV-131-C
• LL/CON/PER/BV-14-C
• LL/CON/PER/BV-145-C
• LL/CON/PER/BV-146-C
• LL/CON/PER/BV-147-C
• LL/CON/PER/BV-15-C
• LL/CON/PER/BV-158-C
• LL/CON/PER/BV-16-C
• LL/CON/PER/BV-17-C
• LL/CON/PER/BV-18-C
• LL/CON/PER/BV-19-C
• LL/CON/PER/BV-20-C
• LL/CON/PER/BV-21-C
• LL/CON/PER/BV-22-C
• LL/CON/PER/BV-23-C
• LL/CON/PER/BV-24-C
• LL/CON/PER/BV-25-C
• LL/CON/PER/BV-26-C
• LL/CON/PER/BV-27-C
• LL/CON/PER/BV-28-C
• LL/CON/PER/BV-29-C
• LL/CON/PER/BV-30-C
• LL/CON/PER/BV-31-C
• LL/CON/PER/BV-32-C
• LL/CON/PER/BV-33-C
• LL/CON/PER/BV-34-C
• LL/CON/PER/BV-40-C
• LL/CON/PER/BV-42-C
• LL/CON/PER/BV-44-C
• LL/CON/PER/BV-45-C
• LL/CON/PER/BV-46-C
• LL/CON/PER/BV-47-C
• LL/CON/PER/BV-48-C
• LL/CON/PER/BV-49-C
• LL/CON/PER/BV-50-C
• LL/CON/PER/BV-51-C
• LL/CON/PER/BV-52-C
• LL/CON/PER/BV-53-C
• LL/CON/PER/BV-77-C
• LL/CON/PER/BV-78-C
• LL/CON/PER/BV-79-C
• LL/CON/PER/BV-80-C
• LL/CON/PER/BV-81-C
• LL/CON/PER/BV-85-C
• LL/CON/PER/BV-86-C

[erbr-ot]

Another run - as above but now with Coded PHY code base compiled in, but functionality 'featured out'
Also with fix to issue for Terminate.
LL/CON/CEN/BI-02-C
LL/CON/CEN/BI-04-C
LL/CON/CEN/BI-05-C
LL/CON/CEN/BI-06-C
LL/CON/CEN/BI-07-C
LL/CON/CEN/BI-08-C
LL/CON/CEN/BI-11-C
LL/CON/CEN/BI-12-C
LL/CON/CEN/BI-13-C
LL/CON/CEN/BI-14-C
LL/CON/CEN/BV-02-C
LL/CON/CEN/BV-03-C
LL/CON/CEN/BV-04-C
LL/CON/CEN/BV-05-C
LL/CON/CEN/BV-07-C
LL/CON/CEN/BV-08-C
LL/CON/CEN/BV-09-C
LL/CON/CEN/BV-10-C
LL/CON/CEN/BV-117-C
LL/CON/CEN/BV-126-C
LL/CON/CEN/BV-128-C
LL/CON/CEN/BV-13-C
LL/CON/CEN/BV-14-C
LL/CON/CEN/BV-146-C
LL/CON/CEN/BV-147-C
LL/CON/CEN/BV-15-C
LL/CON/CEN/BV-16-C
LL/CON/CEN/BV-17-C
LL/CON/CEN/BV-18-C
LL/CON/CEN/BV-19-C
LL/CON/CEN/BV-20-C
LL/CON/CEN/BV-21-C
LL/CON/CEN/BV-22-C
LL/CON/CEN/BV-23-C
LL/CON/CEN/BV-24-C
LL/CON/CEN/BV-25-C
LL/CON/CEN/BV-26-C
LL/CON/CEN/BV-27-C
LL/CON/CEN/BV-28-C
LL/CON/CEN/BV-29-C
LL/CON/CEN/BV-30-C
LL/CON/CEN/BV-31-C - Fails sometimes, but usually not when 'run again' - will investigate
LL/CON/CEN/BV-32-C
LL/CON/CEN/BV-33-C
LL/CON/CEN/BV-34-C
LL/CON/CEN/BV-35-C
LL/CON/CEN/BV-41-C
LL/CON/CEN/BV-43-C
LL/CON/CEN/BV-45-C
LL/CON/CEN/BV-46-C
LL/CON/CEN/BV-47-C
LL/CON/CEN/BV-48-C
LL/CON/CEN/BV-49-C
LL/CON/CEN/BV-50-C
LL/CON/CEN/BV-51-C
LL/CON/CEN/BV-73-C
LL/CON/CEN/BV-74-C - Fails due to some HCI thing
LL/CON/CEN/BV-75-C
LL/CON/CEN/BV-76-C
LL/CON/CEN/BV-77-C - Fails due to some HCI thing
LL/CON/CEN/BV-73-C
LL/CON/CEN/BV-81-C
LL/CON/CEN/BV-82-C
LL/CON/PER/BI-01-C
LL/CON/PER/BI-02-C
LL/CON/PER/BI-04-C
LL/CON/PER/BI-05-C
LL/CON/PER/BI-07-C
LL/CON/PER/BI-08-C
LL/CON/PER/BI-09-C
LL/CON/PER/BI-10-C
LL/CON/PER/BI-11-C
LL/CON/PER/BI-14-C
LL/CON/PER/BI-15-C
LL/CON/PER/BI-16-C
LL/CON/PER/BI-17-C Fails - non LLCP
LL/CON/PER/BI-18-C New Fail - due to miss-hap in Coded PHY hack - 99,9% a non-issue
LL/CON/PER/BI-20-C
LL/CON/PER/BI-21-C
LL/CON/PER/BV-02-C
LL/CON/PER/BV-04-C
LL/CON/PER/BV-05-C
LL/CON/PER/BV-06-C
LL/CON/PER/BV-10-C
LL/CON/PER/BV-109-C
LL/CON/PER/BV-11-C
LL/CON/PER/BV-110-C
LL/CON/PER/BV-12-C
LL/CON/PER/BV-129-C
LL/CON/PER/BV-13-C
LL/CON/PER/BV-131-C
LL/CON/PER/BV-14-C
LL/CON/PER/BV-145-C
LL/CON/PER/BV-146-C
LL/CON/PER/BV-147-C
LL/CON/PER/BV-15-C
LL/CON/PER/BV-158-C
LL/CON/PER/BV-16-C
LL/CON/PER/BV-17-C
LL/CON/PER/BV-18-C
LL/CON/PER/BV-19-C
LL/CON/PER/BV-20-C
LL/CON/PER/BV-21-C
LL/CON/PER/BV-22-C
LL/CON/PER/BV-23-C
LL/CON/PER/BV-24-C
LL/CON/PER/BV-25-C
LL/CON/PER/BV-26-C
LL/CON/PER/BV-27-C
LL/CON/PER/BV-28-C
LL/CON/PER/BV-29-C
LL/CON/PER/BV-30-C
LL/CON/PER/BV-31-C
LL/CON/PER/BV-32-C
LL/CON/PER/BV-33-C
LL/CON/PER/BV-34-C
LL/CON/PER/BV-40-C - New Fail - due to miss-hap in Coded PHY hack - 99,9% a non-issue
LL/CON/PER/BV-42-C
LL/CON/PER/BV-44-C
LL/CON/PER/BV-45-C
LL/CON/PER/BV-46-C
LL/CON/PER/BV-47-C
LL/CON/PER/BV-48-C
LL/CON/PER/BV-49-C
LL/CON/PER/BV-50-C
LL/CON/PER/BV-51-C
LL/CON/PER/BV-52-C
LL/CON/PER/BV-53-C
LL/CON/PER/BV-77-C
LL/CON/PER/BV-78-C
LL/CON/PER/BV-79-C
LL/CON/PER/BV-80-C
LL/CON/PER/BV-81-C
LL/CON/PER/BV-85-C
LL/CON/PER/BV-86-C

[kruithofa]

Table is updated, but still need a complete run to verify that there are no regression failures

[sjanc]

Harmony failing tests (DF excluded):

LL/CON/CEN/BI-11-C - regression 25e6279 (Bluetooth: controller: llcp: fix for failing LL qualification tests)
LL/CON/CEN/BI-12-C - regression 25e6279 (Bluetooth: controller: llcp: fix for failing LL qualification tests)
LL/CON/CEN/BV-146-C - regression 25e6279 (Bluetooth: controller: llcp: fix for failing LL qualification tests)
LL/CON/CEN/BV-147-C - regression 25e6279 (Bluetooth: controller: llcp: fix for failing LL qualification tests)

LL/CON/INI/BV-13-C [OTA] - First connection event too early
LL/CON/INI/BV-25-C [OTA] - First connection event too early
LL/CON/INI/BV-26-C [OTA] - First connection event too early

LL/CON/INI/BV-27-C [OTA] Fails randomly - "AA bit 3 should be clear at least 3 times (actual 1)")
LL/CON/PER/BV-15-C [OTA] Fails randomly - nesn:sn bits mismatch

LL/DDI/ADV/BV-29-C
LL/DDI/ADV/BV-33-C
LL/DDI/ADV/BV-45-C
LL/DDI/ADV/BV-48-C
LL/DDI/ADV/BV-52-C
LL/DDI/ADV/BV-54-C

LL/DDI/SCN/BV-21-C fails ~1/5
LL/DDI/SCN/BV-28-C fails ~1/5
LL/DDI/SCN/BV-34-C fails ~1/5
LL/DDI/SCN/BV-61-C fails ~1/5

LL/DDI/SCN/BV-62-C
LL/DDI/SCN/BV-68-C
LL/DDI/SCN/BV-72-C wrong address type in sync established event
LL/DDI/SCN/BV-73-C wrong address type in sync established event

Below tests fail if LL/DDI/ADV/BV-54-C is executed before them...
LL/CON/PER/BI-09-C pass if run individually
LL/TIM/PER/BV-02-C pass if run individually
LL/TIM/PER/BV-03-C pass if run individually
LL/TIM/PER/BV-04-C pass if run individually
LL/TIM/PER/BV-05-C pass if run individually
LL/TIM/PER/BV-06-C pass if run individually