Failing `toolbar` test

[howard-e]

There's a new failure with the toolbar_toolbar: menuitemradio elements have aria-checked set test. This seems to be affecting new contributions since the start of this week:

toolbar_toolbar: menuitemradio elements have aria-checked set failure

...
not ok 97 - toolbar_toolbar › content/patterns/toolbar/examples/toolbar.html [data-test-id="toolbar-menuitemradio-aria-checked"]: menuitemradio elements have aria-checked set
  ---
    name: AssertionError
    details:
      'Rejected promise returned by test. Reason:': |-
        TimeoutError {
          remoteStacktrace: '',
          message: `Waiting for menu to open␊
          Wait timed out after 10174ms`,
        }
    message: Rejected promise returned by test
    at: Wait timed out after 10174ms
  ...
...
...

Examples:

• minor version bump dependabot PR which should not affect that the test
• new spin button pattern PR which affects no toolbar-based content

[howard-e]

The last build to main over 2 weeks ago was successful.

I also can’t replicate this failure locally with:

• npm run regression or
• npm run regression -- -t test/tests/toolbar_toolbar.js

That makes me think something has changed recently in the OS / browser versions picked up by the GH workflow affecting how it interacts with that component and/or test

[css-meeting-bot]

The ARIA Authoring Practices (APG) Task Force just discussed Failing toolbar test.

The full IRC log of that discussion

<jugglinmike> Topic: Failing toolbar test
<jugglinmike> github: https://github.com//issues/3331
<jugglinmike> howard-e: Toward the end of last week, I noticed a lot of patches were failing the automated tests
<jugglinmike> howard-e: I used a toolbar example as a test
<jugglinmike> howard-e: It seems like, in the test now, for whatever reason, it isn't picking up when items have been checked
<jugglinmike> howard-e: At first, I couldn't reproduce it locally--only on GitHub Actions
<jugglinmike> howard-e: But recently (as of this week), I have been able to reproduce it on my own
<jugglinmike> howard-e: I can't think of anything that would cause this
<jugglinmike> Matt_King: I didn't notice it failing on any of the pull requests that I just merged...
<jugglinmike> howard-e: Those were submitted prior to whenever this failure started
<jugglinmike> Matt_King: So it's a failure that's looking for the "aria-checked" state on a menu item?
<jugglinmike> howard-e: That's right
<jugglinmike> Adam_Page: Could it be a WebDriver update?
<jugglinmike> howard-e: That was my thought. There was an update in this time span, but nothing looked suspicious
<jugglinmike> howard-e: It looks like a timing issue. I'm especially interested to learn why this started now, in the absence of any change in our project
<jugglinmike> howard-e: The continuous integration environment has a lot of dependencies. It may be caused by an updated dependency there
<jugglinmike> Matt_King: We always have the nuclear option of disabling a test. I hate doing that
<jugglinmike> Jem: Please, no!
<jugglinmike> Matt_King: if you have time to look into this during this week, howard-e, that would be great--especially with the pull requests we're expecting
<jugglinmike> howard-e: Sure thing
<jugglinmike> Jem: The error message says "timeout", right? It doesn't successfully open the menu
<jugglinmike> howard-e: That's right
<jugglinmike> zakim, end the meeting

[howard-e]

The last build to main over 2 weeks ago was successful.

I also can’t replicate this failure locally with:

• npm run regression or
• npm run regression -- -t test/tests/toolbar_toolbar.js

I can now replicate this issue with npm run regression -- -t test/tests/toolbar_toolbar.js locally, so will continue the investigation