Bevy Accessibility

[paulGeoghegan]

Hi, so I just started using Bevy today so it's entirely possible that I'm doing something silly but I can't figure out how to make buttons accessible to screen readers. As far as I read Bevy should work with screen readers on most major platforms because of it's access kit integration from earlier this year.
I'm using the button example code like below:
`
//! This example illustrates how to create a button that changes color and text based on its
//! interaction state.

// This lint usually gives bad advice in the context of Bevy -- hiding complex queries behind
// type aliases tends to obfuscate code while offering no improvement in code cleanliness.
#![allow(clippy::type_complexity)]

use bevy::{prelude::*, winit::WinitSettings};

fn main() {
App::new()
.add_plugins(DefaultPlugins)
// Only run the app when there is user input. This will significantly reduce CPU/GPU use.
.insert_resource(WinitSettings::desktop_app())
.add_systems(Startup, setup)
.add_systems(Update, button_system)
.run();
}

const NORMAL_BUTTON: Color = Color::rgb(0.15, 0.15, 0.15);
const HOVERED_BUTTON: Color = Color::rgb(0.25, 0.25, 0.25);
const PRESSED_BUTTON: Color = Color::rgb(0.35, 0.75, 0.35);

fn button_system(
mut interaction_query: Query<
(
&Interaction,
&mut BackgroundColor,
&mut BorderColor,
&Children,
),
(Changed, With),
>,
mut text_query: Query<&mut Text>,
) {
for (interaction, mut color, mut border_color, children) in &mut interaction_query {
let mut text = text_query.get_mut(children[0]).unwrap();
match *interaction {
Interaction::Pressed => {
text.sections[0].value = "Press".to_string();
*color = PRESSED_BUTTON.into();
border_color.0 = Color::RED;
}
Interaction::Hovered => {
text.sections[0].value = "Hover".to_string();
*color = HOVERED_BUTTON.into();
border_color.0 = Color::WHITE;
}
Interaction::None => {
text.sections[0].value = "Button".to_string();
*color = NORMAL_BUTTON.into();
border_color.0 = Color::BLACK;
}
}
}
}

fn setup(mut commands: Commands, asset_server: Res) {
// ui camera
commands.spawn(Camera2dBundle::default());
commands
.spawn(NodeBundle {
style: Style {
width: Val::Percent(100.0),
height: Val::Percent(100.0),
align_items: AlignItems::Center,
justify_content: JustifyContent::Center,
..default()
},
..default()
})
.with_children(|parent| {
parent
.spawn(ButtonBundle {
style: Style {
width: Val::Px(150.0),
height: Val::Px(65.0),
border: UiRect::all(Val::Px(5.0)),
// horizontally center child text
justify_content: JustifyContent::Center,
// vertically center child text
align_items: AlignItems::Center,
..default()
},
border_color: BorderColor(Color::BLACK),
background_color: NORMAL_BUTTON.into(),
..default()
})
.with_children(|parent| {
parent.spawn(TextBundle::from_section(
"Button",
TextStyle {
font: asset_server.load("fonts/FiraSans-Bold.ttf"),
font_size: 40.0,
color: Color::rgb(0.9, 0.9, 0.9),
},
));
});
});
}
`
When I run it using cargo run I get the following screen:

They only thing that NVDA says is app so I'm not sure what is going wrong here.
Any help would be apreciated.
Note:
I'm running Windows 11 with the latest version of NVDA.

[TimJentzsch]

Sorry for the late reply.

I think the formatting of your post is broken, the code is currently very hard to read.
Can you try enclosing it in three backticks (```) on each side, ideally with rs after the first set of backticks to enable syntax highlighting?
Like this:

That would make it a lot easier to help. Thanks!

[paulGeoghegan]

I'll do that now in a little bit. Just FYI I'm blind so the image doesn't help but I'll try my best.

[TimJentzsch]

Sure, no worries.

I'll try to type it without Markdown messing it up:

```rs
// Your code here
```

So it's three backticks, then rs, then a newline, then your code snippet and then a newline and three backticks again :)

[TimJentzsch]

Ok, so I tried to reproduce the problem.
I'm on Linux, so I used the Orca screen reader instead of NVDA.

Indeed, for me Orca doesn't announce anything (not even "App") when I try to run the button example.

I looked through the bevy_a11y and bevy_ui source code to check if anything special needs to be set up for screen reader support.
I tried to manually set the AccessibilityRequested resource to true and to manually set the Focus resource to the button which is spawned, but still the reader falls completely silent in the Bevy app.

Notably, I might also be misusing Orca, as I don't have any screen reader experience.
Additionally, I don't use a US keyboard layout so some of the keybinds don't work well for me.

Maybe @ndarilek knows if there are any additional steps needed, to my knowledge they also work with a screen reader and implemented the access kit integration.

[paulGeoghegan]

I actually tried another example as well for the different types of UI elements. It had buttons, lists, texts, etc. and these were definitely set up using all of the proper accessibility parameters, but again, I got nothing. I also checked windows narrator and the problem persisted, so that's 3 screen readers on 2 different operating systems (I assume) having the same problem.

[ndarilek]

Hmm, interesting, and you're using object nav under Windows and NVDA? Since bevy_ui has no robust concept of focus or keyboard controls, the only way to currently navigate bevy_ui-based UIs with accessibility is via object nav, which should work but is a bit clunky. The goal of the existing accesskit support is just getting it into the engine and making it easy to push updates to the tree for when bevy_ui gains the more advanced concepts needed to make it more pleasant to use.

As far as Linux support goes, you'll need to enable a feature (I forget the exact name now) and I don't even think Orca mainline is even supported. The Linux a11y APIs are going through a bit of a revision ATM and I think AccessKit only supports experimental forks of the necessary APIs. Windows and mac are the only platforms I've tested personally.

[paulGeoghegan]

What exactly do you mean by that? I was just trying to tab or arrow through the items and was getting no output. Hovering a mouse over them also produced nothing. I'm not sure what object nav is, so if you can tell me how to use it, I can try that and see if it works.

[ndarilek]

Sorry, I thought you mentioned being blind as well so I assumed you knew. I misspoke and meant object review, what you get when using the numpad or laptop mode commands. Not sure why my brain farts and calls that "object nav" but it does.

Honestly, unless you're willing to roll up your sleeves and help out, I wouldn't spend a lot of time on bevy_ui right now if accessibility is a hard requirement. It's not a great a11y experience, but it's not a great experience in general right now and is actively being experimented with.

If you want an accessible Bevy UI, your best path forward is probably bevy_egui. The trick is that Bevy, egui, and bevy_egui all have to be released with support for the same accesskit dependency, and that's further complicated by egui and accesskit having essentially the same issue, but with winit. I'm going to make another go at figuring out the state of things and untangling that mess again at the beginning of next year but I could certainly use help doing it. I did some work bumping egui's accesskit version and getting it working with the bevy_egui integration, but then winit 0.29 came out and I kind of ran out of steam.

[paulGeoghegan]

Ah okay I haven't used that much to be honest but yeh I know what you're talking about now.
I have honestly spent a lot of time looking around at different "game engine" options as I haven't found any that I as a blind dev can actually use with much success besides Bevy so I was really hoping to use it.
I would definitely be interested in helping if I can but if I'm completely honest my experience with both Rust and Access kit is limited so I'm not sure how much help I could really be. If you're willing to show me where to get started in the source code then I'm more than willing to go off and do my own research but I can't promise anything.

[ndarilek]

I just polished off vladbat00/bevy_egui#238 to make bevy_egui accessible.

If you check out that repo and run cargo run --example ui you'll get an accessible UI you can tab through. Note that there's currently a bug where you have to alt-tab out of, and back into, the window for the UI to be visible to NVDA. Currently trying to figure out what's going on there. If I can fix it in bevy_egui then we can just update the PR, but bevy fixes likely won't land until the next release in a couple months.

[paulGeoghegan]

Okay so I got a chance to check that out. I think the bug might be because the window isn't getting focus or something because NVDA doesn't even anounce the new window which I believe it does in Bevy at the minute. Even if I just press the windows key and then press escape it will then anounce the window. It's amazing the changes you have already made so again if I can help at all please let me know.