Standard Headless Widgets

[viridia]

Games thrive on novelty: nearly every game ever published has a unique artistic style. Unfortunately, this often results in developers having to re-implement basic UI components such as sliders, checkboxes, and spinners, in order to get the distinctive look that they want. Even worse, building robust, high-quality, cross-platform widgets requires knowledge of accessibility technologies, input mapping techniques, and other esoteric concerns, which many developers may not be experts in.

Headless Libraries

In the web world, this problem is solved by the a number of "headless" UI component libraries which provide high-quality widget implementations with no built-in styling, such as Headless UI and ReaKit. Users can easily add their own custom look and feel on top of these widgets.

Bevy has very recently gained a number of new capabilities (such as the input_focus crate) which facilitates the construction of headless widgets. The next logical step is to build out a selection of a few curated widgets. I refer to these as "headless" or "core" widgets.

Even though the world of UI has thousands of different kinds of widgets, the headless libraries mentioned previously have less than a dozen widgets in them. The reason is because what often distinguishes one kind of widget from another is styling. Since the headless widget library isn't concerned with styling, it does not need to distinguish widgets by style.

Also, many standard widgets are assemblies of smaller widgets: for example, a calendar widget is composed of many buttons. The headless widget libraries generally only provide low-level "atomic" components.

Accessibility Support

A key value of headless widgets is first-class support for accessibility. In Bevy, accessibility is provided by the bevy_a11y crate, which uses AccessKit to integrate with the platform's native OS capabilities for things like speech output. However, these features are not enabled unless you include the proper metadata. For example, when a widget is in a state like disabled or checked, adding in the proper metadata will allow the screen reader to say the words "disabled" or "checked" when the widget is focused.

Unfortunately, most developers aren't familiar with these technologies, many do not even know how to enable the screen reading feature integrated in their OS for testing.

By ensuring that the core widgets fully support the accessibility APIs, we can make it so that developers get most of the benefits "for free".

Platform support

With respect to widget design, platforms fall into three classes:

• Desktop
• Console
• Mobile

Desktop widgets are generally driven by mouse events and keypresses; console widgets by gamepad buttons, and mobile by touch events.

Some kinds of widgets are more "universal" than others. For example, a button or toggle widget works pretty much the same everywhere; the main difference is the type of event used to trigger it. The existence proof here is the standard HTML input fields: you can use <button> on both desktop and mobile, and it will adhere to platform conventions.

However, some widgets are quite different on different platforms. Take for example text input: on a console or mobile device this normally happens by popping up some auxiliary keyboard interface which is part of the OS. Unlike a desktop the text widget doesn't accept keystrokes.

These kinds of behaviors can likely be controlled by feature flags. In rare cases, a widget may not be available on all platforms.

Technical Requirements

Framework Agnostic

It is envisioned that the core widgets will not depend on any particularly templating system or reactive framework. They will be vanilla Bevy UI components that use observers and bubbling events. Keyboard shortcuts will go through the FocusInput mechanism.

Use of observers

Core widgets are implemented using a set of global observers - that is, we don't register a separate observer for each widget instance. There's a plugin which registers the observers (perhaps a separate plugin for each type). Alternatively, we might be able to use register_component_hooks to automatically set up the observers the first time the user inserts the component.

Component States

Widget states will be represented using components. Dynamic styles can be implemented by the user via normal Bevy change detection on the state components.

Controlled vs. Uncontrolled

To maximize flexibility, widgets will be "controlled" rather than "uncontrolled". This is a term from React; a controlled widget does not update its own state, but simply emits events containing the proposed new state, which the receiver can choose to ignore. For example, a checkbox widget, when clicked, emits an event with the new proposed checkbox state, but does not update itself. It's up to the parent component to update the component with the new state.

The choice between controlled and uncontrolled is somewhat of a personal preference, but most professional UI developers I have talked to prefer controlled widgets. These have several advantages:

• It's easier to integrate them with reactive frameworks or external state management.
• It's easier to do input validation if the receiver can simply ignore the proposed change, or flash an error indicator in response to the event.
• It's easier to programmatically change the state, or synchronize state between multiple widgets, if the state is not buried inside the widget.
• It's simple to transform a controlled widget into an uncontrolled one by adding an event handler which applies the proposed state.

However, controlled widgets do have a downside in that they are not self-sufficient, which may confuse novice programmers.

Hover Agnosticism

Hover states: in general, hover highlights are purely stylistic, which means that the headless library need not concern itself with hovering. Now, normally in headless libraries we do have to concern ourselves with "roll-off" behavior - that is, if you click on a button and then move the mouse outside of the button before releasing, there is no "click" event emitted. However, in the case of Bevy, the picking crate already handles this behavior for us, so we don't actually need to do anything.

As a result, the core widgets have no need to integrate with the hover status provided by picking. That is purely up to the style layer provided by the user.

Focus Behavior

Most widgets support shortcut keys when focused.

On desktop, widgets will set focus to themselves when clicked. They will also set the FocusVisible resource to false to hide the focus rect. Console games will generally ignore this, instead showing the focus indicator unconditionally.

Disabled states

Widgets can be disabled by adding an InteractionDisabled marker component. Disabled widgets do not respond to user input, but they can still have keyboard focus - this is important for accessibility, otherwise sight-impaired users cannot "see" the widget.

An entity hook will ensure that the a11y "disabled" metadata is kept up to date.

Widget Types

Widgets will be divided into a number of tiers or cohorts, with the first tier consisting of widgets that are (a) very common, and (b) easy to implement.

Tier 1

• Button - the standard push button. On desktop, Enter and Space will trigger the button when focused.
• Toggle - for checkboxes and toggle switches. On desktop, Enter and Space will toggle the button when focused.
• Choice Spinner - this is a widget that offers a set of choices which you can select by incrementing or decrementing, for example game difficulty: < [ veteran ] >. The value may be edited by pressing buttons or by dragging. On console, this will be edited by the direction buttons, whereas on desktop arrow keys will change the value.
• Number Spinner - like the choice spinner but allows editing of a numeric value, within a range from min to max.

Tier 2

• Slider - a very basic slider that consists of a thumb and a track. More complex sliders which have increment / decrement arrows are built on top of this.
• Scrollbar - similar to a slider, but uses different math (has no min property, but does have a span or visible_range property).
• Text Input - on desktop, this would (hopefully) use the cosmic-edit library to support full-featured editing by the keyboard. On console and mobile, this would use the OS keyboard.

Tier 3

• Disclosure - a toggle widget which shows or hides some other UI content.
• Radio - radio buttons are in the third tier because they have a complex focus behavior (according to the WAI authoring guidelines) and are a challenge. They are also not common in games, as they take up a lot of screen real-estate.
• List Box - lists have similar focus behavior to radio buttons, but are quite common in games.
• Menu / MenuItem - menus are a challenge due to the need for popping up and intelligent positioning of the popup. (This will be easier once we have support for Position::Fixed.)

Open Questions

• One piece of a11y information that cannot be derived automatically is the alt-text, that is, the text that should be read to the user when that widget is focused. Should we emit a warning when a widget lacks alt-text? Maybe as an opt-in feature?
• On console, widgets generally will use the platform conventions for button mapping. For example on Xbox, the A button will trigger a button or toggle. Focus navigation will be determined by the focus system, so the widget need not concern itself with which button changes focus. However, what about the spinner increment / decrement? Which game pad button will we use for that?
  • The obvious answer here is to use an input mapper like LWIM. However, the current thinking is that the event mapper is the last element in the event chain - focused widgets get first crack at button presses before the event mapper gets to see them. The reason for this is to allow widgets such as text input to override keystrokes that might be mapped to global shortcuts.
  • One solution might be to do this using a double-mapping: the widget allows direction button presses to pass through to the event mapper, the event mapper then maps these to "increment / decrement" events, and then re-dispatches the events to the focus widget. With observers, this shouldn't introduce an additional 1-frame delay.
• Where should these widgets live? In bevy_ui? Or a new crate such as bevy_ui_headless or bevy_ui_widgets?
• Even though the functionality of checkbox and toggle switch are identical, they have different ARIA roles ("checkbox" vs. "switch"). This suggests that they should be separate widgets rather than combined into a single "Toggle" widget type.

[viridia]

I have implemented a prototype of the core widgets here: https://github.com/viridia/bevy_core_widgets

@alice-i-cecile @cart @NthTensor I have a bunch of open questions and I could use some feedback and advice.

The first question relates to the "checked" state for both checkboxes and radio buttons. Currently checked is a property of CoreCheckbox and CoreRadio, along with the optional on_click callback handler (which is a SystemId). We need to be able to call AccessibilityNode::set_toggled() whenever the checked state changes, so that the screen reader will respond to changes in the checked state. To do this we need a way to detect component mutations.

• One approach would be to make CoreCheckbox and CoreRadio immutable. However, this is a pain because you'd need to copy over the on_click property as well as the checked property.
• Another approach is to break out the Checked state into its own component, which would be immutable. This could be a required component of both CoreCheckbox and CoreRadio.
• A third approach would be to use a regular ECS system, with change detection, rather than a hook, to synchronize the checked state. However, this means running an extra system every frame.

A second issue is that I'm not happy with the design of CoreRadioGroup. Currently CoreRadioGroup is an optional widget - if you want to do mutual exclusion yourself, and you don't care about keyboard navigation of radio buttons (or want to implement it yourself), you can just use the radio buttons directly. However, the fact that it's optional makes some things more complicated.

CoreRadioGroup listens for ButtonPress events from its children, and handles the mutual exclusion when this happens. However, a radio button which has an on_click handler doesn't produce events, instead it triggers the one-shot system - so the radio group never has a chance to do mutual exclusion. This isn't a problem for reactive apps, because mutual exclusion happens automatically - the individual radio buttons respond to the signal holding the selection value. But for apps that don't use a reactive framework, but which do use callbacks rather than events, this is a problem.

A different approach would be to remove the on_click handler in CoreRadio and move it to CoreRadioGroup - that is, rather than having a separate click handler for each individual radio button, you'd have a single handler for the whole group. This sounds nice, but now the handler needs a way to disambiguate which button was clicked. If each radio button has its own on_click handler, you can just capture a closure variable containing the radio button value, but if it's just a single handler you can't do that. It also means that CoreRadioGroup is no longer optional.

Note that part of the complexity here is that, unlike JavaScript where we can associate an arbitrary value (string, int, enum, or whatever) with each radio button, with Rust we need a known type for this. That is, in most web-based widget libraries, radio buttons contain a selection value, and the output of the group is the value of the currently selected button. But to do this in Bevy would require making radio buttons a generic type, which greatly complicates the registration of observers and hooks.

What I've done is to punt: when a radio button is clicked, the user can get the entity id of the clicked button, and then use that to look up the associated value.. If the user wants to, for example, store the radio value as a custom component on the radio button, or store it in a map, or something else, they can do this. It does make things a bit more cumbersome however. It also means that the mutual exclusion logic is based on entity id rather than on selection value.

[alice-i-cecile]

I would prefer to use change detection with an ordinary system until we're absolutely convinced that it's inadequate. That sort of "check and synchronize" pattern is very common, and I don't think that running an extra system every frame is particularly concerning. If we're worried about overhead, swapping to a Populated query will prevent these systems for running when there's no matching entities. I don't think there's a need for customized behavior here for this part of the system, so sticking to vanilla systems seems fine.

[alice-i-cecile]

I'm not happy with the existing radio button solution: we definitely want to provide more guidance and structure there..

I would personally design this to make CoreRadioGroup mandatory, and then bubble selection events up to the CoreRadioGroup entity, which then goes and deselects all of the other radio buttons. I'm somewhat skeptical of using one-shot systems for callbacks on things like radio buttons. I would prefer a more structured approach where we:

1. Store the selected value in a component on the radio group entity, likely as an struct SelectedValue<T>(Option<T>).
2. Encourage users to add their own components (or values of components) to handle responsive styling and themes.
3. Use name-based lookups in the UI to avoid the need for a billion marker components and play nicer with a scene-driven or dynamic approach.

The SelectedValue component would then end up as the input into a reactive framework of your choice (or as the trigger for a change detection driven one shot system), which could then respond in complex ways within the course of a single frame.

[alice-i-cecile]

Stream-of-thought feedback on https://github.com/viridia/bevy_core_widgets/tree/30dd141de1f2fa6ec12435e03a576e9cf7005ce9, at the time of the linked commit.

1. I really love the core idea of providing a high-quality, fully accessible set of core widgets with all the bells and whistles that everyone can use and restyle. I suspect we should also provide a standard set of styles, which are probably best reused for the editor and other first-party GUI tooling. Vital for our examples, and nice to allow other tooling to easily match our look and feel.
2. One plugin per widget is great: good for organization, really easy to pick and choose what you want.
3. Barriers should similarly be split out into their own sub-plugin.
4. The existing accesskit integration needs to be completely redone: we need to store this information granularly on individual components defined in bevy_ui or bevy_a11y.
5. One-shot systems for callbacks stored on widgets for custom behavior seems like an excellent pattern in general. I really like its use for buttons.
6. Observers are the right choice for figuring out when to call the e.g. on_click callbacks stored in widgets: efficiency isn't a huge concern and worrying less about system ordering and multi-frame patterns is good. This was the core problem that motivation the infamous YesAndLoop run criteria design.
7. This is fairly obvious, but yes, we need to have a separate InteractionDisabled component and can't reuse Disabled for it. It needs to be styled, and disabled buttons should be rendered and interacted with!
8. It may make more sense to use run_system_with in some cases for callbacks, where you want to pass in information about the context or the triggering action. I don't think that makes sense for buttons, but something to bear in mind. Addendum: I see you're already doing this for other widgets!
9. I'm generally rather leary about mixing "arbitrary app effects" with "responsive styling", and shoving them all into on-click etc handlers, because that makes it really hard to avoid duplicating the styling logic. Ideally there's a clear split between "logic that differs for each instance of this widget" and "logic that's shared by all widgets of the same type defined by this opinionated style crate". I think that's manageable, but we need to be really clear with docs and patterns.
10. I'd like to try and reuse simple components across widgets more aggressively when the meaning is clear. Something like CoreButtonPressed could be a unified Pressed component for example. Downstream systems etc can filter for Pressed + CoreButton. This is particularly weird because Hovering gets its own component!
11. Mildly concerned about using hardcoded KeyCodes for navigation due to accessibility and localization issues, but I think they're stable across keyboard layouts and it's a fine place to start.
12. Individual callbacks for checkboxes make sense to me! If you want observer-powered change detection you have to split this out and use immutable components though, as discussed in my other reply.
13. I'm not thrilled at the amount of duplication for keypress vs on-click behavior, which only gets worse when we add gamepads. You're aware of my broad goals to try and unify this by mocking clicks (or some other design) though, so I don't think that's permanent.
14. Using fully public fields like in CoreSlider is great for easy serialization and ease of use, but it's not enforcing invariants like min < max correctly. I suspect we're going to need fallible scene deserialization and private fields sometimes.
15. Splitting out Interaction into a collection of tiny bool-holding components definitely feels like the right design. Adding and removing components is slow, and lumping them together risks nonsense data and messes with change detection.
16. update_cursor_icon is really nice, and something that we should standardize on / communicate upstream. I don't think resetting to CursorIcon::default works for games though; I think that you need a Res<DefaultCursorIcon>.
17. The ValueChange observer-event is very nice. I think that + immutable components should work well.

Key takeaways:

1. Huge thumbs up to a headless widget library.
2. One-shot systems for ad-hoc UI functionality upon interaction is great.
3. We need to be really careful about splitting out one-off interaction behavior from "shared styling" to make sure our users don't abuse the former to do the latter (slow, hard to maintain).
4. IMO we can and should start upstreaming more of the foundations here: things like ValueChanged, Hovering, Pressed, which should mostly live in bevy_ui.
5. Fallible deserialization to maintain invariants / use constructors (rather than just naively setting public fields) is going to be a core design consideration for BSN.
6. bevy_a11y needs to be rewritten to better fit the way bevy_ui wants to work: accesskit should be reading ordinary Bevy component values that devs naturally want to set, rather than tacked on in a megacomponent.