[touch actions] handwriting manipulation type to distinguish panning

[adettenb]

Summary

Opening an issue to discuss options and alternatives to allow developers control over whether handwriting input should be allowed for elements the user agent supports handwriting input with.

Updates

• 11/27/2024 - Including links to the HTML and Editing WG Issues, and CSS explainer, and CSS Chrome Status.

Introduction

Give developers granular per-document and per-element control over which content should (dis)allow handwriting input for elements which the user agent supports handwriting to text input for.

Earlier created a proposal to introduce an HTML handwriting attribute, however upon review touch-action was noted as the preferred mechanism for enabling/disabling touch behaviors.

Currently Android uses the combination of both pan-x pan-y as the mechanism to allow handwriting on eligible fields.
However, this means that scrolling and handwriting are tied together, so turning off panning will turn off both handwriting and touch scrolling.

What this is

• A mechanism for developers to specify whether handwriting, when supported by the user agent, should be allowed.

What this is not

• A mechanism to detect the availability of handwriting support.
• A mechanism to enable handwriting on elements the user agent doesn't support handwriting for.

Use Cases

• Document editor that wants to temporarily disable handwriting input while certain tools are selected, to support using a stylus to seamlessly draw, place, or size non-text content overtop an editable text region.
  • Outlook drawing tools with "text pen"
    
• Application with custom form controls that accept sensitive input, or have a strict but not standardized format that isn't easily input with handwriting to text input.
• Application with custom text inputs or editing experiences that override default browser behaviors by observing and handling input events and editing experiences, doesn't support input method editor (IME) or composition{start|end|update} events, or if for any reason the experience designed by website authors doesn't behave as they intend when handwriting input is available.

Pros

• Matches existing expectations around the touch-action CSS property, developers can choose to enable handwriting or not.
• Requires a relatively small code change, both for Chromium implementation and for web developers to control where handwriting should be allowed.
• Allows developers to individually specify whether stylus handwriting or touch scrolling should be enabled.
  • This wouldn't be possible without using preventDefault to bypass handwriting and then manually handle stylus scrolling with something like scrollTo.
  • For example, developers could specify:
    • touch-action: manipulation to allow stylus handwriting, and both touch scrolling and touch zoom
    • touch-action: pinch-zoom handwriting to allow stylus handwriting and touch zoom, but disallow scrolling
    • touch-action: pinch-zoom pan-x pan-y to allow scrolling and touch zoom, but disallow handwriting

Cons

• ~40% of sites specify touch-action and may need treatment to enable handwriting if used in the context of editable text.
  • Chrome Platform Status (chromestatus.com)
  • Property keywords are opt-in when any are specified (default is opt-out by including all keywords). So sites specifying any keywords other than auto, inherit, or manipulation on editable text or ancestors of them will need to be considered by site developers.
  • i.e., Sites that may want handwriting could unintentionally have it disabled once the experience becomes available, needing to update the site to allow the experience.
  • It's unclear from the chrome status page which keywords are being used or if they're being used in editable context, so not enough information to gauge the full impact.
     

See also:

• HTML WG Issue
• Editing WG Issue
• CSS handwriting keyword explainer
• CSS handwriting keyword Chrome Status

Previous HTML Attribute proposal:

The HTML attribute proposal is a competing standards proposal and has been abandoned in favor of this CSS touch-action standards proposal. However, the discussions around the HTML attribute are still relevant here so including them here.

• HTML handwriting attribute explainer
• HTML handwriting attribute Chrome Status
• HTML handwriting attribute Intent to Prototype
• Enable direct pen and touch to have different touch-action behavior

[flackr]

See also #512

[adettenb]

Hey @flackr, I reviewed the link you posted about pointer-manipulation, and it sounds like it would supplement touch-action?

If I understand correctly, the benefit of the pointer-manipulation property would be that developers can specify which input device type(s) [touch | mouse | stylus] should allow the behavior(s) enabled by touch-action: manipulation (i.e., [pan-[x|y] | pinch-zoom | +handwriting])?
Unless touch-action would be deprecated and replaced by another property with a more appropriate name.

I noticed you mentioned stylus writing in that comment, but I don't see a clear and concise way to separate the two behaviors (scrolling and handwriting) with pointer-manipulation: [auto | none | touch | mouse | stylus], and handwriting doesn't see like it'd fit as a keyword for that property since it's not an input device type.

For this proposal the goal would be to enable developers to specify whether scrolling and/or handwriting are allowed independently. So the developer could allow neither, either, or both (with UA defined precedence, likely handwriting over scrolling).

cc: @mustaqahmed

[mustaqahmed]

For this proposal the goal would be to enable developers to specify whether scrolling and/or handwriting are allowed independently. So the developer could allow neither, either, or both (with UA defined precedence, likely handwriting over scrolling).

I really like both the ideas here: independent control and precedence. Today in PEWG we also discussed the need for a fallback option when the page specifies handwriting but the device doesn't support it. I think both the precedence and the fallback questions could be addressed using a list like this. In any case, I am fine with addressing those questions separately later after adding a pointer-type agnostic handwriting value first.

[smaug----]

I think I'm not fine addressing those issues later. At least we need to consider what all issues there can be if there are no reasonable fallbacks.

[flackr]

If a developer specifies touch-action: handwriting; they have specified (through omission) that panning is not expected / allowed. So if a device supports handwriting it can start handwriting, and otherwise, it will do nothing. This seems fine - they may have added a pointermove listener that records the pen strokes.

This doesn't just apply for devices that don't support handwriting. If you specify touch-action: handwriting at the root, even on a device / platform that supports handwriting only starting near some elements (contenteditable=true / <input> / <textarea>) will actually be able to start writing.

[patrickhlauke]

Dropping this here for interest/context https://www.youtube.com/watch?v=5nELac-Wgo0

[smaug----]

It doesn't seem fine. Since one can't detect whether handwriting is supported or not, part of the page would be just unusable.

[flackr]

It doesn't seem fine. Since one can't detect whether handwriting is supported or not, part of the page would be just unusable.

Whether handwriting is supported or not, that part of the page would not be scrollable. I get that you're making an argument that they set handwriting because they assumed that you'd want to write in that area, but there's no difference in terms of scrolling functionality just as if touch-action: none had been specified. Developer documentation should encourage specifying touch-action: handwriting pan if both actions should be allowed.

There is a separate question of priority. E.g. even if both actions are allowed can you scroll over an area that accepts handwriting and/or can you write over an area that is scrollable.

However, I do think touch-action or a related pointer-action is the right property for this. Developers are already using touch-action: none to state that the browser should not handle swipes as they intend to handle pointer input instead so it's nice to build off of this in a compatible way.

[ogerchikov]

I agree with @flackr's argument that the intention of touch-action: handwriting is to disable scrolling/zooming but allowing text entry using handwriting method on capable devices. On devices that don't support handwriting, this equivalent to touch-action: none.

@smaug---- do you have a use case in mind that would make page unusable?

[patrickhlauke]

do you have a use case in mind that would make page unusable?

I'd imagine a long page with lots of text fields, like a long form ... if each of those inputs (possibly even their containers with some extra padding around them) was set to handwriting, and the device didn't support it, then users would find it very difficult/impossible to scroll/vertically pan the page, as they'd need to try and grab any bits of page between the form fields/containers only

[ogerchikov]

I'd imagine a long page with lots of text fields, like a long form ... if each of those inputs (possibly even their containers with some extra padding around them) was set to handwriting, and the device didn't support it, then users would find it very difficult/impossible to scroll/vertically pan the page, as they'd need to try and grab any bits of page between the form fields/containers only

The same breakage would apply to devices that do support handwriting. Wouldn't it?

[patrickhlauke]

Ah, true. So it's more of a potential developer footgun in general - once you specify that an area is just going to react to handwriting, nothing else will happen (and with the current way that touch-action applies indiscriminately to all pointer types, this would mean no panning with touch either)...

[flackr]

So it's more of a potential developer footgun in general - once you specify that an area is just going to react to handwriting

Once you specify that an area is only going to react to handwriting. It reacts to handwriting by default, nothing needs to be specified. This is why the developer advocacy would point out that you should be saying touch-action: handwriting pan or some other combination.

This is similar to how we react to pinch-zoom by default but if you say touch-action: pan-x you can no longer pinch zoom.

[adettenb]

+1 for touch-action: handwriting pan or similar, combined with UA defined precedence would allow both handwriting and scrolling (for stylus and touch), and devices that don't support handwriting will simply have scrolling. If there were a more granular mechanism like we discussed on call we could also filter specific actions by device type, and provide a fallback mechanism for example.

An unfortunate side-effect is any sites that might want handwriting which already specify touch-action would need to take action to enable this feature, however this is by design for touch-action as far as I can tell.

On the BlinkOn slide #21, I included this concept with a format similar to manipulation: DEVICE_TYPE(ACTION_LIST) ...;.
This was the same idea as the earlier proposed pointer-action, but I include other non-pointer "device types" like "eye" for eye-tracking, "positional-tracking" for VR/AR controllers, and "voice" for microphone input. Some of these input device types were mentioned in another similar proposal.

I'm imagining this CSS manipulation or pointer-action property optionally allowing device filtering and multiple actions per filter. Possibly even allowing the defined order to declare which takes precedence rather than leaving it UA defined. I think UA defined precedence probably makes the most sense though, it'd be much easier to implement at least.
e.g.,

• If an author only wanted "stylus device" handwriting and "touch device" pan/zoom:
  • manipulation: stylus(handwriting) touch(pan zoom) none
• If an author only wanted "stylus device" handwriting and "any device" pan/zoom:
  • manipulation: stylus(handwriting) pan zoom
  • manipulation: stylus(handwriting) any(pan zoom)

For the PR I'll be drafting this week, I plan to simply add handwriting as a keyword with UA defined precedence
I think the other options are worth pursuing in the future, but would likely replace and deprecate touch-action.

[ogerchikov]

@gastonrod has filed issues across multiple working groups (see links above) requesting feedback. Please feel free to leave your comments there.

[flackr]

We discussed this in the call. There was a general concern about the behavior for legacy content (for which handwriting was not something developers knew they should opt into).

• For pre-existing touch-action: none content, developers expect to be disabling and manually handling pointer events. This will work as expected with this proposal as none will disable handwriting.
• For pre-existing touch-action: pan-x (or pan-y) content, developers expect to be disabling and handling scrolling in the other direction. However, it's less clear whether or not they would have wanted to disable handwriting.

There were concerns about cases where you might want a different action for touch and stylus, note that no UA currently will start handwriting on touch so by default touch will scroll where a stylus will start writing. However, you would not be able to specify that touch should pan but a stylus does nothing. For this, we expect that #203 would allow differentiating.

Further, we expect that developers would like to be able to opt into a different default action for devices capable of multiple, #512 suggests some extension for this in the future.

We think that we might be able to defer to these other issues and suggest that authors use new properties there to both fix some of the oddities of the touch-action property and allow for the full expressivity of desired behaviors and for now focus on defining something for touch-action that best handles legacy content.

[flackr]

The other thing to consider here is whether handwriting should only consider the touch-action to the nearest ancestor scroller similar to panning: https://www.w3.org/TR/pointerevents/#determining-supported-direct-manipulation-behavior

If it did, I suspect the majority of legacy cases would "just work" since input fields and textareas are scrolling containers which would implicitly get handwriting re-enabled if not explicitly disabled on them.

[gastonrod]

The way it was laid out in the explainer (and how the affected pages counter was implemented) is that having a parent that disables handwriting between the writable element and the root of the document would disable handwriting, more in line with how pinch-zoom works [1]. For example:

<style>
body {
    touch-action: none;
}
</style>
<div style="overflow: auto;background-color:lightblue;height:300px;width:300px;touch-action: auto;">
    <div style="touch-action: auto;width:500;height:500;" contenteditable="true">
        This element can pan due to the scroller allowing it, but it wouldn't have handwriting.
        The UA doesn't consume pointer events in the rest of this document.
    </div>
</div>

Would disable pinch-zoom and handwriting in the entirety of the page, but the div would still be touch-pannable, because the top level scroller allows it.

I think changing the enablement to work as panning does would be very effective to mitigate how many pages we cause issues for. I'll start discussing this with Olga. Thanks for the comment 👍

[gastonrod]

Following the panning rules seems to be a good idea, as the handwriting keyword will be replacing the current unofficial heuristic that requires pan-x pan-y to enable handwriting.

Another angle this could be tackled from is to use writable elements as panning uses scrollable elements, so the spec would look something like this:

• A direct manipulation interaction for panning is supported if it conforms to the touch-action property of each element between the hit tested element and its nearest inclusive ancestor that is a scroll container (as defined in [CSS-OVERFLOW-3]).
• A direct manipulation interaction for handwriting is supported if it conforms to the touch-action property of each element between the hit tested element and its nearest inclusive ancestor that is an editable element (probably taking some definition from here).

This way, we can have handwriting rules that are independent on whether or not an element is scrollable, which on the surface level at least seem unrelated. Some pros and cons for this approach:

Pros:

• Greatly minimizes the amount of existing scenarios we will remove handwriting for, practically minimizing this backwards compatibility concern (I speculate that much more than making handwriting follow the panning rules).
• We aren't creating an arbitrary unprecedented rule, just taking what exists for panning and adapting it for this new use case.
• Makes scrolling and handwriting independent from each other.

Con:

• Adds one more rule that developers would have to take into consideration if they want no touch actions in their web pages. Not only would they have to explicitly set touch-action: none; for scrollable containers, but also for text editable elements. For example:

<style>
body {
    touch-action: none;
}
</style>
<div style="overflow: auto;background-color:lightblue;height:300px;width:300px;touch-action: auto;">
    <div style="touch-action: auto;width:500;height:500;" contenteditable="true">
        This element can pan due to the scroller allowing it, but it wouldn't have handwriting.
    </div>
</div>
<input type="text">/*this input would have handwriting, but no panning*/</input>

In this page, if authors don't want any touch action in the entire page to be consumed by the UA they would have to add touch-action: none; in both the scrollable div and the input.

[adettenb]

I'm on the fence about restoring the handwriting bit for nested scrollable areas.
I think it might be better to scope restoring the handwriting bit when entering a nested document instead, similar to pinch-zoom.

The main appeal I see for restoring the pan bits is the keyword is directly tied to the behavior enabled by overflow. It encourages developers to disable scrolling for each context that needs to. A developer may want to disable touch scrolling for the main document, but likely wouldn't want to disable scrolling for all widgets/modules/etc. that become scrollable when the user resizes the window or a portion of the document. A scenario that comes to mind is a richly editable infinite canvas or chat application with nested code blocks or something, where the author wants to handle panning/scrolling/resizing content themselves, but when a nested textbox becomes scrollable from user actions (entering text, resizing content/window) then touch scrolling would work for that content unless the developer specified otherwise (e.g., for a more natural text editing experience; selection or caret placement).

However, adding the handwriting keyword is meant to decouple the behavior from pan since handwriting behavior isn't dependent on content being scrollable or related to overflow beyond taking precedence over (suppressing) scrolling when handling stylus events. I worry this could make it more difficult or impossible to prevent handwriting for some widgets containing overflow content. I need to think about this some more and experiment a bit, but I think restoring the handwriting bit based on either overflow or which document the element belongs to may make it impossible for developers to disable handwriting for editable text fields within native Shadow DOM controls?

I'm also curious what the behavior should be for OOP <iframe>?

• Should that be the responsibility of the embedder or the host content?
• Should developers be able to disable handwriting for an <iframe>:
  • ... implicitly through inheritance
  • ... explicitly by specifying touch-action on the outer <iframe> element
  • ... or should that be at the discretion of the content being embedded?

For OOP <iframe>'m I leaning towards the last point, effectively following the example of pinch-zoom and scoping handwriting to the nearest containing document. It might be desirable to allow embedders to explicitly override this for the nested document with <iframe style="touch-action: none;"> or something like that, but the existing rules for pan and pinch-zoom allow the nested document to restore those bits.

---

I think this is worth pursuing though as an experiment. I think @gastonrod's recent histogram CL will help understand the impact for these kinds of scenarios.