text work planning & discussion

[cmyr]

Druid text planning

This is intended as a medium-level sketch for how we will implement multi-line text, text editing, and (eventually) rich text in druid.

This covers a number of things: the storage of text itself, the storage of style and color information, the storage of selection state, and the storage of line breaks and width measurement, as well as the relationships between these, as is required for things like interactive editing.

Representation of text

We are going to adapt the xi-rope crate as our main way of representing text buffers. This is a not-insignificant depedency, but ultimately it feels worthwhile; the design of the rope (using smart pointers under the hood) makes it very well suited to druid's Data model, and xi-rope includes a large API for things like representing and applying edits, diffing, text search, and rich
text spans.

We will use xi-rope's Rope to store our text, and the Spans type to (later) represent rich text information.

These two types will be used together, and will be wrapped up in some druid type, with a name like TextBuffer.

Line breaking and width measurement

In order to display text to the user in a meaningful way, we need to be able to do line breaking and width measurement. A version of this is provided by piet; we will wrap this API in druid, providing our own TextLayout type.

This type will own a copy of the TextBuffer, and will also have a width; it will be responsible for both painting text as well as for converting from points on the screen (e.g. from a mouse click) to offsets in the buffer.

You will create a TextLayout from a TextBuffer and a PietText object (discussion: can we make PietText not be bound to a lifetime?); you will then be able to paint it, using a convenience method looking something like, TextLayout::draw(&self, ctx: &mut PaintCtx, point: Point, env: &Env). You will also be able to query the layout for information like the offset in the buffer that corresponds to a given point in the layout.

When the buffer changes elsewhere, you will update the TextLayout, by calling a method like (e.g.) TextLayout::update_buffer(&mut self, buffer: &TextBuffer) and then the TextLayout will update its internal state. An important part of this design is that the layout will be able to update itself incrementally, by comparing the old and the new buffers; however for our first pass we may just recompute the entire layout from scratch.

This TextLayout type will be the basis for all text that is displayed in druid; we will discourage the use of piet directly.

Selection state and editing

The last major component to manage will be selection state. This is slightly tricky; in xi, for instance, we treat selection state as a property of the 'view' (which corresponds to our TextLayout, above) but this might not work in druid. The main question is whether or not selection state should be part of the data model, or whether it is just part of the view state. I'm of two minds. The main question seems to be whether or not it is reasonable during normal use for the selection state to be modified elsewhere in the application.

I see a number of options: the selection state could be part of the TextBuffer, it could be part of the TextLayout, or it could be independent?

In any case, selection exists at the intersection of the buffer and the layout; it is stored in terms of offsets into the buffer, but it can only be modified with access to the layout (for instance: if we press the 'down' arrow, how do we know what offset in the buffer corresponds to our current horizontal position on the subsequent line?)

Regardless of where the selection is stored, edit operations can be considered as functions of the form, Fn(TextBuffer, Selection, &TextLayout, &EditOp) -> (TextBuffer, Selection). That is: it takes a buffer and a selection, and referencing the layout produces a new buffer and a new selection.

Rich text

TextBuffer will be able to store style (font, color, size) information for regions of its content. This is not going to be fully exposed initially; for the time being a given buffer + layout must use a single style.

As follow-up work, however, we will want to allow proper rich text. This is going to require some additional work in piet, and also a major design decision: should piet handle the layout of rich text (requiring us to expose our TextBuffer and related styling types there) or should we implmenet a lower-level piet api that operates on glyph runs, but which requires more work in druid in order to handle actual line breaking?

This is a complicated question; it makes sense for piet to be as low level an api as is reasonable, but moving to using runs is potentially a reasonable large project, and it may not offer any real efficiency improvements over an approach that just relies on directwrite/coretext/etc to do more of the heavy lifting.

Next steps:

I'm going to play around with this general design, and see if it feels like a reasonable direction.

Open questions:

where does selection state live?: I think it should be part of either TextBuffer or of TextLayout (which owns a copy of the TextBuffer), largely because it makes it easy for TextLayout to also be responsible for drawing the selection. Between these two it comes down to whether or not we want selection state to be part of the data model, or not.

How do we handle layout for rich text? we'll cross this bridge when we get to it.

What do we need in a font api? We didn't touch on this, but an additional aspect of this work is going to be a fleshing out of the piet api for resolving and using fonts.

What other minor changes might we want in the piet api? For instance, can we make PietText not be bound by a lifetime? This may not be necessary, but might be a quality of life improvement?

[ForLoveOfCats]

There are a few things I would love to have exposed by piet which are related to your rich text subtopic. The first is the ability to modify font size, bold, italics, ect of text being passed to new_text_layout on a [index, index] level so that it is still a single layout solution with correct character spacing and such. This one is probably the most work especially with different text shaping backends that may be supported. Secondly draw_text should allow coloring different portions of a text layout in similar fashion. Finally I would like a way to control font fallback with one such usage case being the ability to provide a specific emoji set instead of relying on the system emoji font.

[cmyr]

styling for ranges is definitely one thing we would like to do, although it's going to come after some initial work.

Explicit font fallback is something else that would be nice to have, but is also very platform dependent, and I'm going to punt on that for the time being, but it might be worth opening an issue in piet?

[simonbuchan]

Other cases to think about for text editing are password and credit-card or date inputs, where the text display is separate from the input.

[TheNeikos]

Would RichText also allow for clickable regions with Hover/Active states? This way one could have 'links' in their text to trigger commands etc... Or is that out of scope here?

[JAicewizard]

From what I have found it is currently not possible to load custom fonts into druid. Any upcoming API should allow for loading fonts on the fly, or at the very least at startup.

[cmyr]

yea, the ability to at least bundle fonts with your application will be part of #397.

[dhardy]

For what it's worth, I've recently been working on kas-text for roughly this area. Current status is that multi-line text layout works, also with editing but only over a String buffer (so not appropriate for large documents), HarfBuzz shaping is integrated (optional). BIDI is next on my to-do list, then some fixes for text navigation. Rich text is also intended, but I haven't put much thought into that or how to handle large documents yet.

In my case, I have a single prepared::Text type corresponding roughly to your TextLayout; it contains three things: environment data (default font size, alignment, bounding box), the text (currently just a String), and "prepared" data (contiguous runs divided by break-points, positioned glyphs without wrapping, and finally the lines as slices over the glyph runs with offsets). This allows re-wrapping without re-shaping and is set-up to allow easy BIDI integration, but rich text may be harder.

For font-handling I currently use ab-glyph (from glyph-brush). Supporting other font libraries and glyph-renderers should be possible.

Hopefully we can collaborate on this, since text processing is a complex topic and one that should be addressable in a flexible enough way that works with most GUI toolkits. For my part I've been focussing on hacking things together to meet my design requirements, leaving a lot of clean up and testing work for later.

[raphlinus]

Tristan says on Zulip, and I agree: Godot has implemented bidi, shaping, variable fonts, OpenType features, fallback and more, and all their code is confined to a few branches which might be handy to reference: https://godotengine.org/article/complex-text-layouts-progress-report-2