[resize-observer] ResizeObserver should support fragments

[gregwhitworth]

@plinss Brought up the desire for the API to understand the notion of fragments in the TAG review (w3ctag/design-reviews#187 (comment)). I've decided to make this a seperate issue because I would like to understand this a bit further. I'm assuming that you wouldn't expect to be able to attach an observer to a fragment correct (since we won't know how many fragments will exist in a given layout until after layout, thus hard coding the fragment doesn't make much since)?

So my suspicion is that you're wanting the fragments represented in the resulting entry object, is this correct?

[plinss]

Correct, I haven't spent any time trying to design a specific api surface for this, but anything that exposes all the fragments in the entry rather than presuming one box is what I'm trying to get.

[gregwhitworth]

Thanks - that clarifies it for me. So effectively that would require a change to the specification in two ways:

1. The entry object

This would need to have a notion of fragments although I don't desire to denote that in any way. Taking a scenario such as multi-column - we could adjust the ResizeObserverEntry to a <boxName>: sequence<ResizeObserverSize> which would end up with a result of something like this, again assuming multi-col with two resulting columns and the author observing the contentBox:

{
/* ... rest of object ... */
contentBox: [
        {
           inlineSize: <double>,
           blockSize: <double>
        },
        {
           inlineSize: <double>,
           blockSize: <double>
        }
]
}

This has the added benefit, to @atotic point in the TAG review of the most common use cases currently not being about fragmentation so even if an implementation can't initially support building out the fragments easily or performantly they can add it as demand for the capabilty increases and we don't have to revisit the API or come up with a brand new one to support it.

2. Observing numerous boxes
I'll need to think about this a bit more thoroughly to understand if there is anything that currently needs to change with the how boxes are observed. Regions currently aren't auto generated and thus there would need to be seperate observers attached to the seperate elements.

Now if we assume a new Regions display type that does auto generate a new box, similiar to how multi-col works where the Fragmentainer clones the intial child that requires fragmenting to create a sibling to box to contain the new content. In this scenario I would expect that the Observer would be then attached to each additional fragment and its associated box to observe to ensure that any time that any box's dimensions change the callback is called and the above ResizeObserverEntry is passed.

There may be issues here with queueing up numerous additional Observers, some open questions I have is:

• Are dynamically clonable observers supported by Observer's currently, and would the dynamic fragments need to be Elements in order to attach the observers?
• What are the performance implications within the adjusted ResizeObserver lifecycle of queueing up potentially infinite Observers? I have a feeling we'll run into a situation where it will result in an increase of skipped observations and thus lead to increase of LoopErrors.
• @atotic it would valuable, with your work on LayoutNG to put together a rough dev assessment of supporting fragments in resizeObserver in conjuction with LayoutNG since it seems Firefox will be able to support this.

@plinss Primarily, does the ResizeObserverEntry object above address your concerns with support of fragments and this API not blocking the capality of using RO in conjuction for fragment based scenarios in the future?

Sorry for the book, just trying to think out loud here about the potential issues and the usecases where this may be utilized as I agree I don't want RO to become useless if we light up future scenarios where more fragmentation occurs.

I look forward to hearing your thoughts @atotic & @plinss

[atotic]

Re: Implementing fragments in Chromium

Our current layout engine support for fragments is a giant hack. Our new layout engine, LayoutNG, supports fragments natively. LayoutNG will be able to support fragment size observation.

I'd rather not tackle fragment support in Legacy. We are hoping to ship LayoutNG fragment support within a year.

[atotic]

Re: reporting fragment size as array of sizes

Current proposal: object
borderSize: { inline: ,block: }
mutates to an array if target becomes fragmented
borderSize: [ { inline: ,block: }

My concern is this:

If webdevs wrote their RO handler without taking fragmentation into account like this:

new ResizeObserver(entries => {
  let newBorderWidth = entries[0].borderSize.inline / 10;
},

their code would break if borderSize became an array.

I believe most webdevs will not write fragmentation-compatible code.

To minimize breakage of fragmented components that use ResizeObserver, I propose that developers should explicitly opt-in to fragment support. This can be done by adding a new option to RO.observer fragments: false/true.. If fragments is true, sizes would always be reported as an array.

[gregwhitworth]

@atotic borderSize is not currently shipping anywhere - as such, when authors learn about this change to the API they will undoubtedly either through tutorial or debugging see that it returns an array of entries. And thus their code will be written with borderSize[0].

[gregwhitworth]

Another question, do we want to surface the contentRect dimensions in total alongside the fragments that contribute to these dimensions. I think we should as there may be information that are not surfaced in the fragments (eg: column gaps).

contentBox: {
           inlineSize: <double>
           blockSize: <double>
           fragments: [{ inlineSize: <double>, blockSize: <double> }]
}

[css-meeting-bot]

The CSS Working Group just discussed adding fragmentation support to resize observer, and agreed to the following:

• RESOLVED: Option 1; contentBox is an array of fragment sizes

The full IRC log of that discussion

<heycam> TopiC: adding fragmentation support to resize observer
<gregwhitworth> Github: https://github.com//issues/3673
<heycam> gregwhitworth: plinss brought up that we don't support fragments
<heycam> ... which is valid
<heycam> ... I don't want this API to back us into a corner for future support of some new display type
<heycam> florian: or printing
<heycam> gregwhitworth: in the issue we have this similar entry API shape
<heycam> florian: if you observe a multi col you would have two values in the entry?
<heycam> gregwhitworth: if observing a box that is fragmenting ...
<heycam> florian: ok
<heycam> gregwhitworth: ... this is just turning the contentBox property value from a single object to a list
<heycam> ... the one caveat I have with Houdini causing many fragments is that you'll get callbacks during custom layout, and you probably wouldn't use custom layout here.
<heycam> florian: inline fragmentation
<heycam> ... do this cover that?
<heycam> alex: resize observer doesn't support watching inlines
<heycam> florian: ruby doesn't fragment
<heycam> dbaron: some larger pieces can
<heycam> florian: the rb and rt don't
<heycam> florian: anyway, the API shape here works
<heycam> Rossen: the only thing you're trying to express here is paginated scenarios, such as multicol, in which you say are observing a div
<heycam> ... and the div is broken between two columns
<heycam> ... the representation would be a list of all of the fragments for that div
<heycam> ... of the content box of that div
<heycam> ... and the expected behavior is that you would get a callback when any of them change?
<heycam> gregwhitworth: that's the next issue
<heycam> ... technically I don't think you can observe those
<heycam> ... in multicol it's ok, content rect will change, and you'll get the notification
<heycam> ... I don't think you can't observe the other boxes
<heycam> florian: what other boxes?
<heycam> gregwhitworth: when you add a RO to an element, I don't know if you can add a resize observer to the other fragments
<heycam> dbaron: one way you could see this working is that -- an element has multiple boxes, if you're split over three columns, you have three boxes, if you fire an observer right now, you would produce an array with 3 boxes in it
<heycam> ... if that array were at some future point some different result, now it only has 2 things in it, or 4, or anything changes in that array, you fire another observation
<heycam> ... because something has changed
<heycam> florian: e.g. because it moved
<heycam> stefan: the interesting case if the content box didn't change but the fragments did
<heycam> ... I don't like that idea
<heycam> gregwhitworth: not even sure we can observe just the fragments
<heycam> dbaron: so you do'nt want to fire an event if it only moves between columns?
<heycam> gregwhitworth: did the content rect change? then produce the fragments from that
<heycam> dbaron: but there are 3 content rects
<heycam> [Rossen draws]
<heycam> gregwhitworth: so what the API represents at this point is the fragmentainer ...
<heycam> florian: no fragmentainer is the container around that
<heycam> alex: if I am a web dev, have a concern how they're oging to use the fragmentation feature if at all
<heycam> ... most webdevs will code for a single box
<heycam> ... what happens if I've created a resized observer and suddenly someone puts the target in a multi col
<heycam> ... now the use of resize observer breaks
<heycam> heycam: depends on the exact API shape whether it would break or not
<heycam> alex: not sure what the use cases are exactly, but I can see most authors not thinking about fragmentation
<heycam> Rossen: the few that care about it have it
<heycam> gregwhitworth: I think it's ok to have it in here
<heycam> ... I don't see this scenario happening much based on the main use case for resize observer
<heycam> florian: I think for now we could kind of pretend that if we want to make it easy, we could pretend there's no fragmentation and just expose a single box
<heycam> ... but that's not future proof
<heycam> iank_: we have bugs in our impl with reporting geometry things in fragmented containers
<heycam> gregwhitworth: so the idea was to add an array, then add a single item for now in the array (incorrectly)
<heycam> florian: I think this satisfies plinss and fantasai's concerns
<heycam> rego: now the offsets might be useful...
<heycam> stefan: at some point if you're doing things with multicol and resize observer is not giving you granular enough information, you will use Houdini
<heycam> florian: cases where you might want more detailed info is custom painting, but then you'd use Custom Painting API
<heycam> dbaron: what kind of boxes or elements or whatever can you use resize observer on?
<heycam> Rossen: block elements
<heycam> alex: and SVG
<heycam> florian: grid, multi-col
<heycam> dbaron: tables?
<heycam> ... some things get messier with e.g. column spans in tables
<heycam> alex: watching column span sizes? or cell sizes?
<heycam> dbaron: watching a div with a column span in the middle of it
<heycam> florian: we need to be mindful about which boxes, and which kinds of elements you can observe
<heycam> Rossen: I hope the current spec is already precise as to which elements you're allowed to observe
<heycam> florian: if not it needs to be
<heycam> iank_: still an open question about how to represent the fragments, a separate side structure
<heycam> alex: an array
<heycam> iank_: an additional array?
<heycam> ... the most common thing is people will use only one box
<heycam> ... weshouldn't make them access through an array
<heycam> Rossen: multiple box objects or a bunch of rects?
<heycam> Rossen: people might come back and say that fragments are not really rects
<heycam> iank_: if there's a single fragment, is that still an array with 1 element?
<heycam> gregwhitworth: yes
<heycam> heycam: what about the order of the entryies in the array?
<heycam> Rossen: document order
<heycam> ... the biggest issue with regions is that people complain about search result order
<heycam> gregwhitworth: the common use case for this is people watching border box and watch that change
<heycam> Rossen: but even that can be fragmented
<heycam> alex: the question what elements can we watch
<heycam> ... all elements, but observations don't fire for non-replaced inline elements
<heycam> ... per spec
<heycam> florian: so we need to be a bit more specific about table cells, ...
<heycam> alex: we can
<heycam> iank_: it would be nice if not needing to use the array
<heycam> ... could also make the contentBox attribute polymorphic
<heycam> alex: it would break things as soon as it becomes fragmented
<heycam> gregwhitworth: that's why I would like to have the summation in the entry too
<heycam> florian: what about when one fragment is narrower than another?
<heycam> iank_: Edge and Firefox take the union of the rects
<heycam> gregwhitworth: that would let you handle the common case
<heycam> ... but you would also have the fragments available in a separate property
<heycam> florian: as long as this isn't an excuse to not implement the array
<heycam> Rossen: if I have a tiny column for one part of a div and a large one for the rest of the div
<heycam> ... would you sum up the inline sizes in these single values?
<heycam> iank_: what you report for clientWidth today, which is the union of those values
<heycam> [whiteboard discussion about how to deal with summing up fragment sizes when they're on different pages]
<heycam> gregwhitworth: valid point about not being able to sum them up
<heycam> ... but I'm going back to the main use case of non fragmented boxes you want to observe
<heycam> Rossen: one way to interpret the proposal here is "we have heard your comment about fragmentation and come up with a future proofing of how this could be done"
<heycam> ... "and this or a version of this will solve this problem in a way to satisfy everyone"
<heycam> ... so we could resolve on having the fragments as a list we can add later
<heycam> florian: doesn't help, it's hard to define the separate values for the summed inlineSize/blockSize now
<heycam> Rossen: we can open a separate issue on fragmentation spec for that
<heycam> florian: if we go with the earlier syntax we don't have that problem
<heycam> iank_: we need this defined anyway for clientWidth/clientHeight
<heycam> ... there is blink/webkit vs edge/firefox
<heycam> Rossen: opening an issue on CSSOM and defining how clientWidth/clientHeight work under fragmentation is needed
<heycam> ... and whatever answer that has, will be reflected to be the two numbers here for inlineSize and blockSize on the entry
<heycam> ... if you don't have fragmentation, you don't have the problem
<heycam> ... being able to add fragments even as a v2, with this defined shape, will resolve the issue that peter, me, etc. who care about fragmentation have
<heycam> florian: in the spec right now we say "it's the same as clientWidth" and defer to CSSOM
<heycam> stefan: Intersection Observer is similar, it defers to getBoundingClientRect
<heycam> florian: the "add that later" ...
<heycam> gregwhitworth: realistically nobody is asking me for it
<heycam> Rossen: it's fine to have a v2 with it there right now
<heycam> gregwhitworth: but in terms of blocking on implementations we can't do that
<heycam> florian: the thing I like better with the first alternative, the fragmentation case is rare, when things become weird -- we tell them "wait til v2"
<heycam> gregwhitworth: this is not a "we don't want this", but it's prioritization
<heycam> [discussion about Chrome implementation of fragments]
<heycam> [discussion of whether web devs will notice the inline/block size being duplicated in the single-fragment case]
<heycam> iank_: I was imagining that for a single fragment that fragments is null
<heycam> florian: different approach to the same question
<heycam> ... an author who hasn't thought about fragments
<heycam> ... the thing they haven't thought about happened to be placed in multicol
<heycam> ... does it break more to get the first fragment size, or the union size?
<heycam> gregwhitworth: I would say equally
<heycam> iank_: if we use the max of the inline sizes and the sum of the block sizes that would be closest to being useful
<heycam> florian: regardless, if they just look at one it'll be wrong
<heycam> heycam: just wonder if using names other than inlineSize/blockSize for the summed values would make it look less unsensible
<heycam> florian: so I'm ok with both options prefer the one without the summed inlineSize/blockSize
<heycam> alex: is there any perf difference between the two?
<heycam> iank_: very small. don't have to synthesize the array until you access it
<heycam> straw poll:
<heycam> 1. always return a contentBox which is an array of fragment sizes (usually containing one item currently)
<heycam> 2. return contentBox: { inlineSize: <offsetWidth>, blockSize: <offsetHeight>, fragments: [ { inlineSize, blockSize }, ... ] }
<astearns> main room is about to break
<astearns> (go on a break)
<florian> option 1
<smfr> 1
<Rossen> option 1, 2
<iank_> 2
<gregwhitworth> option 2
<heycam> option 1
<rego> option 1
<rachelandrew> option 1
<dbaron> prefer 2, I think, but not sure
<heycam> result: option 1
<heycam> RESOLVED: Option 1; contentBox is an array of fragment sizes
<heycam> trackbot: end telcon

[gregwhitworth]

To summarize the above - the decision was to go with Option 1 which is:

{
/* ... rest of object ... */
contentBox: [
        {
           inlineSize: <double>,
           blockSize: <double>
        },
        {
           inlineSize: <double>,
           blockSize: <double>
        }
]
}

In addition it was to recommended to add a visual example of a fragmented example and how to combine the fragments to be something useful (I'll open a new issue for this).

[dlibby-]

Addressed with #4529

[Loirooriol]

#4529 made it forwards compatible with fragment support, but didn't add fragment support, so reopening.