[resize-observer] device-pixel-border-box size

[gregwhitworth]

device-pixel-border-box size

device-pixel-border-box size is Element's border-box size in device pixels.
It is always an integer, as there are no fractional device pixels.
It can currently be approximated by Math.round(borderBoxSize * window.devicePixelRatio), but it cannot be computed exactly, because native code uses a different rounding algorithm.

Use case

This unusual size request comes from Chrome's WebGL canvas team. It solves the long standing WebGL developers problem: "How to create HiDPI canvas without moire pattern?".

The existing "best practice" for creating a HiDPI canvas is to set size of canvas context to a multiple of canvas's css width/height. Example:

let cssWidth = 300, cssHeight = 200;
let c = document.querySelector('canvas');
c.style.width = cssWidth + "px";
c.style.height = cssHeight + "px";
let ctx = c.getContext("webgl2");
ctx.width = cssWidth * window.devicePixelRatio;
ctx.height = cssHeight * window.devicePixelRatio;

The webgl context will be HiDPI, one one canvas pixel should correspond to one device pixel.
But, because ctx.width is pixel snapped, ctx.width can differ from "true" device pixel width. This difference can cause visible moire patterns when rendered.

Because of this, WebGL team believes that web platform needs to provide an API for true device pixel width.

Discussion

This size has several interesting differences from others reported by ResizeObserver:

• use case is narrow: only useful for canvas Elements.
• changing only Element's position, not size, can cause device-pixel-size to change.

Q: Does this size belong to ResizeObserver, or should we create a diferent DOM API for it?

I can't think of a clean API that would provide same functionality. Web developers must observe this size, and respond to its changes. ResizeObserver is the only size-observing API. Observing border-box size, and providing "devicePixelSize()" method will not work, because devicePixelSize could change without border-box changing.

Q: Should we observe device-pixel-size on all elements, or just canvas?

Observing device-pixel-size comes with performance cost, because size must be checked when Element's position changes. For all other sizes, we do not need to check size when position changes.
Weak preference: Only allow device-pixel-size observation for canvas.

Q: Should we report device-pixel-size on all elements, or just canvas?

Weak preference: make it canvas-only, because other elements cannot observe this size.

Originally posted by @atotic in #3326 (comment)

[gregwhitworth]

We discussed this today and we decided the following:

device-pixel-border-box size is just another word for border-box but for the canvas element when in a WebGL context for reporting device pixels since they can't be rounded.

In order to keep the API clean, we're recommending removing this as a value and just doing the right thing for canvas by reporting the device pixels alongside the CSS pixels under new properties.

[css-meeting-bot]

The CSS Working Group just discussed Device pixel border box removal from spec.

The full IRC log of that discussion

<emilio> topic: Device pixel border box removal from spec
<astearns> github: https://github.com//issues/3554
<emilio> github: https://github.com//issues/3554
<emilio> gregwhitworth: so, effectively what this comes down to is that this was a size you could watch, for canvas scenarios this'd be useful because CSS pixels get rounded in some way
<emilio> gregwhitworth: proposal is that for <canvas> only you'd provide the device pixel size of the same dimensions
<emilio> Rossen: that's super weird
<emilio> gregwhitworth: I agree, that's why aleks was going to propose this one
<emilio> iank_: I can do an argument
<emilio> gregwhitworth: the use case is there, the question is whether it makes sense to put the information there instead of in the canvas interface
<dbaron> +1 to thinking maybe it belongs on canvas, although interested in hearing the pro argument tomorrow
<emilio> gregwhitworth: we can wait for aleks
<emilio> everyone: Let's move this until tomorrow

[css-meeting-bot]

The CSS Working Group just discussed ResizeObserver device-pixel-border-box.

The full IRC log of that discussion

<gregwhitworth> Topic: ResizeObserver device-pixel-border-box
<gregwhitworth> Github: https://github.com//issues/3554
<gregwhitworth> atotic: I've been working on the spec
<gregwhitworth> atotic: the graphics team came to me - the problem they have is a way to determine the device pixel size of canvas
<gregwhitworth> atotic: why do they need to do this - the reason why they need to do this - is there is no way to do this
<gregwhitworth> atotic: the way the devs do hidpi in canvas it has CSS Pixels and what they do is they create the bitmap size they create the canvas in physical pixels it gets shrunk to CSS pixels so they can draw really fine hairlines
<gregwhitworth> atotic: device pixels are never rounded - there is no doubles or floating point, etc
<gregwhitworth> atotic: how we do this is by pixel snapping, there is no way for webdevs to determine this
<gregwhitworth> atotic: pixel snapping depends on not only pixels but also position
<gregwhitworth> atotic: I asked to add it to canvas, and they said sure but they also need to be notified when its resize
<gregwhitworth> atotic: currently the proposal will be that you will get device pixel inline and border size if its a canvas
<gregwhitworth> fantasai: are you proposing to add it somewhere else
<gregwhitworth> atotic: no
<gregwhitworth> fantasai: so the only way to get this is to add the resizeObserver
<gregwhitworth> atotic: yes
<gregwhitworth> dholbert: does devicePixelRatio?
<gregwhitworth> atotic: they get a bad pattern from hacking this because it can't do pixel snapping correctly
<gregwhitworth> dbaron: under what conditions does Chrome change the device pixel backing size when it's repositioned?
<gregwhitworth> dbaron: atotic is saying that, if you make a subpixel change in the top coordinate of a canvas, if it's 100.25 px tall
<gregwhitworth> dbaron: and it's top is positioned at a pixel aligned position it would round to 100, if it's further down it will be 101
<gregwhitworth> dbaron: do you actually change the backing store
<gregwhitworth> dbaron: the number of image pixels in the canvas
<gregwhitworth> dbaron: I guess thats the height and width
<gregwhitworth> dbaron: they want to change those the attrs based on the device pixel sizes
<gregwhitworth> atotic: the developer determines the size of the backing store, what Chrome does it will copy that bitmap to the CSS size of the canvas, if the ratio is a nice even number then we don't get moar patterns - if we don't
<bkardell_> how is that different from a url background image?
<gregwhitworth> atotic: there is a CSS size which can be fractional - but there is also the real bitmap to paint the canvas and that bitmap is in physical pixels
<bkardell_> " if the ratio is a nice even number then we don't get moar patterns - if we don't"
<bkardell_> that bit
<TabAtkins> s/moar/moire/g
<gregwhitworth> dbaron: bkardell_ is asking why don't you get moire pattern in background images
<gregwhitworth> atotic: it will scale, but background image is fundamentally different
<bkardell_> I can unmute and talk, I just dont want to interupt and don't knwo if this is enough to warrant a q
<gregwhitworth> atotic: we probably do get them but you usually aren't doing hidpi background image or you can't tell
<gregwhitworth> atotic: where they primarily have this issue is in the 3d context and they'll have patterns that look different
<AmeliaBR> q+
<gregwhitworth> bkardell_: the act of painting a thing on a canvas - the bitmap comes from an image and we rescale the image - it shoudln't matter who creates this image?
<gregwhitworth> atotic: you will see artifacts - but they'll create 3 different images
<dbaron> So I understand the use case -- but I'm a little suspicious of making it <canvas>-only, though.
<gregwhitworth> iank_: within painting, we know where the pixel snapping is so we can account for it
<gregwhitworth> atotic: also we're talking about every frame in video games, it happens a lot more
<gregwhitworth> dbaron: I understand the usecase I just don't know if I like that it's canvas only
<florian> q+
<gregwhitworth> atotic: I agree, but there is a performance penalty to calculate them and most people don't care about this except in a canvas context
<dbaron> I'd have expected it to be implemented a different way, but okay...
<gregwhitworth> atotic: if we find that it's useful in other areas it's easy to extend it
<gregwhitworth> AmeliaBR: if this is a canvas thing - why are we doing this here. Especially since this is a really hacky way to do it
<gregwhitworth> atotic: from what I understand from the history, there was a hidpi way in which to do this but the other standard didn't go anywhere
<gregwhitworth> hober: we also saw that people still create the larger one
<gregwhitworth> atotic: I agree RO is an odd place but the dimensions change if you move around you still need to be notified
<gregwhitworth> liam: checking the device pixel size, it seems easy to check every frame
<AmeliaBR> ack me
<gregwhitworth> liam: why would you not check it?
<gregwhitworth> atotic: if you're a game developer maybe
<heycam> q+
<gregwhitworth> frremy: If you want to draw a perfect line in your table you'll need to resnap it
<gregwhitworth> atotic: exactly, you need it or you add it to both specs
<heycam> ack florian
<emilio> q+
<gregwhitworth> florian: another thing - here we know what CSS & device pixel things and very easy to get confused
<gregwhitworth> florian: I don't know exactly what we want to add so let's not add it to too many places because it will get misused
<emilio> ack heycam
<gregwhitworth> atotic: it would only happen on canvas - you would see inlineDevicePixel and blockDevicePixel
<Rossen> gregwhitworth, recaps the resolution from yesterday
<gregwhitworth> heycam: before you were seeing a perf penalty - why not add a new box watching keyword?
<gregwhitworth> heycam: you can still have the device pixel border box of another element
<gregwhitworth> florian: but that was my point - if you make it broadly available they'll abuse it
<gregwhitworth> florian: they'll think they want device pixels but they'll probably be incorrect
<bkardell_> q+
<gregwhitworth> liam: to be clear, the only usecase this solves compared to RO plus having a method on canvas
<bkardell_> q-
<florian> q-
<dholbert> q+
<gregwhitworth> liam: this canvas snapping happens during painting or layout? If it happens at painting you wouldn't want to wait for painting to be done
<emilio> s/liam/emilio
<gregwhitworth> smfr: the only way to really know this you is to do it at painting you have to take transforms into account
<gregwhitworth> Rossen: if you have a scale it's broken
<florian> ScribeScribe: Florian
<gregwhitworth> gregwhitworth: I'm against adding this based on what smfr and liam said
<gregwhitworth> atotic: it will be blurry anyways so it doesn't matter
<emilio> s/liam/emilio again :D
<florian> gregwhitworth: in your use case, you may be right, but if we add this, there may be people who want to use it in the more general case that includes tranforms
<florian> gregwhitworth: so taking all web devs into account, this looks like a partial solution, not good enough
<gregwhitworth> iank_: I'll need to ask Chris, but we may compute the transforms
<gregwhitworth> atotic: resizeObserver doesn't watch transforms
<gregwhitworth> smfr: then this is the wrong API to bolt this onto
<gregwhitworth> atotic: ok, the hidpi with an ancestor that has a transform or something
<gregwhitworth> atotic: that should still work because it will then get transformed
<gregwhitworth> florian: then it's broken because the second you scale it's broken
<gregwhitworth> florian: people will assume they can and it will fail
<gregwhitworth> atotic: I thought one of the issues on the webplatform you don't want to expose zoom because it will break
<gregwhitworth> heycam: this is different
<heycam> s/this/pinch zooming and transforms on ancestors/
<gregwhitworth> atotic: we were talking about the pinch zoom of the page and they would end up with arbitrarily long page
<gregwhitworth> florian: pinch zoom, leaving it out - but transform?
<gregwhitworth> atotic: if we were going to do transforms, there will be a subset of these that should work
<gregwhitworth> atotic: we want this to be a size API not a quad API
<florian> gregwhitworth: I feel this is a chrome only perspective, safari and Firefox don't seem up to it. Should we go back to the github issue?
<gregwhitworth> atotic: RO itself does not work with transforms
<gregwhitworth> fremy: If you're going to draw a canvas you're not trying to do a transform on top of that
<gregwhitworth> fremy: transform is for animations
<gregwhitworth> fremy: why would you intend to do this if you're focused on pixel perfect drawing
<gregwhitworth> liam: I agree it doesn't make sense, it does make sense to center a canvas
<gregwhitworth> florian: offset path will also impact this as it can change the position
<gregwhitworth> atotic: we're going to notify you when the border box changes
<gregwhitworth> atotic: there's nothing that says we're not going to change on the transforms
<gregwhitworth> dholbert: how integral is the movement of the subpixel
<gregwhitworth> atotic: when you're moving it's blurry anyways, it's a video game
<gregwhitworth> dholbert: I'm feeling like a static API is more desirable than an RO API
<bkardell_> it is possible to make a subclass phase 2 of this that was specifically for observing that?
<bkardell_> .. or something
<gregwhitworth> dholbert: the canvas API makes sense to have this
<gregwhitworth> atotic: but people want to get an observation
<fremy> q+
<emilio> ack emilio
<dholbert> q-
<gregwhitworth> AmeliaBR: is it reasonable to trigger RO on something may have changed even if the sizes haven't changed - they then can do a canvas pixel
<gregwhitworth> atotic: I thought of that - but you end up getting a bunch of ROs without knowing why
<gregwhitworth> AmeliaBR: we have a way of adding device-pixel-border-box back?
<gregwhitworth> atotic: where most observers they look at the border box or content box, what is this box and creates a larger turbelance
<gregwhitworth> AmeliaBR: maybe I don't care about those and you're adding a perf hit
<gregwhitworth> heycam: you can rename the type of the box, canvas-bitmap-box if you're worried about
<gregwhitworth> atotic: it kind of locks us into watching the canvas only
<gregwhitworth> fantasai: I think agree with atotic named canvas - so we shouldn't lock that down
<fantasai> s/named/about not naming specific to/
<gregwhitworth> smfr: we simply snap device scale factor - if you're inside the scale transform we don't snap all the time, so we do need to go outside of layout a bit.
<fantasai> fantasai^: because there might be other elements in the future where we want to use it, and we should be able to just re-use the same thing for those cases
<gregwhitworth> atotic: it has a perf hit
<gregwhitworth> smfr: if you drag to another screen would it fire
<gregwhitworth> atotic: yes
<gregwhitworth> smfr: it's not real device pixels, because your not taking transforms into account
<gregwhitworth> chrishtr: its the dims of the texture of the backing store
<gregwhitworth> smfr: what you just said is very impl specific
<gregwhitworth> chrishtr: it's the CSS pixel size snapped and multiplied by the devicePixelRatio without taking into account transforms
<gregwhitworth> chrishtr: it has to take into account zoom
<gregwhitworth> smfr: ours doesn't
<gregwhitworth> chrishtr: the spec says to
<gregwhitworth> dholbert: doesn't it fire after layout?
<gregwhitworth> atotic: yes
<gregwhitworth> dholbert: so after layout
<gregwhitworth> dholbert: it would need to be extended to take into account after the fact
<gregwhitworth> liam: I don't think this a great fit
<gregwhitworth> iank_: it's where all other resizes are triggered, it's actually bad to try to keep it in sync with RO
<myles__> q+
<dbaron> So about 20 minutes ago there was a suggestion to take this back to the github issue... is there something we're actually going to resolve on here?
<gregwhitworth> fantasai: one thing to note if we have a RO if your 100 CSS pixels and I move it to the hidpi screen it's still 100 CSS pixels. And it's a canvas specific issue
<gregwhitworth> fantasai: it would be nice to not get the extra events unless I actually care about the device pixel border box
<gregwhitworth> atotic: yes
<gregwhitworth> fantasai: I think it should be the content box since that's what you're painting into
<dbaron> atotic: you could register to observer device-pixel-content-box
<gregwhitworth> atotic: that was my original proposal
<chrishtr> q+
<xfq> ack my
<gregwhitworth> myles__: so in conjuction with what fantasai was saying then you need to know the pos and size.
<gregwhitworth> fantasai: no only if those pixels change
<florian> gregwhitworth: that's already in the algo
<gregwhitworth> fantasai: you can resize in result of pos, but it may not
<Rossen> ack fremy
<gregwhitworth> fremy: I think it's going to be tricky in a sense, for the usecase your using - have you considered the paint() API?
<dbaron> (sounds like people were leaning towards registering for a separate box name rather than giving these notifications magically in some case)
<gregwhitworth> iank_: we don't have webgl and a bunch of other APIs
<gregwhitworth> fremy: oh ok
<fantasai> Proposal: Register for changes to devicePixelContentBox, get back sizes in device pixels, throws an error if registered on anything other than a canvas element, fires whenever device pixel size changes whether due to change in screen resolution, element position, or element resizing. Does not fire if device pixel size does not change.
<smfr> q+
<dbaron> still doesn't deal with the issues about rectilinear transforms, offset-path, etc.
<Rossen> ack chrishtr
<gregwhitworth> chrishtr: I just wanted to follow up on explicitly defining the API, in the CSS paint worklet callback you can see the device pixel backing size. It helps devs to dinstinguish between the two and can discover it. I think it would be useful to distinguish
<gregwhitworth> atotic: I think we have agreement to watch a seperate box?
<astearns> zakim, close queue
<Zakim> ok, astearns, the speaker queue is closed
<gregwhitworth> smfr: I just want to say that I don't think canvas is the only one, I think other situations may care about the same thing
<gregwhitworth> smfr: I think people will want it in other ways
<astearns> ack smfr
<smfr> s/in other ways/on other elements/
<gregwhitworth> smfr: I was proposing this is the snapped content box multiplied by device pixel ratio
<gregwhitworth> dbaron: but you all agree to ignore transforms?
<gregwhitworth> chrishtr: because it should be a paint based situation for perf
<gregwhitworth> dbaron: I don't think that's actually true
<gregwhitworth> chrishtr: unless overflow it's almost post paint
<gregwhitworth> krit: even SVG?
<AmeliaBR> I like the connection with Paint API. Even comes with a nice name, "paint size": https://drafts.css-houdini.org/css-paint-api/#paintsize
<gregwhitworth> Rossen: my proposed closing of the issue - let's bring the proposal back to github and then bring it back and we can resolve then
<gregwhitworth> atotic: thank you
<TabAtkins> Ooh, "paint size" sounds good.

[dbaron]

I think this is proposing to make observable some details of the Chromium transforms implementation and how it interacts with painting that aren't specified anywhere. I'm not sure we want to make those things observable -- but if we do, we definitely need a spec for them so that other browsers can match them.

[gregwhitworth]

To summarize the very large thread above, the request was placed on @atotic to create a new box with a better name and how the resulting API shape would change, if any.

In addition, there was discussion around when pixel snapping is done which seems to vary to some extent (pinch zoom for example) between engines and will need to be addressed (although not necessarily in the RO spec).

[kenrussell]

The problem of not being able to know exactly how many device pixels a given Canvas covers exists in all browsers, not just Chromium. All browsers' layout engines and compositors do some sort of rounding or snapping, and all of them do it subtly differently. For this reason it is not possible to create pixel-accurate canvas content. There are some longstanding bugs filed against WebGL on this topic:

KhronosGroup/WebGL#2460
KhronosGroup/WebGL#587

In the past the WebGL WG discussed adding properties to the canvas exposing this information, but doing so is sub-optimal. If the surrounding elements of the canvas changed, layout needs to occur in order to compute the values of these properties, so fetching them would induce a synchronous layout, and could induce infinite loops in applications. @grorg pointed this out years ago and there was no good solution.

ResizeObserver is a great innovation and a great place to put these additional observable properties. It allows the application to respond asynchronously, and 100% correctly, to resizing of the canvas.
@atotic prototyped these new observable properties in Chromium and the results were excellent - they finally allowed perfectly pixel-accurate canvas content to be drawn.

Could we please reconsider adding device-pixel-border-box as an observable property? If refinements are needed we're happy to collaborate on them. Thanks.

[gregwhitworth]

@kenrussell I don't believe that's what @dbaron was meaning. Yes pixel snapping occurs and this will vary by browser as it isn't specified anywhere, nor is subpixel rounding. What David was getting at is when pixel snapping is done and if it includes transforms or not, Chromium does not include transforms - thus if the element is transformed the dimensions snapped may be incorrect if it's a position+size altering transform as it will result in different rounding which isn't observed by ResizeObserver which primarily captures Layout adjustments.

That said, while device-pixel-border-box is not currently in the spec, this issue is to work out how to go about it and what the tradeoffs are.

So to put it another way, are you ok with that tradeoff, being able to get device pixels in the general case but some authors may not if they apply certain transforms?

[dbaron]

In particular, it sounds like, in Chromium, the pixels being snapped to are a function of what does or doesn't have a layer in the compositor... which means that this is exposing that, and thus likely causing some sites to depend, for correct behavior (not just for performance), on particular layerization. I think Gecko might pixel snap across rectilinear transforms (translate and scale only), whereas it sounds like Chromium doesn't.

[kenrussell]

Sorry for the delay replying.

It's fine with me personally - and I suspect for most web authors - if this API works most reliably and portably only if no size-altering or rotational transforms are applied to the canvas. That should be the case for applications that are really trying to render 1:1 to device pixels, since I doubt they will apply any transformations to the canvas.

I don't know Chromium's compositor and it would be good to get confirmation from someone on that team that this direction is OK with them. Not sure who on that team watches this repo but I see that @chrishtr can be CC'd; Chris, do you have any feedback or suggestions?

Perhaps we can further specify and standardize the behavior as we get experience with it, and as multiple browsers implement it? It would be great to start somewhere.

[chrishtr]

Re compositing: in Chromium, it is not the case that the pixel snapped size depends on compositing.
It depends only on the local layout size of the element.

Re pixel snapping across transforms: Chromium pixel-snaps across rectilinear translations, but
not scales. It also does not pixel-snap across CSS containment isolation.
(ref: FragmentPaintPropertyTreeBuilder::UpdateForPaintOffsetTranslation in the code)

The pixel-sizing of the canvas depends on pixel snapping, but is not scaled up to include
scales of ancestor transforms.

This means that the device-pixel-border-box will change if certain ancestor transforms change.
device-pixel-border-box also depends on paint offset generally.

I think we should just specify that if a ResizeObserver is configured to observe device-pixel-border-box, it needs to take into account all such sources of rounding and difference like I enumerated
above. This is more expensive, and so should be an opt-in only for developers who need it.

[kenrussell]

The WebGPU community group discussed this at length in a recent face-to-face meeting, with WebGL working group members also present. This ResizeObserver feature is strongly desired for both graphics APIs. People principally involved in the discussion were @atotic (Google), @grorg (Apple), @litherum (Apple), @jdashg (Mozilla) and @kenrussell (Google). Perhaps @Kangz (Google) can provide a link to meeting minutes.

High-level takeaway is that there is general consensus between Safari, Firefox and Chrome that:

1. device-pixel-border-box should not take pinch-zoom into account, because that would allow easy mistakes by web developers. (Envision huge canvases being allocated.)
2. device-pixel-border-box should take page zoom into account.
3. device-pixel-border-box should ignore transforms applied to the element. It can't handle rotations, scales, etc. and shouldn't try to. Developers should know what they're doing when using this box to render 1:1 to device pixels.

(Please correct me if I've misrepresented the conclusions of this discussion.)

Is this feedback sufficient to help move this proposal forward? @atotic pointed out during the meeting that ResizeObserver v0 is implemented in all of Safari, Firefox and Chrome now, and we would collectively love to try prototypes of device-pixel-border-box in all of these browsers.

[smfr]

WebKit's primary objection to this API is that in WebKit, device-pixel snapping is a paint-time operation and the snapped rectangle width is a function of the rectangle origin. So to provide the correct information to ResizeObserver (which fires before paint), we'd have to do a fake paint just to get the right device-pixel size, which we don't want to have to do.

[kdashg]

It should be sufficient for ResizeObserver to be eventually consistent, so a frame of latency would be acceptable, though it sounds like the spec doesn't presently allow this behavior.

FWIW, I'm playing with a pixel-pre-snapping resize: canvas.style.width = canvas.width / window.devicePixelRatio, which appears to work well on at least Windows. It does, of course, depend on an assumed pixel snapping behavior, which may be insufficient to work on all platforms. (But theoretically, the behavior seems sound for intuitive pixel snapping)

device-pixel-border-box does satisfy this use-case, but it seems incredibly specific. Perhaps it needs to be because of layout rules I'm not familiar with, but @dholbert indicated that this should work in at least some cases. I'm following up with our Painting/Compositing people.

[kdashg]

A related discussion on the WebGL repo:
KhronosGroup/WebGL#587

[chrishtr]

It should be sufficient for ResizeObserver to be eventually consistent, so a frame of latency would be acceptable, though it sounds like the spec doesn't presently allow this behavior.

Re "a frame of latency would be acceptable": are you saying you think that it's ok for the canvas to draw wrong for one frame and then fix itself? If so, I don't agree. Fixing this is the point of ResizeObserver after all..

FWIW, I'm playing with a pixel-pre-snapping resize: canvas.style.width = canvas.width / window.devicePixelRatio, which appears to work well on at least Windows. It does, of course, depend on an assumed pixel snapping behavior, which may be insufficient to work on all platforms. (But theoretically, the behavior seems sound for intuitive pixel snapping)

This does not work, because pixel snapping on browsers takes into account position as well as device pixel size. This is necessary in order to have consistent and high-quality rendering when a canvas is contained within layout boxes, and between layout phases of a box. Subpixel layout and pixel snapping are necessary and inherent parts of high-quality DOM rendering.

device-pixel-border-box does satisfy this use-case, but it seems incredibly specific. Perhaps it needs to be because of layout rules I'm not familiar with, but @dholbert indicated that this should work in at least some cases. I'm following up with our Painting/Compositing people.

I don't think this is incredibly specific at all. The reason canvas is special is that it's an immediate-model API that draws directly to a backing, as opposed to CSS+HTML rendering, which is mediated by the browser. Therefore the canvas needs to know exactly how big the backing is.

However, the canvas is not rendered in isolation, but rather is an immediate-mode rendering contained within a browser-mediated DOM that has subpixel rendering, pixel snapping, and render timing that is UA-controlled. For this reason, the developer needs a way for the canvas to participate in the rendering lifecycle, via an appropriate callback once the size of the canvas backing is known. The correct timing of this is once layout is complete and before/during paint.

This is very similar to the basic ResizeObserver use-case, except that the canvas needs to also observe the pixel snapping inputs to paint. Therefore, as Simon mentioned, "part of" paint needs to run, in order to compute paint offset, which is an input to the pixel snapping algorithm. I agree that this is more work, but it's necessary for high-quality rendering, and is not that hard to implement.

[Kangz]

For reference here are @kenrussell's notes of the discussions in the WebGPU CG.

[kdashg]

If part of paint needs to run, it's a tough pill to swallow to duplicate that work, which is why a frame of latency allowance helps.

Canvas cannot possibly stay 1-1 throughout a CSS Animation, so we know we're already working on an imperfect solution.

Ideally, a pre-snapped rect will be exactly N pixels tall regardless of offset, unless we round the edge y0 and y1, instead of y0 and height. Anything else will require spurious resizes, which not only do we not want in webgl, but we don't want in general web content painting. Is my intuition leading me astray here?

[chrishtr]

If part of paint needs to run, it's a tough pill to swallow to duplicate that work, which is why a frame of latency allowance helps.

Chromium's implementation does not duplicate any work. There is a pre-paint rendering lifecycle
phase that is after layout and before paint, and among other things computes paint offset.

Canvas cannot possibly stay 1-1 throughout a CSS Animation, so we know we're already working on an imperfect solution.

A CSS animation for anything other than transform or opacity induces layout as necessary, which runs the ResizeObserver on every frame. Transforms, as mentioned by Ken above, are not taken into account for the device-pixel-border-box, and so is not a problem. Transform animations without layout/ResizeObserver is one motivating reason for this rule, in addition to the ones Ken mentioned.

Ideally, a pre-snapped rect will be exactly N pixels tall regardless of offset, unless we round the edge y0 and y1, instead of y0 and height. Anything else will require spurious resizes, which not only do we not want in webgl, but we don't want in general web content painting. Is my intuition leading me astray here?

Suppose you have a canvas inside of a div of exactly the same CSS pixel size. The border of the div should match up exactly to the edge of the canvas. For this reason, the div and canvas have to have the same rounding algorithm.

As for why the rounding needs to depend on position: consider two sibling divs that have fractional widths and paint offsets and need to touch each other with no gap. In this case the rounding needs to be consistent, and the way to achieve that is to include subpixel accumulation (which in turn depends on fractional paint offset) in the rounding, so that they round the same direction. This is why the rounding depends on paint offset.

[kdashg]

Can we pre-snap the position and the size?
Alternatively, what about getDeviceRects?

Relying on observers for this seems likely to cause unexpected observer callback ordering issues.

[chrishtr]

Can we pre-snap the position and the size?

How would this work?

Alternatively, what about getDeviceRects?

Do you mean a new method exposed to script that returns these rects? The problem is that the canvas doesn't know when to ask, since it doesn't known when it might change due to layout invalidations.

[kenrussell]

@jdashg can you reply to @chrishtr 's questions?

Being able to observe this device-pixel-border-box size would categorically solve a longstanding rendering quality problem on the web platform. Since ResizeObserver is an asynchronous API, it solves problems where providing these attributes on the canvas element forces a synchronous layout.

In Safari, would it eventually be possible to implement this?

Is there any way we can reach consensus that this is a needed feature, and that ResizeObserver is a good place to expose it to applications?

[css-meeting-bot]

The CSS Working Group just discussed device-pixel-border-box size.

The full IRC log of that discussion

<dael> Topic: device-pixel-border-box size
<dael> github: https://github.com//issues/3554#issuecomment-513045449
<dael> chrishtr: I'm the one that added it to the agenda
<dael> chrishtr: Main thing is resolving that this is a good approach to resolving responcive design for canvases.
<dael> chrishtr: Exact details of API can be followed up if agree on first step
<dael> jeff: The main thing on figuring out how this works is when we're thinking about scheduling we get a request at beginning of frameso can render webgl
<dael> jeff: Then main thing is take stock of width and height and that's usually when resize happens. I understand resize is only latter after raff when browser thinks we can take current layout
<dael> jeff: Then resize observer kicks off events and says to canvas you're resized. Issues for WebGL is that time is non-trivial. Late in the frame is hard to respond. You can realize in time for end of frame you rendered wrong and can do minor fix but can't render at new resize if observer after raff
<dael> chrishtr: Design of resize observer is to support this responsive design wehre decentralized action causes layout to change. if resize observer callback thinks sized changed non-canvas can change internal layout. If there's change it causes another layout. non-canvas the frame is twice as long and I think that's unavoidable.
<dael> chrishtr: Key is resizes are uncommon and it's only when user changes browser or orientation. COuld be doing layout animation but one that changes layout in every frame is perf problem
<dael> chrishtr: For webGL canvas it is an unavoidable slowdown
<dael> gregwhitworth: Is your issue that you may potentially miss a frame due to ascyn nature?
<dael> jeff: yeah
<dael> gregwhitworth: In current state you're sync. Feels you prefer have a performance thing where effecting frame vs the inverse
<dael> jeff: Not clear. If you rely on resize to tell you size and can't correct ahead of time it's hard to from the get go render frame with proper size
<dael> dbaron: Almost think for this use case resize-observer feels like wrong thing due to structure. Request animation frame was designed as a callback where can write layout information.
<dael> dbaron: One of the problems here is that raff, you get a raff [echo issues]
<astearns> s/raff/RAF/
<dael> dbaron: RAF is early in cycle and you can do it before layout. Layout as result of resize is after raf. Some things people do will flush layoutanyway and may layout twice. What you want here is to know canvas size and then do WebGL stuff
<dael> dbaron: If you want to know size of canvas can do resize-observer but it's designed to not flush layout and you might not need changes. In your use case you want to know this is the new size of box after alyout this cycle. Way you do that is use a sync method that flushes layout. Assuming 1 canvas
<dael> dbaron: What works for this use case is late in the RAF you get width and height of canvas, do WebGL work, don't use resize-observer. You did minium layout and get new canvas size
<dael> jeff: THat's what I've been centering on the solution for the common case. Where it gets muddy is part of the goal is device-pixel-border-box size. This is predominantely useful for canvas and WebGL.
<dael> jeff: A bing chunk of WebGL cases will have this workflow and not want resize-observer to get late resizes. Want the early layout to get box sizes.
<dael> jeff: What we'd be lacking is a way to sync query the device-pixel-border-box
<dael> jeff: Right now can get bounding client rects but it's not device pixel. That's what we want for WebGL.
<dael> jeff: Sounds like what we want is a a getBoundingDeviceRects or something sync to get pixel size. THen resize-observer is helpful for the other use cases. WebGL where they flush layout uses this othercommand
<bradk> I have to leave early. Bye!
<dael> chrishtr: Two things wrong about desc dbaron . Resize-observer is not async. Delivered immediately and cause another layout on same frame. Second is you can't actually know size of WebGL canvas even if you calls a sync method because there can be a different callback regisered by someone else that dirties layout.
<dael> chrishtr: Cna't do it the way you desc
<dael> dbaron: Those are fixable.
<dael> dbaron: Problem with resizeobserver is jeff's use case is a notification every time, not jsut resize.
<dael> dbaron: There's tricks to put yourself after next RAF. There's always problem of multi-observers.
<emilio> q+
<dael> dbaron: I think fundamentlly people want to do different things. THis prop makes a set of data only available in resize-observer but not in other ways and I don't think we want to put to use only resize-observer and other solutions
<dael> chrishtr: Good to only provide right information in right spot
<dael> dbaron: THink it'swrong spot
<dael> ken: resize-observer solves a lot of problems. Fantastic all browsers are impl v0 so that we can encourage everyone to use that. We'd like to recommend just use device-pixel-border-box if they want it exactly correct. sync apis have a host of problems like going to while loops to make layout settle down. resize-observer callback solves problems
<dael> jeff: It does, but it has remaining problems. If WebGL content wants to render...there's no way to deterministically always try and render correct frame size and not occationally double-paint
<dael> ken: I see that. THe live resizeing case is vanishingly small. You're moving a handle on the webcase. In the steady state you'll render nromally. If you end up double painting during live resize not end of world
<dael> Rossen_: A little under statement. Let's not diminish value of resizing and only put it ina corner. I can see multi frame being in same overall app layout each with own observer and then the drawback will be magnified. Let's not diminish effect of this
<bkardell_> if it was really only based on the size of the window, this feature wouldn't be necessary
<astearns> ack emilio
<dael> emilio: DId we figure out...what this device wants to add is a snapped rect. We figured for some combo of transforms that can create non-rect we also need to do transform calcs to get right device snap rect
<Rossen_> q?
<dael> emilio: It stashes a bunch of paint information onto an API. Want a reminder of what the web reprecussions of that discussion were
<smfr> q+
<dael> gregwhitworth: Talked about that asSanFran F2F. Most people thought okay beca'sescope was to canvas. Use case for canvas we won't have transforms and those types of scales so we don't need to go that far down stack. I think chrishtr said they are doing that. You are stashing, but primary case is you want pixel snapping because we're WebGL and want draw to pixel snapping and want ot know about resize to get perfect frame
<dael> gregwhitworth: Not opposed to dbaron where we can't do this anywhere else. But in this specific case where the resize gets a less clean seam and it's only on resize
<dael> gregwhitworth: To your point you have to shove painting knowledge into it. And you don't have all information.
<dael> chrishtr: I think transforms shouldn't apply at all
<dael> chrishtr: Exact def. or impl of pixel thing is different then other discussion. even if want to do something to instruct dev to always resize at beginning of frame you can't get bounding rect and multiple by pixel. Needs to be fixed or canvas can't render
<dael> jeff: Can do work to estimate eventual pixel boundries. I'm told people who know pixel snapping rules find it sketchy because rules aren't defined and we'd have to guess. Need a library that sniffs for UAs and gets right rules.
<gregwhitworth> they're not defined and vary by UA
<astearns> ack smfr
<dael> chrishtr: That's something where it's not going to be a great solution. Dev will hack around and that's not good
<dael> smfr: Remind people not all AU have pixel snapping before paint. We don't know before we go to paint. Even if added the device pixel we'd have to do a fake paint. THat's non-trivial for WK. And makes forced layout be even more expensive and they're already a source of perf issues. Hesitant to add an API that makes fake paints
<dael> jeff: Maybe alternative for WebGL usecase is if there were a callback following RAF like resize-observer but happens as early in the frame as possible and deterministicly every frame might be more workable if people object to a sync call for pixel layout
<dael> jeff: Solves double paints, gives webGL a way to know this is final size as long as nothing else changes it. As long a snothing else changes it is a problem we can't ever get around. might be the way forward there.
<smfr> https://html.spec.whatwg.org/multipage/webappapis.html#update-the-rendering
<dael> smfr: HTML 5 has rendering steps. Where in those would callback fire?
<dael> jeff: I don't know
<dael> dbaron: Not sure I believe rendering steps in HTML5 make sense. I have an open issue to fix some of it.
<dael> dbaron: I support basic idea from Jeff. I think there's a similar Google proposal. Requires people have separate callbacks to separate writing layout and reading layout. In a refresh you want all writing before all reading
<dael> smfr: GOod idea except APIs we have make it easy to do wrong things. That only works is everyone does reading and writing at same time. Need a whole new set ofDOM APIs that read without forcing layout
<dael> dbaron: resize-observer feels designed for this, but it's a very particular solution
<dael> chrishtr: It's providing a responsive decentralized layout at one time.
<dael> chrishtr: What you desc about post layout callback if you have arbitrary code running it dirties layout
<dael> jeff: Not solvable
<dael> chrishtr: Resize-observer solves it. If you dirty layout should you layout again? If yes it's same as resize-observer.
<dael> jeff: Keep in mind my objections to have resize-observer isn't determatively fired. If we're trying to solve WebGL problem here, the solution shouldn't be half baked. There are problems with as exists resize-observer.
<dael> jeff: Main problem is we can't hit our frametimes if we get resize-observer late. Solutions are sync call or get device-pixel size and other is a resize call that always happens early in theframe. THat solves WebGL problem.
<dael> jeff: Getting confident WebGL problem isn't solved by current PR
<drousso> gtg :(
<dael> ken: USing resize-observer is simplier to explain. It gets 80-90% to solution. I realize poss of duplicate renders but that resize-observer takes into account relayouts that may happen is I think good design. I would advocate adding device-pixel-border-box to resize observer API as a pretty good solution. We can iterate on it in the future to maybe get alyout earlier and less change of dup. frames
<dael> jeff: I have an 80% solution in hand already
<dael> ken: But it doesn'timpl the real pixel snapping the browsers do. Need access at app level
<dael> jeff: But my JS hack sounds like a better solution then the API
<dael> chrishtr: I don't htink that's true. resize-observer 100% solves get size of canavs
<dael> jeff: Do you think my rec. to have deterministic frame that says we finished layout is unworkable.
<dael> gregwhitworth: That's what it does.
<dael> jefBut we can't hit frame times because takes 12ms
<bkardell_> you always get 1 yes
<dael> dbaron: jeff wants a notification if it does or doesn't change size. For resize-observer when you register a new one do you always get one notifiation?
<dael> ??: Yeah
<dael> dbaron: What happens if you re-register resize observer every frame? Does that solve?
<dael> jeff: Depends how early it fires. Is it designed to fire as soon as possible?
<dael> gregwhitworth: Following layout for as accurate as poss. TO get pixel snapping need to get into paint. I don't understand as early as possible, has to be late to understand layout
<dael> dbaron: Wants it before he does WebGL painting to canvas
<tantek> sounds like maybe you want to both register a RO and a RAF callback, RO for the first notification, and RAF for no-resize notifications, and RO when there is an actual resize
<dael> gregwhitworth: Whichi s fine if not respond to resize. THat's what RAF does now
<dael> dbaron: What Jeff wants si to do layout, know what size results, and then paint to canvas
<dael> gregwhitworth: It's a callback to say hey we changed
<dael> dbaron: He want sto paint when no resize too
<tantek> painting when no resize is what RAF is for right?
<dael> chrishtr: yOu're desc a post-layout animation callback which doesn't exist. One objection is it gets in theway of running layout off main thread. resize prob has same problem
<gregwhitworth> tantek: that's my confusion
<gregwhitworth> if you don't want resize calls you add it to RAF
<gregwhitworth> and what chrishtr just said if you want both
<dael> chrishtr: Can get same thing except potentially slower during resize with resize-observer but need request animation callback as well.
<tantek> hah literally what I just said in IRC
<gregwhitworth> chrishtr: added it to TPAC
<dael> astearns: Nearly at time. I don't think we will resolve today, but good to have back and forth
<tantek> jeff, does that solve your use-case? using *both* RO and RAF?
<gregwhitworth> sorry, meant that to be a statement - not to you
<dael> astearns: Interesting to have fuller proposal for the post layout callback. I htink should continue in GH and come back.
<dael> many people: thanks

[dbaron]

One note is that I think (although being sure requires executing the entire ResizeObserver spec in my head) calling ResizeObserver.observe() on the canvas element in question on every requestAnimationFrame should lead to a reliable ResizeObserver notification every cycle, whether the canvas has resized or not. That might be a path forward for @jdashg's use case.

[chrishtr]

One note is that I think (although being sure requires executing the entire ResizeObserver spec in my head) calling ResizeObserver.observe() on the canvas element in question on every requestAnimationFrame should lead to a reliable ResizeObserver notification every cycle, whether the canvas has resized or not. That might be a path forward for @jdashg's use case.

Just to repeat what I said in the call also: that would indeed work and be fine, as long as you could observe the device-pixel border box of the canvas. Implementation for the developer would be:

function observerCallback(entry, observer) {
  let devicePixelBorderBoxRect = entry[0].devicePixelBorderBox;
  render(devicePixelBorderBoxRect); // WebGL rendering
  observer.unobserve(myCanvas);
  observer.observe(myCanvas, {box: 'device-pixel-border-box'});
}

var observer = new ResizeObserver(observerCallback);
observer.observe(myCanvas, {box: 'device-pixel-border-box'});

@jdashg would this meet your needs?

[atotic]

@christr: I think you meant

function observerCallback(entry, observer) {
  let devicePixelBorderBoxRect = entry[0].devicePixelBorderBox;
  render(devicePixelBorderBoxRect); // WebGL rendering
}
function rafCallback() {
  observer.unobserve(myCanvas);
  observer.observe(myCanvas, {box: 'device-pixel-border-box'});
}
var observer = new ResizeObserver(observerCallback);
document.requestAnimationFrame(rafCallback);

@dbaron that is very clever!

[chrishtr]

Yes that would work (@chrishtr btw :))

One typo about requestAnimationFrame though, new code below:

function observerCallback(entry, observer) {
  let devicePixelBorderBoxRect = entry[0].devicePixelBorderBox;
  render(devicePixelBorderBoxRect); // WebGL rendering
}
function rafCallback() {
  observer.unobserve(myCanvas);
  observer.observe(myCanvas, {box: 'device-pixel-border-box'});
  requestAnimationFrame(rafCallback);
}
var observer = new ResizeObserver(observerCallback);
requestAnimationFrame(rafCallback);

[juj]

function rafCallback() {
observer.unobserve(myCanvas);
observer.observe(myCanvas, {box: 'device-pixel-border-box'});
requestAnimationFrame(rafCallback);
}

This feels like an abuse of the observer model, and I think also causes running garbage collection pressure from {box: 'device-pixel-border-box'} each frame. (Wasm applications strive to run garbage free to avoid JS performance issues). :/

How do browsers like if rendering would always take place outside the rAF function?

Does the following achieve the same?

function observeCanvasSizeChange(canvas) {
  function observerCallback(entry, observer) {
    let devicePixelBorderBoxRect = entry[0].devicePixelBorderBox;
    canvas.deviceWidth = devicePixelBorderBoxRect.width;
    canvas.deviceHeight = devicePixelBorderBoxRect.height;
  }
  var observer = new ResizeObserver(observerCallback);
  observer.observe(canvas, {box: 'device-pixel-border-box'});
}

function rafCallback() {
  render(myCanvas.deviceWidth, myCanvas.deviceHeight);
  requestAnimationFrame(rafCallback);
}
observeCanvasSizeChange(myCanvas);
requestAnimationFrame(rafCallback);

or does that also have the problem that rendering may occur to wrong(unsynchronized) size e.g. under continuous DOM size animation?

[chrishtr]

This feels like an abuse of the observer model,

Maybe, maybe not. The example script I gave, which fleshed out the idea @dbaron suggested, shows that there is a way to avoid the double-WebGL-render problem @jdashg raised in situations where a resize occurs. And if it turns out to be a very useful mode, we could formalize that in a new API.

Regarding GC: if it turns out to be a real problem for a WASM app of the future, it can definitely be solved pretty easily with an API tweak.

How do browsers like if rendering would always take place outside the rAF function?

There is nothing at all that forces developers to render within a rAF. In fact, many popular frameworks, such as React, don't actually render during rAF, in part to have more time to render than rAF allows, as rAF often tries to fire as late as it can to minimize latency to the screen.

As the Chromium rendering lead, I would say my view is that "do all rendering within rAF" is not a feasible solution for many cases, especially in complex apps, due to the issue of wasting CPU time being idle. (*)

or does that also have the problem that rendering may occur to wrong(unsynchronized) size e.g. under continuous DOM size animation?

It has several problems; one of them is the wrong-size issue you mentioned. There is also the issue of needing to poll (or constantly rAF, which amounts to the same thing) just in case sizes change, and also that extra forced layouts can occur because deviceWidth would force layout, and other rAF callbacks may dirty layout also. These are mentioned in more detail in comments above in this issue.

(*) Aside that may be useful:

To address the framework use-case mentioned above, postAnimationCallback is a new callback being proposed that is intended to be run after rendering is complete, so that reading back layout is more likely to be free, and to support the use cases of starting rendering of the next frame as soon as possible.

However, this callback happens "post-commit", which means that the rendering display list has already been sent to the browser for display to the screen. Doing it post-commit is important to avoid postAnimationCallback ending up being the same as ResizeObserver, and thereby accidentally delaying frames.

[greggman]

Sorry to butt in here but I just wanted to point out the HTML snipped above is probably not the best practice for a full window canvas.

If you want a canvas to fill its container, in this case the <body> then you should set the canvas to display: block. Canvas defaults to display: inline which ends up adding extra space at the end as is the reason people often get a scrollbar and then they end up hacking around their mistake by adding overflow hidden etc...

As well if you're using 100% and you don't set the size of the body and html you won't get 100% height. Example

These are 2 recommended ways to get a full page canvas

1. use vw an vh

<html>
<style>
body { margin: 0; }
#canvas { width: 100vw; height: 100vh; display: block; }
</style>
<body>
  <canvas id="canvas"></canvas>
</body>
</html>

Example

This will give you a fullscreen canvas with no scrollbar on all desktop browsers.

The problem comes in on mobile where you have to decide what you want full page to mean. Both Chrome and Safarai (and Firefox?) decided that to deal with browsers on mobile hiding and showing the address bar and/or other UI then 100vh = the size when the UI is at its smallest and 100% = the size actually displayed (smaller when UI is visible, larger when not)

To get that to work cross browsers you have to do this

<html>
<style>
html, body { height: 100%; }
body { margin: 0; }
#canvas { width: 100%; height: 100%; display: block; }
</style>
<body>
  <canvas id="canvas"></canvas>
</body>
</html>

Example

Which is more correct depends on the use case. if the canvas is some background element over which content scrolls then you probably want 100vh. If the canvas is just supposed to fill the screen then you probably want 100%

[css-meeting-bot]

The CSS Working Group just discussed ResizeObserver Device Pixel Border Box.

The full IRC log of that discussion

<fantasai> Topic: ResizeObserver Device Pixel Border Box
<fantasai> ScribeNick: fantasai
<fantasai> chrishtr: device-pixel-border-box is the actual device pixel bounds of a canvas element
<fantasai> chrishtr: including pixel-snapping feature which is ued by browsers to avoid blurriness by snapping to pixel grid
<fantasai> chrishtr: fundamental problem with no complete solution
<fantasai> chrishtr: authors that use canvas
<fantasai> chrishtr: have no reliable method to determine on-screen pixel size of the canvase
<fantasai> chrishtr: if off by one pixel due to pixel-snapping, rounding, or other issue
<fantasai> chrishtr: will end up with wrong or blurry results
<fantasai> chrishtr: pixel-snapping is intentionally not specced to allow UAs to do their best
<fantasai> chrishtr: acocunt for varying rendering architecture
<fantasai> chrishtr: and evolution
<fantasai> chrishtr: so no way to reliably find this size
<fantasai> chrishtr: proposal is to add a box that observes device pixel border box
<fantasai> chrishtr: to ResizeObserver
<fantasai> chrishtr: which will notify if that box changes
<fantasai> chrishtr: will happen at similar timing as other ResizeObserver
<astearns> github: https://github.com//issues/3554
<fantasai> chrishtr: there were two main objections to adding this
<fantasai> chrishtr: one was raised by Jeff Gilbert
<fantasai> chrishtr: had a long discussion with him and Kem Russel, our WebGL expert (?)
<fantasai> chrishtr: have a compromise proposal that both of us agree to
<heycam> s/Kem Russel/Ken Russell/
<fantasai> chrishtr: objection he raised was that if you have a WebGL-centric application
<fantasai> chrishtr: e.g. full-screen game that uses WebAssembly and only DOM in minimal way
<fantasai> chrishtr: want to have a continuous requestAnimationFrame loop
<fantasai> chrishtr: drawing the canvas
<fantasai> chrishtr: in that model where you are canvas-centric
<fantasai> chrishtr: it still lives in a DOM shell or browser window that has a size
<fantasai> chrishtr: need to observe that size
<fantasai> chrishtr: where pixel snapping will work
<fantasai> chrishtr: but ... not in JS
<heycam> s/.../in Web Assembly/
<fantasai> chrishtr: most convenient to query device border box directly from canvas during rAF loop
<fantasai> chrishtr: rather than by observer
<fantasai> chrishtr: if layout was dirty at the time during making the query, it will force layout and other things in order to compute pixel snapping
<fantasai> chrishtr: in taht use case maybe it's OK
<fantasai> chrishtr: other use case, which I was most focused on
<fantasai> chrishtr: you have canvas element embedded in a DOM-centric website
<fantasai> chrishtr: maybe photo-app or ad or widget,
<fantasai> chrishtr: potential multi-actor scenario
<fantasai> chrishtr: want to avoid layout thrashing
<fantasai> chrishtr: cases for which ResizeObserver was designed
<fantasai> chrishtr: this makes most sense
<fantasai> chrishtr: compromise proposal is that we just have both APIs
<fantasai> chrishtr: The End! :D
<fantasai> jeff: ....
<fantasai> jgilbert: ... moving forward with both propsoals works out
<fantasai> jgilbert: having it be in ResizeObserver also makes sense
<fantasai> jgilbert: like you went over, there was some concerns about it covering espeically more easily some more trivial cases
<fantasai> jgilbert: having both lets people pick and choose appropriate API
<fantasai> jgilbert: default should be to use ResizeObserver, most reliable
<fantasai> jgilbert: kindof like getClientBOundingRect() , need to be careful if calling that often
<fantasai> jgilbert: maybe warn in UA
<fantasai> jgilbert: if called too often
<fantasai> jgilbert: but getDeviceClientRect() API alone ...
<fantasai> chrishtr: 2nd objection smfr raised
<fantasai> chrishtr: pixel-snapping requires more information than just layout in today's engines
<fantasai> chrishtr: in Chrome requires pre-paint step
<fantasai> chrishtr: in Safari requires paint
<fantasai> chrishtr: don't have solution to that problem
<fantasai> chrishtr: just note readback method for rAF
<fantasai> chrishtr: canvas.getThing
<fantasai> chrishtr: also has to run the same steps
<fantasai> Rossen_: but that's not blocking to accept the proposla
<fantasai> smfr: in some ways make sit worse
<fantasai> smfr: because 2 places we have to do this painting calculation
<fantasai> Rossen_: was asking if this second objection is blocking accepting the proposal
<fantasai> smfr: so to clarify the proposal is having API to return the box on canvas and also in ResizeObserver
<fantasai> smfr: I'm not a fan of doing partial paitn to get this info
<fantasai> smfr: I would expect for games , especially in full-screen, to position on ? boundaries
<fantasai> smfr: surprised if using fractional pixel positioning
<fantasai> jgilbert: for full-screen, unlikely, but for partial screen almost always get this
<fantasai> jgilbert: both on Windows and on Android, I believe, Windows will happily give you 1.6574 device pixel ratio
<fantasai> jgilbert: you just have to deal with it
<fantasai> jgilbert: you end up trying to reverse-engineer what pixel-snapping is to figure that out
<fantasai> jgilbert: pre-snap a rect to a non-integre CSS size , if that works out to integer pixels... then no artifacts? but it's a huge mess
<fantasai> smfr: other case that's confusing is on our iPhone's that have retina displays
<fantasai> smfr: already 2.7? scaling factors
<fantasai> smfr: so can cause hairlines to disappear aetc.
<fantasai> smfr: seems like also on windows, too
<fantasai> smfr: getting pixel perfection, seems like OS is doing scaling behind your back, how can you expect it to work?
<fantasai> jgilbert: out of the box cases where if you play around with virtual scaling
<fantasai> jgilbert: on a Mac and play with scaling on a retina screen
<fantasai> jgilbert: no way to get 1-1 device pixel
<fantasai> myles: even worse, the default ratio is not 1-1 or 2-1
<fantasai> jgilbert: it really depends on how the OS is doing this sort of scaling
<fantasai> jgilbert: I'll add that what you end up with, for instance in Mac, when you have this OS-level virtual resolution scaling
<fantasai> jgilbert: can't get one to one
<fantasai> jgilbert: if can't get 1-1 in application, can still get moire effects
<fantasai> jgilbert: can't entirely eliminate scaling artifacts, but can do better
<fantasai> jgilbert: than naively try to grid on the screen and hope it looks good
<fantasai> mattwoodrow: difference is on Windows the scaling isn't hidden
<fantasai> mattwoodrow: can attempt to match
<fantasai> jgilbert: windows 10 implements scaling that llows 1-1 rendering, not virtual scaling as default
<fantasai> jgilbert: Mac doesn't expose effective scaling, just says 2x all the time
<fantasai> jgilbert: Android says 3x all the time
<Rossen_> q?
<fantasai> ??: regarldess of what operating systems do, if application renders pixel -perfect
<fantasai> ??: Result will be better than if it wasn't
<fantasai> ??: I think we agree on that
<fantasai> myles: if goal is to get hairlines, even if you get a little closer to hairline, can still disappear
<fantasai> ??: Display scaling might give you smoother color if checkerboard is pixel-perfect before the scaling
<fantasai> ??: and if you have checkerboard before scaling, then less smooth design
<fantasai> s/??/mstage/
<jgilbert> s/mstage/mstange/
<fantasai> mstange: still discussing value of getting an accurate box? what's status?
<fantasai> jgilbert: if we decide that we don't want to give ppl this box, then ?
<fantasai> jgilbert: do we want to get into that? or do we want to take it as a given that we're trying to give ppl most correct idea of device pixel than we can?
<fantasai> mstange: what we would need to do in Firefox to get this result
<fantasai> mstange: Firefox takes into account the full zoom, and takes into account CSS Transforms
<fantasai> mstange: space in which we snap is established by the closest non-animated transform or the root
<fantasai> mstange: we only know this info during painting
<fantasai> mstange: so would need to run more steps before giving info
<fantasai> smfr: this device border box would not be affected by transforms
<fantasai> smfr: so would have to do *special* calculation
<fantasai> jgilbert: 3D transforms no idea what to fee dback
<fantasai> jgilbert: pixel-snapped bounding box?
<fantasai> atotic: if you're doing transforms won't be pixel-perfect anyway, so don't worry about it
<fantasai> atotic: your'e also talking about implementation that you need to get this nformation during pre-painting
<fantasai> atotic: remember you had this information ??
<fantasai> atotic: Might be ok to deliver incorrect information and not have to paint
<fantasai> myles: there's a chance of you never get correct info
<mstange> s/closest non-animated transform/closest animated ancestor transform/
<fantasai> atotic: I think you want, once things have settled down want accurate box size
<fantasai> atotic: if animating, don't care so much
<fantasai> jgilbert: deliver informmation lately
<fantasai> jgilbert: would be nice if we could try so that you could do 1-1 perfect every time within certain set of constraints
<fantasai> jgilbert: be nice if we didn't immediately settle on a 1 frame late policy
<fantasai> jgilbert: matching what native APIs are able to do
<fantasai> jgilbert: don't want to suffer if dont' have to suffer on native
<fantasai> myles: if it is one frame late then the exact timeline that we drop frames is exposed to the Web so don't think we want
<fantasai> myles: similarly, worried that we'd have to reverse-engineer chrome's pixel-snapping strategy
<fantasai> jgilbert: shouldn't have to do more normalziation than you do today, I think
<fantasai> jgilbert: if trying to use this to get anti-moire, in order to do this today have to ??
<fantasai> smfr: one frame late version also problem of which rectangle to trust
<fantasai> chrishtr: not late if layout occured
<fantasai> chrishtr: only late if you have a threaded animation
<fantasai> smfr: thought it would be late because we woudl collect info at paint time
<fantasai> chrishtr: I don't think we shoudl do that, disagree with atotic
<fantasai> chrishtr: think we can do it therefore we should
<fantasai> chrishtr: in cases where layout has occured
<fantasai> chrishtr: agree with point about threaded animations and not syncing with JS thread
<fantasai> smfr: want to tie together two of previous points, first that if we implement this paint day device pxel border box, will be more expensive for us
<fantasai> smfr: secondly because of physical mismatch, wouldn't have some user benefit
<fantasai> smfr: display is scaling anyway
<fantasai> smfr: can we get examples?
<fantasai> smfr: want to try on Apple devices, see what would happen if looks right
<fantasai> atotic: moire patter
<fantasai> atotic: if you're moving canvas around the screen, the moire is animated. looks really bad
<fantasai> smfr: please give us some tests
<fantasai> chrishtr: I think time is up?
<fantasai> chrishtr: propose to add the feature?
<fantasai> Rossen_: does the group feel there's enough merit to add this?
<fantasai> Rossen_: would be adding both APIs
<fantasai> Rossen_: is there any objection?
<fantasai> smfr: I dont' think we can accept new API without evidence that it's so much better that it's worth the extra cost
<fantasai> fremy: you can agree with the API and just return some approximate result if you feel it's good enough
<fantasai> myles: your'e saying implement the API without implementing teh API
<fantasai> fremy: might not be useful on your device
<fantasai> fremy: but might be useful on Android
<fantasai> fremy: so just return the result
<fantasai> AmeliaBR: Not having an API is better than having API with bad results
<fantasai> AmeliaBR: if a particular browser has particular concerns about implementing the particular API
<jgilbert> fantasai: in this case it would be, if you as an author were trying to do this thing, and you had this browser gives me the actual DPR, and this one gives me some approximation of the size
<jgilbert> ... I still need to size my box either way
<jgilbert> ... if my choice is I can get teh actual device pixel size from Chrome, but calcaulate it myself from widht/height props in WebKit, and get an approx result, then I don't know it might be better to have an API that does that calculation for me
<jgilbert> ... then I only need to write one code flow
<fantasai> AmeliaBR: might make sense, but have existing cases where you can't trust the API
<fantasai> florian: In general I agree with your statement, this doesn't seem to be such a case
<fantasai> Rossen_: I woudld like to end this topic
<fantasai> Rossen_: going to call for objections to adding the API, if we have objections we'll deal with them and have additional conversatiosn with the TAG and whatnot
<fantasai> Rossen_: any objections to having these APIs?
<fantasai> smfr: Can we wait until we ahve the examples?
<fantasai> astearns: I might object because seems like we need more data
<fantasai> astearns: so I will object on Safari's behalf
<fantasai> Rossen_: so back to you to get test cases, we'll discuss again on telecon

[kdashg]

My manual testing demo page: https://jdashg.github.io/misc/webgl/device-pixel-tester.html

[chrishtr]

Here is a demo page that clearly shows the issue:

https://output.jsbin.com/gemipizazi/1

If you load this in Chrome Canary with the --enable-use-zoom-for-dsf command-line flag (*), and resize the browser window vertically, there will be a number of visual artifacts in the rendering, that appear or disappear according to the different pixel snapping that happens with different viewport heights.

If you load the same demo in Safari Technical Preview with the experimental ResizeObserver turned on, and resize the browser vertically, you'll see the same kind of artifacts (but different ones, because the pixel snapping algorithm depends on the browser).

If you apply a hacked-up prototype of the device pixel border box observation feature to Chromium via patching this CL then the rendering artifacts are gone.

I tested these on a Retina MacBook Pro.

(*) This flag turns on pixel snapping at device pixel granularity, rather than CSS pixel granularity. This flag has already been shipped in Chrome on all platforms except for Mac, and will ship on Mac soon. It's also a necessary feature for high quality output, but orthogonal & complementary to the issue being discussed here.

[chrishtr]

I have added this issue to the agenda for next week's teleconference.

[atotic]

Another demo revision: https://jsbin.com/soqexor

• viewport tag for mobile
• demo button that automatically cycles through different sizes (fractional, and whole), demonstrating the existing problem.

[css-meeting-bot]

The CSS Working Group just discussed device-pixel-border-box size, and agreed to the following:

• RESOLVED: Add API to get content box height and width in device pixel sizes to resize observer. Only return when requested and applies to all element.

The full IRC log of that discussion

<dael> Topic: device-pixel-border-box size
<dael> github: https://github.com//issues/3554#issuecomment-538465771
<dael> astearns: Discussed device-pixel-border-box size at TPAC
<dael> astearns: I recall we still weren't sure how change will effect Safari. Example have now been provided. Do we have enough information on impact to Safari?
<dael> smfr: Saw a couple of examples. I think Chris said he ran the tests on Macbook pro retina with the fixes and it made it look better. Can't say without doing work in webkit, but I think sufficient proof this is useful. On non-apple it's well known it's necessary
<dael> astearns: Any other new information?
<dael> ??: Nothing new except demos and version of chrome that fixes
<astearns> s/??/chrishtr/
<dael> smfr: I'm surprised you could get good results. Most macbooks have default scaling. Surprised you got good results on a machine with that behavior
<dael> chrishtr: It's not the newest
<drousso> presnet+
<dael> [audio problems
<dael> chrishtr: Tested on not absolute newest, but a macbook pro retina. I think it does have value and OS scaling if it's outside of control of application those situations can't be fixed
<dael> smfr: What I'm interested in understanding is if on hardware with builtin scaling suc htat you never get pixel perfect is there improvement in device-pixel-border-box ? Is there value in making it optional and allow UA to not provide if it thinks it can provide better. Or do we make it required and force UA to do this when it's not going to get better
<dael> myles: Easy to test if you change resolution of OS in your macbook pro
<dael> astearns: Sounds to me like we don't have an objection to adding device-pixel-border-box size to the API, but may want another issue about behavior of API possibly making it optional?
<dael> chrishtr: Would app fallback be multiply css pixel width and height by device-pixel ratio if UA doesn't supply?
<dael> smfr: Impl will alreayd have to multiply by device-pixel-ratio. No, does multiply. Okay.
<dael> chrishtr: That was the point, rounding or flooring can't get consistent
<dael> smfr: Okay, then makes optional thing annoying
<dael> chrishtr: I don't know if built in scaling is higher quality but we got really good results on macbook pros even with non-1 to 1 scaling.
<dael> smfr: Okay
<dael> s/ chrishtr / ken
<dael> smfr: Not objecting to adding. Question if needs to be rectangle or just a size since left and top don't mean anything
<dael> chrishtr: Use case for canvas sizing top left not nec
<dael> smfr: We don't have a dom size object
<dael> chrishtr: Yep. More consistent to have a rect, rect does have meaning. Not used in canvas case
<dael> ken: Position of rect a little confusing if you consider overflow area
<dael> chrishtr: Does mean where it is on device window though
<astearns> s/ken/myles/
<dael> chrishtr: It's used within the native texture if there's not special scaling smfr referred to. If texture is scrolled it's still...it doesn't take into account transforms and scrolls
<dael> myles: Values off the top of the screen?
<dael> chrishtr: I think would be fine to have size only for this reason
<dael> chrishtr: top left doesn't seem to mean much, want to know canvas size
<dael> astearns: Array of 2 values or are you minting a new size objection?
<dael> chrishtr: I don't know. Maybe new size object?
<astearns> s/objection/object/
<dael> smfr: If it always integral?
<dael> chrishtr: Yeah
<dael> smfr: Maybe you jsut have two long on the entry
<dael> chrishtr: THat are just width and height
<dael> chrishtr: As relates to desire to add the eq method to getBoundingClientRect it would change to get device pixel width and height
<dael> smfr: If we agree to resizeObserver do we still need?
<dael> chrishtr: Don't think so, but ?? from Mozilla thought case was strong so I was okay adding
<dael> smfr: Which case was it?
<astearns> s/??/Jeff Gilbert/
<dael> chrishtr: You have a full screen where you don't want resizeObserver and you want a slight jump on resize frames
<dael> ken: Jeff also had concerns that it fires late and application would fall behind a frame which is legit concern
<dael> myles: You would need to executecode every frame with this. resizeObserver means code only called when need to be.
<fantasai> s/executecode/execute heavy calculations/
<dael> ken: Agree, but in experience from a dev on FF stack we felt it was important to aleviate concern. And FF intends to not recompute if layout isn't dirty
<tantek> regrets+
<dael> smfr: I don't want every getBoundingClientRect to be more expensive
<dael> chrishtr: Adding a new thing
<dael> smfr: Okay, okay. Should discuss sep.
<dael> chrishtr: Okay with me. Is Jeff Gilbert on? Want ot make sure he's okay to separate.
<dael> chrishtr: If he's not, maybe tentatively do that.
<dael> ken: I think Jeff would object. Not to represent his opinion, but I think he would.
<dael> chrishtr: Great to resolve device-pixel-border-box size makes sense and then follow-up on the other one
<dael> chrishtr: To make progress.
<dael> ken: Want progress too
<dael> astearns: smfr you would rather continue new API discussion or resolve to add it and continue discussion?
<dael> smfr: Prefer to fork into separate issue
<dael> fantasai: Question, says doing device-pixel-border-box size. What if person wants size of padding or content box?
<dael> chrishtr: Those other boxes have use cases and tracked via other issues on the spec
<dael> florian: But not at device pixel level the others
<dael> chrishtr: No. Only use case for device pixels is canvas
<dael> fantasai: But why wouldn't people using canvas want to paint inside the border?
<AmeliaBR> Wouldn't it be the content-box that is important for the canvas? Since that's what they're using in their code?
<dael> chrishtr: Canvas has a certain size and border is outside of that. Browser does that for you.
<dael> fantasai: boder box size includes border so if it's not included it's not a border box
<dael> chrishtr: It's the device pixel box size in case of canvas. It's actual size of canvas
<dael> fantasai: Then it's the content box and we should call it that and not border box
<dael> chrishtr: Right
<Zakim> fantasai, you wanted to ask about padding/content boxes
<dael> ken: Sounds good
<dael> astearns: Other discussion?
<dael> fantasai: Summary of proposal?
<dael> astearns: Add an API to get canvas height and width to resizeObserver
<dael> chrishtr: Actual device width and height of the canvas. When changes resizeObserver fires.
<dael> fantasai: I want to be clear. Adding and API which is only available on canavas element. Reflects pixel size of content box of canvas element
<dael> fantasai: And it has a name that is not device-pixel-border-box
<dael> chrishtr: Yes
<dael> smfr: Available for any element you resizeObserve not jsut canvas
<dael> fantasai: Have had various discussion on only canvas or all and want to be clear
<dael> chrishtr: Only use case I know is canvas. Could do it for all
<dael> smfr: I wonder if this should be something the user of the API request.
<dael> chrishtr: For any API you ask for what you want and you're saying only available on canvas?
<dael> smfr: ONly want browser to do computation if author asks for data
<dael> chrishtr: For sure. Part of draft spec to observe multiple types of boxes. You say what to observe. I agree browser should only do this if needed
<dael> astearns: Further question of if someone requests this on not canvas what happens
<dael> chrishtr: Allow or not allow is both okay. In terms of APIs implicity maybe better on all
<dael> smfr: Can imagine on something like video. Better on all
<dael> chrishtr: No impl difficulty from allowing it
<dael> fantasai: Add API to get content box height and width in device pixel sizes to resize observer. ONly return when requested and applies to all.
<dael> chrishtr: Can bikeshed name
<dael> florian: Not obj, but want to be cautios on all elements. pixels vs device pixels people can get confused about and I have mild concern making this too easily available people will try and use it.
<dael> astearns: A lot of things discussing now should be separate issues once we have draft text in place.
<dael> astearns: objections to adding this?
<dael> RESOLVED: Add API to get content box height and width in device pixel sizes to resize observer. Only return when requested and applies to all element.
<dael> astearns: Thanks chrishtr
<dael> chrishtr: I appriciate all the feedback
<dael> ken: Thanks for discussion
<dael> fantasai: Want to discuss optionality
<dael> astearns: Add an issue for that

[TremayneChrist]

So this will not be provided as a box option, but as a getter in the ResizeObserverEntry object?

If so, I see a problem with this when the scaling changes across multiple screens - The css pixel will stay the same, however, the device pixels will be different. This means that the resize observer will not fire, when it probably should?

https://jsbin.com/gamerul

[kenrussell]

It will be provided as a box option. The summary above didn't completely capture the discussion.

To summarize the conclusions from the meeting:

• ResizeObserver.observe() will accept a new option, device-pixel-content-size. (@fantasai pointed out that observing the border box was not what was desired. Also, @smfr pointed out that it doesn't need to be a rect, since the position is not useful.)
• A new property will be added to ResizeObserverEntry, devicePixelContentSize, which will be populated if device-pixel-content-size is being observed for the target element. If populated, it will contain a sequence of two longs, representing the width and height of the element, in device pixels.

@chrishtr @atotic is this correct?

[atotic]

Yes. device-pixel-content-size will be an option, and the size will be reported on ResizeObserverEntry. Size could be defined as:

interface ResizeObserverSize {
    readonly attribute unrestricted double inlineSize;
    readonly attribute unrestricted double blockSize;
} ;

Side note: #3673 discussed how to handle fragments. Current resolution is that the sizes will be reported as an array. ResizeObserverEntry per spec would look like this:

{
    target: Element
    contentRect: {....}
    devicePixelContentSize: [{ inlineSize: ..., blockSize: ... }]
}

[dlibby-]

Addressed by #4476