Support chunks in HTTP2

[soulgalore]

Patrick at WebPageTest has added a super useful feature for websites that use HTTP2. He now includes when chunks of data are sent from the server in the HAR. Instead of just having those long blocks (start/stop) you can now see when chunks of data are sent (the chunks have darker color)

The response has extra data:

 "_chunks": [
 {
   "bytes": 385,
   "ts": 1361.457
},
{
   "bytes": 1300,
   "ts": 1367.293
},

Here's one full example:
https://gist.github.com/soulgalore/2a45fb2ecce8d8e039fd587ef7cb437c

I think this would be cool to support.

[radum]

Has anyone tackled this yet? I was thinking to have a stab at it, and by the looks of it shouldn't ben anything that dramatic of a change.

Would be great to talk a bit about what will be the best way of doing it before we get started.

But as a summary, I think the easiest way to tackle this will be to paint on top of the existing rect the coming in chunks as new SVG rects with different color. Some math will need to be involved to calculate the position and width of each but we can sort it out.

@micmro what do you think?

[soulgalore]

I haven't done anything, it would be super cool if you had a go @radum . I created an issue for Firefox to also implement it https://bugzilla.mozilla.org/show_bug.cgi?id=1493579 some time ago.

[micmro]

This would be great. Thanks a lot @radum!

Yes I think painting over it (e.g. with alpha transparency so you would not need to worry about the underlying colour) sounds like be best way to go.

The loading rect is created here, but you could also add it in the row-setup if you think this would work better.

[radum]

@soulgalore Do you know where I can find some info on the _chunks object? Not sure what the TS stands for exactly.

EDIT: OK I got it. Basically each chunk object has a ts that shows when the chunk finished to download, and is relative to the start of the waterfall. And bytes which amounts the total number of bytes received.

So I managed to draw the lines for when each chunk is finished, but the width of each bar that needs to be drawn is determined by the available bandwidth, info that I don't think we have in the har file. Do we get that from sitespeed instead?

WPT sets that value in connectivity.ini and is used to define the amount of bandwidth for each connection type, like 3g fast, or cable and so on. If we can get that info, we could draw the correct bars.

[soulgalore]

@radum adding it so its in the HAR from sitespeed.io is easy, I can do that, and also make sure we inject it into the HAR if you drive WPT from sitespeed.io but it would be sweat to have it directly in the HAR from WebPageTest. I'll have a look the coming weeks.

[radum]

@soulgalore Here is a quick demo on how it looks with chunk bars defaulted to 1px width.

The problem with bandwidth is that Sitespeed.io as WPT can be run without traffic-shaping locally. Or is done via external tools like throttle. So we won't really know the bandwidth unless we check it. I know we can pass the values we have for each connectivity.profile but I am not sure how reliable that will be.

We can always just show the narrow bars and that is at least some sort of indication on how chunks were received, but not as good as it could be.

[soulgalore]

Looks great @radum ! Let me do a fix so that when the throttling is known (in WPT and sitespeed.io when it's done internally), I can just add the info to the meta section of the HAR following the standard of https://github.com/sitespeedio/throttle/blob/main/bin/index.js#L33-L36 , that will be enough right? Then it will work firn for HARs from WebPageTest. For sitespeed.io, hopefully the Firefox team will add it soonish (I can ping the issue). For Chromium it needs to be implemented in Chrome-HAR, I haven't looked into the trace log from Chrome what info we get and how much work it is to include.

[radum]

Yes that should be enough. Basically all we need to know is the down value. As WPT does it using the bwIn value.

Having that info then the calculation is simple. From each chunk object ts will give us the timestamp when the chunk was finished to download and using ts - (bytes / (bandwidth / 8.0)) will give use the timestamp when it started to download, and this is our width of each chunk block.

[micmro]

This looks awesome - thanks a lot!
Are you planning to make this progressive? E.g. just show a 1px bar for HARs that don't contain any bandwidth information?

[radum]

Yeah that is the idea @micmro. Also I need to add some safe guards to make sure the chunks bars will not be bigger than the receive block, or some chunks will show data for headers and those don't really need to be shown and so on.

[radum]

Hello, @soulgalore @micmro just wanted to say I haven't forgotten about this, is just I am struggling to find time these days to finish it. Also there is quite a bit of TS involved 😁 and I need to navigate around that also.

I hope to get some free time in the next month or so.

[micmro]

Thanks for the heads-up @radum. Please also feel free to ping me if I can help in any way.

[soulgalore]

@radum I can help you and try to finish it if you still have the code? :) It doesn't matter is your code is rough, I can try to tune it if you want. I also wants this + plan to have a go to implement the _chunks part for Chrome in Browsertime/sitespeed.io.

[radum]

@soulgalore I have raised #282 with the initial code. Also moved the conversation on that PR.

[micmro]

Released with 2.11.0

I had played around if sizing the chunks as percentage of the total size would create a similar result as WPT, but it's often not as the bandwidth can change :(. One change I made is making the chunk <g/> transparent to cursor events so they don't affect the tooltip.

[radum]

I tried using % myself and yeah its not the same. In theory we can still improve PerfCascade to receive the throughput available and use it. And perhaps sitespeed.io can calculate it at run time and pass it over. @soulgalore not sure how easy and practical that would be across platforms.

[micmro]

Alternately sitespeed could introduce an `endTs` or so flag to indicate when the chunk was downloaded, as bandwidth might change atnany time during download?

…

On Wed, 24 Nov 2021, 3:14 am Radu Micu, ***@***.***> wrote: I tried using % myself and yeah its not the same. In theory we can still improve PerfCascade to receive the throughput available and use it. And perhaps sitespeed.io can calculate it at run time and pass it over. @soulgalore <https://github.com/soulgalore> not sure how easy and practical that would be across platforms. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#237 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAM2IBWJ7FFCRFW2TOCWI4TUNSNHDANCNFSM4FWVTAUA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.