download:plugins much slower after merging HTTP proxy support

[ccorn]

Context

After your awesome efforts resulting in commits 8ba808a (#11112) and 9a10b6b (#11201), there was a brief phase during which I could enjoy reliable download:plugins functionality with a slow (1Mbps) connection, at least when using the --no-parallel option.

As you may remember, when resolving extension packs, a lot of metadata gets downloaded (about 1,3 GB for 73 plugins). Without compression that would take 3 hours over a 1Mbps connection. But apparently the transfer uses compression, which speeds up the transfer by a factor of 6, reducing the time to 30 minutes.

Issue

Alas, with the master branch at v1.26.0, download:plugins --no-parallel keeps a 1Mbps download connection maxed out for 3 hours. Therefore I suspect that compression has been inhibited.

The culprit seems to be

• d276e5a Support http proxies (Support http proxies #11043)

And no, I do not have any proxy setup.

How to reproduce

1. Checkout and build 116a40c
2. rm -rf plugins; time yarn download:plugins --no-parallel
3. Checkout and build the successor d276e5a
4. rm -rf plugins; time yarn download:plugins --no-parallel
5. Note that the second duration is about 6 times longer, consistent with lack of compression.

Requests

1. I have not examined the actual API request headers, so further investigation to identify the cause would be prudent.
2. To catch such unintended side effects in the future, an automated test case for the CI would be helpful. Not about the time taken, but about the presence of compression request headers (or whatever the cause turns out to be).
3. And, yes, a bug fix would be nice. Particularly because a factor 6 increase in traffic seems not desirable for open-vsx.org right now.

[zhaomenghuan]

I also think that the download plugins is very slow now. Even if it has been downloaded, it is still very slow to skip it when theia download:plugins is executed again.

[ccorn]

Even if it has been downloaded, it is still very slow to skip it when theia download:plugins is executed again.

That is because the metadata for all versions is requested even if the extension has been downloaded already. Ideally, the skip test would occur before the metadata request and prevent the latter if it turns out to be unnecessary.

[ccorn]

A temporary workaround for users is to replace the extension pack entry in theiaPlugins with the actual plugin list, as was the case for releases up to 1.24.0. Because those entries have their version specified explicitly and therefore do not need metadata scans, and the vsix data itself is (presumably) compressed already, therefore the speed of that phase is not affected.

[zhaomenghuan]

@ccorn I recently changed the way that the plugins declare the vscode plugins one by one to the vscode-builtin-extensions-pack package. The speed is too slow. I decided to temporarily fall back to the previous declarations one by one.