It's way too big! (Working on making aegir smaller)

[mkg20001]

Just take a look at that:

maciej@maciej-pc:~$ time npm i -g aegir@13.0.7
/usr/bin/aegir -> /usr/lib/node_modules/aegir/cli.js

> pre-push@0.1.1 install /usr/lib/node_modules/aegir/node_modules/pre-push
> node install.js


> uglifyjs-webpack-plugin@0.4.6 postinstall /usr/lib/node_modules/aegir/node_modules/uglifyjs-webpack-plugin
> node lib/post_install.js

npm WARN ajv-keywords@2.1.1 requires a peer of ajv@^5.0.0 but none is installed. You must install peer dependencies yourself.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.1.3 (node_modules/aegir/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.1.3: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

+ aegir@13.0.7
added 1678 packages in 430.014s

real	7m10.646s
user	1m31.776s
sys	0m7.068s
maciej@maciej-pc:~$ du -hs /usr/lib/node_modules/aegir
215M	/usr/lib/node_modules/aegir

AEgir, a development CLI is 200mb. I repeat: AEgir a CLI is as big as say, ubuntu core, an IoT OS!

This is way to large and makes up for about 70% up to 90% of the size of the node_modules folders.

Additionally on my usually slow HDD it's a pain to install because it takes way too long.

Possible Solutions:

• Bundle and minify the aegir executable as one single file
  • (pro) Would still require native modules to be installed seperatly, but still a significant reduction in module count
  • (pro) Would drop all unused code
  • (maintaince/con) This bundle should be auto-published and auto-updated as some deps might have in-range updates available before the next release, which requires a new bundle of the current version to be published
  • (con?) The resulting bundle file might be huge (20mb) and I don't know if the V8 parser likes that (cli might get slower)
• Split the lint and test commands into a much lighter "aegir-light" version which transparently calls the globally installed "full aegir" when commands such as aegir release are executed
  • (pro) Would make the module smaller and the cli would load faster
  • (con) This would stil require a significant amount of modules to be installed and would make it much harder to maintain this module
  • (con?) Both versions (global & local) would have to match

[mkg20001]

Bundle and minify the aegir executable as one single file

Working on a pr for that: #216

[mkg20001]

Update: The webpack method is more work than I thought it would be, mostly because webpack isn't made for that sort of thing.

Possibly the best way would be to write a version of https://github.com/zeit/pkg that would output a standalone/bundle JS file instead of nodejs executable. The current methods pkg is using to emulate a virtual filesystem (require("fs") patching, etc.) should work as-is with a js file. Unsure about the require wrapper, it might be that some internal functions are specifically exposed. This is a bit complicated to write but could make a big impact across a vast landscape of projects because it would remove a lot of code (and file loading time) per cli project.

Also it would be great if I could get somebody else's opinion on this endeavour as well...

[mkg20001]

I've started to work on the pkg-js module here: https://github.com/testtest232/pkg
I'll move that to a proper repo moved to https://github.com/mkg20001/pkg-js and rebase it after I'll get it to do it's job (Also already had a pkg fork in my personal namespace so I forked it to my test org)

[mkg20001]

Submitted a few issues upstream which also affect original pkg: vercel/pkg#528 vercel/pkg#529

[mkg20001]

I'm still waiting to get some feedback on this.
The goal is to make a big single-file bundle (with native deps still managed by npm, single-file because that will have positive impact on performance on HDDs) that could also be minified to save even more space.
That big bundle could also be placed into a custom, global directory so there is only one single instance of every version of aegir on a dev's machine (The downloading could be handled similary as in go-ipfs-dep package, just with a global folder)

Does this sound like a good idea? For me at least it makes sense to do this because of the reasons described in the issue description above.

[mkg20001]

Btw, here are some initial measurments:

aegir node_modules node v10

307M	node_modules/

bundled with pkg for node v10 amd64 linux

196M	aegir

bundled with pkg-bundle for any node version

4.0K	aegir.js
128M	aegir.js.data

bundled with pkg-bundle for any node version + minified with terser

4.0K	aegir.js
106M	aegir.js.data

[mkg20001]

I've published not-yet-fully-functional bundles here: https://aegir.mkg20001.io/15.2.0
Compressed size is only 24MB. This means: 1) There is lots of duplicate code 2) The install time could be greatly reduced as npm wouldn't have to do a thousands requests and download 300mb, instead it only would download 24mb in a single request.
If needed I can publish the scripts for bundle-building.

[mkg20001]

I created a module (https://npm.im/aegir-bundle) that uses the above mentioned bundles. Still wip, but aegir docs already works 🎉

[victorb]

@mkg20001 thanks for the work so far on this and sorry for not providing any feedback sooner.

Am I understanding correctly that the final archive ends up being 24MB? If that's so, it's amazing! Is this because of dead-code elimination and minification?

I'm not sure how the suggested installation flow would be here. Would it be to download the archive, untar it, then run npm install to get any native modules? Can we make all of that automatic? I guess it would involve moving all the things set as dependencies here to dev-dependencies, then have a post-install script that downloads the archive, untars it and then runs install again.

[mkg20001]

@victorbjelkholm The 24mb are after compression, the extracted files are ~100mb but minification is currently still disabled until everything starts working as expected so they might possibly be smaller. Most of the current gain is because unused files aren't bundled.

The install flow work like this: The user installs aegir-bundle which exposes a wrapper binary called aegir that checks if the aegir bundle is already installed at $HOME/.aegir/<version> and if it isn't it downloads that. Afterwards it requires the bundle file which then should operate like the aegir cli would normally do. The aegir version can be specified via package.json property aegir

Edit:

Would it be to download the archive, untar it, then run npm install to get any native modules?

Yes, although currently the native modules aren't getting installed because, well, it's still WIP