Complete VM freeze while running the following gulp task

[marcofranssen]

Hi,

Our development VM has 4GB of memory and 2CPU assigned at 75% execution cap. We are running the following gulp script to transform our svg to png. However 8 out of 10 times it completely kills the VM. The only solution is to just force a poweroff and reboot.

It is ran on a debian wheezy VM.

I really hope there is someone with a solution for this issue.

gulp.task('svg2png', ['clean-dest', 'build-iconfont'], function () {
      return gulp.src(iconsInput)
          .pipe(svg2png())
          .pipe(gulp.dest(iconOutput));
});

The input are 109 SVGs. Is there a way to make it work without freezing the whole OS?

[domenic]

I have no idea how this could possibly work given that svg2png is not a transform stream so .pipe() should not impact it. In fact calling it with no arguments should just error.

[marcofranssen]

It works because we use gulp-svg2png. Also filed the issue at that repo.
akoenig/gulp-svg2png#16

I suspect it is because running out of memory, but I can't prove as I won't be able to check because of the freeze.

[marcofranssen]

Just updated the issue over there: akoenig/gulp-svg2png#16

I guess we need some kind of parameter in order to make it work on this side to mention it should reuse an existing phantomjs or something like that.

I have the feeling that for each svg2png conversion a new phantonjs browser is started. My OS runs out of memory and freezes as you can see in my latest output from top, just before the freeze.

top - 15:56:55 up  1:17,  2 users,  load average: 164.86, 123.16, 56.83
Tasks: 246 total,   1 running, 245 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.7 us,  2.7 sy,  0.0 ni,  0.0 id, 95.8 wa,  0.0 hi,  0.8 si,  0.0 st
KiB Mem:   4061492 total,  3975792 used,    85700 free,        0 buffers
KiB Swap:  2047996 total,  1976932 used,    71064 free,    72248 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
12420 mongodb   20   0  256m 4856 1740 S   3.4  0.1   1:55.45 mongod
19962 elastics  20   0 1570m  36m 1352 S   0.9  0.9   1:14.36 java
   22 root      20   0     0    0    0 D   0.5  0.0   0:04.58 kswapd0
    3 root      20   0     0    0    0 S   0.2  0.0   0:03.70 ksoftirqd/0
 2081 root      20   0 93580  676  496 S   0.2  0.0   0:17.37 VBoxService
 3839 couchbas  20   0  142m  788  788 S   0.2  0.0   0:09.67 moxi
 3840 couchbas  20   0  285m 4760 1820 S   0.2  0.1   3:40.52 memcached
 4732 root      20   0 90816  376  376 D   0.2  0.0   0:00.01 winbindd
11731 www-data  20   0  366m 1904 1652 S   0.2  0.0   0:03.61 php
17809 root      20   0     0    0    0 S   0.2  0.0   0:02.03 kworker/1:1
17984 root      20   0     0    0    0 S   0.2  0.0   0:00.22 kworker/0:1
21872 root      20   0 1203m  23m 2448 D   0.2  0.6   0:02.39 phantomjs
21875 root      20   0 1203m  31m 2448 D   0.2  0.8   0:02.40 phantomjs
21876 root      20   0 1203m  35m 2444 D   0.2  0.9   0:02.35 phantomjs
21877 root      20   0 1204m  28m 2444 D   0.2  0.7   0:02.34 phantomjs
21879 root      20   0 1203m  26m 2448 D   0.2  0.7   0:02.34 phantomjs
21883 root      20   0 1204m  25m 2448 D   0.2  0.6   0:02.25 phantomjs
21900 root      20   0 1203m  22m 2444 D   0.2  0.6   0:02.34 phantomjs
21902 root      20   0 1203m  29m 2448 D   0.2  0.7   0:02.41 phantomjs
21903 root      20   0 1203m  21m 2444 D   0.2  0.5   0:02.36 phantomjs
21916 root      20   0 1204m  24m 2448 D   0.2  0.6   0:02.42 phantomjs
21917 root      20   0 1203m  49m 2448 D   0.2  1.2   0:02.42 phantomjs
21918 root      20   0 1203m  34m 2448 D   0.2  0.9   0:02.26 phantomjs
21921 root      20   0 1203m  21m 2448 D   0.2  0.5   0:02.32 phantomjs
21922 root      20   0 1203m  25m 2444 D   0.2  0.6   0:02.38 phantomjs
21923 root      20   0 1203m  37m 2448 D   0.2  1.0   0:02.30 phantomjs
21928 root      20   0 1203m  32m 2448 D   0.2  0.8   0:02.34 phantomjs
21929 root      20   0 1203m  48m 2444 D   0.2  1.2   0:02.37 phantomjs
21935 root      20   0 1203m  29m 2448 D   0.2  0.7   0:02.41 phantomjs
21938 root      20   0 1203m  45m 2448 D   0.2  1.2   0:02.43 phantomjs
21939 root      20   0 1203m  29m 2448 D   0.2  0.7   0:02.37 phantomjs
21942 root      20   0 1203m  42m 2444 D   0.2  1.1   0:02.30 phantomjs
21943 root      20   0 1203m  38m 2448 D   0.2  1.0   0:02.39 phantomjs
21944 root      20   0 1203m  17m 2448 D   0.2  0.4   0:02.35 phantomjs
21945 root      20   0 1203m  26m 2448 D   0.2  0.7   0:02.40 phantomjs
21947 root      20   0 1203m  27m 2444 D   0.2  0.7   0:02.40 phantomjs
21948 root      20   0 1203m  24m 2444 D   0.2  0.6   0:02.40 phantomjs
21949 root      20   0 1203m  27m 2448 D   0.2  0.7   0:02.41 phantomjs
21953 root      20   0 1139m  12m 5264 D   0.2  0.3   0:02.35 phantomjs
21955 root      20   0 1203m  18m 2444 D   0.2  0.5   0:02.33 phantomjs
21958 root      20   0 1204m  35m 2444 D   0.2  0.9   0:02.48 phantomjs
21960 root      20   0 1203m  21m 2444 D   0.2  0.5   0:02.37 phantomjs
21961 root      20   0 1203m  44m 2444 D   0.2  1.1   0:02.39 phantomjs
21965 root      20   0 1204m  32m 2448 D   0.2  0.8   0:02.27 phantomjs
21967 root      20   0 1204m  42m 2444 D   0.2  1.1   0:02.40 phantomjs
21975 root      20   0 1203m  23m 2448 D   0.2  0.6   0:02.40 phantomjs
22200 root      20   0 31124 1496  888 R   0.2  0.0   0:02.20 top
    1 root      20   0 10648  640  608 S   0.0  0.0   0:02.90 init
    2 root      20   0     0    0    0 S   0.0  0.0   0:00.00 kthreadd
    5 root      20   0     0    0    0 S   0.0  0.0   0:00.00 kworker/u:0
    6 root      rt   0     0    0    0 S   0.0  0.0   0:00.11 migration/0

[domenic]

Yeah that's an option. Another option is to just limit the number of conversions you do in parallel. It looks like gulp-svg2png has a "concurrency" parameter you could use?

[marcofranssen]

I checked that one out. However it just slows down the time I run out of memory. As you can see above each phantomjs instance takes quite a big amount of memory. It would be way better if your code uses one phantomjs. I think karma does something similar. When we run karma there is only one phantomjs spawned.

[domenic]

What happens if you set concurrency to 1?

[marcofranssen]

They recently released an update. It indeed has that option now, however the library now works even without limiting the concurrency just fine. It seems they fixed it.

I tried before without updating as I didn't knew they recently released an update, so the concurrency option I put in was just not used.

Anyway thanks for your suggestion.

[domenic]

Oh cool! I'm curious to see what they did; I'll check it out.

I'm still open to an option to reuse PhantomJS instances, but designing the API will be tricky so we'd need someone who's willing to do some brainstorming with me plus implementation work on that, probably in a separate issue.