Puma won't restart sometime after deploy

[prashantvithani]

Puma: 3.8.2
capistrano3-puma: 3.1.1

My system configuration contains 3 servers, there's always one or two servers, in which Puma won't restart after deployment. Though, the deployment log says Command restart sent success.

Then, I logged in to all servers, and ran ps aux | grep puma. The output was weird.

Server 1: puma works fine, restarted successfully at 10:06 with release 20170726095950

deploy   21379  0.0  3.5 2119668 274772 ?      Sl   10:06   0:01 puma 3.8.2 (unix:///var/www/fbdash/shared/tmp/sockets/puma.sock) [20170726095950]                                                        
deploy   21384  0.7  8.0 2266296 618124 ?      Sl   10:06   0:35 puma: cluster worker 0: 21379 [20170726095950]                                                                                           
deploy   21387  0.9  7.9 2269380 607296 ?      Sl   10:06   0:46 puma: cluster worker 1: 21379 [20170726095950]                                                                                           
deploy   21390  1.0  7.4 2271436 569760 ?      Sl   10:06   0:48 puma: cluster worker 2: 21379 [20170726095950]                                                                                           
deploy   21394  0.8  7.8 2203948 603500 ?      Sl   10:06   0:41 puma: cluster worker 3: 21379 [20170726095950]                                                                                           
deploy   25204  0.0  0.0  10484  2260 pts/2    S+   11:26   0:00 grep --color=auto puma

Server 2: Same as server 1, puma works fine, restarted successfully at 10:06 with release 20170726095950

deploy   21379  0.0  3.5 2119668 274772 ?      Sl   10:06   0:01 puma 3.8.2 (unix:///var/www/fbdash/shared/tmp/sockets/puma.sock) [20170726095950]                                                        
deploy   21384  0.7  8.0 2266296 618124 ?      Sl   10:06   0:35 puma: cluster worker 0: 21379 [20170726095950]                                                                                           
deploy   21387  0.9  7.9 2269380 607296 ?      Sl   10:06   0:46 puma: cluster worker 1: 21379 [20170726095950]                                                                                           
deploy   21390  1.0  7.4 2271436 569760 ?      Sl   10:06   0:48 puma: cluster worker 2: 21379 [20170726095950]                                                                                           
deploy   21394  0.8  7.8 2203948 603500 ?      Sl   10:06   0:41 puma: cluster worker 3: 21379 [20170726095950]                                                                                           
deploy   25204  0.0  0.0  10484  2260 pts/2    S+   11:26   0:00 grep --color=auto puma

Server 3: Two of the clusters stopped (Didn't restart at all), still pointing at old release 20170726080157

deploy   22364  0.0  3.6 2120888 279128 ?      Sl   09:37   0:00 puma 3.8.2 (unix:///var/www/fbdash/shared/tmp/sockets/puma.sock) [20170726080157]                                                                                                                             
deploy   22369  0.1  9.3 2405448 719184 ?      Sl   09:37   0:11 puma: cluster worker 0: 22364 [20170726080157]                                                                                                                                                                
deploy   22378  0.2 10.9 2538956 838948 ?      Sl   09:37   0:15 puma: cluster worker 3: 22364 [20170726080157]                                                                                                                                                                
deploy   30422  0.0  0.0  10484  2160 pts/2    S+   11:26   0:00 grep --color=auto puma

This is causing all requests that are directed to this server to timeout with error code HTTP 408.

---

I've also tried cap production puma:stop followed by cap production puma:start, but it's still not working. I had to exlicitly kill the all the old processes with kill -9 PIDs, and then send cap production puma:start

Any idea why this is happening?

[jacek213]

I have similar issue (one server only). Puma 3.9.1, capistrano3-puma (3.1.1).

On first deploy after changes, restart fails when command puma:restart is called. If I run deploy once again, it gets up as puma is dead at the moment and puma:start is called instead of puma:restart. It happens each time.

In puma_error.log I see:

=== puma startup: 2017-08-07 21:00:36 +0200 ===
/home/deploy/.rvm/rubies/ruby-2.3.3/lib/ruby/2.3.0/rubygems.rb:241:in `bin_path': can't find gem puma (>= 0.a) (Gem::GemNotFoundException)
        from /home/deploy/apps/alfalek_rails/shared/bundle/ruby/2.3.0/bin/puma:22:in `<main>'

[pgericson]

A bit more info on why this is failing:

Where there is a process running/(pid file exists) then line 68 is called, which fails where it try to call pumactl with a -S state file. It don't have the same parameters as the puma:start task. That is why we get a log entry in the puma_error.log

capistrano-puma/lib/capistrano/tasks/puma.rake

Lines 66 to 72 in 220e81c

	if test "[ -f #{fetch(:puma_pid)} ]" and test :kill, "-0 $( cat #{fetch(:puma_pid)} )"
	# NOTE pid exist but state file is nonsense, so ignore that case
	execute :pumactl, "-S #{fetch(:puma_state)} -F #{fetch(:puma_conf)} #{command}"
	else
	# Puma is not running or state file is not present : Run it
	invoke 'puma:start'
	end