Server crashes when trying to restart pending build

[espenaf]

Running server and worker on CentOS 8 via podman. While trying to restart a build which is stuck in pending state the following error is thrown in the server, and the container crashes:

2021-02-19T12:55:06.662+0100        DEBUG        PUT /api/v1/builds/job/restart (code=200 dt=17.194709ms written=3 B remote=10.0.0.218)        {"type": "http"}
2021-02-19T12:55:06.662+0100        INFO        processing job 2, sending to worker 51043ced...        {"type": "scheduler"}
panic: runtime error: index out of range [1] with length 1
goroutine 289 [running]:
github.com/bleenco/abstruse/server/scheduler.(*scheduler).startJob(0xc00022a480, 0xc00015a300, 0xc0005b6f00)
         /app/server/scheduler/scheduler.go:294 +0x1757
 created by github.com/bleenco/abstruse/server/scheduler.(*scheduler).process
         /app/server/scheduler/scheduler.go:262 +0x1f5

The build config is added manually via the UI, and has the following setting

image: gcr.io/distroless/java:8

script:
  - mvn compile jib:build

[jkuri]

hi @espenaf I opened pull request which fixes pulling docker images from third party registries such of a grc.io but in my case does not fix your issue. I investigated a bit more into this and seems like gcr.io/distroless/java:8 image does not include any shell which is needed for abstruse to execute commands. Can you please try with official java image from docker hub and tell me if that works?

[espenaf]

Ahh. Did not know the standard jib container build images did not have a shell.

Switched to using docker.io/adoptopenjdk/maven-openjdk11 now, but still the same issue, but maybe your #503 also fixes pulls from docker.io?

[jkuri]

try to use just adoptopenjdk/maven-openjdk11

[jkuri]

PR will be merged later after review from someone.

[espenaf]

Getting the same error:

2021-03-02T12:27:08.567+0100        INFO        scheduling job 1 from build 1...        {"type": "scheduler"}
2021-03-02T12:27:08.575+0100        INFO        job 1 scheduled        {"type": "scheduler"}
2021-03-02T12:27:08.575+0100        INFO        processing job 1, sending to worker 1fe4088e...        {"type": "scheduler"}
2021-03-02T12:27:08.579+0100        DEBUG        PUT /api/v1/builds/trigger (code=200 dt=212.018867ms written=3 B   remote=10.0.0.237)        {"type": "http"}
 panic: runtime error: index out of range [1] with length 1
 goroutine 154 [running]:
 github.com/bleenco/abstruse/server/scheduler.(*scheduler).startJob(0xc000408090, 0xc000112300, 0xc000345500)
     /app/server/scheduler/scheduler.go:294 +0x1757
 created by github.com/bleenco/abstruse/server/scheduler.(*scheduler).process
     /app/server/scheduler/scheduler.go:262 +0x1f5

The build status is now stuck in "running" with no value in the output, and i can`t stop or restart the build. If i do a restart it crashes with the error above. Stop has no effect.

[jkuri]

thanks for reporting this, I'll try to reproduce that and make a fix. btw build status is stuck because server is down after this.

[espenaf]

If i restart the server the build will still be stuck in running state. Trying to stop it then will have no effect in the UI, while the server logs a call to /api/v1/builds/stop with return code=200, and if then try to restart, the server crashes again.