Use fixed version of jenkins vs. latest version of jenkins

[heiderich]

The Dockerfile advises the user to download the Jenkins war file from
http://mirrors.jenkins.io/war-stable/latest/jenkins.war

In contrast, the Makefile contains the URL
http://mirrors.jenkins.io/war-stable/2.176.1/jenkins.war

Is there a special reason for this discrepancy?

[rbehrends]

The Dockerfile comment is in error here. It is simply, as so often with comments, that it didn't get updated along with the actual code.

[heiderich]

Is there a specific reason to pin the jenkins version instead of using the latest one?

I am not sure where one would have to expect problems when changing the jenkins version while keeping the same "jenkins volume" (the one mentioned in https://github.com/oscar-system/oscar-ci/pull/4/files). Do you know @rbehrends?

[rbehrends]

Well, I originally had the latest version (as indicated by the Dockerfile comment), but then it was pointed out to me (don't remember who it was) that there might be inadvertent versioning issues if there are slight incompatibilities between versions. So, better to pin the version and update it manually after it's been verified to work rather than having such a surprise at random.

Edit: Just to be clear, the jenkins.war file is not stored on /var/jenkins_home, and hence there might be an inadvertent divergence between the jenkins.war file and the Jenkins configuration if the Jenkins container is updated to a newer version. That's what I'm talking about.

[rbehrends]

I've updated the comment accordingly.

[heiderich]

I understand that concern and was expecting occasional surprises. We are facing the usual trade-off between being up-to-date (including occasional surprises) and stability. The main question seems to be how Jenkins deals with configurations / workspaces created with older versions of Jenkins. I have too little experience with Jenkins to answer this.

The downside of fixing the version is the risk that it does not get updated for a long time, because it "just works". In this scenario updates require a manual interaction (currently a change in the Makefile). I just fear that we might forget updating it regularly. In view of possible security concerns this worries me a little. A simple "make" followed by a new build of the docker image would not use a new of Jenkins as long as the jenkins.war file is not deleted in the oscar-ci directory on the host system. So an update of Jenkins on a given host system would require deleting the jenkins.war file. I would hope that an administrator would be expecting possible incompatibilities. For this reason I would personally tend to use the latest version in order to detect possible incompatibilities between our CI files and Jenkins at an early stage.

But as I mentioned above, I completely understand the concern. If stability of Jenkins is the main priority it is probably the best to pin the Jenkins version.

[rbehrends]

I'm personally pretty agnostic on the matter and went with what people wanted (and by people I mean the completely unscientific random sample on the Oscar Slack channel :) ). I'm up for having it be the latest version by default if e.g. security concerns are the bigger issue.

Right now, I've updated the comment and Makefile; you can use make rebuild JENKINS_VER=latest to grab the latest version (or plugin whatever version you want instead).

[fingolfin]

I do see the potential advantage in getting quick security updates when using the latest version -- although that only works if we redownload the jenkins.war regularly (actually, I'd expect the Dockerfile to simply download it each time -- why exactly do we set it up so that an external download of the .war is needed? It's not as if we plan to rebuild the docker image regularly, do we?)

Thing is, to me, one of the motivations for using Docker and scripting everything is that we are able to fully replicate the build server setup in different places. Which is great for being able to prepare changes on a test instance before deploying them to production; or for moving the whole thing to a new server; and in general for debugging.

But using a non-fixed version completely defeats that. Bugs may appear in one place and not another because of different Jenkins versions. For "local debugging purposes", that's not so much a problem. But just think what happens if I test a change locally, and it works fine with the latest Jenkins; but when I deploy it, it totally breaks production, because that still uses an older jenkins.war that just happened to be around.

Another aspect of this general concern is that using this "latest" version means that people building the same Dockerfile but on different dates may get different containers. And since docker caches layers of the container which did not change, and since it has no way of detecting that the .war changed, it may not rebuild the container even if you download a newer .war file.

Of course this can all be dealt with: one "just" needs to remember to compare the Jenkins versions; and if one notices a discrepancy, test whether it matters. And remember to force Docker to fully rebuild the Docker image, and so on.

But all in all, Jenkins releases are not that frequent. I'd prefer if we simply set a recurring calendar reminder that pops up every 3 month, to remind us to check for Jenkins updates and update a hardcoded Jenkins version in the Docker container manually.

[heiderich]

@fingolfin mentioned some valid points. Using an external downloader and the docker caches cause problems.

In my personal opinion reproducibility of builds is not a major concern for this very project, as I expect that there will be relatively few containers running.

How about downloading the .war file in the Dockerfile and add --no-cache=true to the docker run command in the Makefile? I think this should address the problem of an old copy of a downloaded .war file gets picked up, which can easily happen in the current setup.