README.md

An example nix project with C++ and Python

This project consists of 2 simple example applications that model a writer and a reader to/from a postgres database:

The "server" application (C++) listens on TCP socket 1300 and writes into the database whatever it receives. The "webserver" application (Python) is either a script that just prints the database content, or a webserver script (both included) that listens on port 8000 and lists the latest 10 database "messages".

Both applications are relatively simple and stupid as they are mere demonstration vehicles for showing:

• how to setup a reproducible working environment with toolchain and library dependencies etc. with a single command using nix
• how to build the application(s) with different compilers, different library versions, and select dynamic/static linking with a single parametrized command using nix
• how to make the build process reproducible so that it still works in a few years on completely different linux environments, by pinning all versions of the toolchain down using nix
• how to automatically create minimal docker images (so minimal that they contain no base distro image) using nix
• how to run completely automated integration test scenarios in VMs using nix

Dependencies

At compile time, the applications need the following:

• server
  • C++ compiler toolchain (GCC or clang)
  • cmake (meta) build system
  • boost library
  • libpqxx library (C++ postgres interface)
  • gtest unit test framework
• webserver
  • python version 3
  • flask library (webserver)
  • psycopg2 library (postgres interface)

The applications can be compiled (the C++ app) and both be run without nix at all. But in that case you have to install/configure all dependencies yourself - with nix, everything will be handled automagically.

Build

In order to build the server app, do the following:

$ cd $project/server
$ nix-build
# You may now run ./result/bin/messagedb-server

In order to package the python app into a wrapper script that can be called like any binary, do the following:

$ cd $project/python_client
$ nix-build
# You may now run ./result/bin/mdb-webserver

The python wrapper calls python with the dependencies in place, leaving no way to accidentally call it with the wrong python interpreter. (This way of packaging a python app is completely optional, nix does not force you to do so)

nix can be obtained here. There is always the quick and unsafe curl | sh way to install, but there are also instructions on the project website that include binary verification.

Please note that tab completion in the following examples works out of the box on NixOS, while users of other distros have to install it manually. https://github.com/hedning/nix-bash-completions

Build in different configurations

The server app can be compiled in different configurations:

• Compiler (can be easily extended/customized.)
  • GCC 7.3.0 (That is the default choice in the current package list. Can be easily parametrized, of course!)
  • GCC 8.2.0
  • clang 5.0.2
  • clang 6.0.1
  • clang 7.0.0
• boost versions 1.6.3, 1.6.4, 1.6.5, 1.6.6, 1.6.7, 1.6.8
• Binary linkage
  • static (single-file with no further runtime dependencies)
  • dynamic

To select one configuration, run:

$ cd $project
$ nix-build release.nix -A <tab-complete>
integrationtest-mdb-server-clang7-boost166         integrationtest-mdb-server-static-clang8-boost169  mdb-server-gcc8-boost166
integrationtest-mdb-server-clang7-boost167         integrationtest-mdb-server-static-gcc7-boost166    mdb-server-gcc8-boost167
integrationtest-mdb-server-clang7-boost168         integrationtest-mdb-server-static-gcc7-boost167    mdb-server-gcc8-boost168
integrationtest-mdb-server-clang7-boost169         integrationtest-mdb-server-static-gcc7-boost168    mdb-server-gcc8-boost169
integrationtest-mdb-server-clang8-boost166         integrationtest-mdb-server-static-gcc7-boost169    mdb-server-static-clang7-boost166
integrationtest-mdb-server-clang8-boost167         integrationtest-mdb-server-static-gcc8-boost166    mdb-server-static-clang7-boost167
integrationtest-mdb-server-clang8-boost168         integrationtest-mdb-server-static-gcc8-boost167    mdb-server-static-clang7-boost168
integrationtest-mdb-server-clang8-boost169         integrationtest-mdb-server-static-gcc8-boost168    mdb-server-static-clang7-boost169
integrationtest-mdb-server-gcc7-boost166           integrationtest-mdb-server-static-gcc8-boost169    mdb-server-static-clang8-boost166
integrationtest-mdb-server-gcc7-boost167           mdb-server-clang7-boost166                         mdb-server-static-clang8-boost167
integrationtest-mdb-server-gcc7-boost168           mdb-server-clang7-boost167                         mdb-server-static-clang8-boost168
integrationtest-mdb-server-gcc7-boost169           mdb-server-clang7-boost168                         mdb-server-static-clang8-boost169
integrationtest-mdb-server-gcc8-boost166           mdb-server-clang7-boost169                         mdb-server-static-gcc7-boost166
integrationtest-mdb-server-gcc8-boost167           mdb-server-clang8-boost166                         mdb-server-static-gcc7-boost167
integrationtest-mdb-server-gcc8-boost168           mdb-server-clang8-boost167                         mdb-server-static-gcc7-boost168
integrationtest-mdb-server-gcc8-boost169           mdb-server-clang8-boost168                         mdb-server-static-gcc7-boost169
integrationtest-mdb-server-static-clang7-boost166  mdb-server-clang8-boost169                         mdb-server-static-gcc8-boost166
integrationtest-mdb-server-static-clang7-boost167  mdb-server-docker                                  mdb-server-static-gcc8-boost167
integrationtest-mdb-server-static-clang7-boost168  mdb-server-docker-static                           mdb-server-static-gcc8-boost168
integrationtest-mdb-server-static-clang7-boost169  mdb-server-gcc7-boost166                           mdb-server-static-gcc8-boost169
integrationtest-mdb-server-static-clang8-boost166  mdb-server-gcc7-boost167                           mdb-webservice
integrationtest-mdb-server-static-clang8-boost167  mdb-server-gcc7-boost168                           mdb-webservice-docker
integrationtest-mdb-server-static-clang8-boost168  mdb-server-gcc7-boost169

$ nix-build release.nix -A mdb-server-gcc7-boost166
# You may now run ./result/bin/messagedb-server

Generate docker images for both apps

Compiling and installing the docker images is simple. The step of the docker image creation does not even require you to have docker on your system.

server:

$ cd $project
$ nix-build release.nix -A mdb-server-docker
$ du -sh $(readlink -f result)
46M     /nix/store/bnd9gzdgmrvdxd7cdrm7fxr39db8phfk-docker-image-mdb-server.tar.gz

The docker image with the static version of the library is a bit smaller:

$ nix-build release.nix -A  mdb-server-docker-static
$ du -sh $(readlink -f result)
22M     /nix/store/58lpaf0yh6skjdxdd0qfc4vrw9nj1j76-docker-image-mdb-server.tar.gz

Please note that this can of course be further reduced, but then the example would go out of scope and we have a nice starting point already.

The webserver docker image:

$ nix-build release.nix -A mdb-webservice-docker
$ du -sh $(readlink -f result)
60M	/nix/store/l4ipb3p325yaflsdna8h7nijla61vwrz-docker-image-mdb-webservice.tar.gz

What's particularly interesting about these docker images compared to conventionally created images is, that they only contain the run-time dependencies they need. This means that for the statically linked binary, the corresponding docker image, in addition to bash and its run-time dependencies, only contains the binary itself. Likewise, for the dynamically linked binary, and the python webserver, the docker images contain exactly the needed dependencies, e.g. Python 3.xyz for the webserver. In contrast, building a Docker image traditionally, e.g., with a Docker file built upon a Ubuntu/Debian/Fedora base image, would quickly result in an image size around hundreds or MB.

Integration tests

There is only one integration test in this example. It does the following

• Create a complete Linux VM that contains at least:
  • A preconfigured postgres DB (with role and DB for test user)
  • both applications

Running the test does the following:

• Boot the VM
• wait for postgres to start
• then start the server app
• then start the webservice app
• feed some data into the VM via the server app
• check if the DB is in the right state
• check if the webserver's output is correct

This integration test scenario can easily be run with all different build configurations of the server app. The NixOS distribution tests many of its packages with such integration tests. These are however much more complex by running full herds of VMs in virtual networks, to some extent even with the X desktop activated and OCR etc. - out of scope for this example.

To run the integration tests, do:

$ nix-build release.nix -A integrationtest-mdb-server-<tab-complete>
integrationtest-mdb-server-clang7-boost166         integrationtest-mdb-server-gcc7-boost169           integrationtest-mdb-server-static-clang8-boost168
integrationtest-mdb-server-clang7-boost167         integrationtest-mdb-server-gcc8-boost166           integrationtest-mdb-server-static-clang8-boost169
integrationtest-mdb-server-clang7-boost168         integrationtest-mdb-server-gcc8-boost167           integrationtest-mdb-server-static-gcc7-boost166
integrationtest-mdb-server-clang7-boost169         integrationtest-mdb-server-gcc8-boost168           integrationtest-mdb-server-static-gcc7-boost167
integrationtest-mdb-server-clang8-boost166         integrationtest-mdb-server-gcc8-boost169           integrationtest-mdb-server-static-gcc7-boost168
integrationtest-mdb-server-clang8-boost167         integrationtest-mdb-server-static-clang7-boost166  integrationtest-mdb-server-static-gcc7-boost169
integrationtest-mdb-server-clang8-boost168         integrationtest-mdb-server-static-clang7-boost167  integrationtest-mdb-server-static-gcc8-boost166
integrationtest-mdb-server-clang8-boost169         integrationtest-mdb-server-static-clang7-boost168  integrationtest-mdb-server-static-gcc8-boost167
integrationtest-mdb-server-gcc7-boost166           integrationtest-mdb-server-static-clang7-boost169  integrationtest-mdb-server-static-gcc8-boost168
integrationtest-mdb-server-gcc7-boost167           integrationtest-mdb-server-static-clang8-boost166  integrationtest-mdb-server-static-gcc8-boost169
integrationtest-mdb-server-gcc7-boost168           integrationtest-mdb-server-static-clang8-boost167

The result of such integration tests is a HTML/XML log document in the result folder.

Nix CI

NixOS comes with a service for continuous integration, called hydra. It does not look as polished as other CIs and its integration is very github centric (because nix and NixOS source code and package lists are hosted on github), but it is generally extensible with plugins.

A screenshot of a private hydra instance building this project: