Benchmarking Getting Started

This is the getting started guide for __benchmarking__. 

If you're interested in development please visit the getting started guide for development.

You will need three computers (or virtual machines) networked together to fill the three benchmark roles of database server, load generation server, and web framework server.

Note: If you're setting up the infrastructures for the benchmarking using cloud services, the 3 computers / virtual machines should be connected via private network and any public access should be restricted (or only open necessary ports for admin use, such as SSH and only via key-based authentication) to ensure the accuracy of the benchmarking results as well as your own network security.

Linux Server Setup

When setting up a Linux server for benchmarking, you can follow the same steps as setting up a Linux development environment.

Note: If testing a pull request or doing development, it is usually adequate to only use one computer. In that case, your server, client, and database IPs will be 127.0.0.1.

Benchmark Suite Deployment

Required Host Roles

• App (aka web framework) Server
  • Runs the web framework containers
  • Operating system: Ubuntu Server 16.04 LTS 64-bit
• Database Server
  • Runs database containers (e.g. MySQL, PostgreSQL, MongoDB, etc).
  • Operating system: Ubuntu Server 16.04 LTS 64-bit
• Client (aka load generation) Server
  • Runs the benchmark client container
  • Operating system: Ubuntu Server 16.04 LTS 64-bit

Manual Deployment

Prerequisites

Before you get started, the following are the steps you must take on each machine (App, Database, Client) to run the benchmarking suite:

1. Docker installed (you can verify this via docker run hello-world)
2. /lib/systemd/system/docker.service needs ExecStart to have an additional flag: -H [machine's ip]:2375
3. sudo systemctl daemon-reload
4. sudo service docker restart (you can verify this via sudo service docker status and see the newly added -H [machine's ip]:2375 flag)

Since the tests can run for several hours, it helps to set everything up so that once the tests are running, you can leave the machines unattended and don't need to be around to enter ssh or sudo passwords.

Running the Benchmarking Suite

Let us assume make some assumptions for the example:

1. You cloned the repository into the home directory of user techempower
2. Your App machine has the IP 10.0.0.1
3. Your Database machine has the IP 10.0.0.2
4. Your Client machine has the IP 10.0.0.3

The following command, given the prerequisites here, would run a benchmark for all the tests in the codebase.

docker run \
  --network=host \
  --mount type=bind,source=/home/techempower/FrameworkBenchmarks,target=/FrameworkBenchmarks \
  techempower/tfb \
  --server-host 10.0.0.1 \
  --database-host 10.0.0.2 \
  --client-host 10.0.0.3 \
  --network-mode host \
  --quiet