Running the worker on Linux

Propedeutical steps

Read Running the worker: overview before installing the worker.

Minimal worker setup

To run the worker on Ubuntu 18.04 install the required packages and clone fishtest with git:

sudo apt update
sudo apt install -y python3 python3-requests git build-essential
git clone https://github.com/glinscott/fishtest.git

Run the worker with:

python3 fishtest/worker/worker.py YOUR_USERNAME YOUR_PASSWORD --concurrency NUMBER_OF_CORES

Worker setup script for Ubuntu

Use this script:

• to create a separate user to run the worker for security reasons
• to setup a python virtual environment
• to clone fishtest with git
• to write a bash script to start the worker in the cli
• (optional) to configure a systemd unit to run the worker as a service; write a bash script to configure the worker concurrency; to enable the auto startup

Write your fishtest credentials and the number of cores to be dedicated to fishtest at the start of the script, and run the script with:

sudo -H bash setup_worker.sh 2>&1 | tee setup_worker.sh.log`

#!/bin/bash
# setup_worker.sh
# to setup a fishtest worker on Ubuntu 18.04, simply run: 
# sudo -H bash setup_worker.sh 2>&1 | tee setup_worker.sh.log

# replace YOUR_USERNAME and YOUR_PASSWORD with your fishtest username and password
# and NUMBER_OF_CORES with the number of cores to be contributed
# the double quotes deal with symbols in username or password: don't delete them
usr_name="YOUR_USERNAME"
usr_pwd="YOUR_PASSWORD"
n_cores="NUMBER_OF_CORES"


# install required packages
apt update && apt full-upgrade -y && apt autoremove -y && apt clean
apt install -y python3 python3-venv git build-essential 

# new linux account used to run the worker
worker_user='fishtest'
# create user for fishtest
useradd -m -s /bin/bash ${worker_user}

# add the bash variable for the python virtual env
sudo -i -u ${worker_user} << 'EOF'
echo export VENV=${HOME}/fishtest/worker/env >> .profile
EOF

# download fishtest
sudo -i -u ${worker_user} << EOF
git clone --single-branch --branch master https://github.com/glinscott/fishtest.git
cd fishtest
git config user.email "you@example.com"
git config user.name "your_name"
EOF

# fishtest worker setup and first start to write the "fishtest.cfg" configuration file
# this uses the fish.exit file to stop the worker after writing the configuration file 
sudo -i -u ${worker_user} << EOF
python3 -m venv \${VENV}
\${VENV}/bin/python3 -m pip install --upgrade pip setuptools wheel
\${VENV}/bin/python3 -m pip install requests

arch_cpu=x86-64

if [ "$(g++ -Q -march=native --help=target | grep mbmi2 | grep enabled)" ] ; then
  if [ "$(g++ -Q -march=native --help=target | grep march | grep 'znver[12]')" ] ; then
    arch_cpu=x86-64-modern
  else
    arch_cpu=x86-64-bmi2
  fi
elif [ "$(g++ -Q -march=native --help=target | grep mpopcnt | grep enabled)" ] ; then
  arch_cpu=x86-64-modern
fi
echo "CXXFLAGS='-march=native' make profile-build -j ARCH=\${arch_cpu} COMP=gcc" > \${HOME}/fishtest/worker/custom_make.txt

touch \${HOME}/fishtest/worker/fish.exit
\${VENV}/bin/python3 \${HOME}/fishtest/worker/worker.py --concurrency ${n_cores} ${usr_name} ${usr_pwd}
rm -f \${HOME}/fishtest/worker/fish.exit
EOF

# write a script to start the worker from a cli with the option to change the number of cores
cat << EOF0 > worker_start.sh
#!/bin/bash
if [ \${#} -gt 1 ]; then
  echo "usage: bash \${0} [<n_cores>]"
  echo "       <n_cores>: new number of cores (optional)"
  exit 1
elif [ \${#} -eq 1 ]; then
  new_param="--concurrency \${1}"
fi

sudo -i -u ${worker_user} << EOF
\\\${VENV}/bin/python3 \\\${HOME}/fishtest/worker/worker.py \${new_param}
EOF
EOF0
chown ${SUDO_USER}:${SUDO_USER} worker_start.sh

echo
echo "setup fishtest-worker as a service"
echo "stop here with windows subsystem for linux"
read -p "press <Enter> to continue or <CTRL+C> to exit ..."

# install fishtest-worker as systemd service
# start/stop the worker with:
# sudo systemctl start fishtest-worker
# sudo systemctl stop fishtest-worker
# check the log with:
# sudo journalctl -u fishtest-worker.service
# the service uses the worker configuration file "fishtest.cfg"

# get the worker_user $HOME
worker_user_home=$(sudo -i -u ${worker_user} << 'EOF'
echo ${HOME}
EOF
)

cat << EOF > /etc/systemd/system/fishtest-worker.service
[Unit]
Description=Fishtest worker
After=multi-user.target

[Service]
Type=simple
StandardOutput=file:${worker_user_home}/fishtest/worker/worker.log
StandardError=inherit
ExecStart=${worker_user_home}/fishtest/worker/env/bin/python3 ${worker_user_home}/fishtest/worker/worker.py
User=${worker_user}
WorkingDirectory=${worker_user_home}/fishtest/worker

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload

# write a script to change the default number of cores
# useful for the fishtest-worker service
cat << EOF0 > worker_config.sh
#!/bin/bash
if [ \${#} -ne 1 ]; then
  echo "usage: \${0} <n_cores>"
  echo "       <n_cores>: new number of cores (mandatory)"
  exit 1
fi

sudo -i -u ${worker_user} << EOF
touch \\\${HOME}/fishtest/worker/fish.exit
\\\${VENV}/bin/python3 \\\${HOME}/fishtest/worker/worker.py --concurrency \${1}
rm -f \\\${HOME}/fishtest/worker/fish.exit
EOF
EOF0
chown ${SUDO_USER}:${SUDO_USER} worker_config.sh

echo
echo "start fishtest-worker service"
read -p "press <Enter> to continue or <CTRL+C> to exit ..."
systemctl start fishtest-worker.service

echo
echo "enable fishtest-worker service auto start"
read -p "press <Enter> to continue or <CTRL+C> to exit ..."
systemctl enable fishtest-worker.service

How to choose the number of cores

Option --concurrency refers to the number of cores dedicated to the worker, the safest max setting is to use the number of physical cores leaving one core for the OS. If you do not know the number of physical cores on your system, execute:

lscpu | awk '/^Core\(s\) per socket:/ {cores=$NF}; /^Socket\(s\):/ {sockets=$NF}; END{print cores*sockets}'

if lscpu is not installed a portable way to report the total (physical + hyperthreaded) core is

getconf _NPROCESSORS_ONLN

Running the worker on RHEL/CentOS

If the default version of GCC should not support C++11, you can install the developer toolset, which adds a newer version of GCC. It installs side-by-side seamlessly, simply follow the instruction for your OS:

https://www.softwarecollections.org/en/scls/rhscl/devtoolset-7/

Then, create a file called launch_worker.sh, with these commands:

#!/bin/bash
# usage: fishtest.sh  [<n_cores> <username> <password>]
# <n_cores>: number of cores to be used in fishtest. Suggested max value = n. physical cores-1
# <username>: username on fishtest (to be enclosed in quote if contains special character)
# <password>: password on fishtest (to be enclosed in quote if contains special character)
# The three parameters are mandatory only for the first execution

if [ $# -gt 0 ]; then
  sudo -i -u fishtest << EOF
source scl_source enable devtoolset-7
python3 fishtest-master/worker/worker.py --concurrency $1 "$2" "$3"
EOF
else
  sudo -i -u fishtest << EOF
source scl_source enable devtoolset-7
python3 fishtest-master/worker/worker.py
EOF
fi

And use that to launch the worker.

Install several GCC versions on Ubuntu

Launch the following script to have several versions of GCC on Ubuntu alongside the default one: install some GCC versions using the Ubuntu Toolchain Uploads repository and manage the different GCC versions using update-alternatives. This works also for Windows Subsystem for Linux.

#!/bin/bash
# launch this script using "sudo"

# install the default building tools
apt update
apt install -y build-essential software-properties-common

# add the repository "ubuntu-toolchain-r/test"
apt update
add-apt-repository -y ppa:ubuntu-toolchain-r/test
apt update

# install other GCC versions
apt install -y gcc-5 g++-5
apt install -y gcc-6 g++-6
apt install -y gcc-7 g++-7

# configure the alternatives for gcc and g++, setting a higher priority to a newer version (ymmv)
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 50 --slave /usr/bin/g++ g++ /usr/bin/g++-5
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-6 60 --slave /usr/bin/g++ g++ /usr/bin/g++-6
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 70 --slave /usr/bin/g++ g++ /usr/bin/g++-7

# check or change the alternatives configuration
update-alternatives --config gcc