Failsafe node red setup

For controlling critical parts of the home like the central heating, some protection against failures is required. This page describes setting up a cluster with two Raspberry Pis, where one of them runs the home automation while the other Raspberry Pi watches the first one and can take over the home automation functionality if the first Raspberry Pi fails.

Both Raspberry Pis are model 4B with 4GB RAM. Each of them is connected to a separate 128GB USB3 SSD, to the Raspberry Pi USB C Power Supply, and to a LAN switch via Ethernet cables. The Pis boot from 16GB A1 SD cards whith Raspbian Lite. The Power supplies are plugged into Tasmota-enabled power plugs to enable power cycling non-responsive Raspberry Pis. A 12V fan cools the whole setup with low fan speed (powered by the 5V Header Pin of one of the Raspberry Pis).

Before first bootup

For each Raspberry Pi,

• Mount SD card boot partition on PC
  • create empty file "ssh" on that partition,
  • prevent SSD from using UAS by editing cmdlineline.txt on this partition as described here: Raspberry Pi Forum: If you have a Raspberry Pi 4 and are getting bad speeds transferring data to/from USB3.0 SSDs, read this. My cmdline.txt starts like this after the change: usb-storage.quirks=152d:0579:u .
• Mount SD card root partition, change host name in /etc/hostname and /etc/hosts to recognizable host names.
• Attach USB3 SSD to PC, create 8GB linux-swap partition and 120GB ext4 data partition on the SSD.

I tried running rootfs directly from the SSD following https://gist.github.com/lucabelluccini/9a11c48dcf1d627bbcbd8213f6de3eae but did not succeed. The Raspberry Pis would not show up on the network and I do not have a micro HDMI cable to debug the boot process. Therefore I am keeping the rootfs on SD and will move some directories to the SSD later.

Configure base system

Switch on power. The devices appear in the router's device list.

• ssh into each of them
  • passwd
  • ssh-keygen
  • cat >.ssh/authorized_keys
  • Set PasswordAuthentication to no in /etc/ssh/sshd_config.
  • sudo mkdir /data
  • modify /etc/fstab to mount the SSD to /data and to enable the swap partition (adapt the identifiers! consult blkid):

LABEL=data            /data           ext4    defaults,noatime  0       1
PARTUUID=9473b04f-01  none            swap    sw                0       0

• Reboot and ssh again into each of them
• To move /var and /home to /data,
  • Find all running services and terminate most of them so that hopefully nothing writes to /var:
    • sudo systemctl list-units --type service --state running
    • sudo systemctl stop alsa-state.service avahi-daemon.service bluetooth.service cron.service dbus.service dhcpcd.service getty@tty1.service hciuart.service rng-tools.service rsyslog.service ssh.service systemd-journald.service systemd-logind.service systemd-timesyncd.service systemd-udevd.service triggerhappy.service wpa_supplicant.service
    • sudo swapoff /var/swap
    • sudo mv /var /data/var
    • sudo ln -s /data/var/ /var
    • sudo mv /home /data/home
    • sudo ln -s /data/home/ /home
• Reboot and ssh again into each of them
  • Set time zone, wifi zone, and memory split in raspi-config:
    • sudo raspi-config
  • Update installed software:
    • sudo apt update
    • sudo apt install aptitude
    • sudo aptitude
  • Avoid relying on the DHCP server as a single point of failure by configuring static IP addresses outside the DHCP range:
    • Uncomment the static IP address example in file /etc/dhcpcd.conf and adapt.

After a final reboot, the Raspberry Pis can now be reached through their static IP addresses. The IP address of the MQTT server and of any web interfaces to control the home should not change when the reserve Raspberry Pi takes over because the main Raspberry Pi fails. Linux allows to add additional IP addresses to a network card:

sudo ip address add 192.168.110.252 dev eth0:1

would add that address to the Ethernet network interface. That IP address would need to be added by the reserve Raspberry Pi when it takes over because the main Raspberry Pi has failed. It can be statically assigned to the ethernet interface on the main Raspberry Pi.

On the main Raspberry Pi, I am extending /etc/rc.local to assign the interface (just add the line above before the line containing exit, without sudo).

Edit /etc/systemd/timesyncd.conf, set NTP to point to the home router.

Set up node-red

• sudo apt install git mosquitto mosquitto-clients nodered influxdb
• Set up password protection for MQTT:
  • mosquitto_passwd -c mqttpasswd username
  • sudo cp mqttpasswd /etc/mosquitto/
  • Extend /etc/mosquitto/mosquitto.conf with

allow_anonymous false
password_file /etc/mosquitto/mqttpasswd

• Restart mosquitto:
  • sudo systemctl restart mosquitto
• Start node-red:
  • sudo systemctl enable nodered
  • sudo systemctl start nodered
• Enable node-red projects:
  • Edit file /home/pi/.node-red/settings.js, search for projects, change the enabled setting from false to true.
  • Restart node-red.