Move all live databases into cache and only have snapshots in lib / backup

[ahayzen]

As sometimes we get the following when running rsync

file has vanished: "/var/lib/docker-compose-runner/bitwarden/config/vault.db-journal"

This also reduces the amount of random binary data in the backups.

[ahayzen]

Related to this we had a backup failure of a machine because of live databases

vm-test-run-lab-bitwarden-test> lab # [  105.714577] backup-machines-start[1871]: sending incremental file list
vm-test-run-lab-bitwarden-test> lab # [  105.718901] backup-machines-start[1871]: file has vanished: "/var/lib/docker-compose-runner/bitwarden/config/vault.db-journal"
vm-test-run-lab-bitwarden-test> lab # [  105.724930] backup-machines-start[1871]: ./
vm-test-run-lab-bitwarden-test> lab # [  105.726136] backup-machines-start[1871]: bitwarden/
vm-test-run-lab-bitwarden-test> lab # [  105.726528] backup-machines-start[1871]: bitwarden/config/
vm-test-run-lab-bitwarden-test> lab # [  105.731257] backup-machines-start[1871]: bitwarden/config/identity.pfx
vm-test-run-lab-bitwarden-test> lab # [  105.733585] backup-machines-start[1871]:           4.23K 100%    0.00kB/s    0:00:00            4.23K 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=9/13)
vm-test-run-lab-bitwarden-test> lab # [  105.735295] backup-machines-start[1871]: bitwarden/config/vault-snapshot-2.db
vm-test-run-lab-bitwarden-test> lab # [  105.736730] backup-machines-start[1871]:          12.29K 100%   11.72MB/s    0:00:00           12.29K 100%   11.72MB/s    0:00:00 (xfr#2, to-chk=8/13)
vm-test-run-lab-bitwarden-test> lab # [  105.738586] backup-machines-start[1871]: bitwarden/config/vault.db
vm-test-run-lab-bitwarden-test> lab # [  105.750292] backup-machines-start[1871]:          32.77K   6%   31.25MB/s    0:00:00          532.48K 100%   19.53MB/s    0:00:00 (xfr#3, to-chk=7/13)
vm-test-run-lab-bitwarden-test> lab # [  105.751570] backup-machines-start[1871]: bitwarden/config/data-protection/
vm-test-run-lab-bitwarden-test> lab # [  105.757090] backup-machines-start[1871]: bitwarden/config/data-protection/key-7e18ad2d-0583-4eba-ba8c-7d064f88fc64.xml
vm-test-run-lab-bitwarden-test> lab # [  105.757822] backup-machines-start[1871]:           1.00K 100%   37.56kB/s    0:00:00            1.00K 100%   37.56kB/s    0:00:00 (xfr#4, to-chk=4/13)
vm-test-run-lab-bitwarden-test> lab # [  105.763508] backup-machines-start[1871]: bitwarden/config/logs/
vm-test-run-lab-bitwarden-test> lab # [  105.766327] backup-machines-start[1871]: bitwarden/config/logs/admin.log
vm-test-run-lab-bitwarden-test> lab # [  105.769326] backup-machines-start[1871]:           5.87K 100%  150.85kB/s    0:00:00            5.87K 100%  150.85kB/s    0:00:00 (xfr#5, to-chk=3/13)
vm-test-run-lab-bitwarden-test> lab # [  105.769915] backup-machines-start[1871]: bitwarden/config/logs/api.log
vm-test-run-lab-bitwarden-test> lab # [  105.776230] backup-machines-start[1871]:              50 100%    1.28kB/s    0:00:00               50 100%    1.28kB/s    0:00:00 (xfr#6, to-chk=2/13)
vm-test-run-lab-bitwarden-test> lab # [  105.777306] backup-machines-start[1871]: bitwarden/config/logs/identity.log
vm-test-run-lab-bitwarden-test> lab # [  105.778568] backup-machines-start[1871]:              55 100%    1.41kB/s    0:00:00               55 100%    1.41kB/s    0:00:00 (xfr#7, to-chk=1/13)
vm-test-run-lab-bitwarden-test> lab # [  105.779575] backup-machines-start[1871]: bitwarden/config/logs/notifications.log
vm-test-run-lab-bitwarden-test> lab # [  105.780987] backup-machines-start[1871]:             604 100%   15.52kB/s    0:00:00              604 100%   15.52kB/s    0:00:00 (xfr#8, to-chk=0/13)
vm-test-run-lab-bitwarden-test> lab # [  105.915778] backup-machines-start[1871]: rsync warning: some files vanished before they could be transferred (code 24) at main.c(1338) [sender=3.3.0]
vm-test-run-lab-bitwarden-test> lab # [  105.925517] sudo[1869]: pam_unix(sudo:session): session closed for user root
vm-test-run-lab-bitwarden-test> lab # [  105.937175] systemd[1]: backup-machines.service: Main process exited, code=exited, status=24/n/a
vm-test-run-lab-bitwarden-test> lab # [  105.943531] systemd[1]: backup-machines.service: Failed with result 'exit-code'.
vm-test-run-lab-bitwarden-test> lab # Job for backup-machines.service failed because the control process exited with error code.
vm-test-run-lab-bitwarden-test> lab # [  105.956928] systemd[1]: Failed to start backup-machines.service.

[ahayzen]

Change our data structure so

/mnt/data/ # for static structured data
/var/cache/  # for live databases
/var/lib/  # for snapshots of databases / auto backups from services

Then backup of machines takes /var/lib and places into /mnt/data/backup.

/mnt/data/app/ has any data that needs to be saved but is not as structured eg immich's sync data as that cannot be put into /mnt/data/camera/<devicename>/<year>/*.jpg due to storage template limitations.

[ahayzen]

Note how jellyfin currently has no snapshot of databases