Zaloha2.sh

Zaloha2.sh is the next step in development of Zaloha.sh, the small and simple directory synchronizer (a BASH script).

Why a new repository and program name?

• Some design changes break backward compatibility (see full list of changes below).
• To keep the original Zaloha.sh intact for the conservative user.
• All functionalities of the original Zaloha.sh are contained in Zaloha2.sh too.
• For new deployments use Zaloha2.sh.

Documentation

Full documentation is available both online as well as inside of Zaloha2.sh.

All other documentation and explanatory items from the original Zaloha.sh repository are valid for Zaloha2.sh too (Article, Interactive Flowchart, Usage Examples, Screenshot, Simple Demo, Performance Data).

New feature 1: The Remote Backup Mode

In the Remote Backup Mode, the source directory is available locally, and the backup directory is on a remote backup host that can be reached via SSH/SCP. This mode is activated by the option --backupUserHost.

The Remote Backup Mode explained in eight sentences

• The FIND scan of the source directory occurs locally (no change against Zaloha.sh).
• The script for FIND scan of the backup directory (file 220) is prepared locally.
• The script is then copied to the remote backup host via SCP and executed via SSH.
• The obtained CSV metadata (file 320) is copied back from the remote backup host via SCP.
• The CSV metadata is compared by a sequence of sorts and AWK processing steps (occurs locally).
• The results (= prepared synchronization actions) are presented to the user for confirmation.
• If the user confirms, the synchronization actions are executed (via SSH and SCP).
• A non-interactive regime is available as well.

Usage Example of the Remote Backup Mode

# Establish the SSH master connection
ssh -nNf -o ControlMaster=yes -o ControlPath='~/.ssh/cm-%r@%h:%p' 'user@backuphost'

# Run Zaloha2.sh
./Zaloha2.sh --sourceDir="test_source_local"     \
             --backupDir="test_backup_remote"    \
             --backupUserHost='user@backuphost'  \
             --sshOptions='-o ControlMaster=no -o ControlPath=~/.ssh/cm-%r@%h:%p'     \
             --scpOptions='-o ControlMaster=no -o ControlPath=~/.ssh/cm-%r@%h:%p -T'

# Terminate the SSH master connection
ssh -O exit -o ControlPath='~/.ssh/cm-%r@%h:%p' 'user@backuphost'

New feature 2: The Remote Source Mode

In the Remote Source Mode, the source directory is on a remote source host that can be reached via SSH/SCP, and the backup directory is available locally. This mode is activated by the option --sourceUserHost.

The Remote Source Mode explained in eight sentences

• The script for FIND scan of the source directory (file 210) is prepared locally.
• The script is then copied to the remote source host via SCP and executed via SSH.
• The obtained CSV metadata (file 310) is copied back from the source host via SCP.
• The FIND scan of the backup directory occurs locally (no change against Zaloha.sh).
• The CSV metadata is compared by a sequence of sorts and AWK processing steps (occurs locally).
• The results (= prepared synchronization actions) are presented to the user for confirmation.
• If the user confirms, the synchronization actions are executed.
• A non-interactive regime is available as well.

Usage Example of the Remote Source Mode

# Establish the SSH master connection
ssh -nNf -o ControlMaster=yes -o ControlPath='~/.ssh/cm-%r@%h:%p' 'user@sourcehost'

# Run Zaloha2.sh
./Zaloha2.sh --sourceDir="test_source_remote"     \
             --backupDir="test_backup_local"    \
             --sourceUserHost='user@sourcehost'  \
             --sshOptions='-o ControlMaster=no -o ControlPath=~/.ssh/cm-%r@%h:%p'     \
             --scpOptions='-o ControlMaster=no -o ControlPath=~/.ssh/cm-%r@%h:%p -T'

# Terminate the SSH master connection
ssh -O exit -o ControlPath='~/.ssh/cm-%r@%h:%p' 'user@sourcehost'

New feature 3: Compare the contents of files via SHA-256 hashes

The option --byteByByte cannot be used in the Remote Source and Remote Backup Modes, because the CMP command needs local access to both compared files.

For the remote modes, a new option --sha256 has been introduced. This option causes both FIND scans to additionally invoke SHA256SUM on each encountered file. Note that the SHA-256 hashes of the files are calculated on the hosts where the files are located. Also no contents of files are transferred over the network, just the SHA-256 hashes. The SHA-256 hashes are then compared to detect files that appear identical but their contents differ.

Performance tuning in the Remote Backup and Remote Source Modes

• The option --noRestore switches off creation of the restore scripts. If you do not need the restore scripts, use this option to shorten the analysis phase.
• The option --mawk instructs Zaloha to use mawk instead of (usually) GNU awk. mawk is an AWK implementation based on a bytecode interpreter and is significantly faster. To utilize mawk, it must be installed on the local system.
• The option --findParallel instructs Zaloha to run the FIND scans of the source and backup directories in parallel. As these scans run on different hosts, this will further save time.
• Last, scp can be tuned for higher speed by choosing suitable ciphers and compression levels. See scp documentation for details. Use Zaloha's --scpOptions option to pass additional command-line options to scp commands invoked by Zaloha. Faster scp will shorten both the analysis phase (faster copying of CSV metadata over the network) as well as the actual execution phase.
• Eventually use the High performance SSH/SCP – HPN-SSH, that removes the performance limitation due to statically defined internal flow control buffers.

Obtain Zaloha2.sh

The simplest way: Under the green button "Code" above, choose "Download ZIP". From the downloaded ZIP archive, extract Zaloha2.sh and make it executable (chmod u+x Zaloha2.sh).

Add-on script Zaloha2_Snapshot.sh

The script for hardlink-based snapshots has been adapted to Zaloha2.sh: Zaloha2_Snapshot. This allows to create Time Machine-like backup solutions.

Full list of changes and new features of Zaloha2.sh

Zaloha.sh	Zaloha2.sh
	New option --sourceUserHost to activate the Remote Source Mode via SSH/SCP
	New option --backupUserHost to activate the Remote Backup Mode via SSH/SCP
	New option --sshOptions to pass additional command-line options to SSH in the remote modes
	New option --scpOptions to pass additional command-line options to SCP in the remote modes
	New option --findParallel to run the local and remote FIND scans in parallel in the remote modes
Option --metaDir	In Remote Backup Mode: allows to place the Zaloha metadata directory on the remote backup host to a different location than the default.
	New option --metaDirTemp: In the remote modes, Zaloha needs a temporary Metadata directory too. This option allows to place it to a different location than the default.
Shellscript 610	In Remote Backup Mode: executed on the remote side
Shellscript 620	Split to 621 (pre-copy), 622 (copy), 623 (post-copy). In Remote Backup Mode: 621 and 623 are executed on the remote side. In both remote modes, 622 contains SCP commands instead of CP commands.
Shellscript 630	Split to 631 (pre-copy), 632 (copy), 633 (post-copy). In Remote Source Mode: 631 and 633 are executed on the remote side. In both remote modes, 632 contains SCP commands instead of CP commands.
Shellscript 640	In Remote Backup Mode: executed on the remote side
Shellscript 650	Split to 651 (pre-copy), 652 (copy), 653 (post-copy). In Remote Backup Mode: 651 and 653 are executed on the remote side. In both remote modes, 652 contains SCP commands instead of CP commands.
Restore script 810	In the remote modes: contains SCP commands instead of CP commands
	New option --sha256 for comparing the contents of files via SHA-256 hashes
CSV data model of 16 columns	Extended to 17 columns to accommodate the SHA-256 hashes in new separate column 13 (original columns 13+14+15+16 shifted to 14+15+16+17)
	New check for falsely detected hardlinks: SHA-256 hash differs
Option --hLinks	Renamed to --detectHLinksS (more descriptive option name)
Option --touch	Renamed to --extraTouch (more descriptive option name)
Option --noExec1Hdr	Renamed to --no610Hdr
Option --noExec2Hdr	Replaced by finer-grained options --no621Hdr, --no622Hdr and --no623Hdr
Option --noExec3Hdr	Replaced by finer-grained options --no631Hdr, --no632Hdr and --no633Hdr
Option --noExec4Hdr	Renamed to --no640Hdr
Option --noExec5Hdr	Replaced by finer-grained options --no651Hdr, --no652Hdr and --no653Hdr
	New Sanity Check for column 6 not alphanumeric
	More stringent directories hierarchy check
	More tolerant check of zero modification times of files
	Minor code improvements and optimizations
Code size 76 kB	Code size 105 kB
Docu size 78 kB	Docu size 92 kB

License

MIT License