Bupstash v0.12.0

We are proud to bring you the next iteration of bupstash, this update contains a large amount of
performance work as well as important bug fixes.

As an example of the potential performance improvements, here is a synthetic benchmark snapshotting the linux
kernel source code on a development machine:

Benchmark 1: bupstash-0.11.1 put --no-send-log /tmp/linux
  Time (mean ± σ):      5.885 s ±  0.091 s    [User: 6.099 s, System: 1.766 s]
  Range (min … max):    5.701 s …  6.016 s    10 runs
 
Benchmark 2: bupstash-0.12.0 put --no-send-log /tmp/linux
  Time (mean ± σ):      1.884 s ±  0.014 s    [User: 7.334 s, System: 1.340 s]
  Range (min … max):    1.862 s …  1.908 s    10 runs
 
Summary
  'bupstash-0.12.0 put --no-send-log /tmp/linux' ran
    3.12 ± 0.05 times faster than 'bupstash put --no-send-log /tmp/linux'

You read that right, a 3x speed improvement! It is also important to keep in mind bupstash is often limited
by disk, network and send-log speeds rather than the cpu and ram performance that this benchmark measures.

New features

• The deduplication rolling hash algorithm has been improved and is now 30 to 50 percent faster.
• Those using a nightly rust compiler can enable SIMD (even faster) rolling hash implementations.
• A multithreaded bupstash put implementation has been added that can read, hash, compress, encrypt files in parallel.
• Bupstash diff and bupstash restore can now use multiple threads when computing changes on the local disk.
• New flags have been added to bupstash put and other commands to tune how bupstash uses cpu threads.

Notable Bug fixes

• A bug that caused bupstash serve to not detect io errors in certain situations has been fixed.
  Only the server side needs to be updated, so users of bupstash.io managed repositories do not need to take
  further action.

Incompatibilities

• It is likely your repositories will grow in size if they contain data chunks from previous
  versions of bupstash, this can be fixed by cycling older data out over time, or recreating your backups.
• Because the chunks generated by bupstash will change, expect increased bandwidth usage until
  the new chunks are uploaded.
• bupstash is now biased to produce 2 MiB data chunks on average instead of 1 MiB chunks (in the future this may become configurable).
• This version of bupstash produces chunks too large for previous versions of bupstash serve.

Supporting bupstash

Bupstash.io managed repositories are in open beta and anyone can create an account.
If you enjoy bupstash then please consider creating a managed repository at https://bupstash.io/managed.html
to support the project.

Another great way to help the project is to just tell your friends to give bupstash a try.

Bupstash v0.11.1

This release is a primarily a bug fix and maintenance release in preparation for an upcoming bupstash 1.0 release.
While this release tweaks the repository format, the network protocol remains unchanged and interoperable.

New features

• Added a new --ignore-permission-errors option to bupstash put.
• Added a new advanced environment variable BUPSTASH_KEEP_WAL=1 for
  'bupstash serve' See WAL section below.

Notable Bug fixes

• Fixed some cases where non utf8 paths caused bupstash to reject files.
• Fixed a bug that caused using --pick on directories larger 15-20GiB to fail.

Incompatibilities

• The repository format has changed, repositories will be automatically migrated
  by bupstash and cannot be used with older versions of bupstash after migration
  (access across the network is still compatible).
• Bupstash now performs upload checkpoints based on elapsed time instead of upload byte count.
• BUPSTASH_CHECKPOINT_BYTES has been replaced by BUPSTASH_CHECKPOINT_SECONDS.

WAL

Bupstash now writes all changes it makes to the repository to a WAL (write ahead log)
file for crash recovery purposes. If the 'bupstash serve' command is run with the new
BUPSTASH_KEEP_WAL=1 environment variable set, then WAL entries are accumulated in the
repository wal directory instead of being removed when they are no longer needed.

The main purpose of the wal directory is incremental replication of the repository
metadata and history. Future bupstash versions may include tools to interact with
these WAL files to do operations like point in time recovery or recovery from external storage engines.

For now this feature is only for advanced users with specialist use cases.

Signed releases

From this point on git tags and downloads will be signed by a gpg key from developers at the
bupstash.io domain. The PGP signing keys can be found at https://bupstash.io/doc/man/bupstash-authors.html
or via the source code repository itself in the file doc/man/bupstash-authors.7.md.

Supporting bupstash

Bupstash.io managed repositories now are in open beta and anyone can create an account.
If you enjoy bupstash then please consider creating a managed repository at https://bupstash.io/managed.html
to support the project.

One handy way of using bupstash.io is in conjunction with bupstash sync and an external drive to
keep a local and remote copy of your backups for extra assurance.

Another great way to help the project is to just tell your friends to give bupstash a try.

Bupstash v0.11.0

This release adds new commands, improves repository compression, reduces bandwidth usage, and improves support for paths that are invalid unicode.

New features:

• Added the new bupstash sync command for efficiently transferring data between repositories.
• Added the new --allow-sync permission to bupstash serve.
• Added the new bupstash exec-with-locks command for running commands with exclusive repository write access.
• Added the zstandard compression algorithm, decreasing repository sizes 30 percent in some benchmarks.
• Added an optional --compression flag has been added to specify the compression algorithm and level during puts.

Improvements:

• The put and restore command can now preserve sparse file holes, improving the accuracy of backups.
• The restore command now restores mtime timestamps on restored files.
• FreeBSD support has been improved.

Notable Bug fixes:

• Support for paths that are not valid unicode but are still valid unix paths has been improved.
• * no longer matches / in put exclusions.

Incompatibilities:

• Repositories will be automatically migrated by bupstash and cannot be used with older versions of bupstash after migration.
• The bupstash remote protocol version has been incremented to support new features and does not interoperate with older bupstash versions.
• The --no-compression flag has been removed (use --compression=none instead).
• The list-contents command json output now uses 'null' instead of undefined values for missing fields.
• The list-contents command json output will emit paths as byte arrays if they are not representable as json strings.
• The send log now stores chunk addresses in more situations so will grow in size in many cases.
• Put exclusions have slightly different matching semantics and should be reviewed.

Bupstash v0.10.3

Bugfix release.

• Fix bug causing 'bupstash serve' remote error messages to be unreliably displayed.
• Fix bug where 'bupstash put' ignored a user specified name in certain situations.
• Minor fixes to documentation.

Bupstash v0.10.2

Bupstash v0.10.2

Hot fix release, the bugs fixed caused inconvenient error messages, but do not
risk any user data.

Bug fixes:

• Fix a bug that caused bupstash get --pick to succeed, but print an error message.
• Fix a bug that caused bupstash restore --pick to sometimes fail with an error message.

Bupstash v0.10.1

Bupstash v0.10.1

This release[1] adds new features, performance improvements, and also addresses
some flaws in the repository format. Be sure to read the incompatility
notes to understand the limitations the upgrade adds to snapshots created
by older versions of bupstash.

New features:

• The bupstash restore command for efficiently restoring snapshots has been added.
• The bupstash put command now takes advantage of more parallelism while saving directories.
• The bupstash put command now supports verbose logging via --verbose and --print-file-actions.
• The bupstash diff command now supports diffs between snapshots and the local filesystem.
• The bupstash list-contents and bupstash diff commands are faster on huge snapshots.
• The bupstash get command does improved validation of remote data as it is fetched.
• The client side send log and query caches are now protected against silent corruption via checksums.
• The client now does more validation on item ids.
• Commands that show a progress bar now have a --no-progress flag, with --quiet remaining a catch all shorthand.

Bug fixes:

• Deduplication has been improved by addressing a flaw in the repository format.
• A bug in the diff command that could lead to bad diff results has been fixed.
• A bug preventing snapshots from having a custom name has been fixed.

Incompatibilities:

• To support the new bupstash restore command, the network protocol has changed. Clients and
  servers must both upgrade bupstash in order to connect to eachother.
• Due to the repository format tweaks, snapshots created with older versions of bupstash
  no longer support bupstash diff, the --pick option, and also do not support the new bupstash restore
  command. Old format snapshots are still accessible via bupstash get.
• The send log and query cache format has changed, so these caches will be invalidated automatically.
• The new version of bupstash will automatically upgrade your repository when run. After this upgrade
  occurs, older versions of bupstash will not be able to access the repository.

[1] v0.10.1 is the same as 0.10.0, but has fixed compilation on OpenBSD and fixed rust lints for the newly released rust version.

Bupstash v0.10.0

Bupstash v0.10.0

This release adds new features, performance improvements, and also addresses
some flaws in the repository format. Be sure to read the incompatility
notes to understand the limitations the upgrade adds to snapshots created
by older versions of bupstash.

New features:

• The bupstash restore command for efficiently restoring snapshots has been added.
• The bupstash put command now takes advantage of more parallelism while saving directories.
• The bupstash put command now supports verbose logging via --verbose and --print-file-actions.
• The bupstash diff command now supports diffs between snapshots and the local filesystem.
• The bupstash list-contents and bupstash diff commands are faster on huge snapshots.
• The bupstash get command does improved validation of remote data as it is fetched.
• The client side send log and query caches are now protected against silent corruption via checksums.
• The client now does more validation on item ids.
• Commands that show a progress bar now have a --no-progress flag, with --quiet remaining a catch all shorthand.

Bug fixes:

• Deduplication has been improved by addressing a flaw in the repository format.
• A bug in the diff command that could lead to bad diff results has been fixed.
• A bug preventing snapshots from having a custom name has been fixed.

Incompatibilities:

• To support the new bupstash restore command, the network protocol has changed. Clients and
  servers must both upgrade bupstash in order to connect to eachother.
• Due to the repository format tweaks, snapshots created with older versions of bupstash
  no longer support bupstash diff, the --pick option, and also do not support the new bupstash restore
  command. Old format snapshots are still accessible via bupstash get.
• The send log and query cache format has changed, so these caches will be invalidated automatically.
• The new version of bupstash will automatically upgrade your repository when run. After this upgrade
  occurs, older versions of bupstash will not be able to access the repository.

Bupstash v0.9.1

Minor release improving performance during huge backups.

• The default put checkpoint is now every ~20GiB.
• Send log files are now updated in a more efficient way.

Bupstash v0.9.0

One more step on the journey to 1.0.

• Simplify the repository format such that it does not depend on sqlite3 anymore. The repository will automatically
  migrate itself.
• Better support filesystems that only support POSIX style locks such as NFSv3.
• Improve the efficiency of the send log when not using a stat cache.
• Migrate to server side timestamps to prevent client timestamp spoofing (which would interfere with intrusion auditing).
• Fix a case where the send-log stat cache would be invalidated too eagerly.

Bupstash v0.8.0

Another exciting bupstash release for your enjoyment.

Major new features:

• Add bupstash diff command to quickly view a summary of changes between snapshots.
• Remove automatic retry, instead bupstash put attempts to finish all snapshots, even if smeared by concurrent modification.
• Add --allow-list permission that is separate from --get to 'bupstash serve' so you can allow list/list-contents without giving access to snapshot data.
• Speed improvements in many cases for the directory storage backend.

Minor changes and fixes:

• Add blake3 hashes to content index so file contents can be diffed without a full fetch.
• Fix incorrect progress bar accounting.
• Improve smoothness of progress bar.
• Only fetch xattrs on files and directories.
• Support snapshotting device nodes on openbsd.
• Minor improvements to documentation.