I had been using borg the last years as my backup solution. I have used restic at one of my previous employers, but stopped right after that.

But with rustic on its way and borg2 also in development, there are two new candidates to compare to.

I previously only compared restic and borg against each other. This was interesting, because restic was multithreaded, while borg was not. But borg was equally fast over a 200Mbit/s(?) line, because it compressed the data. And also did a test-shoot with rustic earlier this year.

Evaluation process

• Each solution will backup the same directory.
• Once onto local disk.
• Another time onto local disk (without any change in data set)

I’m not going to cover restore speeds right now or remote backup speeds. This will happen later.

Dataset

I’m compressing my TODO folder. This contains a mix of video, audio, PDF files and also some code projects. So it’s rather random dataset.

If you’re having a data set of only pre-compressed data (like mp4 videos etc.), you’re maybe even better disabling the compression completely.

Stats about the data:

• 9.9GB data
• compressible to 7.9GB with tar c <folder> | zstd -T0 -12
• 9591 directories
• 43423 files
• 35 symlinks

Hardware

I’m testing this on a Lenovo T14 Gen 1 Laptop with an Intel i5-10310U, 16GB RAM on a WD SN730 512GB NVMe disk.

Metrics

We’re going to compare the following things:

• Final Repo Size
• Time
• Other observations, especially the usage.

Tests

restic

Installation seems easy from the NixOS 24.05 repos:

$ nix-env -iA nixos.restic
[...]
$ restic version
restic 0.16.5 compiled with go1.22.6 on linux/amd64

In the meantime, there is 0.17 released. But according to the release notes, there is no relevant change for our test.

The docs are very fine. Everything listed on the start page: https://restic.net/

After checking my terminal log, I had been finished installing the package, reading the docs and doing the full snapshot in under 10 Minutes.

$ restic init -p ~/backup-tests/password -r ~/backup-tests/repos/restic
created restic repository c6e0c717c7 at /home/bebe/backup-tests/repos/restic

Please note that knowledge of your password is required to access
the repository. Losing your password means that your data is
irrecoverably lost.

The initial backup:

$ restic -p ~/backup-tests/password -r ~/backup-tests/repos/restic backup ~/00todo
repository c6e0c717 opened (version 2, compression level auto)
created new cache in /home/bebe/.cache/restic
no parent snapshot found, will read all files


Files:       43423 new,     0 changed,     0 unmodified
Dirs:         9593 new,     0 changed,     0 unmodified
Added to the repository: 8.676 GiB (7.548 GiB stored)

processed 43423 files, 9.654 GiB in 1:49
snapshot 55af1c78 saved

The second backup:

$ restic -p ~/backup-tests/password -r ~/backup-tests/repos/restic backup ~/00todo
repository c6e0c717 opened (version 2, compression level auto)
using parent snapshot cb2f57d8
[0:00] 100.00%  1 / 1 index files loaded

Files:           0 new,     0 changed, 43423 unmodified
Dirs:            0 new,     2 changed,  9591 unmodified
Added to the repository: 737 B (581 B stored)

processed 43423 files, 9.654 GiB in 0:03
snapshot 045d8fa0 saved

Final repo size: 7.6 GB

Backup Time: 1:49 minutes

Re-Backup Time: 0:03 minutes

This is less than zstd’s compressed size? How?

They apparently added compression back in 2022 for version >=0.14. Wow!

In my last evaluation in 2019, this was not available yet.

Rustic

$ nix-env -iA nixos.rustic-rs
[...]
$ rustic version
error: unrecognized subcommand 'version'

Usage: rustic [OPTIONS] <COMMAND>

For more information, try '--help'.
$ rustic --version
rustic 0.7.0

Consistency, Consistency consistency. Why no rustic version?

$ rustic init -p ~/backup-tests/password -r ~/backup-tests/repos/rustic
[INFO] using no config file, none of these exist: /home/bebe/.config/rustic/rustic.toml, /etc/rustic/rustic.toml, ./rustic.toml
[INFO] key ca7d4762 successfully added.
[INFO] repository 6312b2ea successfully created.
[INFO] using cache at /home/bebe/.cache/rustic/6312b2ea9dc3ae28697588a6a758358a7d2ba86269b8116de420c2ce25f39a4e

$ rustic -p ~/backup-tests/password -r ~/backup-tests/repos/rustic backup ~/00todo
[INFO] using no config file, none of these exist: /home/bebe/.config/rustic/rustic.toml, /etc/rustic/rustic.toml, ./rustic.toml
[INFO] repository local:/home/bebe/backup-tests/repos/rustic: password is correct.
[INFO] using cache at /home/bebe/.cache/rustic/6312b2ea9dc3ae28697588a6a758358a7d2ba86269b8116de420c2ce25f39a4e
[00:00:00] reading index...               ████████████████████████████████████████          0/0           [00:00:00] getting latest snapshot...     ████████████████████████████████████████          0/0           [INFO] using no parent
[INFO] starting to backup "/home/bebe/00todo"...
[00:01:48] backing up...                  ████████████████████████████████████████   9.65 GiB/9.65 GiB   91.37 MiB/s  (ETA 0s)Files:       43458 new, 0 changed, 0 unchanged
Dirs:        9594 new, 0 changed, 0 unchanged
Added to the repo: 7.5 GiB (raw: 8.7 GiB)
processed 43458 files, 9.7 GiB
snapshot fcc18715 successfully saved.
[INFO] backup of "/home/bebe/00todo" done.

$ rustic -p ~/backup-tests/password -r ~/backup-tests/repos/rustic backup ~/00todo
[INFO] using no config file, none of these exist: /home/bebe/.config/rustic/rustic.toml, /etc/rustic/rustic.toml, ./rustic.toml
[INFO] repository local:/home/bebe/backup-tests/repos/rustic: password is correct.
[INFO] using cache at /home/bebe/.cache/rustic/6312b2ea9dc3ae28697588a6a758358a7d2ba86269b8116de420c2ce25f39a4e
[00:00:00] reading index...               ████████████████████████████████████████          1/1           [00:00:00] getting latest snapshot...     ████████████████████████████████████████          1/1           [INFO] using parent fcc18715
[INFO] starting to backup "/home/bebe/00todo"...
[00:00:02] backing up...                  ████████████████████████████████████████   9.65 GiB/9.65 GiB   3.28 GiB/s   (ETA 0s)Files:       0 new, 0 changed, 43458 unchanged
Dirs:        0 new, 52 changed, 9542 unchanged
Added to the repo: 131.4 kiB (raw: 431.0 kiB)
processed 43458 files, 9.7 GiB
snapshot 78aec230 successfully saved.
[INFO] backup of "/home/bebe/00todo" done.

I’m sorry rustic for blaming your consistency. Everything I did in these commands was changing restic to rustic, actually just s/e/u/. Nice!

Final repo size: 7.6 GB

Backup Time: 1:48 minutes

Re-Backup Time: 0:02 minutes

The progress bar was very nice. It turns out, this is not good to later paste it, but you can disable it via --no-progress.

It seemed like there was less CPU load on my box in comparison to restic.

borg (1)

$ nix-env -iA nixos.borgbackup
[...]
$ borg --version
borg 1.4.0

For the next tests, we’re going to use these variables defined in the environment

$ export BORG_PASSCOMMAND="cat $HOME/backup-tests/password"
$ export BORG_REPO=~/backup-tests/repos/borg1

$ borg init -e repokey

IMPORTANT: you will need both KEY AND PASSPHRASE to access this repo!

Key storage location depends on the mode:
- repokey modes: key is stored in the repository directory.
- keyfile modes: key is stored in the home directory of this user.

For any mode, you should:
1. Export the borg key and store the result at a safe place:
   borg key export           REPOSITORY encrypted-key-backup
   borg key export --paper   REPOSITORY encrypted-key-backup.txt
   borg key export --qr-html REPOSITORY encrypted-key-backup.html
2. Write down the borg key passphrase and store it at safe place.

There is something nice about borg, there is a benchmark for it:

$ borg benchmark crud :: ~/00todo
C-Z-BIG         293.14 MB/s (10 * 100.00 MB all-zero files: 3.41s)
R-Z-BIG         102.11 MB/s (10 * 100.00 MB all-zero files: 9.79s)
U-Z-BIG        1832.28 MB/s (10 * 100.00 MB all-zero files: 0.55s)
D-Z-BIG        6882.84 MB/s (10 * 100.00 MB all-zero files: 0.15s)
C-R-BIG          73.05 MB/s (10 * 100.00 MB random files: 13.69s)
R-R-BIG          89.44 MB/s (10 * 100.00 MB random files: 11.18s)
U-R-BIG        1501.64 MB/s (10 * 100.00 MB random files: 0.67s)
D-R-BIG        4460.61 MB/s (10 * 100.00 MB random files: 0.22s)
C-Z-MEDIUM      217.60 MB/s (1000 * 1.00 MB all-zero files: 4.60s)
R-Z-MEDIUM      103.81 MB/s (1000 * 1.00 MB all-zero files: 9.63s)
U-Z-MEDIUM     2697.52 MB/s (1000 * 1.00 MB all-zero files: 0.37s)
D-Z-MEDIUM     6107.01 MB/s (1000 * 1.00 MB all-zero files: 0.16s)
C-R-MEDIUM       75.88 MB/s (1000 * 1.00 MB random files: 13.18s)
R-R-MEDIUM       95.92 MB/s (1000 * 1.00 MB random files: 10.43s)
U-R-MEDIUM     2615.30 MB/s (1000 * 1.00 MB random files: 0.38s)
D-R-MEDIUM     4151.86 MB/s (1000 * 1.00 MB random files: 0.24s)
C-Z-SMALL        22.92 MB/s (10000 * 10.00 kB all-zero files: 4.36s)
R-Z-SMALL        54.77 MB/s (10000 * 10.00 kB all-zero files: 1.83s)
U-Z-SMALL        46.98 MB/s (10000 * 10.00 kB all-zero files: 2.13s)
D-Z-SMALL       317.41 MB/s (10000 * 10.00 kB all-zero files: 0.32s)
C-R-SMALL        18.36 MB/s (10000 * 10.00 kB random files: 5.45s)
R-R-SMALL        50.92 MB/s (10000 * 10.00 kB random files: 1.96s)
U-R-SMALL        47.54 MB/s (10000 * 10.00 kB random files: 2.10s)
D-R-SMALL       174.93 MB/s (10000 * 10.00 kB random files: 0.57s)

This fills the repo now with 2GB of data. We’re going to delete this again. I’m not even sure what this will tell me and how I can interpret the data. Yes, it’s faster, but…. ?

$ borg create -v --progress --stats ::test1 ~/00todo
Creating archive at "/home/bebe/backup-tests/repos/borg1::test1"
------------------------------------------------------------------------------
Repository: /home/bebe/backup-tests/repos/borg1
Archive name: test1
Archive fingerprint: a9026ed20535df4c7ea339d66fd302ceb43aece100b79a6a68e4576505a9985d
Time (start): Tue, 2024-10-01 11:03:41
Time (end):   Tue, 2024-10-01 11:06:44
Duration: 3 minutes 2.16 seconds
Number of files: 43423
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               10.37 GB              9.13 GB              8.43 GB
All archives:               10.37 GB              9.13 GB              8.43 GB

                       Unique chunks         Total chunks
Chunk index:                   29377                46306
------------------------------------------------------------------------------

$ borg create -v --progress --stats ::test2 ~/00todo
Creating archive at "/home/bebe/backup-tests/repos/borg1::test2"
------------------------------------------------------------------------------
Repository: /home/bebe/backup-tests/repos/borg1
Archive name: test2
Archive fingerprint: 2cb9b3ff58df2572284635c88cad7fcb9d28cb1afb55f44abc755c13bd933da4
Time (start): Tue, 2024-10-01 11:07:21
Time (end):   Tue, 2024-10-01 11:07:30
Duration: 9.04 seconds
Number of files: 43423
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               10.37 GB              9.13 GB              3.70 kB
All archives:               20.75 GB             18.26 GB              8.43 GB

                       Unique chunks         Total chunks
Chunk index:                   29378                92612
------------------------------------------------------------------------------

As expected, takes much longer to do the backups, but IMO still feasible.

Final repo size: 7.9 GB

Backup Time: 3:02 minutes

Re-Backup Time: 0:09 minutes

The repo size totally contradicts my tests from 2019, where borg had compression without multithreading and restic the exact opposite. So it took longer for borg when having a high speed repository (like local disk), but the disk usage was lower. With decreasing bandwidth, borg became much better.

Now, borg is even worse in compression ratio.

borg1 (tuned compression ration)

I did the same as above, re-initializing the borg1 repo and using everything with -C auto,zstd,12. This seems to be legit for me, since:

1. The auto flag checks if the data is compressible at all. One of the main benefits from lz4 over zstd.
2. The zstd,12 compression having zstd with -12 compression. In my experience this is best compression/CPU cycle trade off.

$ borg init -e repokey
[... see above ...]

$ borg create -v --progress --stats -C auto,zstd,12 ::test1 ~/00todo
Creating archive at "/home/bebe/backup-tests/repos/borg1::test1"
------------------------------------------------------------------------------
Repository: /home/bebe/backup-tests/repos/borg1
Archive name: test1
Archive fingerprint: 8a0a2494ac5821967a1331465bf08d27c98e3cd2ed3bb5fcc9cd8c7bbe9b00d1
Time (start): Tue, 2024-10-01 11:32:31
Time (end):   Tue, 2024-10-01 11:37:53
Duration: 5 minutes 21.91 seconds
Number of files: 43423
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               10.37 GB              8.61 GB              8.05 GB
All archives:               10.37 GB              8.61 GB              8.05 GB

                       Unique chunks         Total chunks
Chunk index:                   29389                46310
------------------------------------------------------------------------------

Backing up another time:

$ borg create -v --progress --stats -C auto,zstd,12 ::test2 ~/00todo
Creating archive at "/home/bebe/backup-tests/repos/borg1::test2"
------------------------------------------------------------------------------
Repository: /home/bebe/backup-tests/repos/borg1
Archive name: test2
Archive fingerprint: 12b970a2645a221ea9b56e7cd865b32143c7ade7de48c30dfe36195f2c10b726
Time (start): Tue, 2024-10-01 11:39:18
Time (end):   Tue, 2024-10-01 11:39:27
Duration: 8.70 seconds
Number of files: 43423
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               10.37 GB              8.61 GB              3.00 kB
All archives:               20.75 GB             17.22 GB              8.05 GB

                       Unique chunks         Total chunks
Chunk index:                   29390                92620
------------------------------------------------------------------------------

Final repo size: 7.6 GB

Backup Time: 5:21 minutes

Re-Backup Time: 0:09 minutes

The repo size seems to be reasonable size now. But inital backup time doubled.

As far as I can see, there is no multi threaded zstd compression possible with borg. I wonder, what the reason is, because there is no point not having it. zstd is a C module in python and not requiring the GIL locked.

borg2 (beta)

We’re going to use the current beta, 2.0.0b11.

Installation is not so easy for NixOS, since this is not in nixpkgs yet. So we donwload it from the GitHub Releases page, but:

$ ./borg-linux-glibc236
Could not start dynamically linked executable: ./borg-linux-glibc236
NixOS cannot run dynamically linked executables intended for generic
linux environments out of the box. For more information, see:
https://nix.dev/permalink/stub-ld

… nope. pip install "borgbackup==2.0.0b11" doesn’t work either. The development headers are not reachable.

… Fast forward two hours dealing with Nix(OS) issues, we’re here:

$ nix --extra-experimental-features "nix-command flakes" run github:thiagokokada/nix-alien -- ./borg-linux-glibc236 --version
borg-linux-glibc236 2.0.0b11

There we go! The nix-alien bootstrap overhead takes utterly long, but it seems there is no performance penalty after the program launched.

$ nix --extra-experimental-features "nix-command flakes" run github:thiagokokada/nix-alien -- ./borg-linux-glibc236 repo-create -e authenticated-blake2
Enter new passphrase:
Enter same passphrase again:
Do you want your passphrase to be displayed for verification? [yN]: y
Your passphrase (between double-quotes): "asdf1234"
Make sure the passphrase displayed above is exactly what you wanted.

IMPORTANT: you will need both KEY AND PASSPHRASE to access this repo!

Key storage location depends on the mode:
- repokey modes: key is stored in the repository directory.
- keyfile modes: key is stored in the home directory of this user.

For any mode, you should:
1. Export the borg key and store the result at a safe place:
   borg key export -r REPOSITORY           encrypted-key-backup
   borg key export -r REPOSITORY --paper   encrypted-key-backup.txt
   borg key export -r REPOSITORY --qr-html encrypted-key-backup.html
2. Write down the borg key passphrase and store it at safe place.

Reserve some repository storage space now for emergencies like 'disk full'
by running:
    borg repo-space --reserve 1G

$ borg2 create -v --progress --stats -C auto,zstd,12 ::test1 ~/00todo
Enter passphrase for key /home/bebe/backup-tests/repos/borg2:
Creating archive at "/home/bebe/backup-tests/repos/borg2"
Repository: /home/bebe/backup-tests/repos/borg2
Archive name: ::test1
Archive fingerprint: 267f32affd17b7c5c180ff39aeb659a264590338d304d19f0de8f11f5479a797
Time (start): Tue, 2024-10-01 13:55:50 +0200
Time (end):   Tue, 2024-10-01 14:01:16 +0200
Duration: 5 minutes 26.443 seconds
Number of files: 43423
Original size: 10.37 GB
Deduplicated size: 9.35 GB
Time spent in hashing: 32.000 seconds
Time spent in chunking: 41.310 seconds
Added files: 43423
Unchanged files: 0
Modified files: 0
Error files: 0
Files changed while reading: 0
Bytes read from remote: 0
Bytes sent to remote: 0
Saving files cache
Saving chunks cache
Saving cache config

$ borg2 create -v --progress --stats -C auto,zstd,12 ::test2 ~/00todo
Enter passphrase for key /home/bebe/backup-tests/repos/borg2:
Creating archive at "/home/bebe/backup-tests/repos/borg2"
Repository: /home/bebe/backup-tests/repos/borg2
Archive name: ::test2
Archive fingerprint: 95ee4376539281d512c02e95ce4f4165720c5167cb1067482829ed3ae0b08a4b
Time (start): Tue, 2024-10-01 14:01:41 +0200
Time (end):   Tue, 2024-10-01 14:03:17 +0200
Duration: 1 minutes 36.194 seconds
Number of files: 43423
Original size: 10.37 GB
Deduplicated size: 376 B
Time spent in hashing: 35.296 seconds
Time spent in chunking: 40.077 seconds
Added files: 43423
Unchanged files: 0
Modified files: 0
Error files: 0
Files changed while reading: 0
Bytes read from remote: 0
Bytes sent to remote: 0
Saving files cache
Saving chunks cache
Saving cache config

Final repo size: 7.7 GB

So, the initial run took a percent longer, bug the zero difference backup took about 10 times longer. What happened? I don’t know. But the second create reports time for chunking and hashing, which should not have been required at all.

I don’t know the reason, but it seems there are some weird performance penalties for borg 2.0. Or it’s simply a bug.

Conclusio

• restic performs insanely good with compression
• rustic seems to be a bit faster on local disk and consume less CPU for compression
• borg 1 still performs as good/bad as in the previous years. It’s still single threaded.
• borg 2.0 beta has got some weird issues in the process. The user interface looks a bit cleaner. It’s still single threaded.

I’m currently writing some automated tests to compare it with almost all compressions on my local internet line.