rsync segfaults with *** buffer overflow detected ***: terminated

[pinpox]

Describe the bug

I'm running on nixos-unstable and rsync segfaults with the error below. I tried the same rsync version on archlinux and it seems to work fine there, so the error is nixpkgs-specific.

 rsync --verbose \
-e ssh\ -l\ root\ -T \
-FD \
--checksum \
--times \
--perms \
--recursive \
--links \
--delete-excluded \
--mkpath  . .
sending incremental file list
*** buffer overflow detected ***: terminated
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(231) [Receiver=3.2.7]
zsh: abort (core dumped)  noglob rsync --verbose -e ssh\ -l\ root\ -T -FD --checksum --times --perms

The rsync above being run has the hash /nix/store/i9dgn6ymv732q5gzw669y6y9zxsv3x2z-rsync-3.2.7/bin/rsync

Steps To Reproduce

Steps to reproduce the behavior:

1. Run the command above in any directory.
2. Observe output

Expected behavior

The command succeeds as it did before on nixos and still does on other OS's

Screenshots

If applicable, add screenshots to help explain your problem.
Here is a screenshot of the output on arch for comparison

Additional context

Add any other context about the problem here.

Notify maintainers

@ehmry @kampfschlaefer @ivan

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.60, NixOS, 23.11 (Tapir), 23.11.20231029.0cbe9f6`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.17.1`
 - channels(root): `""`
 - channels(pinpox): `"home-manager"`
 - nixpkgs: `/nix/store/35dcag44a0ymww0vy0s4jjgxwpv9g62d-source`

[pinpox]

I also tried installing rsync with nix inside an archlinux container, which also works. So the issue seems to be NixOS specific

[root@arch-container tmp]# nix run --extra-experimental-features 'nix-command flakes' 'nixpkgs#rsync' -- --verbose -e ssh\ -l\ root\ -T -FD --checksum --times --perms --recursive --links --delete-excluded --mkpath  . .
sending incremental file list

sent 39 bytes  received 12 bytes  102.00 bytes/sec
total size is 0  speedup is 0.00

Same hash aswell:

[root@arch-container tmp]# nix build  --extra-experimental-features 'nix-command flakes' 'nixpkgs#rsync'

[root@arch-container tmp]# ls -l ./result
lrwxrwxrwx 1 root root 55 Nov  2 17:12 ./result -> /nix/store/i9dgn6ymv732q5gzw669y6y9zxsv3x2z-rsync-3.2.7

[splack]

As a workaround, I use this where I need rsync. Overlay it for a system wide fix.

rsync = pkgs.rsync.overrideAttrs (_: _: {
  hardeningDisable = ["fortify"];
});

[ehmry]

@pinpox rebuild with hardeningDisable = ["fortify"] and see if that "fixes" your problem. We need to to know if Arch is building with hardening or the segfault is just a silent defect there.

[illustris]

[illustris@illustris-thinkpad:~/src/nixpkgs/.worktree/master]$ git checkout e86152986c6e6c563ced037c91403318d5a8b447
Previous HEAD position was dcdd9518fe59 Merge pull request #256887 from LeSuisse/glibc-2.37-39
HEAD is now at e86152986c6e glibc: 2.37-39 -> 2.38-0
[illustris@illustris-thinkpad:~/src/nixpkgs/.worktree/master]$ ./check.sh
+ set -e
+ NIXPKGS=/home/illustris/src/nixpkgs/.worktree/master
+ cd /dev/shm/test
+ nix run /home/illustris/src/nixpkgs/.worktree/master#rsync -- --verbose -e 'ssh -l root -T' -FD --checksum --times --perms --recursive --links --delete-excluded --mkpath . .
sending incremental file list
*** buffer overflow detected ***: terminated
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(231) [Receiver=3.2.7]
./check.sh: line 18: 1977400 Aborted                 (core dumped) nix run $NIXPKGS#rsync -- --verbose -e ssh\ -l\ root\ -T -FD --checksum --times --perms --recursive --links --delete-excluded --mkpath . .
+ false
[illustris@illustris-thinkpad:~/src/nixpkgs/.worktree/master]$ git checkout e86152986c6e6c563ced037c91403318d5a8b447~1
Previous HEAD position was e86152986c6e glibc: 2.37-39 -> 2.38-0
HEAD is now at dcdd9518fe59 Merge pull request #256887 from LeSuisse/glibc-2.37-39
[illustris@illustris-thinkpad:~/src/nixpkgs/.worktree/master]$ ./check.sh
+ set -e
+ NIXPKGS=/home/illustris/src/nixpkgs/.worktree/master
+ cd /dev/shm/test
+ nix run /home/illustris/src/nixpkgs/.worktree/master#rsync -- --verbose -e 'ssh -l root -T' -FD --checksum --times --perms --recursive --links --delete-excluded --mkpath . .
sending incremental file list

sent 38 bytes  received 12 bytes  100.00 bytes/sec
total size is 0  speedup is 0.00

e861529

This is the change that broke it. Disabling hardening as suggested by @splack works.

[pinpox]

@splack @ehmry Maybe I'm doing it wrong, but the workaround is not working for me. The hash of the overriden rsync is /nix/store/5ndkb1mvj2m7dnz1jy3rk7nvc482a4dk-rsync-3.2.7/bin/rsync for me, still getting the same error. The relevant part of that script is:

                          deploy-flake =

                            let
                              rsync-workaround = pkgs.rsync.overrideAttrs (_: _: {
                                hardeningDisable = [ "fortify" ];
                              });
                            in
                            {

                              deps = [ "check-vars" ];
                              desc = "Deploy flake repository to: ${hostName}";
                              cmds = [
                                ''echo "Deploying flake to: {{.HOSTNAME}}"''
                                ''
                                  source_path={{.LOCAL_FLAKE_SOURCE}}
                                  if test -d "$source_path"; then
                                    source_path=$source_path/
                                  fi
                                  ${rsync-workaround}/bin/rsync \
                                  --verbose \
                                  -e {{.REMOTE_COMMAND}}\ -l\ {{.REMOTE_USER}}\ -T \
                                  -FD \
                                  --checksum \
                                  --times \
                                  --perms \
                                  --recursive \
                                  --links \
                                  --delete-excluded \
                                  --mkpath \
                                  ${optionalString useSudo ''--rsync-path="{{.REMOTE_SUDO_COMMAND}} {{.REMOTE_SUDO_OPTS}} rsync"''} \
                                  $source_path {{.REMOTE_USER}}\@{{.REMOTE_HOST}}:{{.REMOTE_CONFIG_DIR}}
                                ''
                              ];
                            };

[splack]

@pinpox I made a flake to demonstrate the workaround. https://git.lanvise.com/lanvise/rsync

Your snippet looks like it should work. Maybe reduce it to a simple flake to fix the inputs?

[pinpox]

@splack I can confirm your flake works. The rsync-test gives me the overflow error, while rsync-test-soft works properly. I havn't been able to get it to work on my system configuration yet though.

Since the cause of the error seems clear or is at least reproducible, how can we properly fix this? A workaround is nice, but I heavily depend on rsync for configuration and deployments and would be in favor of a proper, nixpkgs-wide fix if possible. I can submit PR's if necessary or take further steps to get this fixed, but would need some guidance as I don't really know here to start.

[Mic92]

Applied a patches that fixes the crash for me: #265846