File has vanished (Similar to issue #18)

[NatureMuncher]

I run four BackupPC systems backing up a handful of clients each and have been getting these errors on two of the hosts for several months:

From host identified as "f-az" on client "db8-az"

incr backup started for directory /
Xfer PIDs are now 5893
This is the rsync child about to exec /usr/bin/rsync_bpc
Xfer PIDs are now 5893,5897
xferPids 5893,5897
[ skipped 83 lines ]
file has vanished: "/usr/lib/.build-id/5e/f%2f/fusr/flib/f.build-id/f5e/attrib"
file has vanished: "/usr/lib/.build-id/b3/f%2f/fusr/flib/f.build-id/fb3/attrib"
file has vanished: "/usr/lib/firmware/3com/f%2f/fusr/flib/ffirmware/f3com/attrib"
file has vanished: "/usr/lib/firmware/advansys/f%2f/fusr/flib/ffirmware/fadvansys/attrib"
file has vanished: "/usr/lib/firmware/amd-ucode/f%2f/fusr/flib/ffirmware/famd-ucode/attrib"
[ skipped 27 lines ]
Done: 0 errors, 6 filesExist, 15685422 sizeExist, 3480429 sizeExistComp, 0 filesTotal, 0 sizeTotal, 83 filesNew, 460150033 sizeNew, 371747412 sizeNewComp, 621521 inode

Number of files: 288,082 (reg: 271,192, dir: 11,347, link: 5,535, special: 8)
Number of created files: 26 (reg: 26)
Number of deleted files: 0
Number of regular files transferred: 95
Total file size: 23,859,454,943 bytes
Total transferred file size: 476,422,270 bytes
Literal data: 91,275,589 bytes
Matched data: 384,559,866 bytes
File list size: 5,199,215
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 562,483
Total bytes received: 98,890,812
sent 562,483 bytes  received 98,890,812 bytes  57,338.31 bytes/sec
total size is 23,859,454,943  speedup is 239.91
DoneGen: 0 errors, 0 filesExist, 0 sizeExist, 0 sizeExistComp, 276735 filesTotal, 23859454943 sizeTotal, 0 filesNew, 0 sizeNew, 0 sizeNewComp, 621468 inode
rsync warning: some files vanished before they could be transferred (code 24) at main.c(1685) [generator=3.1.3.0]
rsync_bpc exited with benign status 24 (6144)
That means the client rsync had errors on some files.  Please check the XferLOG.
It likely means that rsync's delete cleanup (which deletes files on the backup
server that are no longer on the client) was skipped.  You should fix the error(s)
that rsync can run cleanly.  You can also specify the --ignore-errors option
which will still do the delete even if there are rsync errors, but do that with caution.
Xfer PIDs are now 
incr backup 213 complete, 276735 files, 23859454943 bytes, 7 xferErrs (0 bad files, 0 bad shares, 7 other)
BackupExpire: cntFull = 11, cntIncr = 6, firstFull = 0, firstIncr = 8, oldestIncr = 11.9838657407407, oldestFull = 141.025520833333
Removing filled backup 109
Xfer PIDs are now 11290
BackupPC_backupDelete: removing #109
BackupPC_backupDelete: No prior backup for merge
Xfer PIDs are now 11290,12498
BackupPC_refCountUpdate: host db8-az got 0 errors (took 48 secs)
Xfer PIDs are now 11290
Xfer PIDs are now 
Finished BackupPC_backupDelete, status = 0 (running time: 446 sec)
Xfer PIDs are now 
BackupExpire: cntFull = 10, cntIncr = 6, firstFull = 0, firstIncr = 7, oldestIncr = 11.9890277777778, oldestFull = 113.03068287037
Running BackupPC_refCountUpdate -h db8-az on db8-az
Xfer PIDs are now 12670
BackupPC_refCountUpdate: host db8-az got 0 errors (took 2 secs)
Xfer PIDs are now 
Finished BackupPC_refCountUpdate (running time: 2 sec)
Xfer PIDs are now 

From host identified as "f-az" on client "o8-az"

[ skipped 15 lines ]
file has vanished: "/usr/lib/.build-id/5a/f%2f/fusr/flib/f.build-id/f5a/attrib"
file has vanished: "/usr/lib/.build-id/5c/f%2f/fusr/flib/f.build-id/f5c/attrib"
file has vanished: "/usr/lib/.build-id/5d/f%2f/fusr/flib/f.build-id/f5d/attrib"
file has vanished: "/usr/lib/.build-id/5f/f%2f/fusr/flib/f.build-id/f5f/attrib"
file has vanished: "/usr/lib/.build-id/65/f%2f/fusr/flib/f.build-id/f65/attrib"
file has vanished: "/usr/lib/.build-id/c5/f%2f/fusr/flib/f.build-id/fc5/attrib"
[ skipped 26 lines ]
Done: 0 errors, 2 filesExist, 35 sizeExist, 48 sizeExistComp, 0 filesTotal, 0 sizeTotal, 22 filesNew, 806064920 sizeNew, 777075184 sizeNewComp, 189671 inode

Number of files: 84,345 (reg: 68,007, dir: 10,882, link: 5,449, special: 7)

From host identified as "f-ev" on client "fd8-a"

file has vanished: "/etc/alternatives/f%2f/fetc/falternatives/attrib"
[ skipped 104 lines ]
Done: 0 errors, 13 filesExist, 1730988 sizeExist, 1546568 sizeExistComp, 0 filesTotal, 0 sizeTotal, 43 filesNew, 52405157 sizeNew, 18028323 sizeNewComp, 622993 inode

rsync versions on the servers:

[Cent-7:root@f-ev ~]# rpm -qa | grep -ie rsync
rsync-bpc-3.1.3.0-1.el7.x86_64
rsync-3.1.2-10.el7.x86_64

[Cent-7:user@f-az ~]$ rpm -qa | grep -ie rsync
rsync-bpc-3.1.3.0-1.el7.x86_64
rsync-3.1.2-10.el7.x86_64

rsync versions on the clients:

ssh db8-az 'rpm -qa | grep -ie rsync'
rsync-3.1.3-12.el8.x86_64

ssh o8-az 'rpm -qa | grep -ie rsync'
rsync-3.1.3-12.el8.x86_64

ssh fd8-a 'rpm -qa | grep -ie rsync'
rsync-3.1.3-12.el8.x86_64

@craigbarratt - when you suggested enabling verbose logging in issue #18 you said 'the backup seems quite small'. I'm not sure how mine compares so wanted to wait before going to the next step.
Thank you!

[ediazrod]

do you create a snapshot before to make the backup I see similar errors if don't use the backup