The --extra-files <file...> command is unintuitive

[Qubasa]

Hello,

I just wanted to point out that the --extra-files <file...> command is a bit unintuitive, I had to look it up on Github. A better way would be --extra-files <file> <dest>.

[Prince213]

To anyone who is looking at this, #316 (or #325) has been merged and provides a solution to the problem.

[clhodapp]

@Prince213 Unless I'm missing something, it's still counter-intuitive because you don't get to specify the source and destination, as described in the initial report.

[Prince213]

@Prince213 Unless I'm missing something, it's still counter-intuitive because you don't get to specify the source and destination, as described in the initial report.

It's not an exact solution. Instead, the entire path will be copied to the destination, so you can just create the directory structure and put files there. Maybe we can add another option for individual files.

[bogorad]

I'm trying to copy my local /persist/* to a new remote host, into the /persist directory. zfs filesystem is created, and tank/persist is a mounted dataset on the new system. this command has no effect:

 nix run github:nix-community/nixos-anywhere -- --flake .#test root@nixos-live.lan --extra-files /persist

ah, now I get it! it actually copies files from /persist/* to /mnt/*. so I'll have to fiddle with softlikns. ugh. this sucks.

wow, it's even worse: tar can't unpack if /mnt/persist already exists! i made a directory /home/chuck/persist/ and created a softlink to /persist inside of it.

 ls -la /home/chuck/persist/
total 4
drwxr-xr-x  2 chuck users  3 Dec 13 21:36 .
drwx------ 15 chuck users 41 Dec 13 21:35 ..
lrwxrwxrwx  1 chuck users  8 Dec 13 21:36 persist -> /persist

now, this:

 nix run github:nix-community/nixos-anywhere -- --flake .#test root@nixos-live.lan --phases disko,install --disko-mode disko --extra-files /home/chuck/persist --debug

fails:

+ [[ -n /home/chuck/persist ]]
+ step Copying extra files
+ echo '### Copying extra files ###'
### Copying extra files ###
+ tar -C /home/chuck/persist -cpf- .
+ runSsh 'tar -C /mnt -xf- --no-same-owner'
+ ssh -t -i /tmp/tmp.sdRZUUPgiu/nixos-anywhere -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@nixos-live.lan 'tar -C /mnt -xf- --no-same-owner'
Pseudo-terminal will not be allocated because stdin is not a terminal.
Warning: Permanently added 'nixos-live.lan' (ED25519) to the list of known hosts.
tar: ./persist: Cannot open: File exists
tar: Exiting with failure status due to previous errors
+ rm -rf /tmp/tmp.sdRZUUPgiu

would've been so much easier if you added <dest> as OP suggested!

p.s. in case someone finds this for keyword "impermanence" "sops "zfs" - here's an (inelegant) solution:

1. run nixos-anywhere in phases. first, --phase disko
2. then copy files to "/persist' using whatever method. I transfer all persistent stuff, and generate ssh keys on host and copy them to dest. that's another quirk of nxios-ahywhere: you can tell it which ssh key to use for connecting, but it still copies the host key.
3. then do --phase install

[sedlund]

I'm trying to copy my local /persist/* to a new remote host, into the /persist directory. zfs filesystem

Use

mkdir /target && cp -av /persist /target

Then pass --extra-files /target

If /persist and / are on the same zpool this will use copy on write.

Everything in /target will be copied on top of / on the new system.

[bogorad]

If /persist and / are on the same zpool this will use copy on write.

They are not, and how would that help even if they were? Passing /persist is not the problem.

 zfs list
NAME           USED  AVAIL  REFER  MOUNTPOINT
tank          6.06G  47.2G   670K  /
tank/home      493M  47.2G   493M  /home
tank/nix      5.55G  47.2G  5.55G  /nix
tank/persist  7.23M  47.2G  7.23M  /persist

Everything in /target will be copied on top of / on the new system.

well, that doesn't solve my problem - I need contents of local /persist/* (which is a dataset on zfs) to be copied to remote /persist/ (which is also a dataset on zfs). The problem is that tar, which you switched to from ssh/scp, can't overwrite existing directories.

[sedlund]

The tar command used is:

nixos-anywhere/src/nixos-anywhere.sh

Line 589 in f9d38ca

tar -C "$extraFiles" -cpf- . | runSsh "tar -C /mnt -xf- --no-same-owner"

tar copies soft links by default (doesn't follow), so you cannot do what you tried as the error shows.

The -C option changes directory into the provided path then creates an archive of everything inside of it.

Effectively[1] this is:

scp -r /persist/. root@remote:/

which is not:

scp -r /persist root@remote:/

The contents of the path (not the directory) you pass to --extra-files overwrites /

If /persist and / are on the same zpool this will use copy on write.

They are not, and how would that help even if they were? Passing /persist is not the problem.

 zfs list
NAME           USED  AVAIL  REFER  MOUNTPOINT
tank          6.06G  47.2G   670K  /
tank/home      493M  47.2G   493M  /home
tank/nix      5.55G  47.2G  5.55G  /nix
tank/persist  7.23M  47.2G  7.23M  /persist

this shows / and /persist are on the same zpool tank.

my suggestion would place a copy of /persist inside /target which the command would cd into and copy to the remote host.

For those without a COW filesystem one could use cp with the --link option to create hard links:

cp --archive --verbose --link /persist /target

which will create hard links (not using more storage) which tar will copy.

[1] scp follows symlinks

[bogorad]

all of that is correct but misses the point entirely. I can read bash :)

the tar archive contains something like

persist/file1
persist/file2
...
persist/file999

when tar -xf- happens on the target machine, it fails because /persist already exists there. this is the problem, not how to get files tar'd, my method of symlinking also works fine.

I solved it by deconstructing the process and using phases, then using

tar -C /persist -cpf- . | exec_ssh "$DEST" "tar -C /mnt/persist/ -xf- --no-same-owner" \
  || die "Failed to copy /persist"

between the disko and install phases.

Also, I mentioned another problem with nixos-anywhere. To be able to use SOPS I need to control the ssh_host_ed25519_key file. The only option nixos-anywhere provides is to copy host files. so, I copy the files between phases.

If only nixos-anywhere fixed those two problems (allow <dest> and allow choosing which ssh_host_ed25519_key to copy to host, I wouldn't need this.

[sedlund]

To be able to use SOPS I need to control the ssh_host_ed25519_key file

This is documented using the --extra-files option here:

https://github.com/nix-community/nixos-anywhere/blob/main/docs/howtos/secrets.md#example-decrypting-an-openssh-host-key-with-pass