🐛 delta causes fancy unicode symbols such as Nerd Font Icons to be displayed as escaped in angle brackets

[vorburger]

$ git clone https://github.com/vorburger/vorburger-dotfiles-bin-etc.git
$ git checkout 975d4c977d41d8dd096067fa6d0588edee2c9fc7
$ ./fonts-install.sh
$ git --no-pager log | head -12

Note the funky "font symbol" I've used in those 2 commit messages (the fancy 'A' - see it?), this is from https://github.com/ryanoasis/nerd-fonts/, which my ./fonts-install.sh that's invoked above installed (if you then also appropriately configure to use that [patched!] Fira Code Nerd Font in your Terminal).

Now check out what delta seems to do:

$ git --no-pager log | delta

commit 975d4c977d41d8dd096067fa6d0588edee2c9fc7 (HEAD -> main, origin/main, origin/HEAD)
Author: Michael Vorburger <SPAM@vorburger.ch>
Date:   9 minutes ago

    Fonts: Install <U+F031> Fonts in *-gui.sh instead of *install.sh

commit 7f9b65a35974881c0712f1fab386bb8edf9a6f8d
Author: Michael Vorburger <SPAM@vorburger.ch>
Date:   32 minutes ago

    Fonts: Fira Code != Fira Code NERD <U+F031>

So that "fancy" 'A' got replaced by <U+F031>- even though my (appropriately configured) Terminal could display it.

Is there a way to configure delta to pass it through? For reference, check out this out for good ol' less:

$ git --no-pager log | less
commit 975d4c977d41d8dd096067fa6d0588edee2c9fc7 (HEAD -> main, origin/main, origin/HEAD)
Author: Michael Vorburger <SPAM@vorburger.ch>
Date:   11 minutes ago

    Fonts: Install <U+F031> Fonts in *-gui.sh instead of *install.sh

commit 7f9b65a35974881c0712f1fab386bb8edf9a6f8d
Author: Michael Vorburger <SPAM@vorburger.ch>
Date:   34 minutes ago

    Fonts: Fira Code != Fira Code NERD <U+F031>

so same, it also does this (unwanted) "escaping", but less --raw-control-chars does the trick:

$ git --no-pager log | less --raw-control-chars

Of course, one would want to have the cake and eat it too, and still have fancy "ANSI coloring" as well... 😄

[dandavison]

Hi @vorburger, what encoding does the text use that you're sending to delta? delta currently only guarantees to correctly handle utf-8. (ref #188)

[vorburger]

@dandavison My git's output should be in UTF-8, based on this:

$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

[vorburger]

@dandavison My git's output should be in UTF-8, based on this:

@dandavison actually, forget about guessing what encoding git is sending to delta - a friend of mine pointed out this is simpler to reproduce, debug and illustrate; here is how it (for me, on a Fedora 39 Workstation e.g. in GNOME Terminal = VTE, with $TERM set to screen-256color and other settings from my dotfiles) behaves:

$ python3 -c 'print("\uF031")'
# Prints the "fancy" Font Symbol 'A'

$ python3 -c 'print("\uF031")' | less
<U+F031>

$ python3 -c 'print("\uF031")' | less --raw-control-chars
# Prints the "fancy" Font Symbol 'A'

$ python3 -c 'print("\uF031")' | delta
<U+F031>

It looks like it's something particular with those special https://github.com/ryanoasis/nerd-fonts/ icon symbol Unicode Codepoints? Because a good ol' standard Smiley Emoji ☺(U+263A) works for me, check this out:

$ python3 -c 'print("\u263A")'
☺

$ python3 -c 'print("\u263A")' | delta
☺

$ python3 -c 'print("\u263A")' | less
☺

[nogweii]

I've run into the same issue with Nerd Fonts, but a bit surprisingly emoji work just fine. (I used them for the file labels.)

[injust]

delta already supports this. But it's a bit of a headache to understand what needs to be configured so delta doesn't overwrite your less options, which is being done here:

delta/src/utils/bat/output.rs

Lines 132 to 134 in 5d538b2

	let mut p = Command::new(less_path);
	if args.is_empty() || replace_arguments_to_less {
	p.args(vec!["--RAW-CONTROL-CHARS"]);

@vorburger already figured out that you need to pass --raw-control-chars to less (instead of the default --RAW-CONTROL-CHARS). But you should also read the caveat in man less:

-r or --raw-control-chars
    Causes "raw" control characters to be displayed.  The default is to display control characters using the caret notation; for example, a control-A (octal 001) is
    displayed as "^A" (with some exceptions as described under the -U option).  Warning: when the -r option is used, less cannot keep track of the actual appearance of
    the screen (since this depends on how the screen responds to each type of control character).  Thus, various display problems may result, such as long lines being
    split in the wrong place.
    USE OF THE -r OPTION IS NOT RECOMMENDED.

Basically, delta will pass --RAW-CONTROL-CHARS if the pager is less and you are not passing your own options to less.
You need to do one of these to avoid this behaviour:

• Set delta.pager to less --raw-control-chars in .gitconfig
• Set DELTA_PAGER='less --raw-control-chars' and unset delta.pager in .gitconfig

Notably, if you pass options to less by setting the LESS or PAGER environment variables, delta will ignore this and will set --RAW-CONTROL-CHARS, overriding your environment variable. This includes if you set core.pager to LESS='--raw-control-chars' delta.

(edit: See @th1000s's reply below for the actually correct solution)

[dandavison]

Nice instructions @injust! If you ever feel like improving the manual or README in this area that would of course be fantastic also.

[injust]

Unfortunately, it turns out that --raw-control-chars breaks navigation (pressing n/N will print Pattern not found). I haven't figured out why exactly (or if it breaks anything else).

(edit: See reply below for the actually correct solution)

[th1000s]

Nerd Fonts use the Private Use Area of Unicode, and less as of version 633 (from May 2023) treats this whole area as control characters (because there could be Dragons, or Klingons). But it also introduces the LESSUTFCHARDEF env var to mark areas as "printable" again. The less manual has this example, which sets the whole area as printable (and as single-width):

export LESSUTFCHARDEF='E000-F8FF:p,F0000-FFFFD:p,100000-10FFFD:p'

Setting this variable (unless already present) would restore the pre-633 behavior, especially since "there was a bug where only the two characters U+E000 and U+F8FF were treated as control characters [in prior less versions]". I think this could be done for all versions, older ones just ignore it.

The open Nerd Fonts issue is Add documentation for "less" LESSUTFCHARDEF (Private Use Area characters definition) option: nerd-fonts/ryanoasis#1337. If Nerd Fonts mark double-width :w sections then delta could use this to calculate the glyph width (relevant for side-by-side diffs).