wezterm support

[wez]

I saw this tweet: https://twitter.com/NyaosOrg/status/1497486187279175681
and found:

go-readline-ny/moji.go

Lines 15 to 33 in 48b319a

	var isWezTerm = os.Getenv("WEZTERM_EXECUTABLE") != ""
	var (
	// SurrogatePairOk is true when the surrogated pair unicode is supported
	// If it is false, <NNNN> is displayed instead.
	SurrogatePairOk = isWindowsTerminal || isWezTerm
	// ZeroWidthJoinSequenceOk is true when ZWJ(U+200D) is supported.
	// If it is false, <NNNN> is displayed instead.
	ZeroWidthJoinSequenceOk = isWindowsTerminal
	// VariationSequenceOk is true when Variation Sequences are supported.
	// If it is false, <NNNN> is displayed instead.
	VariationSequenceOk = isWindowsTerminal
	// ModifierSequenceOk is false, SkinTone sequence are treated as two
	// character
	ModifierSequenceOk = isWindowsTerminal
	)

I'd like to make some suggestions and invite you to please open issues with wezterm if things aren't working as you expect!

Detecting wezterm

Firstly, I would suggest looking at these two environment variables if you'd like to detect when you are running inside a local wezterm:

; env | egrep '^TERM_'
TERM_PROGRAM=WezTerm
TERM_PROGRAM_VERSION=20220213-203140-4a1c4b55

If you'd like to support the case where you are running in a remote ssh session, you could use a sequence like XTVERSION to probe the terminal version:

1. Emit \x1b[>q
2. Read the response string, which for wezterm looks like: ^[P>|WezTerm 20220213-203140-4a1c4b55^[\
3. That read operation should have a timeout in case the remote terminal doesn't respond to this sequence.

Variation Sequences

These are supported by wezterm when the unicode version is set to enable them.

https://wezfurlong.org/wezterm/config/lua/config/unicode_version.html#unicode-version-escape-sequence describes escape sequences that you can use to enable them for your application

Modifier Sequence

wezterm should support all skin tone and other modifiers out of the box: we use harfbuzz for this and have explicit unit tests to verify that these are shaped correctly.

If you find examples where this isn't true, please file an issue!

ZWJ

These should be supported, but I'm not sure what your criteria for conformance is for these. I would love to hear more!

[wez]

One other thing I'd like to note: the nightly build of wezterm has improved support for handling some of these things: I would recommend trying that!

[hymkor]

@wez

Thank you. I am glad that you have been interested.

I will update the code to detect WezTerm by comparing %TERM_PROGRAM% and "WezTerm" and enable ZWJ , Variation Sequences and Modifier Sequence in WezTerm.

My readline library has to know how many cells the cursor moves when characters are output into terminal, however, sometimes the cursor position differs from that expected.

Since I am making the sample to tell you, would you wait a moment ?

[hymkor]

@wez :

When ZWJ is output, the cursor position changes during sleeping 1 second.
Therefore, it is difficult to support ZWJ on WezTerm.

package main

import (
	"fmt"
	"time"
)

func main() {
	farmer := "\U0001F468\u200D\U0001F33E"
	fmt.Printf("(a) %s%s%s:OK\n", farmer, farmer, farmer)
	fmt.Printf("(b) %s\b\b%s\b\b%s:NG\n", farmer, farmer, farmer)
	fmt.Printf("(c) %s\b%s\b%s:NG\n", farmer, farmer, farmer)
	fmt.Printf("(d) %s\b\b\b\b\b%s\b\b\b\b\b%s:OK\n", farmer, farmer, farmer)
	fmt.Println("------ When sleep 1 second after three farmer are drawn ------")
	fmt.Printf("(x) %s%s%s", farmer, farmer, farmer)
	time.Sleep(time.Second)
	fmt.Printf("\b\b\b\b\b: NG ???\n")
}

The compiled executable of the test code above:farmer.zip

On WezTerm

With (a),(b),(c), and (d), I considered the MAN-FARMER-EMOJI can be erased with 5 BACKSPACES on WezTerm.

However, the case (x) , during sleeping 1 second, the cursor position changes and 5 BACKSPACES can not erase any farmers.

On Windows Terminal

When I run this code on Windows Terminal, the screenshot is below :

[wez]

That's an interesting test case!
I translated it to this shell script:

#!/bin/zsh

man=$(echo -e "\U0001F468")
ear_of_rice=$(echo -e "\U0001F33E")
farmer=$(echo -e "${man}\u200D${ear_of_rice}")

echo -e "$farmer$farmer$farmer."
echo -e "$farmer\b\b$farmer\b\b$farmer."
echo -e "$farmer\b$farmer\b$farmer."
echo -e "$farmer\b\b\b\b\b$farmer\b\b\b\b\b$farmer."
echo -ne "$farmer"
sleep 1
echo -e "\b\b\b\b\b."
echo "DONE"

and ran it with a variety of different terminal emulators on unix systems:

xterm:

kitty:

gnome terminal:

contour:

wezterm (on linux, and also wezterm ssh linuxhost when run from Windows):

wezterm (version: 20220225-082033-57059992, using the ConPTY layer on windows; ssh.exe to connect to remote linux, and also wsl.exe for local linux):

Terminal.app:

iTerm:

Windows Terminal: (using C:/Windows/System32/OpenSSH/ssh.exe to connect to linux):

[wez]

As you can see, all of them behave differently!

The backspace control is defined as moving the cursor position left one cell. The test case presented above makes an assumption about the relationship of bytes (or codepoints) and cells (graphemes) that cannot be relied upon for terminals that support emoji sequences.

In wezterm's implementation, the input text is recognized as graphemes and each grapheme is assigned to the cell at the current cursor position. If the cursor is positioned in the middle of a double-width cell and new text is output, wezterm will first "invalidate" the double-wide cell by replacing it with a blank cell. For a sequence like "MAN ZWJ EAR-OF-RICE" there are three constituent codepoints so you might think that we can "just remove the ZWJ EAR-OF-RICE bits" when invalidating, but the sequences can be more complex: \u{1F469}\u{1F3FF}\u{200D}\u{1F91D}\u{200D}\u{1F469}\u{1F3FC} (women holding hands, dark skin tone, medium light skin tone) and \u{1f3f4}\u{e0067}\u{e0062}\u{e0065}\u{e006e}\u{e0067}\u{e007f} (England flag) for example, so there isn't an obvious or universal way to reason about this kind of edit operation.

Regardless of whether wezterm's implementation is right or wrong:

I think the ideal situation would be that everyone agreed on using grapheme clustering (per http://www.unicode.org/reports/tr29/) to count the cell position and use that to manage the cursor position and update text. I don't know if you can rely on that working everywhere today!

My recommendation would be for your line editor to prefer to repaint the entire line rather than perform a minimal move/update in order to have a better chance at having the result look readable. There's still a chance that the cursor position will look weird in some terminals.

I would also recommend that you resist the urge to code special knowledge of any given terminal's positioning quirks without giving your users a way to turn it off, as terminal emulator development is currently moving rather quickly and you might find that your line editor is broken when bugs are fixed in the terminal emulator!

[hymkor]

My recommendation would be for your line editor to prefer to repaint the entire line rather than perform a minimal move/update in order to have a better chance at having the result look readable. There's still a chance that the cursor position will look weird in some terminals.

Thank you.

Yes, I agree. I would make the mode to repaint the entire line. However, I would not erase the mode to move/update in minimum because I have some slow machines (Windows 8.1 in Virtual Machine ).