Long-press haptics

[devycarol]

We are so back.

This branch closes #1859 by adding long-press haptics to the keyboard.
EDIT: this branch also closes #2021, provided we keep the removal of the haptic cooldown. I came back at a really good time! What a wild coincidence.

Behavior changes:

• When key-press vibration is enabled, a long-press vibration will occur when long-pressing a key.
  • Includes keys with pop-up panels.
  • Includes keys with long-press actions like space -> languages, shift -> caps lock, and symbols -> numpad.
  • Does not include repeating keys
  • Does not include keys that do nothing significant when long-pressed such as symbols-shift or square-root.
• Long-pressing a word in the suggestions strip now uses the "long press" vibration instead of the "key press" one.
  • No longer makes a key-press sound.
• I noticed the change to gesture haptics. I don't like how strong it is, so I reduced the strength to "clock tick." This seems to be in line with my best guess at what Gboard uses. @eranl, what do you think of this?
  • Also made it so those haptics don't happen when the gesture action will do nothing (when we can detect such a thing)
  • Currently, this means they won't happen when doing virtual left/right arrow keys for bad input fields. We can revise this later, but I would want to suppress those virtual keypresses for good input fields before adding haptics to the bad ones. There's currently no easy way to tell the difference, and these revisions have already stretched the scope of this PR.

Technical notes:

• I've created an enum class for different types of haptic feedback that can hopefully make extending things like this easier moving forward. You can see I have some commented constants of haptics that could be implemented later.
• There's a special case with the shift -> caps lock long-press. That's the one long-press that did have haptics since it's implemented using a dummy key-press. We want to use the long-press vibration strength, not the key-press one. HapticEvent.NO_HAPTICS is used to ensure this.

p.s.: didn't we want longer long-press duration for the spacebar -> language menu gesture? did that change at some point? it seems to just have the regular duration. i'd prefer 1.5x personally.

[devycarol]

Just noticed the discussion in #1867. I figure with the reduced strength it's okay to remove that 100ms haptic cooldown, so I've gone ahead and done that. I also fear that could interfere with the long-press vibration of a user who's using the 100ms long-press timeout.

Now that I know that was there, I noticed I really don't like how it would skip vibrations when typing super quickly 😅

If you want me to, I can add it back and have it only apply to gesture-move haptic events. A cooldown of 20ms or 50ms is possible, too. I really like consistently being able to feel each and every cursor nudge when I'm trying to be precise with the space swipe.

[devycarol]

Also caught a bug with custom vibration duration! Custom duration will apply to key-presses and the long-press vibration. It will not apply to gesture movement.

LOL.

Maybe this should be fine-tuned with a duration multiplier. 🤷

[devycarol]

@CloisteredCaiman I'll mention you as well since you seemed rather excited about #1867's changes.

Gesture vibration is still here, it is using a weaker vibration strength and no longer has a cooldown currently.

[eranl]

• When key-press vibration is enabled, a long-press vibration will occur when long-pressing a key.

Questions:

• Is that a second vibration or a delayed single one?
• Are you also making the timing of the short press vibration consistent (on key down vs. key up)?

• I noticed the change to gesture haptics. I don't like how strong it is, so I reduced the strength to "clock tick." This seems to be in line with my best guess at what Gboard uses. @eranl, what do you think of this?

Sounds good to me.

p.s.: didn't we want longer long-press duration for the spacebar -> language menu gesture? did that change at some point? it seems to just have the regular duration. i'd prefer 1.5x personally.

Looks like these are the only exceptions.

[devycarol]

It is a second vibration, slightly stronger than the first.

I'm not changing short press vibration behavior at this time. I like how it is on the main keyboard, but I can see vibrate-on-touch being an annoyance when scrolling through a long toolbar.

[Helium314]

Thanks and welcome back!

When testing everything works fine, except for move gestures. Here it seems both CLOCK_TICK and TEXT_HANDLE_MOVE are too weak to be noticeable on my test phone (Galaxy A3, LineageOS 17).
Regarding allowCustomDuration: alternatively we could allow custom duration, but reduce it to e.g. half the duration? Or maybe wait for feedback from users who have a very high custom duration.

Looking through, I noticed I'm not quite happy about the different ways of calling audio & haptic feedback. (note that some of this was already there previously and I didn't actively notice)

• in some places we call KeyboardActionListener with a specific haptic feedback event, in other places we call the AudioAndHapticFeedbackManager directly
  • I don't like it, but it's done because it's a code / text input and not a key press and probably nothing reasonable to be done about it
• in PointerTracker we implicitly expect onLongPressKey to do haptic feedback (and nothing else?), while for onPressKey it's explicit
  • in my opinion this should be done in a more consistent way, or alternatively call the AudioAndHapticFeedbackManager directly. Then at least we have the same inconsistency we already have, just in one more place.
• KeyboardActionListener is calling AudioAndHapticFeedbackManager or LatinIME for audio & haptic feedback
  • not necessarily part of this PR, but it could be moved from LatinIME to KeyboardActionListener

p.s.: didn't we want longer long-press duration for the spacebar -> language menu gesture? did that change at some point? it seems to just have the regular duration. i'd prefer 1.5x personally.

I don't remember changing it, and for me it's clearly longer than a "normal" long press. I think it's using some system default as opposed to the configurable duration of other key presses.

[Helium314]

Currently, this means they won't happen when doing virtual left/right arrow keys for bad input fields. We can revise this later, but I would want to suppress those virtual keypresses for good input fields before adding haptics to the bad ones. There's currently no easy way to tell the difference, and these revisions have already stretched the scope of this PR.

You could allow it for InputType.TYPE_NULL, which is typically used for terminal emulators. I guess this covers a considerable part of bad input fields.

[Helium314]

Note that this check can be rather slow, as it doesn't use cached data. On my phone it's usually fine, but in some wasteful apps (e.g. Logseq) it usually takes around 10 ms, and 30-100 ms when there is a lot of text.

[devycarol]

seriously, how can you mess up an input field so badly? or firefox is another example. what ungodly hacks are they doing to their EditTexts that causes the input connection to be janky and broken? the default EditText is clean and perfect with things like this. i don't get it.

[Helium314]

I have the impression this in most cases there is JavaScript involved in some way, e.g. Logseq essentially is JS in a WebView.

[devycarol]

I'm reticent to make any further structural changes in this PR since it's already a rather significant refactor of haptic handling. Is there anything you'd like me to change before merging?

It looks like future action items are:

• weakness of TEXT_HANDLE_MOVE and CLOCK_TICK on some devices
  • do i have it right that you can't feel it at all? the goal is to be very subtle, with the same haptics as manually dragging the caret pin around a text field. if your system vibrations are on and that's new to you, it may be an issue with your OS and/or device..
• consider replacing allowCustomDuration with a multiplier for custom durations
  • see if i can find defaults for AOSP or Pixel or whatever and maybe base it on that
  • alternatively, give each vibration its own config. replace the custom vibration duration setting with a dialog with a slider for each haptic type. plus a warning that certain options may burn out your vibration motor so be careful.
• structural consistency of haptic handling
• expensiveness of noTextAfterCursor()
• The unrelated issues that were brought up:
  • only do virtual arrows for things like keypresses on InputType.TYPE_NULL and related cases rather than smash the arrow keys any time we reach the end of a good input field. results in buggy behavior when using MS Teams and other cases where the arrow key escapes the input field for some reason.
  • maybe change spacebar long-press duration to 1.5x. i find i accidentally bring up the language menu a lot now that i use the space-move gesture more since i'm more comfortable with it.

[Helium314]

do i have it right that you can't feel it at all?

No (also commented on that in #1035). Also tried on my other phone, here CLOCK_TICK is noticeable every now and then, while TEXT_HANDLE_MOVE is not. Maybe in older Android Versions the intensity of TEXT_HANDLE_MOVE and CLOCK_TICK is extremely weak?
I tried allowing custom vibration duration for swipe, and even 2 ms are somewhat noticeable on one phone, and 1 ms on the other one where I notice CLOCK_TICK. Interestingly 1 ms is more noticeable than CLOCK_TICK...

[Helium314]

see if i can find defaults for AOSP or Pixel or whatever and maybe base it on that

For Android 9 (I guess 10 of the other phone is the same), both CLOCK_TICK and TEXT_HANDLE_MOVE result in VibrationEffect.EFFECT_TICK, but TEXT_HANDLE_MOVE doesn't have a fallback "if a hardware specific implementation doesn't exist" (VibrationEffect.java in android_frameworks_base).
So that would explain why I don't notice anything. This is likely going to affect many older devices (at least the ones released before Android 8).

alternatively, give each vibration its own config

That's probably too much, unless you (or other users) actually want it.

[Helium314]

structural consistency of haptic handling

That's definitely for later, and not really necessary at all.

expensiveness of noTextAfterCursor()

This is something that should be addressed. Ideas:

• we already have the text after cursor in the case of onMoveCursorHorizontally, and could re-use it (and I guess this is used much more often than vertical move)
• in case of audio & haptic feedback disabled, we don't need to do the check

[devycarol]

That's probably too much, unless you (or other users) actually want it.

I don't lol.

[Helium314]

My suggestion for horizonal cursor move

• expensiveness of noTextAfterCursor: make gestureMoveForwardHaptics(textAfterCursor: Boolean? = null) with the check textAfterCursor ?: !connection.noTextAfterCursor(), in onMoveCursorHorizontally call it with gestureMoveForwardHaptics(text.isNotEmpty()). I expect fixing this for horizonal movement is enough.
• not calling feedback on moveSteps == 0: this is also happening for emojis, but I think users will still want vibration when moving the cursor over emojis. I think we still should do the feedback if text is not empty.

On the cursor move vibration availability:
Users of old devices where for whatever reason there is no hardware implementation for TEXT_HANDLE_MOVE, and thus no vibration at all, should have a way of enabling vibration on cursor move.

• using CLOCK_TICK for everyone: might work, but since the effects could be different not just per Android version but even for each device model, it might not be the vibration users are used to from other keyboard apps. Did you test how CLOCK_TICK feels compared to TEXT_HANDLE_MOVE?
• allowing a custom duration with multiplier: I just tried it with a mulitplier of 10 (minimum duration 1) and it seems pretty reasonable

Btw what are your plans with the other (commented) HapticEvents?