Add inline emoji search

[eranl]

Adding focused emoji suggestions for strings preceded by :.
~s in the search string are treated as word separators.
Each word is looked up separately, and the suggestions are combined according to their total match against all words.
Includes all characters except white space in the emoji search string when initially typed, in order to support search strings with symbols and multiple words.
However, when restarting a search after moving the cursor etc., only the enclosing word characters are considered, in order to keep it simple. In most cases, the search string will likely be just one word. Is this inconsistency acceptable?
Copied some dictionary support logic from #1542.
Added logic for falling back to a country-less locale when loading a dictionary, which was necessary e.g. for the Lao (Laos) language. Should that logic be moved somewhere else?
Made a minor change to RichInputConnection.isCursorTouchingWord, so that it doesn't consider emojis as part of a word, in order to support an emoji search right after an existing emoji. This also helps with regular suggestions after an emoji, which are currently missing.

Partially addresses #259. More to come...

[Helium314]

~s in the search string are treated as word separators.

Is there any chance we could use space without breaking other stuff? Having to use ~ feels strange, and that's probably even worse for non-technical users.
Though I understand that having a convenient way to go back to normal mode is also desirable...

when restarting a search after moving the cursor etc., only the enclosing word characters are considered, in order to keep it simple. In most cases, the search string will likely be just one word. Is this inconsistency acceptable?

Didn't yet test, because a lot of shown emojis are not available on my Android 10 device. But I think it's ok.

Added logic for falling back to a country-less locale when loading a dictionary, which was necessary e.g. for the Lao (Laos) language.

Why is it necessary?

(I'll wait with review until I'm done with #1542)

[Edit] Maybe autocorrect would be an idea? It would be more like select first emoji on exiting search mode, and maybe shouldn't be implemented using the complicated autocorrect logic.

[eranl]

Is there any chance we could use space without breaking other stuff? Having to use ~ feels strange, and that's probably even worse for non-technical users.

Agreed on the problem. The other ideas I had were using newline or multiple whitespace as exit indicators, but they seemed unfriendly and the latter brittle. What's your take?
Is it possible/desirable to change the action key or comma key label and action to something like done during a search?

Added logic for falling back to a country-less locale when loading a dictionary, which was necessary e.g. for the Lao (Laos) language.

Why is it necessary?

The Lao (Laos) language's locale in HeliBoard is lo_LA, and the dictionary is for lo.

Maybe autocorrect would be an idea? It would be more like select first emoji on exiting search mode, and maybe shouldn't be implemented using the complicated autocorrect logic.

Good idea. Can we use the existing autocorrect setting or do we need a separate one?

[Helium314]

Is there any chance we could use space without breaking other stuff? Having to use ~ feels strange, and that's probably even worse for non-technical users.

Agreed on the problem. The other ideas I had were using newline or multiple whitespace as exit indicators, but they seemed unfriendly and the latter brittle. What's your take? Is it possible/desirable to change the action key or comma key label and action to something like done during a search?

Repurposing the action key could work. It could change the icon to clarify the select-emoji action.

The Lao (Laos) language's locale in HeliBoard is lo_LA, and the dictionary is for lo.

Isn't there already something very similar in DictionaryFactory? I think it's about finding the dictionary with the best match locale from assets.

Maybe autocorrect would be an idea? It would be more like select first emoji on exiting search mode, and maybe shouldn't be implemented using the complicated autocorrect logic.

Good idea. Can we use the existing autocorrect setting or do we need a separate one?

Good question. Maybe we can just have it always enabled?

[eranl]

Repurposing the action key could work. It could change the icon to clarify the select-emoji action.

Done. Had to change the order of InputTypeUtils.getImeOptionsActionIdFromEditorInfo, to make this work when the action is Enter.
Are there any restrictions on actionId? Do I need to handle the case of an existing custom action, and restore it after a search?

Isn't there already something very similar in DictionaryFactory? I think it's about finding the dictionary with the best match locale from assets.

Yes, but the logic for cached dicts is different. I pushed this logic down to getCachedDictsForLocale, since I think it could help with other dictionaries too. Let me know if that's ok.

Good idea. Can we use the existing autocorrect setting or do we need a separate one?

Good question. Maybe we can just have it always enabled?

I used the existing setting. When Done is pressed, if autocorrect is on, the default suggestion is accepted, otherwise the search string is committed.

Made a minor change to RichInputConnection.isCursorTouchingWord, so that it doesn't consider emojis as part of a word, in order to support an emoji search right after an existing emoji. This also helps with regular suggestions after an emoji, which are currently missing.

I moved this change to StringUtils.endsWithWordCodepoint and updated the test.

InputLogic is very complex, and there's a good chance I didn't handle all cases. Can you recommend a set of tests to perform?

[eranl]

Changes:

• Only do emoji search and show setting if toolbar is in compatible mode.
• Auto-show emoji suggestions.

[eranl]

Made changes to handleBackspaceEvent that enable restarting suggestions when cursor is in the middle of a word. This helps with both regular suggestions and emoji ones. Updated test.
Unify suggestion strip update mechanisms. Removed the handleMessage-based one.

[eranl]

Replaced calls to isSuggestionsEnabledPerUserSettings() with needsToLookupSuggestions() in several places, in order to fix autocorrect. This helps with both regular autocorrect and emoji.
Fixed mSuggestionsEnabledPerUserSettings value.
Moved setting.

[eranl]

Added logic for falling back to a country-less locale when loading a dictionary, which was necessary e.g. for the Lao (Laos) language.

Reverted, based on this.

[Helium314]

Oh, and the whole glide typing process may shows suggestions during input, maybe that should also be emojis?

I'm afraid now that we might need to considerably change the approach of getting the suggestions, which is awful at this stage. Unfortunately I also didn't think of trying glide typing earlier...

[eranl]

What are the statistics of glide typing? How often does a user have to pick a correction?

How many users use glide typing? Would it be acceptable to not support it for the initial inline emoji search release?

Oh, and the whole glide typing process may shows suggestions during input, maybe that should also be emojis?

That's an interesting idea. Something like:

1. get the top four (?) glide typing results
2. search emojis for each one and multiply the word score by the emoji score
3. merge them

It gets complicated with multiple words though - we'd need to try all word combinations.

Also, with multiple words, how do we keep track of the inline emoji search state? Since each word gets committed separately, I can't think of any natural way to do that, so we might need to introduce an external state flag, which carries the risk of inconsistencies.

[Helium314]

What are the statistics of glide typing? How often does a user have to pick a correction?

No idea in general. I guess it very much depends on the language and layout, as probably every language has some frequent words that use similar glide typing gestures.
Very rough guess for myself is maybe 10% need to be corrected, and in maybe half the time what I want is in the suggestions.

How many users use glide typing?

No idea, and I wouldn't know how to actually find out. Rough guess maybe 30 to 50%?

Would it be acceptable to not support it for the initial inline emoji search release?

In principle yes, but in my understanding, adding support for glide typing requires fundamental changes on how to get the suggestions.
The thing is that I don't want to add a feature that will need considerable changes very soon anyway.

That's an interesting idea. Something like

The multiple word thing could be an issue. Also I'm not sure whether users would want an emoji preview, or a word. Maybe for the first word it could be an emoji preview, and then just the word?
But anyway, this is something we can postpone / add later (and only if users explicitly want it).

Also, with multiple words, how do we keep track of the inline emoji search state?

I think we should make things work the same whether it's one or more words.
One possibility would be using composed data only for the last (currently entered) word. So we get all the existing words, and the composed data, which can be directly used in getSuggestionResults.
The existing words probably can't be in the word composer any more, so we'd need to ask mConnection to get the committed text after the last :. But with this I understand your concern about need for a flag and potential inconsistencies.

Anyway, since this PR will take a bit more work I'll remove it from my plans for 3.3.
What's left now (to finally get out a new release):

• Fix detection of keys on wide shifted popups #1789
• Fix detection of keys on emoji popups with long descriptions #1790
• release beta
• find a way to solve / improve New emoji dictionary causes strange corrections #1826 (ideally during beta, so either the fix or adjusted dictionaries are available in 3.3)

(I really feel it's necessary for me to only focus on very few thing for now, so no touching anything else)

Then find a way to continue here.

[eranl]

The multiple word thing could be an issue. Also I'm not sure whether users would want an emoji preview, or a word. Maybe for the first word it could be an emoji preview, and then just the word? But anyway, this is something we can postpone / add later (and only if users explicitly want it).

What do you suggest we do for now? For each word, use just the top glide suggestion (the one in composed data), ignoring the rest, and suggest emojis based on that?

But with this I understand your concern about need for a flag and potential inconsistencies.

Any alternative?

Anyway, since this PR will take a bit more work I'll remove it from my plans for 3.3.

Makes sense.

[smalltheif]

I think it's uncommon for people to type a colon after a space in normal typing.

Maybe I misunderstand your statement and just for a quick indication, in French-language typography, it's the rule to add a space before a colon.

[eranl]

Thanks for the info. Do you think that a colon following a space is not a good enough marker for starting an inline emoji search then? Should it be space, colon, and then a non-space, as @Helium314 suggested?

[smalltheif]

Do you think that a colon following a space is not a good enough marker for starting an inline emoji search then? Should it be space, colon, and then a non-space, as @Helium314 suggested?

Yes I think it would be preferable to start the inline emoji search with the marker space, colon, and then a non-space compared to just space, colon.
However it seems that some people like to get emoji suggestions without having to type a colon so if it possible we should aim for inline emoji search with space, colon, and then a non-space and just space.

[eranl]

It gets complicated with multiple words though - we'd need to try all word combinations.

I tried this approach, by keeping track of all glide word suggestions and translating the glide suggestion scores to weights such that they sum up to one per glide-typed word. However, I then realized that:

• glide suggestion scores are too similar, resulting in noise
• glide and regular typing can be mixed within a single emoji search

we'd need to ask mConnection to get the committed text after the last :.

So I ended up using this approach, abandoning the WordComposer approach, relying on mConnection.getTextBeforeCursor instead. I used the internal action as the mode marker.

Yes I think it would be preferable to start the inline emoji search with the marker space, colon, and then a non-space compared to just space, colon.

Done, with a minor twist - as before, the character before the colon can be any non-word character,

However it seems that some people like to get emoji suggestions without having to type a colon

This is unrelated - it's about emoji suggestions in regular typing, which is already supported.

maybe this autospace should be disabled during inline emoji search.

This is a catch - we can't tell if the user intended to type a normal word after the colon, or whether they intended to start an emoji search. I guess we can add a setting for this, or rely on SpacingAndPunctuations.isUsuallyPrecededBySpace(':').
There's a similar case when you try to start an emoji search right after punctuation (with either glide or normal typing, regardless of the autospace before word setting).

#1892 is required for preventing frequent triggering of inline emoji search while typing when the edit box contains an inline emoji search marker sequence.

[Helium314]

@eranl when normally entering emojis this works really nice!
What I noticed:

• In emoji search mode, empty suggestions (or punctuation suggestions if enabled) are shown after pressing space, until the next input. When pressing the action key now, the emoji on the action key is gone, but otherwise nothing happens.
• If there is text directly after a :, moving the cursor after this always seems to enable emoji search mode (which could easily be unwanted), but the action key does nothing. When selecting an emoji, all text since the emoji search start is replaced, so this part seems to work. This might be related to Reduce unnecessary input resets #1892?

[eranl]

when normally entering emojis

What do you mean by normally?

• In emoji search mode, empty suggestions (or punctuation suggestions if enabled) are shown after pressing space, until the next input

I can't reproduce this.

When pressing the action key now, the emoji on the action key is gone, but otherwise nothing happens.

In autocorrect mode, pressing the action key replaces the search string with the first suggestion. However, autocorrect mode was turned off when mWordComposer.isResumed(). I just removed that limitation, which might have been related to the previous implementation's reliance on mWordComposer (I don't remember the details. It's been over three months. 🙂).

• If there is text directly after a :, moving the cursor after this always seems to enable emoji search mode (which could easily be unwanted)

When the input is reset, if there is an emoji search start sequence (non-word character, colon, non-white-space character) in the buffer preceding the cursor, emoji search mode is entered. Isn't that what we want?
However, in autocorrect mode, with my latest commit, this now means that there's no easy way out of it if unintended.

I can see a few possible solutions:

1. disable autocorrect for emoji search
2. disable autocorrect for emoji search when mWordComposer.isResumed() (revert my latest commit)
3. limit emoji search triggering on input reset to some number of characters/words between the start sequence and the cursor

We may also want to change action button icon to ⏹️, 🔙, ✖️, Done, or something similar when autocorrect is off for any reason, to make it clear that you're exiting emoji search mode without selecting any emoji.

[eranl]

Just noticed that typing a time (e.g. 7:34) triggers a search, so I tweaked the search start sequence to be:

1. non-word, non-digit character
2. colon
3. non-white-space character

[Helium314]

What do you mean by normally?

Entering search mode and some characters (or maybe delete something again or move the cursor), then confirming with the action key.

Isn't that what we want?

I'd say it depends. Detecting cases where we don't want it can be tricky.
E.g. you have some longer text containing an emoji search sequence somewhere in the middle. You want to normally edit it, maybe append text. But a few lines (or sentences) before there is :h written for whatever reason. Now you'll be stuck in emoji search mode, and have to scan the previous (I think) 1024 characters for the :h.

Maybe some characters should stop emoji search mode. Or possibly: prevent emoji search mode from starting, but don't stop it if you're currently in search mode.
This could be newline, some punctuation characters (preceded by word characters?), probably quite a bunch of different things.
I suggest to move the logic to a separate method so we can use unit tests to check for (un)wanted behavior. Something like getInlineEmojiSearchString(textBeforeCursor) or getInlineEmojiSearchString(textBeforeCursor, isEmojiSearchActive).

We may also want to change action button icon to ⏹️, 🔙, ✖️, Done, or something similar when autocorrect is off for any reason, to make it clear that you're exiting emoji search mode without selecting any emoji.

Sorry I still haven't fully checked this (and no time right now). In this case, will the emoji search text be deleted, or will it stay?

[eranl]

Maybe some characters should stop emoji search mode. Or possibly: prevent emoji search mode from starting, but don't stop it if you're currently in search mode. This could be newline, some punctuation characters (preceded by word characters?), probably quite a bunch of different things.

My concern is that it might get too complex for users to understand and for developers to maintain. Maybe a search string length limit would be good enough?

I suggest to move the logic to a separate method so we can use unit tests to check for (un)wanted behavior. Something like getInlineEmojiSearchString(textBeforeCursor) or getInlineEmojiSearchString(textBeforeCursor, isEmojiSearchActive).

There's another factor to consider: typing vs. cursor moving. While typing, we only look for the search-start sequence, and then maintain the search mode. Only when you move the cursor, do we have to look back in the buffer to determine search mode.

We may also want to change action button icon to ⏹️, 🔙, ✖️, Done, or something similar when autocorrect is off for any reason, to make it clear that you're exiting emoji search mode without selecting any emoji.

In this case, will the emoji search text be deleted, or will it stay?

It stays.

[Helium314]

My concern is that it might get too complex for users to understand and for developers to maintain. Maybe a search string length limit would be good enough?

Well, we can just leave it and see whether this matches user expectations.
Then I'd rather put it in the experimental section for maybe 2 or 3 releases though.

There's another factor to consider: typing vs. cursor moving. While typing, we only look for the search-start sequence, and then maintain the search mode. Only when you move the cursor, do we have to look back in the buffer to determine search mode.

If you can reasonably move logic so unit tests can be used, I'd very much recommend to do so. Otherwise tuning this functionality could become very time consuming.

It stays.

⏹️ seems good to me. With ✖️ I'd expect the text to be deleted, and 🔙 has way too small text.

[eranl]

Well, we can just leave it and see whether this matches user expectations. Then I'd rather put it in the experimental section for maybe 2 or 3 releases though.

Makes sense.

If you can reasonably move logic so unit tests can be used, I'd very much recommend to do so.

Let me know if this is what you had in mind.

⏹️ seems good to me.

Done. On my device it's orange, BTW.

in autocorrect mode, this now means that there's no easy way out of it if unintended.

I can see a few possible solutions:
1. disable autocorrect for emoji search
2. disable autocorrect for emoji search when mWordComposer.isResumed()
3. limit emoji search triggering on input reset to some number of characters/words between the start sequence and the cursor

What's your take on this?