Search results need match-length weighting

[hinerm]

The search results should prefer pages with shorter lengths, or take a penalty for each character that doesn't match the search string.

For example:

The page simply titled TrackMate should be the first hit here:

Same for Fiji:

[ctrueden]

• typesense docs for the indexer
• typesense docs for searching from clients
• The ImageJ search uses the typesense-instantsearch-adapter.js, plain version.
• It is possible to customize the widget to filter the results using a transform

So ways forward might be:

• Change how we index the content
• Change the search API call we use for retrieving the content
• Postprocess the results in way along the lines of what's discussed above

[ctrueden]

I opted for option 3, "Postprocess the results in way along the lines of what's discussed above". For now it just sorts each page of results by its edit distance to the query string. @hinerm Feel free to implement something fancier if you wish.

[hinerm]

@ctrueden it's not quite clear to me how search is working.. for example, here's the result for fij:

oddly the Fiji site doesn't pop up until I expand the search string to fiji

Some times the results are just confusing, like if I search for ij why would it match Virtual above, say fiji:

I don't think this is something we should devote time to manually tuning.. but maybe there's another off-the-shelf option?

[ctrueden]

I think the search is amazingly better than anything we have ever had, and I think it's not worth our time to prototype anything else. But if you really want to, you could try Meilisearch. Here is a comparison chart of the most popular search systems.

I am also personally satisfied with 0b20222, because it makes it so that when you type the complete word (e.g. "fiji" or "trackmate"), then that's the page that comes up on top, as it should. It's true that word fragments, and in particular fragments that do not correspond to the start of words, do not always come up first, because sometimes they aren't in the top 10 results. One way to mitigate that would be to increase the results-per-page to 20 or 25, increasing the likelihood that such pages are in the first batch, and then get sorted to the top by the transform function.

The other thing you could do is post on their issue tracker and/or StackOverflow, to get help from people who know this system. I'd definitely recommend we do that before trying to switch to a whole new search infrastructure.

As for ij matching virtual over fiji: I agree that that is unfortunate; we could try decreasing the "number of typos" setting from 1 to 0 so that that problem doesn't happen. To keep fuzzy search available as an option, we could add a "Fuzzy search" checkbox or even a "fuzziness" slider that defaults to 0, but can be increased to 1, 2, or 3, in cases where you aren't finding what you're looking for or aren't sure of the spelling.

[ctrueden]

With 37eea26, I made the following improvements:

• Increase results per page from 10 to 16.
• Penalize inexact/fuzzy keyword matches below exact ones, and non-beginning-of-word fragment matches below beginning-of-word ones.
• Adjust the layout so it fits at the same height as before (so that mobile search does not go off the bottom of the screen on common phone screen dimensions).
• Make the layout less crappy at narrow/mobile widths.
• Improve vertical alignment of search result rows.

Now, searching for ij displays "IJ Blob" as the first match, and all matches whose titles do not include ij in them are sorted below those that do.

With 685ce71, the results sorting is now case-insensitive.

After these changes, fij does bring up the Fiji page first. @hinerm Could you please test it again, and let me know if you are satisfied now? My only remaining concern personally is 16 is slightly too tall on my phone browser, so I might reduce it to 15 instead.

[imagejan]

I like the search results very much now. I think it would be nice if the URL slug of the page would also be searched (in particular for user pages where you might want to search for the github handle, e.g. ctrueden or imagejan, but also for other pages maybe, if you only remember part of the URL that you read somewhere, but it doesn't fit the page title verbatim).