[Bug] Can't visit more than ten Websites with DuckDuckGo

[NXTler]

Greetings,
I discovered that you can't visit more than ten websites when using DuckDuckGo as your search engine. I suspect that this is due to DuckDuckGo not loading more than ten links when not specifically clicking the "show more results" button.

[NXTler]

Tinkered a bit with the WebSearch Selenium code and could confirm that this is due to DuckDuckGo only giving 10 results initially.
I was able to work around this issue by enabling autoloading in the link parameters and then scroll to the bottom with some waiting time.

Here is how the modified link looks now: https://duckduckgo.com/?kl=wt-wt&kp=-2&kav=1&kf=-1&kac=-1&k1=-1&kv=n&q=

• kav=-1 enables Auto-load Results
• kf=-1 turns off Favicons, resulting in faster loading speed
• kac=-1 turns off Auto-suggestions, they are not needed so why not disable them
• k1=-1 disables Advertisements, probably helps with getting higher quality results
• kv=-1 disables page numbers, also not needed

Here is the scrolling and waiting code I inserted after await driver.wait(until.elementLocated(By.id('web_content_wrapper')), config.TIMEOUT);:

        await driver.executeScript('window.scrollTo(0, document.body.scrollHeight)');
        await driver.sleep(1000);

This is just a quick and dirty fix to the issue, we probably need to introduce a max links parameter and repeatedly scroll until the desired amount is reached. Though, we would need to adjust how a timeout is detected.

[Cohee1207]

kf=-1 turns off Favicons, resulting in faster loading speed
kac=-1 turns off Auto-suggestions, they are not needed so why not disable them
k1=-1 disables Advertisements, probably helps with getting higher quality results

Good suggestions about those. I think I could add those even without the scrolling hack.

Additional features I can suggest too:

• kbh=-1 disable page titles (unused)
• ko=-1 disable page header (faster load)

[NXTler]

Are instant answers used? Else we could turn off headers completely with -2.

[NXTler]

Where did you find the kbh parameter, haven't seen it documented anywhere?
We might be able to disable some more stuff, as I don't feel like spamming DuckDuckGo with useless requests.

[Cohee1207]

Where did you find the kbh parameter, haven't seen it documented anywhere?

You can set it via DDG preferences page.

[NXTler]

Found a few that might be interesting:

• kz=-1 disables the quick answer boxes
• kat=-1 turns off location specific results
• ksn=5 increases the text snipped size to 5 lines
• kbg=-1 hides the AI chat Icon
• kbe=0 turns off AI answers
• kpsb=-1 hides the always private reminder

[NXTler]

Here is what I came up with for now:

        for (let NumberOfLinks = 0, Attempts = 0; 30 >= NumberOfLinks && 3 >= Attempts;) {
            await driver.executeScript('window.scrollTo(0, document.body.scrollHeight)');
            await driver.sleep(1000);
            const links = await driver.findElements(By.css('[data-testid="result-title-a"]'));
            if (NumberOfLinks >= links.length) {
                Attempts++
            } else {
                NumberOfLinks = links.length;
            }
        }

I'm no developer, so I don't know if this is a good/proper approach. 30 is just a placeholder right now and should be replaced with the visit count set by the extension, however I have no clue how to do that.

This is the link I'm currently using for testing, and it seems very fast:
https://duckduckgo.com/?kl=wt-wt&kp=-2&kav=1&kf=-1&kac=-1&kbh=-1&ko=-1&k1=-1&kv=n&kz=-1&kat=-1&kbg=-1&kbe=0&kpsb=-1&q=

[Cohee1207]

We're still talking about the plugin, but the real limit is here:

https://github.com/SillyTavern/Extension-WebSearch/blob/main/settings.html#L172

[NXTler]

I already saw that, but it seemed to work fine when I entered 30 in the settings.
I mean, the plugin also needs to know how much it should load, that's why I did it like this.

[NXTler]

Worked a bit more on it, the visit count isn't limited anymore and directly controls how many links the previous code aims for. The whole thing seems fully functional. Is it okay when I do the PRs only fixing this for DuckDuckGo?

[Cohee1207]

Technically this was and still is a provider limitation. You can uncap the limit on the UI, but all the other providers except DDG via a plugin would not benefit from that.

[NXTler]

Yeah, that's why I'm asking if that would be fine to do. The only potential solution I could find for Google, is by making use of the &num= and/or the &start= link parameters.

• &num= Determines the number of links per page up to 100
• &start= Can be used to start the search at a different page.

Google search isn't working for me, as it instantly detects that a bot is doing the search. So someone else would need to implement this.