Repeated searches results in anomaly detection

[sarfudheen]

Description

When using DuckDuckGo's API for querying, adding the ss_mkt parameter seems to trigger anomaly detection. Specifically, if ss_mkt=FR (or any other country code variant) is appended to the request URL, the response includes a script with window.execDeep = function() {DDG.deep.anomalyDetectionBlock({...})}, which appears to be blocking or altering the query response.

Steps to Reproduce

1. Use the following working URL to perform a query (e.g., what is weather today):

https://links.duckduckgo.com/d.js?q=what%20is%20weather%20today&kl=wt-wt&l=us-en&p=1&s=0&a=h_&dl=en&ct=FR&df=m&vqd=4-130790525484318490675696036619647809147&bing_market=en-US&p_ent=&dp=a2VtvYQbwBUAnJjgW_GROFNV_mc6SfdjMK9PLqH42pqZ3fsCcOsNuDJagf0YUf4NazjDeNsiE8uQBCPRHf0rDV9gnMMek-we5JwQZ87pMeFh4-7e5KVEkLAYafYyGqW7.FOFBUBx1to6nRs7UUN6QSA&sjs=forecast&perf_id=2d03f98f2d7a55da&parent_perf_id=7fe2f5d308add94a&host_region=eun&sp=1&dfrsp=1&bpa=1&baa=1&bcca=1&btaa=1&wrap=1&aps=0&bccaexp=b&biaexp=b&btaaexp=b&eclsexp=b&litexp=b&msvrtexp=b

2. Add the ss_mkt parameter, e.g., ss_mkt=FR, to the URL:

https://links.duckduckgo.com/d.js?q=what%20is%20weather%20today&kl=wt-wt&l=us-en&p=1&s=0&a=h_&dl=en&ct=FR&df=m&vqd=4-130790525484318490675696036619647809147&bing_market=en-US&p_ent=&dp=a2VtvYQbwBUAnJjgW_GROFNV_mc6SfdjMK9PLqH42pqZ3fsCcOsNuDJagf0YUf4NazjDeNsiE8uQBCPRHf0rDV9gnMMek-we5JwQZ87pMeFh4-7e5KVEkLAYafYyGqW7.FOFBUBx1to6nRs7UUN6QSA&sjs=forecast&perf_id=2d03f98f2d7a55da&parent_perf_id=7fe2f5d308add94a&host_region=eun&sp=1&dfrsp=1&bpa=1&baa=1&bcca=1&btaa=1&wrap=1&aps=0&bccaexp=b&biaexp=b&btaaexp=b&eclsexp=b&litexp=b&msvrtexp=b&ss_mkt=FR
3. Observe the response which includes window.execDeep = function() {DDG.deep.anomalyDetectionBlock({...})} instead of the expected data.

Expected Behavior

Adding the ss_mkt parameter should not trigger anomaly detection, and the query should return results as expected.

Actual Behavior

Adding ss_mkt=FR results in the DuckDuckGo response containing a script with DDG.deep.anomalyDetectionBlock, which seems to block or modify the response.

Environment

• Query Tool: DuckDuckGo API
• Affected Parameter: ss_mkt

Additional Information

This anomaly is observed specifically when setting the ss_mkt parameter to a value such as FR. Other values may produce similar results.

Request

If the current DuckDuckGo API does not support the ss_mkt parameter, please consider removing or disabling it to avoid triggering anomaly detection blocks.

[flaviogoncalves]

I have the same issue using langchain and the duck duck search library, please consider removing the parameter from the code.

[Tomas2D]

Temporary fix

import { search, SafeSearchType } from 'duck-duck-scrape';

const results = await search('node.js', {
  safeSearch: SafeSearchType.STRICT
}, {
  uri_modifier: (rawUrl) => {
    const url = new URL(rawUrl);
    url.searchParams.delete("ss_mkt");  // remove the parameter
    return url.toString();
  }
);

[flaviogoncalves]

Thanks for the suggestion. I've forked the project and made a fix replacing ss_mkt by bing_market. I have to prepare a pull request, but super busy this week.

[sarfudheen]

Hello,
Is there any reference where the have mentioned about replacing ss_mkt by bing_market?.

For me replacing ss_mkt by bing_market flag does not give any answer.

Thank you.

[Snazzah]

Looking at doing a DDG request now does seem the query parameter was replaced with bing_market, also assuming the values were somewhat changed since the default is "us" with what im receiving being "en-US". Don't know of other changes but that seems to be the most obvious one.

@flaviogoncalves If you just make a PR and summarize your changes I can review and merge

[dzek69]

It seems that new version is not fixing the issue. I can do one query per like ~15-30 minutes

[Snazzah]

Not sure how else to fix this other than adding common browser headers to each request, I don't know what else I really can do to solve this. You might just need to deal with proxies whenever you hit an anomaly. Unless it's something within the URL I'm forming thats causing it to trigger more, I don't think there is a good solution since I can't really know whats triggering it.

At the moment it seems like you can do a request, but doing more (probably from the same IP) would result in anomaly detection.