Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Proxy Rotation Setup to Reduce CAPTCHA Chances #65

Open
fasihhussain00 opened this issue Dec 28, 2024 · 0 comments · May be fixed by #66
Open

Feature Request: Proxy Rotation Setup to Reduce CAPTCHA Chances #65

fasihhussain00 opened this issue Dec 28, 2024 · 0 comments · May be fixed by #66

Comments

@fasihhussain00
Copy link

implementing a Proxy Rotation feature to minimize the likelihood of encountering CAPTCHAs during web scraping tasks. By rotating proxies frequently, we can reduce the chances of IP addresses being flagged and blocked, which often leads to CAPTCHA challenges.

here is the example code

const scraper = async () => {
    // create a proxy list
    const proxies = [
        'http://160.86.242.23:8080',
        'http://200.60.145.167:8084',
        // ...,
        'http://188.166.229.121:80',
    ];

    // randomize the proxies per request
    const randomProxy = proxies[Math.floor(Math.random() * proxies.length)];

    // launch a browser instance with the
    // --proxy-server flag enabled
    const browser = await puppeteer.launch({
        args: [`--proxy-server=${randomProxy}`],
    });
    // open a new page in the current browser context
    const page = await browser.newPage();

    // visit the target page
    await page.goto('https://httpbin.org/ip');

    // extract the IP the request comes from
    // and print it
    const body = await page.waitForSelector('body');
    const ip = await body.getProperty('textContent');
    console.log(await ip.jsonValue());

    await browser.close();
};
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant