Bypass proxy / pass specific proxy tier for requests

[jensmeichler]

I have a use case where I want to have a crawler running permanently. This crawler has a tieredProxyList set up that it will iterate over in case some of them don't work. For scraping some pages I don't want to use proxies to reduce the amount of money I am spending on them (When I scrape my own page I don't want to proxy, but I want to use the same logic / handlers. Is it possible to specify either the proxy that should be used for specific requests? Or maybe even the proxy tier?

Basic Setup:

        const proxyConfiguration = new ProxyConfiguration({tieredProxyUrls: [{'proxyTier1'], ['proxyTier2']]});

        const crawler = new PlaywrightCrawler(
            {
                keepAlive: true,
                proxyConfiguration: proxyConfiguration,
                // ...
            },
        );

        // ...

        crawler.addRequests(requestsWhereWeWantProxies);
        crawler.addRequests(requestsWhereWeDontWantProxies);

It would be nice to be able to do something like:

        crawler.addRequests(requestsWhereWeWantProxies);
        crawler.addRequests(requestsWhereWeDontWantProxies.map((request) => ({...request, proxy: null}));

or

        const proxyConfiguration = new ProxyConfiguration({tieredProxyUrls: [{'proxyTier1'], ['proxyTier2'], [null]]});

        // ...

        crawler.addRequests(requestsWhereWeWantProxies);
        crawler.addRequests(requestsWhereWeDontWantProxies.map((request) => ({...request, proxyTier: 2}));

TL;DR;
Is it possible to bypass proxies for specific requests?

[janbuchar]

Hi @jensmeichler and thanks for your interest in Crawlee! I guess you could experiment with the newUrlFunction option of ProxyConfiguration - that gives you total control over what proxy should be used for a request.

If you want to keep the tiered proxies functionality, that could be done by keeping another instance of ProxyConfiguration and delegating to that from your newUrlFunction.

Bear in mind that this is just an idea, no guarantees 🙂

[jensmeichler]

That indeed helps! I already played around with the newUrlFunction, but thought I would lose the tieredProxy feature then.
I'll try using a second instance of the ProxyConfiguration to use both tieredProxy and bypassing proxy for certain domains.

Thank you very much!