View all icrawler analysis

How to use the icrawler.utils.ProxyPool function in icrawler

To help you get started, we’ve selected a few icrawler examples, based on popular ways it is used in public projects.

Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.

hellock / icrawler / icrawler / crawler.py View on Github

def set_proxy_pool(self, pool=None):
        """Construct a proxy pool

        By default no proxy is used.

        Args:
            pool (ProxyPool, optional): a :obj:`ProxyPool` object
        """
        self.proxy_pool = ProxyPool() if pool is None else pool