View all icrawler analysis

How to use the icrawler.utils.thread_pool.Worker function in icrawler

To help you get started, we’ve selected a few icrawler examples, based on popular ways it is used in public projects.

Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.

hellock / icrawler / icrawler / utils / thread_pool.py View on Github

def __init__(self, *args, **kwargs):
        super(Worker, self).__init__(*args, **kwargs)
        self.daemon = True
        self.quit = False

hellock / icrawler / icrawler / utils / thread_pool.py View on Github

def init_workers(self, *args, **kwargs):
        self.workers.clear()
        for i in range(self.thread_num):
            worker = Worker(
                target=self.worker_exec,
                name='{}-{:03d}'.format(self.name, i + 1),
                args=args,
                kwargs=kwargs)
            self.workers.append(worker)