How to use the scrapelib.cache function in scrapelib

To help you get started, we’ve selected a few scrapelib examples, based on popular ways it is used in public projects.

Secure your code as it's written. Use Snyk Code to scan source code in minutes - no build needed - and fix issues immediately.

github associatedpress / geomancer / geomancer / mancers / base.py View on Github external
retry_attempts=5,
                 retry_wait_seconds=1,
                 header_func=None, 
                 cache_dir=CACHE_DIR,
                 api_key=None):
        
        super(BaseMancer, self).__init__(raise_errors=raise_errors,
                                             requests_per_minute=requests_per_minute,
                                             retry_attempts=retry_attempts,
                                             retry_wait_seconds=retry_wait_seconds,
                                             header_func=header_func)
        
        # We might want to talk about configuring an S3 backed cache for this
        # so we don't run the risk of running out of disk space.
        self.cache_dir = cache_dir
        self.cache_storage = scrapelib.cache.FileCache(self.cache_dir)
        self.cache_write_only = False
        
        # If subclass declares that an API Key is required and an API Key is not given, 
        # raise an ImportError
        if self.api_key_required and not self.api_key:
            raise ImportError('The %s mancer requires an API key and is disabled.' % self.name)