Package Health Score

45 / 100

security
Security review needed
popularity
Limited
maintenance
Inactive
community
Sustainable

Popularity

Limited

GitHub Stars: 268
Forks: 49
Contributors: 6

Direct Usage Popularity

Based on project statistics from the GitHub repository for the PyPI package scrapy-deltafetch, we found that it has been starred 268 times.

Security

Security review needed

Security and license risk for latest version

Release Date: Sep 20, 2021
Direct Vulnerabilities: 0
C
0
H
0
M
0
L
Indirect Vulnerabilities: 0
C
0
H
0
M
0
L
License Risk: 0
H
2
M
0
L

All security vulnerabilities belong to production dependencies of direct and indirect packages.

License

BSD-2-Clause

Security Policy

We found a way for you to contribute to the project! Looks like scrapy-deltafetch is missing a security policy.

You can connect your project's repository to Snyk to stay up to date on security alerts and receive automatic fix pull requests.

Keep your project free of vulnerabilities with Snyk

Maintenance

Inactive

Commit Frequency

No Recent Commits

Open Issues: 14
Open PR: 5
Last Release: 3 years ago
Last Commit: 3 years ago

Further analysis of the maintenance status of scrapy-deltafetch based on released PyPI versions cadence, the repository activity, and other data points determined that its maintenance is Inactive.

An important project maintenance signal to consider for scrapy-deltafetch is that it hasn't seen any new versions released to PyPI in the past 12 months, and could be considered as a discontinued project, or that which receives low attention from its maintainers.

As a healthy sign for on-going project maintenance, we found that the GitHub repository had at least 1 pull request or issue interacted with by the community.

Community

Sustainable

Readme: Yes
Contributing.md: No
Code of Conduct: No
Contributors: 6
Funding: No

This project has seen only 10 or less contributors.

We found a way for you to contribute to the project! Looks like scrapy-deltafetch is missing a Code of Conduct.

Embed Package Health Score Badge

Keep your project healthy

Check your requirements.txt

Ensure all the packages you're using are healthy and well-maintained

Snyk Vulnerability Scanner

Get health score & security insights directly in your IDE

Package

Python Versions Compatibility: ==3.5.*, ==3.6.*, ==3.7.*, ==3.8.*, ==3.9.*

Age: 8 years
Latest Release: 3 years ago
Dependencies: 1 Direct / 40 Total
Versions: 9
Maintainers: 4
Wheels: OS Independent

Readme

scrapy-deltafetch

This is a Scrapy spider middleware to ignore requests to pages seen in previous crawls of the same spider, thus producing a "delta crawl" containing only new requests.

This also speeds up the crawl, by reducing the number of requests that need to be crawled, and processed (typically, item requests are the most CPU intensive).

DeltaFetch middleware uses Python's dbm package to store requests fingerprints.

Installation

Install scrapy-deltafetch using pip:

$ pip install scrapy-deltafetch

Configuration

Add DeltaFetch middleware by including it in SPIDER_MIDDLEWARES in your settings.py file:
```
SPIDER_MIDDLEWARES = {
    'scrapy_deltafetch.DeltaFetch': 100,
}
```
Here, priority 100 is just an example. Set its value depending on other middlewares you may have enabled already.
Enable the middleware using DELTAFETCH_ENABLED in your settings.py:
```
DELTAFETCH_ENABLED = True
```

Usage

Following are the different options to control DeltaFetch middleware behavior.

Supported Scrapy settings

DELTAFETCH_ENABLED — to enable (or disable) this extension
DELTAFETCH_DIR — directory where to store state
DELTAFETCH_RESET — reset the state, clearing out all seen requests

These usually go in your Scrapy project's settings.py.

Supported Scrapy spider arguments

deltafetch_reset — same effect as DELTAFETCH_RESET setting

Example:

$ scrapy crawl example -a deltafetch_reset=1

Supported Scrapy request meta keys

deltafetch_key — used to define the lookup key for that request. by default it's Scrapy's default Request fingerprint function, but it can be changed to contain an item id, for example. This requires support from the spider, but makes the extension more efficient for sites that many URLs for the same item.
deltafetch_enabled - if set to False it will disable deltafetch for some specific request

scrapy-deltafetch dependencies

scrapy

FAQs

What is scrapy-deltafetch?

Scrapy middleware to ignore previously crawled pages. Visit Snyk Advisor to see a full health score report for scrapy-deltafetch, including popularity, security, maintenance & community analysis.

Is scrapy-deltafetch popular?

The python package scrapy-deltafetch receives a total of 776 weekly downloads. As such, scrapy-deltafetch popularity was classified as limited. Visit the popularity section on Snyk Advisor to see the full health analysis.

Is scrapy-deltafetch well maintained?

We found indications that scrapy-deltafetch is an Inactive project. See the full package health analysis to learn more about the package maintenance status.

Is scrapy-deltafetch safe to use?

While scanning the latest version of scrapy-deltafetch, we found that a security review is needed. A total of 0 vulnerabilities or license issues were detected. See the full security scan results.

Last updated on 23 November-2024, at 02:56 (UTC).

Build a secure application checklist

Select a recommended open source package

Minimize your risk by selecting secure & well maintained open source packages

DONE

Scan your app for vulnerabilities

Scan your application to find vulnerabilities in your: source code, open source dependencies, containers and configuration files

SCAN NOW

Fix identified vulnerabilities

Easily fix your code by leveraging automatically generated PRs

AUTO FIX

Monitor for new issues

New vulnerabilities are discovered every day. Get notified if your application is affected

MONITOR APP

scrapy-deltafetch

Scrapy middleware to ignore previously crawled pages For more information about how to use this package see README

Ensure you're using the healthiest python packages

Package Health Score

Popularity

Direct Usage Popularity

Security

Security and license risk for latest version

Maintenance

Commit Frequency

Community

Embed Package Health Score Badge

Keep your project healthy

Check your requirements.txt

Snyk Vulnerability Scanner

Package

Readme

scrapy-deltafetch

Installation

Configuration

Usage

Supported Scrapy settings

Supported Scrapy spider arguments

Supported Scrapy request meta keys

scrapy-deltafetch dependencies

FAQs

What is scrapy-deltafetch?

Is scrapy-deltafetch popular?

Is scrapy-deltafetch well maintained?

Is scrapy-deltafetch safe to use?

Build a secure application checklist

Select a recommended open source package

Scan your app for vulnerabilities

Fix identified vulnerabilities

Monitor for new issues

Package Health Score

Direct Usage Popularity

Security and license risk for latest version

Commit Frequency

Embed Package Health Score Badge

Keep your project healthy

Check your requirements.txt

Snyk Vulnerability Scanner

Readme

scrapy-deltafetch

Installation

Configuration

Usage

Supported Scrapy settings

Supported Scrapy spider arguments

Supported Scrapy request meta keys

scrapy-deltafetch dependencies

FAQs

What is scrapy-deltafetch?

Is scrapy-deltafetch popular?

Is scrapy-deltafetch well maintained?

Is scrapy-deltafetch safe to use?

Build a secure application checklist

Select a recommended open source package

Scan your app for vulnerabilities

Source Code

Open Source

Container

Config

Fix identified vulnerabilities

Monitor for new issues