colly

1.2.0 Opens a new window with list of versions in this module.

Elegant Scraper and Crawler Framework for Golang For more information about how to use this package see README

Latest version published 5 years ago

GitHub

Package Health Score

77 / 100

security
No known security issues
popularity
Key ecosystem project
maintenance
Inactive
community
Active

Popularity

Key ecosystem project

Imported By: 5.03K
GitHub Stars: 22.07K
Forks: 1.71K
Contributors: 100

Usage Popularity

Based on project statistics from the GitHub repository for the Golang package colly, we found that it has been 22,066 times.

The popularity score for Golang modules is calculated based on the number of stars that the project has on GitHub as well as the number of imports by other modules.

Security

No known security issues

Security risks

Direct Vulnerabilities: 0
C
0
H
0
M
0
L

Indirect Vulnerabilities: Check with Snyk

Release Date

Feb 13, 2019

License

Apache-2.0

Security Policy

We found a way for you to contribute to the project! Looks like colly is missing a security policy.

You can connect your project's repository to Snyk to stay up to date on security alerts and receive automatic fix pull requests.

Keep your project free of vulnerabilities with Snyk

Maintenance

Inactive

Commit Frequency

Open Issues: 146
Open PR: 36
Last Release: 5 years ago
Last Commit: 29 days ago

Further analysis of the maintenance status of github.com/gocolly/colly based on released golang versions cadence, the repository activity, and other data points determined that its maintenance is Inactive.

An important project maintenance signal to consider for github.com/gocolly/colly is that it hasn't seen any new versions released to golang in the past 12 months, and could be considered as a discontinued project, or that which receives low attention from its maintainers.

As a healthy sign for on-going project maintenance, we found that the GitHub repository had at least 1 pull request or issue interacted with by the community.

Community

Active

Readme.md: Yes
Contributing.md: Yes
Code of Conduct: No
Contributors: 100
Funding: No

With more than 10 contributors for the github.com/gocolly/colly repository, this is possibly a sign for a growing and inviting community.

We found a way for you to contribute to the project! Looks like github.com/gocolly/colly is missing a Code of Conduct.

Embed Package Health Score Badge

Keep your project healthy

Check your go.mod

Ensure all the packages you're using are healthy and well-maintained

Readme

Colly

Lightning Fast and Elegant Scraping Framework for Gophers

Colly provides a clean interface to write any kind of crawler/scraper/spider.

With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.

Features

Clean API
Fast (>1k request/sec on a single core)
Manages request delays and maximum concurrency per domain
Automatic cookie and session handling
Sync/async/parallel scraping
Caching
Automatic encoding of non-unicode responses
Robots.txt support
Distributed scraping
Configuration via environment variables
Extensions

Example

func main() {
	c := colly.NewCollector()

	// Find and visit all links
	c.OnHTML("a[href]", func(e *colly.HTMLElement) {
		e.Request.Visit(e.Attr("href"))
	})

	c.OnRequest(func(r *colly.Request) {
		fmt.Println("Visiting", r.URL)
	})

	c.Visit("https://go-colly.org/")
}

See examples folder for more detailed examples.

Installation

Add colly to your go.mod file:

module github.com/x/y

go 1.14

require (
        github.com/gocolly/colly/v2 latest
)

Bugs

Bugs or suggestions? Visit the issue tracker or join #colly on freenode

Other Projects Using Colly

Below is a list of public, open source projects that use Colly:

greenpeace/check-my-pages Scraping script to test the Spanish Greenpeace web archive.
altsab/gowap Wappalyzer implementation in Go.
jesuiscamille/goquotes A quotes scraper, making your day a little better!
jivesearch/jivesearch A search engine that doesn't track you.
Leagify/colly-draft-prospects A scraper for future NFL Draft prospects.
lucasepe/go-ps4 Search playstation store for your favorite PS4 games using the command line.
yringler/inside-chassidus-scraper Scrapes Rabbi Paltiel's web site for lesson metadata.
gamedb/gamedb A database of Steam games.
lawzava/scrape CLI for email scraping from any website.
eureka101v/WeiboSpiderGo A sina weibo(chinese twitter) scraper
Go-phie/gophie Search, Download and Stream movies from your terminal
imthaghost/goclone Clone websites to your computer within seconds.
superiss/spidy Crawl the web and collect expired domains.
docker-slim/docker-slim Optimize your Docker containers to make them smaller and better.
seversky/gachifinder an agent for asynchronous scraping, parsing and writing to some storages(elasticsearch for now)
eval-exec/goodreads crawl all tags and all pages of quotes from goodreads.

If you are using Colly in a project please send a pull request to add it to the list.

Contributors

This project exists thanks to all the people who contribute. [Contribute].

Backers

Thank you to all our backers! 🙏 [Become a backer]

License

FAQs

What is colly?

Elegant Scraper and Crawler Framework for Golang. Visit Snyk Advisor to see a full health score report for colly, including popularity, security, maintenance & community analysis.

Is colly popular?

The golang package colly receives a total of ? weekly downloads. As such, colly popularity was classified as a key ecosystem project. Visit the popularity section on Snyk Advisor to see the full health analysis.

Is colly well maintained?

We found indications that colly is an Inactive project. See the full package health analysis to learn more about the package maintenance status.

Is colly safe to use?

The golang package colly was scanned for known vulnerabilities and missing license, and no issues were found. Thus the package was deemed as safe to use. See the full health analysis review.

Last updated on 25 April-2024, at 01:46 (UTC).

Build a secure application checklist

Select a recommended open source package

Minimize your risk by selecting secure & well maintained open source packages

DONE

Scan your app for vulnerabilities

Scan your application to find vulnerabilities in your: source code, open source dependencies, containers and configuration files

SCAN NOW

Fix identified vulnerabilities

Easily fix your code by leveraging automatically generated PRs

AUTO FIX

Monitor for new issues

New vulnerabilities are discovered every day. Get notified if your application is affected

MONITOR APP

colly

Elegant Scraper and Crawler Framework for Golang For more information about how to use this package see README

Ensure you're using the healthiest golang packages

Package Health Score

Popularity

Usage Popularity

Security

Security risks

Maintenance

Commit Frequency

Community

Embed Package Health Score Badge

Keep your project healthy

Check your go.mod

Readme

Colly

Features

Example

Installation

Bugs

Other Projects Using Colly

Contributors

Backers

Sponsors

License

FAQs

What is colly?

Is colly popular?

Is colly well maintained?

Is colly safe to use?

Build a secure application checklist

Select a recommended open source package

Scan your app for vulnerabilities

Fix identified vulnerabilities

Monitor for new issues

Package Health Score

Usage Popularity

Security risks

Commit Frequency

Embed Package Health Score Badge

Keep your project healthy

Check your go.mod

Readme

Colly

Features

Example

Installation

Bugs

Other Projects Using Colly

Contributors

Backers

Sponsors

License

FAQs

What is colly?

Is colly popular?

Is colly well maintained?

Is colly safe to use?

Build a secure application checklist

Select a recommended open source package

Scan your app for vulnerabilities

Source Code

Open Source

Container

Config

Fix identified vulnerabilities

Monitor for new issues