View amundsen-databuilder on Snyk Open Source Advisor

Package Health Score

81 / 100

security
Security review needed
popularity
Popular
maintenance
Sustainable
community
Active

Explore Similar Packages

Popularity

Popular

GitHub Stars: 4.27K
Forks: 943
Contributors: 220

Direct Usage Popularity

Based on project statistics from the GitHub repository for the PyPI package amundsen-databuilder, we found that it has been starred 4,270 times.

Security

Security review needed

Security and license risk for latest version

Release Date: Dec 7, 2023
Direct Vulnerabilities: 0
C
0
H
0
M
0
L
Indirect Vulnerabilities: 0
C
0
H
0
M
0
L
License Risk: 0
H
5
M
0
L

All security vulnerabilities belong to production dependencies of direct and indirect packages.

License

Apache-2.0

Security Policy

Yes

You can connect your project's repository to Snyk to stay up to date on security alerts and receive automatic fix pull requests.

Keep your project free of vulnerabilities with Snyk

Maintenance

Sustainable

Commit Frequency

Open Issues: 24
Open PR: 9
Last Release: 5 months ago
Last Commit: 14 days ago

Further analysis of the maintenance status of amundsen-databuilder based on released PyPI versions cadence, the repository activity, and other data points determined that its maintenance is Sustainable.

We found that amundsen-databuilder demonstrates a positive version release cadence with at least one new version released in the past 12 months.

As a healthy sign for on-going project maintenance, we found that the GitHub repository had at least 1 pull request or issue interacted with by the community.

Community

Active

Readme: Yes
Contributing.md: Yes
Code of Conduct: Yes
Contributors: 220
Funding: No

A good and healthy external contribution signal for amundsen-databuilder project, which invites more than one hundred open source maintainers to collaborate on the repository.

Embed Package Health Score Badge

Keep your project healthy

Check your requirements.txt

Ensure all the packages you're using are healthy and well-maintained

Snyk Vulnerability Scanner

Get health score & security insights directly in your IDE

Package

Python Versions Compatibility: >=3.7

Age: 5 years
Latest Release: 5 months ago
Dependencies: 25 Direct / 50 Total
Versions: 173
Maintainers: 1
Wheels: OS Independent

Readme

Amundsen

Amundsen is a data discovery and metadata engine for improving the productivity of data analysts, data scientists and engineers when interacting with data. It does that today by indexing data resources (tables, dashboards, streams, etc.) and powering a page-rank style search based on usage patterns (e.g. highly queried tables show up earlier than less queried tables). Think of it as Google search for data. The project is named after Norwegian explorer Roald Amundsen, the first person to discover the South Pole.

Amundsen is hosted by the LF AI & Data Foundation. It includes three microservices, one data ingestion library and one common library.

amundsenfrontendlibrary: Frontend service which is a Flask application with a React frontend.
amundsensearchlibrary: Search service, which leverages Elasticsearch for search capabilities, is used to power frontend metadata searching.
amundsenmetadatalibrary: Metadata service, which leverages Neo4j or Apache Atlas as the persistent layer, to provide various metadata.
amundsendatabuilder: Data ingestion library for building metadata graph and search index. Users could either load the data with a python script with the library or with an Airflow DAG importing the library.
amundsencommon: Amundsen Common library holds common codes among microservices in Amundsen.
amundsengremlin: Amundsen Gremlin library holds code used for converting model objects into vertices and edges in gremlin. It's used for loading data into an AWS Neptune backend.
amundsenrds: Amundsenrds contains ORM models to support relational database as metadata backend store in Amundsen. The schema in ORM models follows the logic of databuilder models. Amundsenrds will be used in databuilder and metadatalibrary for metadata storage and retrieval with relational databases.

Documentation

Community Roadmap

We want your input about what is important, for that, add your votes using the 👍 reaction:

Requirements

Python >= 3.8
Node v12

User Interface

Please note that the mock images only served as demonstration purpose.

Landing Page: The landing page for Amundsen including 1. search bars; 2. popular used tables;
Search Preview: See inline search results as you type
Table Detail Page: Visualization of a Hive / Redshift table
Column detail: Visualization of columns of a Hive / Redshift table which includes an optional stats display
Data Preview Page: Visualization of table data preview which could integrate with Apache Superset or other Data Visualization Tools.

Getting Started and Installation

Please visit the Amundsen installation documentation for a quick start to bootstrap a default version of Amundsen with dummy data.

Supported Entities

Tables (from Databases)
Dashboards
ML Features
People (from HR systems)

Supported Integrations

Table Connectors

Amundsen can also connect to any database that provides dbapi or sql_alchemy interface (which most DBs provide).

Table Column Statistics

Pandas Profiling

Dashboard Connectors

ETL Orchestration

Apache Airflow

Get Involved in the Community

Want help or want to help? Use the button in our header to join our slack channel.

Contributions are also more than welcome! As explained in CONTRIBUTING.md there are many ways to contribute, it does not all have to be code with new features and bug fixes, also documentation, like FAQ entries, bug reports, blog posts sharing experiences etc. all help move Amundsen forward. If you find a security vulnerability, please follow this guide.

Architecture Overview

Please visit Architecture for Amundsen architecture overview.

Resources

Blog Posts and Interviews

Amundsen - Lyft's data discovery & metadata engine (April 2019)
Software Engineering Daily podcast on Amundsen (April 2019)
How Lyft Drives Data Discovery (July 2019)
Data Engineering podcast on Solving Data Discovery At Lyft (Aug 2019)
Open Sourcing Amundsen: A Data Discovery And Metadata Platform (Oct 2019)
Adding Data Quality into Amundsen with Programmatic Descriptions by Sam Shuster from Edmunds.com (May 2020)
Facilitating Data discovery with Apache Atlas and Amundsen by Mariusz Górski from ING (June 2020)
Using Amundsen to Support User Privacy via Metadata Collection at Square by Alyssa Ransbury from Square (July 14, 2020)
Amundsen Joins LF AI as New Incubation Project (Aug 11, 2020)
Amundsen: one year later (Oct 6, 2020)

Talks

Disrupting Data Discovery {slides, recording} (Strata SF, March 2019)
Amundsen: A Data Discovery Platform from Lyft {slides} (Data Council SF, April 2019)
Disrupting Data Discovery {slides} (Strata London, May 2019)
ING Data Analytics Platform (Amundsen is mentioned) {slides, recording } (Kubecon Barcelona, May 2019)
Disrupting Data Discovery {slides, recording} (Making Big Data Easy SF, May 2019)
Disrupting Data Discovery {slides, recording} (Neo4j Graph Tour Santa Monica, September 2019)
Disrupting Data Discovery {slides} (IDEAS SoCal AI & Data Science Conference, Oct 2019)
Data Discovery with Amundsen by Gerard Toonstra from Coolblue {slides} and {talk} (BigData Vilnius 2019)
Towards Enterprise Grade Data Discovery and Data Lineage with Apache Atlas and Amundsen by Verdan Mahmood and Marek Wiewiorka from ING {slides, talk} (Big Data Technology Warsaw Summit 2020)
Airflow @ Lyft (which covers how we integrate Airflow and Amundsen) by Tao Feng {slides and website} (Airflow Summit 2020)
Data DAGs with lineage for fun and for profit by Bolke de Bruin {website} (Airflow Summit 2020)
Solving Data Discovery Challenges at Lyft with Amundsen, an Open-source Metadata Platform by Tao Feng (Data+AI summit Europe 2020)
Data Discovery at Databricks with Amundsen by Tao Feng and Tianru Zhou (Data+AI summit NA 2021)

Community meetings

Community meetings are held on the first Thursday of every month at 9 AM Pacific, Noon Eastern, 6 PM Central European Time. Link to join

Upcoming meetings & notes

You can the exact date for the next meeting and the agenda a few weeks before the meeting in this doc.

Notes from all past meetings are available here.

Who uses Amundsen?

Here is the list of organizations that are officially using Amundsen today. If your organization uses Amundsen, please file a PR and update this list.

Contributors ✨

Thanks goes to these incredible people:

amundsen-databuilder dependencies

FAQs

What is amundsen-databuilder?

Amundsen Data builder. Visit Snyk Advisor to see a full health score report for amundsen-databuilder, including popularity, security, maintenance & community analysis.

Is amundsen-databuilder popular?

The python package amundsen-databuilder receives a total of 38,619 weekly downloads. As such, amundsen-databuilder popularity was classified as a popular. Visit the popularity section on Snyk Advisor to see the full health analysis.

Is amundsen-databuilder well maintained?

We found indications that amundsen-databuilder maintenance is sustainable demonstrating some project activity. We saw a total of 220 open source contributors collaborating on the project. See the full package health analysis to learn more about the package maintenance status.

Is amundsen-databuilder safe to use?

While scanning the latest version of amundsen-databuilder, we found that a security review is needed. A total of 0 vulnerabilities or license issues were detected. See the full security scan results.

Last updated on 27 April-2024, at 01:51 (UTC).

Build a secure application checklist

Select a recommended open source package

Minimize your risk by selecting secure & well maintained open source packages

DONE

Scan your app for vulnerabilities

Scan your application to find vulnerabilities in your: source code, open source dependencies, containers and configuration files

SCAN NOW

Fix identified vulnerabilities

Easily fix your code by leveraging automatically generated PRs

AUTO FIX

Monitor for new issues

New vulnerabilities are discovered every day. Get notified if your application is affected

MONITOR APP