pyspark-data-sources

v0.1.5

Custom Spark data sources for reading and writing data in Apache Spark, using the Python Data Source API For more information about how to use this package see README

Latest version published 4 months ago

License: Apache-2.0

PyPI

GitHub

Package Health Score

79 / 100

security
No known security issues
popularity
Small
maintenance
Sustainable
community
Active

Popularity

Small

GitHub Stars: 40.24K
Forks: 28.4K
Contributors: 340

Direct Usage Popularity

Based on project statistics from the GitHub repository for the PyPI package pyspark-data-sources, we found that it has been starred 40,242 times.

Security

No known security issues

Security and license risk for latest version

Release Date: Sep 5, 2024
Direct Vulnerabilities: 0
C
0
H
0
M
0
L
Indirect Vulnerabilities: 0
C
0
H
0
M
0
L
License Risk: 0
H
0
M
0
L

All security vulnerabilities belong to production dependencies of direct and indirect packages.

License

Apache-2.0

Security Policy

Yes

You can connect your project's repository to Snyk to stay up to date on security alerts and receive automatic fix pull requests.

Keep your project free of vulnerabilities with Snyk

Maintenance

Sustainable

Commit Frequency

Open Issues: 0
Open PR: 248
Last Release: 4 months ago
Last Commit: 22 hours ago

Further analysis of the maintenance status of pyspark-data-sources based on released PyPI versions cadence, the repository activity, and other data points determined that its maintenance is Sustainable.

We found that pyspark-data-sources demonstrates a positive version release cadence with at least one new version released in the past 12 months.

As a healthy sign for on-going project maintenance, we found that the GitHub repository had at least 1 pull request or issue interacted with by the community.

Community

Active

Readme: Yes
Contributing.md: Yes
Code of Conduct: Yes
Contributors: 340
Funding: No

A good and healthy external contribution signal for pyspark-data-sources project, which invites more than one hundred open source maintainers to collaborate on the repository.

Embed Package Health Score Badge

Keep your project healthy

Check your requirements.txt

Ensure all the packages you're using are healthy and well-maintained

Snyk Vulnerability Scanner

Get health score & security insights directly in your IDE

Package

Python Versions Compatibility: <4.0,>=3.9

Age: 11 months
Latest Release: 4 months ago
Dependencies: 0 Direct
Versions: 6
Maintainers: 0
Wheels: OS Independent

Readme

pyspark-data-sources

This repository showcases custom Spark data sources built using the new Python Data Source API for the upcoming Apache Spark 4.0 release. For an in-depth understanding of the API, please refer to the API source code. Note this repo is demo only and please be aware that it is not intended for production use. Contributions and feedback are welcome to help improve the examples.

Installation

pip install pyspark-data-sources[all]

Usage

Install the pyspark 4.0 preview version: https://pypi.org/project/pyspark/4.0.0.dev1/

pip install "pyspark[connect]==4.0.0.dev1"

Or use Databricks Runtime 15.2 or above.

Batch Data Sources

from pyspark_datasources.github import GithubDataSource

# Register the data source
spark.dataSource.register(GithubDataSource)

spark.read.format("github").load("apache/spark").show()

See more here: https://allisonwang-db.github.io/pyspark-data-sources/.

Streaming Data Sources

from pyspark_datasource.weather import WeatherDataSource

# Register the data source
spark.dataSource.register(WeatherDataSource)

# Get an API key from tomorrow.io
api_key = ""
sites = """[
    (37.7749, -122.4194),    # San Francisco
    (40.7128, -74.0060),     # New York City
]"""

# Ingest the weather data.
df = (
    spark.readStream.format("weather")
        .option("locations", sites)
        .option("apikey", api_key)
        .load()
)

# Write to the console.
df.writeStream.format("console").trigger(availableNow=True).start()

Contributing

We welcome and appreciate any contributions to enhance and expand the custom data sources. If you're interested in contributing:

Add New Data Sources: Want to add a new data source using the Python Data Source API? Submit a pull request or open an issue.
Suggest Enhancements: If you have ideas to improve a data source or the API, we'd love to hear them!
Report Bugs: Found something that doesn't work as expected? Let us know by opening an issue.

Need help or have questions? Don't hesitate to open a new issue, and we'll do our best to assist you.

Development

poetry shell

Build docs

mkdocs serve

FAQs

What is pyspark-data-sources?

Custom Spark data sources for reading and writing data in Apache Spark, using the Python Data Source API. Visit Snyk Advisor to see a full health score report for pyspark-data-sources, including popularity, security, maintenance & community analysis.

Is pyspark-data-sources popular?

The python package pyspark-data-sources receives a total of 180 weekly downloads. As such, pyspark-data-sources popularity was classified as small. Visit the popularity section on Snyk Advisor to see the full health analysis.

Is pyspark-data-sources well maintained?

We found indications that pyspark-data-sources maintenance is sustainable demonstrating some project activity. We saw a total of 340 open source contributors collaborating on the project. See the full package health analysis to learn more about the package maintenance status.

Is pyspark-data-sources safe to use?

The python package pyspark-data-sources was scanned for known vulnerabilities and missing license, and no issues were found. Thus the package was deemed as safe to use. See the full health analysis review.

Last updated on 25 December-2024, at 01:23 (UTC).

Build a secure application checklist

Select a recommended open source package

Minimize your risk by selecting secure & well maintained open source packages

DONE

Scan your app for vulnerabilities

Scan your application to find vulnerabilities in your: source code, open source dependencies, containers and configuration files

SCAN NOW

Fix identified vulnerabilities

Easily fix your code by leveraging automatically generated PRs

AUTO FIX

Monitor for new issues

New vulnerabilities are discovered every day. Get notified if your application is affected

MONITOR APP

Package Health Score

Direct Usage Popularity

Security and license risk for latest version

Commit Frequency

Embed Package Health Score Badge

Keep your project healthy

Check your requirements.txt

Snyk Vulnerability Scanner

Readme

pyspark-data-sources

Installation

Usage

Batch Data Sources

Streaming Data Sources

Contributing

Development

Build docs

FAQs

What is pyspark-data-sources?

Is pyspark-data-sources popular?

Is pyspark-data-sources well maintained?

Is pyspark-data-sources safe to use?

Build a secure application checklist

Select a recommended open source package

Scan your app for vulnerabilities

Source Code

Open Source

Container

Config

Fix identified vulnerabilities

Monitor for new issues