View modin on Snyk Open Source Advisor

Package Health Score

94 / 100

security
No known security issues
popularity
Influential project
maintenance
Healthy
community
Active

Explore Similar Packages

Popularity

Influential project

GitHub Stars: 9.89K
Forks: 653
Contributors: 130

Direct Usage Popularity

Based on project statistics from the GitHub repository for the PyPI package modin, we found that it has been starred 9,892 times.

Security

No known security issues

Security and license risk for latest version

Release Date: Sep 11, 2024
Direct Vulnerabilities: 0
C
0
H
0
M
0
L
Indirect Vulnerabilities: 0
C
0
H
0
M
0
L
License Risk: 0
H
0
M
0
L

All security vulnerabilities belong to production dependencies of direct and indirect packages.

License

Apache-2.0

Security Policy

We found a way for you to contribute to the project! Looks like modin is missing a security policy.

You can connect your project's repository to Snyk to stay up to date on security alerts and receive automatic fix pull requests.

Keep your project free of vulnerabilities with Snyk

Maintenance

Healthy

Commit Frequency

Open Issues: 634
Open PR: 33
Last Release: 2 months ago
Last Commit: 2 months ago

Further analysis of the maintenance status of modin based on released PyPI versions cadence, the repository activity, and other data points determined that its maintenance is Healthy.

We found that modin demonstrates a positive version release cadence with at least one new version released in the past 3 months.

As a healthy sign for on-going project maintenance, we found that the GitHub repository had at least 1 pull request or issue interacted with by the community.

Community

Active

Readme: Yes
Contributing.md: No
Code of Conduct: Yes
Contributors: 130
Funding: No

A good and healthy external contribution signal for modin project, which invites more than one hundred open source maintainers to collaborate on the repository.

Embed Package Health Score Badge

Keep your project healthy

Check your requirements.txt

Ensure all the packages you're using are healthy and well-maintained

Snyk Vulnerability Scanner

Get health score & security insights directly in your IDE

Package

Python Versions Compatibility: >=3.9

Age: 6 years
Latest Release: 2 months ago
Dependencies: 0 Direct
Versions: 101
Maintainers: 7
Wheels: OS Independent

Not sure how to use modin?

To help you get started, we've collected the most common ways that modin is being used within popular public projects.

modin-project / modin / ci / benchmarks / groupby_benchmark.py View on Github

from utils import time_logger


parser = argparse.ArgumentParser(description="groupby benchmark")
parser.add_argument("--path", dest="path", help="path to the csv data file")
parser.add_argument("--logfile", dest="logfile", help="path to the log file")
args = parser.parse_args()
file = args.path
file_size = os.path.getsize(file)

if not os.path.exists(os.path.split(args.logfile)[0]):
    os.makedirs(os.path.split(args.logfile)[0])

logging.basicConfig(filename=args.logfile, level=logging.INFO)

df = pd.read_csv(file)
blocks = df._block_partitions.flatten().tolist()
ray.wait(blocks, len(blocks))

with time_logger(
    "Groupby + sum aggregation on axis=0: {}; Size: {} bytes".format(file, file_size)
):
    df_groupby = df.groupby("1")
    blocks = df_groupby.sum()._block_partitions.flatten().tolist()
    ray.wait(blocks, len(blocks))

with time_logger("Groupby mean on axis=0: {}; Size: {} bytes".format(file, file_size)):
    blocks = df_groupby.mean()._block_partitions.flatten().tolist()
    ray.wait(blocks, len(blocks))

View more ways to use modin

Scale your pandas workflows by changing one line of code

Dev Community & Support	Forums	Socials	Docs

What is Modin?

Modin is a drop-in replacement for pandas. While pandas is single-threaded, Modin lets you instantly speed up your workflows by scaling pandas so it uses all of your cores. Modin works especially well on larger datasets, where pandas becomes painfully slow or runs out of memory. Also, Modin comes with the additional APIs to improve user experience.

By simply replacing the import statement, Modin offers users effortless speed and scale for their pandas workflows:

In the GIFs below, Modin (left) and pandas (right) perform the same pandas operations on a 2GB dataset. The only difference between the two notebook examples is the import statement.

The charts below show the speedup you get by replacing pandas with Modin based on the examples above. The example notebooks can be found here. To learn more about the speedups you could get with Modin and try out some examples on your own, check out our 10-minute quickstart guide to try out some examples on your own!

Installation

From PyPI

Modin can be installed with pip on Linux, Windows and MacOS:

pip install "modin[all]" # (Recommended) Install Modin with Ray and Dask engines.

If you want to install Modin with a specific engine, we recommend:

pip install "modin[ray]" # Install Modin dependencies and Ray.
pip install "modin[dask]" # Install Modin dependencies and Dask.
pip install "modin[mpi]" # Install Modin dependencies and MPI through unidist.

To get Modin on MPI through unidist (as of unidist 0.5.0) fully working it is required to have a working MPI implementation installed beforehand. Otherwise, installation of modin[mpi] may fail. Refer to Installing with pip section of the unidist documentation for more details about installation.

Note: Since Modin 0.30.0 we use a reduced set of Ray dependencies: ray instead of ray[default]. This means that the dashboard and cluster launcher are no longer installed by default. If you need those, consider installing ray[default] along with modin[ray].

Modin automatically detects which engine(s) you have installed and uses that for scheduling computation.

From conda-forge

Installing from conda forge using modin-all will install Modin and three engines: Ray, Dask and MPI through unidist.

conda install -c conda-forge modin-all

Each engine can also be installed individually (and also as a combination of several engines):

conda install -c conda-forge modin-ray  # Install Modin dependencies and Ray.
conda install -c conda-forge modin-dask # Install Modin dependencies and Dask.
conda install -c conda-forge modin-mpi # Install Modin dependencies and MPI through unidist.

Note: Since Modin 0.30.0 we use a reduced set of Ray dependencies: ray-core instead of ray-default. This means that the dashboard and cluster launcher are no longer installed by default. If you need those, consider installing ray-default along with modin-ray.

Refer to Installing with conda section of the unidist documentation for more details on how to install a specific MPI implementation to run on.

To speed up conda installation we recommend using libmamba solver. To do this install it in a base environment:

conda install -n base conda-libmamba-solver

and then use it during istallation either like:

conda install -c conda-forge modin-ray --experimental-solver=libmamba

or starting from conda 22.11 and libmamba solver 22.12 versions:

conda install -c conda-forge modin-ray --solver=libmamba

Choosing a Compute Engine

If you want to choose a specific compute engine to run on, you can set the environment variable MODIN_ENGINE and Modin will do computation with that engine:

export MODIN_ENGINE=ray  # Modin will use Ray
export MODIN_ENGINE=dask  # Modin will use Dask
export MODIN_ENGINE=unidist # Modin will use Unidist

If you want to choose the Unidist engine, you should set the additional environment variable UNIDIST_BACKEND. Currently, Modin only supports MPI through unidist:

export UNIDIST_BACKEND=mpi # Unidist will use MPI backend

This can also be done within a notebook/interpreter before you import Modin:

import modin.config as modin_cfg
import unidist.config as unidist_cfg

modin_cfg.Engine.put("ray")  # Modin will use Ray
modin_cfg.Engine.put("dask")  # Modin will use Dask

modin_cfg.Engine.put('unidist') # Modin will use Unidist
unidist_cfg.Backend.put('mpi') # Unidist will use MPI backend

Note: You should not change the engine after your first operation with Modin as it will result in undefined behavior.

Which engine should I use?

On Linux, MacOS, and Windows you can install and use either Ray, Dask or MPI through unidist. There is no knowledge required to use either of these engines as Modin abstracts away all of the complexity, so feel free to pick either!

Pandas API Coverage

pandas Object	Modin's Ray Engine Coverage	Modin's Dask Engine Coverage	Modin's Unidist Engine Coverage
`pd.DataFrame`
`pd.Series`
`pd.read_csv`	✅	✅	✅
`pd.read_table`	✅	✅	✅
`pd.read_parquet`	✅	✅	✅
`pd.read_sql`	✅	✅	✅
`pd.read_feather`	✅	✅	✅
`pd.read_excel`	✅	✅	✅
`pd.read_json`	✳️	✳️	✳️
`pd.read_`	✴️	✴️	✴️

Some pandas APIs are easier to implement than others, so if something is missing feel free to open an issue!

More about Modin

For the complete documentation on Modin, visit our ReadTheDocs page.

Scale your pandas workflow by changing a single line of code.

Note: In local mode (without a cluster), Modin will create and manage a local (Dask or Ray) cluster for the execution.

To use Modin, you do not need to specify how to distribute the data, or even know how many cores your system has. In fact, you can continue using your previous pandas notebooks while experiencing a considerable speedup from Modin, even on a single machine. Once you've changed your import statement, you're ready to use Modin just like you would with pandas!

Faster pandas, even on your laptop

The modin.pandas DataFrame is an extremely light-weight parallel DataFrame. Modin transparently distributes the data and computation so that you can continue using the same pandas API while working with more data faster. Because it is so light-weight, Modin provides speed-ups of up to 4x on a laptop with 4 physical cores.

In pandas, you are only able to use one core at a time when you are doing computation of any kind. With Modin, you are able to use all of the CPU cores on your machine. Even with a traditionally synchronous task like read_csv, we see large speedups by efficiently distributing the work across your entire machine.

import modin.pandas as pd

df = pd.read_csv("my_dataset.csv")

Modin can handle the datasets that pandas can't

Often data scientists have to switch between different tools for operating on datasets of different sizes. Processing large dataframes with pandas is slow, and pandas does not support working with dataframes that are too large to fit into the available memory. As a result, pandas workflows that work well for prototyping on a few MBs of data do not scale to tens or hundreds of GBs (depending on the size of your machine). Modin supports operating on data that does not fit in memory, so that you can comfortably work with hundreds of GBs without worrying about substantial slowdown or memory errors. With cluster and out of core support, Modin is a DataFrame library with both great single-node performance and high scalability in a cluster.

Modin Architecture

We designed Modin's architecture to be modular so we can plug in different components as they develop and improve:

Other Resources

Getting Started with Modin

Modin Community

Learn More about Modin

Frequently Asked Questions (FAQs)
Troubleshooting Guide
Development Guide
Modin is built on many years of research and development at UC Berkeley. Check out these selected papers to learn more about how Modin works:
- Flexible Rule-Based Decomposition and Metadata Independence in Modin (VLDB 2021)
- Dataframe Systems: Theory, Architecture, and Implementation (PhD Dissertation 2021)
- Towards Scalable Dataframe Systems (VLDB 2020)

Getting Involved

modin.pandas is currently under active development. Requests and contributions are welcome!

For more information on how to contribute to Modin, check out the Modin Contribution Guide.

License

Apache License 2.0

FAQs

What is modin?

Modin: Make your pandas code run faster by changing one line of code. Visit Snyk Advisor to see a full health score report for modin, including popularity, security, maintenance & community analysis.

Is modin popular?

The python package modin receives a total of 269,075 weekly downloads. As such, modin popularity was classified as an influential project. Visit the popularity section on Snyk Advisor to see the full health analysis.

Is modin well maintained?

We found that modin demonstrated a healthy version release cadence and project activity. It has a community of 130 open source contributors collaborating on the project. See the full package health analysis to learn more about the package maintenance status.

Is modin safe to use?

The python package modin was scanned for known vulnerabilities and missing license, and no issues were found. Thus the package was deemed as safe to use. See the full health analysis review.

Last updated on 18 November-2024, at 16:44 (UTC).

Build a secure application checklist

Select a recommended open source package

Minimize your risk by selecting secure & well maintained open source packages

DONE

Scan your app for vulnerabilities

Scan your application to find vulnerabilities in your: source code, open source dependencies, containers and configuration files

SCAN NOW

Fix identified vulnerabilities

Easily fix your code by leveraging automatically generated PRs

AUTO FIX

Monitor for new issues

New vulnerabilities are discovered every day. Get notified if your application is affected

MONITOR APP

Package Health Score

Explore Similar Packages

Direct Usage Popularity

Security and license risk for latest version

Commit Frequency

Embed Package Health Score Badge

Keep your project healthy

Check your requirements.txt

Snyk Vulnerability Scanner

Not sure how to use modin?

Scale your pandas workflows by changing one line of code

Dev Community & Support

Forums

Socials

Docs

What is Modin?

Installation

From PyPI

From conda-forge

Choosing a Compute Engine

Which engine should I use?

Pandas API Coverage

More about Modin

Scale your pandas workflow by changing a single line of code.

Faster pandas, even on your laptop

Modin can handle the datasets that pandas can't

Modin Architecture

Other Resources

Getting Started with Modin

Modin Community

Learn More about Modin

Getting Involved

License

FAQs

What is modin?

Is modin popular?

Is modin well maintained?

Is modin safe to use?

Build a secure application checklist

Select a recommended open source package

Scan your app for vulnerabilities

Source Code

Open Source

Container

Config

Fix identified vulnerabilities

Monitor for new issues