Package Health Score

69 / 100

security
No known security issues
popularity
Limited
maintenance
Healthy
community
Sustainable

Popularity

Limited

GitHub Stars: 112
Forks: 7
Contributors: 3

Direct Usage Popularity

Based on project statistics from the GitHub repository for the PyPI package megatron-energon, we found that it has been starred 112 times.

Security

No known security issues

Security and license risk for latest version

Release Date: Oct 24, 2024
Direct Vulnerabilities: 0
C
0
H
0
M
0
L
Indirect Vulnerabilities: 0
C
0
H
0
M
0
L
License Risk: 0
H
0
M
0
L

All security vulnerabilities belong to production dependencies of direct and indirect packages.

License

Unrecognized

Security Policy

We found a way for you to contribute to the project! Looks like megatron-energon is missing a security policy.

You can connect your project's repository to Snyk to stay up to date on security alerts and receive automatic fix pull requests.

Keep your project free of vulnerabilities with Snyk

Maintenance

Healthy

Commit Frequency

Open Issues: 8
Open PR: 4
Last Release: 14 days ago
Last Commit: 2 months ago

Further analysis of the maintenance status of megatron-energon based on released PyPI versions cadence, the repository activity, and other data points determined that its maintenance is Healthy.

We found that megatron-energon demonstrates a positive version release cadence with at least one new version released in the past 3 months.

As a healthy sign for on-going project maintenance, we found that the GitHub repository had at least 1 pull request or issue interacted with by the community.

Community

Sustainable

Readme: Yes
Contributing.md: No
Code of Conduct: No
Contributors: 3
Funding: No

This project has seen only 10 or less contributors.

We found a way for you to contribute to the project! Looks like megatron-energon is missing a Code of Conduct.

Embed Package Health Score Badge

Keep your project healthy

Check your requirements.txt

Ensure all the packages you're using are healthy and well-maintained

Snyk Vulnerability Scanner

Get health score & security insights directly in your IDE

Package

Python Versions Compatibility: >=3.8

Age: 8 months
Latest Release: 14 days ago
Dependencies: 8 Direct
Versions: 6
Maintainers: 1
Wheels: OS Independent

Readme

Megatron's multi-modal data loader

Megatron Energon

Report Bug · Request Feature

DISCLAIMER: This package contains research code. APIs may change.

What is this?

Megatron Energon is the multi-modal data loader of Megatron (you can also use it independently).

It's best at

loading large training data to train large multi-modal models
blending many different datasets together
distributing the work across many nodes and processes of a cluster
ensuring reproducibility and resumability
adapting easily to various types of data samples and processing

Try using it together with Megatron Core.

Quickstart

Megatron Energon is a pip-installable python package that offers

dataset-related classes that you can import in your project
a command line utility for data preprocessing and conversion

This document is just a quick start. Please also check out the documentation.

Installation

pip install megatron-energon

pip install git+https://github.com/NVIDIA/Megatron-Energon.git

NOTE: We encourage you to install the package (and not just import a local copy). This will ensure you have all the needed dependencies and that you can use the command line tool.

For more details on installing this package, see here.

Usage of command line tool

After installation, the command energon will be available.

Here are some examples for things you can do:

Command	Description
`energon prepare DATASET_ROOT`	Take an existing WebDataset and add the required yaml files to turn it into an energon-compatible dataset
`energon lint DATASET_ROOT`	Verify that the dataset complies with the energon dataset format and that all samples are loadable

Usage of the library

To get started, pick a WebDataset-compliant dataset and run energon prepare DATASET_ROOT on it, to run the interactive assistant and create the .nv-meta folder.

Once done, try to load it from your Python program:

from megatron.energon import get_train_dataset, get_loader


train_loader = get_loader(get_train_dataset(
    '/my/dataset/path',
    batch_size=32,
    shuffle_buffer_size=None,
    max_samples_per_sequence=None,
))

for batch in train_loader:
    # Do something with batch
    # Infer, gradient step, ...
    pass

For more details, read the documentation.

Most likely, you'll need your own task encoder.

megatron-energon dependencies

braceexpand click numpy pyyaml s3fs torch tqdm webdataset

FAQs

What is megatron-energon?

Megatron's multi-modal data loader. Visit Snyk Advisor to see a full health score report for megatron-energon, including popularity, security, maintenance & community analysis.

Is megatron-energon popular?

The python package megatron-energon receives a total of 189 weekly downloads. As such, megatron-energon popularity was classified as limited. Visit the popularity section on Snyk Advisor to see the full health analysis.

Is megatron-energon well maintained?

We found that megatron-energon demonstrated a healthy version release cadence and project activity. It has a community of 3 open source contributors collaborating on the project. See the full package health analysis to learn more about the package maintenance status.

Is megatron-energon safe to use?

The python package megatron-energon was scanned for known vulnerabilities and missing license, and no issues were found. Thus the package was deemed as safe to use. See the full health analysis review.

Last updated on 3 November-2024, at 00:25 (UTC).

Build a secure application checklist

Select a recommended open source package

Minimize your risk by selecting secure & well maintained open source packages

DONE

Scan your app for vulnerabilities

Scan your application to find vulnerabilities in your: source code, open source dependencies, containers and configuration files

SCAN NOW

Fix identified vulnerabilities

Easily fix your code by leveraging automatically generated PRs

AUTO FIX

Monitor for new issues

New vulnerabilities are discovered every day. Get notified if your application is affected

MONITOR APP

megatron-energon

Megatron's multi-modal data loader For more information about how to use this package see README

Ensure you're using the healthiest python packages

Package Health Score

Popularity

Direct Usage Popularity

Security

Security and license risk for latest version

Maintenance

Commit Frequency

Community

Embed Package Health Score Badge

Keep your project healthy

Check your requirements.txt

Snyk Vulnerability Scanner

Package

Readme

Megatron's multi-modal data loader

Megatron Energon

What is this?

Quickstart

Installation

Usage of command line tool

Usage of the library

megatron-energon dependencies

FAQs

What is megatron-energon?

Is megatron-energon popular?

Is megatron-energon well maintained?

Is megatron-energon safe to use?

Build a secure application checklist

Select a recommended open source package

Scan your app for vulnerabilities

Fix identified vulnerabilities

Monitor for new issues

Package Health Score

Direct Usage Popularity

Security and license risk for latest version

Commit Frequency

Embed Package Health Score Badge

Keep your project healthy

Check your requirements.txt

Snyk Vulnerability Scanner

Readme

Megatron's multi-modal data loader

Megatron Energon

What is this?

Quickstart

Installation

Usage of command line tool

Usage of the library

megatron-energon dependencies

FAQs

What is megatron-energon?

Is megatron-energon popular?

Is megatron-energon well maintained?

Is megatron-energon safe to use?

Build a secure application checklist

Select a recommended open source package

Scan your app for vulnerabilities

Source Code

Open Source

Container

Config

Fix identified vulnerabilities

Monitor for new issues