pytorch_optimizer

v3.2.0

optimizer & lr scheduler & objective function collections in PyTorch For more information about how to use this package see README

Latest version published 26 days ago

License: Apache-2.0

PyPI

GitHub

Package Health Score

81 / 100

security
No known security issues
popularity
Recognized
maintenance
Healthy
community
Active

Popularity

Recognized

GitHub Stars: 237
Forks: 19
Contributors: 9

Direct Usage Popularity

Based on project statistics from the GitHub repository for the PyPI package pytorch_optimizer, we found that it has been starred 237 times.

Security

No known security issues

Security and license risk for latest version

Release Date: Oct 28, 2024
Direct Vulnerabilities: 0
C
0
H
0
M
0
L
Indirect Vulnerabilities: 0
C
0
H
0
M
0
L
License Risk: 0
H
0
M
0
L

All security vulnerabilities belong to production dependencies of direct and indirect packages.

License

Apache-2.0

Security Policy

Yes

You can connect your project's repository to Snyk to stay up to date on security alerts and receive automatic fix pull requests.

Keep your project free of vulnerabilities with Snyk

Maintenance

Healthy

Commit Frequency

Open Issues: 7
Open PR: 0
Last Release: 26 days ago
Last Commit: 15 days ago

Further analysis of the maintenance status of pytorch_optimizer based on released PyPI versions cadence, the repository activity, and other data points determined that its maintenance is Healthy.

We found that pytorch_optimizer demonstrates a positive version release cadence with at least one new version released in the past 3 months.

As a healthy sign for on-going project maintenance, we found that the GitHub repository had at least 1 pull request or issue interacted with by the community.

Community

Active

Readme: Yes
Contributing.md: Yes
Code of Conduct: Yes
Contributors: 9
Funding: No

This project has seen only 10 or less contributors.

Embed Package Health Score Badge

Keep your project healthy

Check your requirements.txt

Ensure all the packages you're using are healthy and well-maintained

Snyk Vulnerability Scanner

Get health score & security insights directly in your IDE

Package

Python Versions Compatibility: >=3.8

Age: 3 years
Latest Release: 26 days ago
Dependencies: 0 Direct
Versions: 72
Maintainers: 1
Wheels: OS Independent

Readme

pytorch-optimizer


Build
Quality
Package
Status
License

pytorch-optimizer is optimizer & lr scheduler collections in PyTorch. I just re-implemented (speed & memory tweaks, plug-ins) the algorithm while based on the original paper. Also, It includes useful and practical optimization ideas.
Currently, 77 optimizers (+ bitsandbytes, qgalore, torchao), 16 lr schedulers, and 13 loss functions are supported!

Highly inspired by pytorch-optimizer.

Getting Started

For more, see the documentation.

Most optimizers are under MIT or Apache 2.0 license, but a few optimizers like Fromage, Nero have CC BY-NC-SA 4.0 license, which is non-commercial. So, please double-check the license before using it at your work.

Installation

$ pip3 install pytorch-optimizer

From v2.12.0, v3.1.0, you can use bitsandbytes, q-galore-torch, torchao optimizers respectively! please check the bnb requirements, q-galore-torch installation, torchao installation before installing it.

From v3.0.0, drop Python 3.7 support. However, you can still use this package with Python 3.7 by installing with --ignore-requires-python option.

Simple Usage

from pytorch_optimizer import AdamP

model = YourModel()
optimizer = AdamP(model.parameters())

# or you can use optimizer loader, simply passing a name of the optimizer.

from pytorch_optimizer import load_optimizer

optimizer = load_optimizer(optimizer='adamp')(model.parameters())

# if you install `bitsandbytes` optimizer, you can use `8-bit` optimizers from `pytorch-optimizer`.

from pytorch_optimizer import load_optimizer

opt = load_optimizer(optimizer='bnb_adamw8bit')
optimizer = opt(model.parameters())

Also, you can load the optimizer via torch.hub.

import torch

model = YourModel()
opt = torch.hub.load('kozistr/pytorch_optimizer', 'adamp')
optimizer = opt(model.parameters())

If you want to build the optimizer with parameters & configs, there's create_optimizer() API.

from pytorch_optimizer import create_optimizer

optimizer = create_optimizer(
    model,
    'adamp',
    lr=1e-3,
    weight_decay=1e-3,
    use_gc=True,
    use_lookahead=True,
)

Supported Optimizers

You can check the supported optimizers with below code.

from pytorch_optimizer import get_supported_optimizers

supported_optimizers = get_supported_optimizers()

or you can also search them with the filter(s).

&gt;&gt;&gt; get_supported_optimizers('adam*')
['adamax', 'adamg', 'adammini', 'adamod', 'adamp', 'adams', 'adamw']

&gt;&gt;&gt; get_supported_optimizers(['adam*', 'ranger*'])
['adamax', 'adamg', 'adammini', 'adamod', 'adamp', 'adams', 'adamw', 'ranger', 'ranger21']

Optimizer	Description	Official Code	Paper	Citation
AdaBelief	Adapting Step-sizes by the Belief in Observed Gradients	github		cite
AdaBound	Adaptive Gradient Methods with Dynamic Bound of Learning Rate	github		cite
AdaHessian	An Adaptive Second Order Optimizer for Machine Learning	github		cite
AdamD	Improved bias-correction in Adam			cite
AdamP	Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights	github		cite
diffGrad	An Optimization Method for Convolutional Neural Networks	github		cite
MADGRAD	A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic	github		cite
RAdam	On the Variance of the Adaptive Learning Rate and Beyond	github		cite
Ranger	a synergistic optimizer combining RAdam and LookAhead, and now GC in one optimizer	github		cite
Ranger21	a synergistic deep learning optimizer	github		cite
Lamb	Large Batch Optimization for Deep Learning	github		cite
Shampoo	Preconditioned Stochastic Tensor Optimization	github		cite
Nero	Learning by Turning: Neural Architecture Aware Optimisation	github		cite
Adan	Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models	github		cite
Adai	Disentangling the Effects of Adaptive Learning Rate and Momentum	github		cite
SAM	Sharpness-Aware Minimization	github		cite
ASAM	Adaptive Sharpness-Aware Minimization	github		cite
GSAM	Surrogate Gap Guided Sharpness-Aware Minimization	github		cite
D-Adaptation	Learning-Rate-Free Learning by D-Adaptation	github		cite
AdaFactor	Adaptive Learning Rates with Sublinear Memory Cost	github		cite
Apollo	An Adaptive Parameter-wise Diagonal Quasi-Newton Method for Nonconvex Stochastic Optimization	github		cite
NovoGrad	Stochastic Gradient Methods with Layer-wise Adaptive Moments for Training of Deep Networks	github		cite
Lion	Symbolic Discovery of Optimization Algorithms	github		cite
Ali-G	Adaptive Learning Rates for Interpolation with Gradients	github		cite
SM3	Memory-Efficient Adaptive Optimization	github		cite
AdaNorm	Adaptive Gradient Norm Correction based Optimizer for CNNs	github		cite
RotoGrad	Gradient Homogenization in Multitask Learning	github		cite
A2Grad	Optimal Adaptive and Accelerated Stochastic Gradient Descent	github		cite
AccSGD	Accelerating Stochastic Gradient Descent For Least Squares Regression	github		cite
SGDW	Decoupled Weight Decay Regularization	github		cite
ASGD	Adaptive Gradient Descent without Descent	github		cite
Yogi	Adaptive Methods for Nonconvex Optimization		NIPS 2018	cite
SWATS	Improving Generalization Performance by Switching from Adam to SGD			cite
Fromage	On the distance between two neural networks and the stability of learning	github		cite
MSVAG	Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients	github		cite
AdaMod	An Adaptive and Momental Bound Method for Stochastic Learning	github		cite
AggMo	Aggregated Momentum: Stability Through Passive Damping	github		cite
QHAdam	Quasi-hyperbolic momentum and Adam for deep learning	github		cite
PID	A PID Controller Approach for Stochastic Optimization of Deep Networks	github	CVPR 18	cite
Gravity	a Kinematic Approach on Optimization in Deep Learning	github		cite
AdaSmooth	An Adaptive Learning Rate Method based on Effective Ratio			cite
SRMM	Stochastic regularized majorization-minimization with weakly convex and multi-convex surrogates	github		cite
AvaGrad	Domain-independent Dominance of Adaptive Methods	github		cite
PCGrad	Gradient Surgery for Multi-Task Learning	github		cite
AMSGrad	On the Convergence of Adam and Beyond			cite
Lookahead	k steps forward, 1 step back	github		cite
PNM	Manipulating Stochastic Gradient Noise to Improve Generalization	github		cite
GC	Gradient Centralization	github		cite
AGC	Adaptive Gradient Clipping	github		cite
Stable WD	Understanding and Scheduling Weight Decay	github		cite
Softplus T	Calibrating the Adaptive Learning Rate to Improve Convergence of ADAM			cite
Un-tuned w/u	On the adequacy of untuned warmup for adaptive optimization			cite
Norm Loss	An efficient yet effective regularization method for deep neural networks			cite
AdaShift	Decorrelation and Convergence of Adaptive Learning Rate Methods	github		cite
AdaDelta	An Adaptive Learning Rate Method			cite
Amos	An Adam-style Optimizer with Adaptive Weight Decay towards Model-Oriented Scale	github		cite
SignSGD	Compressed Optimisation for Non-Convex Problems	github		cite
Sophia	A Scalable Stochastic Second-order Optimizer for Language Model Pre-training	github		cite
Prodigy	An Expeditiously Adaptive Parameter-Free Learner	github		cite
PAdam	Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks	github		cite
LOMO	Full Parameter Fine-tuning for Large Language Models with Limited Resources	github		cite
AdaLOMO	Low-memory Optimization with Adaptive Learning Rate	github		cite
Tiger	A Tight-fisted Optimizer, an optimizer that is extremely budget-conscious	github		cite
CAME	Confidence-guided Adaptive Memory Efficient Optimization	github		cite
WSAM	Sharpness-Aware Minimization Revisited: Weighted Sharpness as a Regularization Term	github		cite
Aida	A DNN Optimizer that Improves over AdaBelief by Suppression of the Adaptive Stepsize Range	github		cite
GaLore	Memory-Efficient LLM Training by Gradient Low-Rank Projection	github		cite
Adalite	Adalite optimizer	github		cite
bSAM	SAM as an Optimal Relaxation of Bayes	github		cite
Schedule-Free	Schedule-Free Optimizers	github		cite
FAdam	Adam is a natural gradient optimizer using diagonal empirical Fisher information	github		cite
Grokfast	Accelerated Grokking by Amplifying Slow Gradients	github		cite
Kate	Remove that Square Root: A New Efficient Scale-Invariant Version of AdaGrad	github		cite
StableAdamW	Stable and low-precision training for large-scale vision-language models			cite
AdamMini	Use Fewer Learning Rates To Gain More	github		cite
TRAC	Adaptive Parameter-free Optimization	github		cite
AdamG	Towards Stability of Parameter-free Optimization			cite
AdEMAMix	Better, Faster, Older	github		cite
SOAP	Improving and Stabilizing Shampoo using Adam	github		cite

Supported LR Scheduler

You can check the supported learning rate schedulers with below code.

from pytorch_optimizer import get_supported_lr_schedulers

supported_lr_schedulers = get_supported_lr_schedulers()

or you can also search them with the filter(s).

&gt;&gt;&gt; get_supported_lr_schedulers('cosine*')
['cosine', 'cosine_annealing', 'cosine_annealing_with_warm_restart', 'cosine_annealing_with_warmup']

&gt;&gt;&gt; get_supported_lr_schedulers(['cosine*', '*warm*'])
['cosine', 'cosine_annealing', 'cosine_annealing_with_warm_restart', 'cosine_annealing_with_warmup', 'warmup_stable_decay']

LR Scheduler	Description	Official Code	Citation
Explore-Exploit	Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule		cite
Chebyshev	Acceleration via Fractal Learning Rate Schedules		cite
REX	Revisiting Budgeted Training with an Improved Schedule	github	cite
WSD	Warmup-Stable-Decay learning rate scheduler	github	cite

Supported Loss Function

You can check the supported loss functions with below code.

from pytorch_optimizer import get_supported_loss_functions

supported_loss_functions = get_supported_loss_functions()

or you can also search them with the filter(s).

&gt;&gt;&gt; get_supported_loss_functions('*focal*')
['bcefocalloss', 'focalcosineloss', 'focalloss', 'focaltverskyloss']

&gt;&gt;&gt; get_supported_loss_functions(['*focal*', 'bce*'])
['bcefocalloss', 'bceloss', 'focalcosineloss', 'focalloss', 'focaltverskyloss']

Loss Functions	Description	Official Code	Citation
Label Smoothing	Rethinking the Inception Architecture for Computer Vision		cite
Focal	Focal Loss for Dense Object Detection		cite
Focal Cosine	Data-Efficient Deep Learning Method for Image Classification Using Data Augmentation, Focal Cosine Loss, and Ensemble		cite
LDAM	Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss	github	cite
Jaccard (IOU)	IoU Loss for 2D/3D Object Detection		cite
Bi-Tempered	The Principle of Unchanged Optimality in Reinforcement Learning Generalization		cite
Tversky	Tversky loss function for image segmentation using 3D fully convolutional deep networks		cite
Lovasz Hinge	A tractable surrogate for the optimization of the intersection-over-union measure in neural networks	github	cite

Useful Resources

Several optimization ideas to regularize & stabilize the training. Most of the ideas are applied in Ranger21 optimizer.

Also, most of the captures are taken from Ranger21 paper.


Adaptive Gradient Clipping	Gradient Centralization	Softplus Transformation
Gradient Normalization	Norm Loss	Positive-Negative Momentum
Linear learning rate warmup	Stable weight decay	Explore-exploit learning rate schedule
Lookahead	Chebyshev learning rate schedule	(Adaptive) Sharpness-Aware Minimization
On the Convergence of Adam and Beyond	Improved bias-correction in Adam	Adaptive Gradient Norm Correction

Adaptive Gradient Clipping

This idea originally proposed in NFNet (Normalized-Free Network) paper. AGC (Adaptive Gradient Clipping) clips gradients based on the unit-wise ratio of gradient norms to parameter norms.

code : github
paper : arXiv

Gradient Centralization

Gradient Centralization (GC) operates directly on gradients by centralizing the gradient to have zero mean.

code : github
paper : arXiv

Softplus Transformation

By running the final variance denom through the softplus function, it lifts extremely tiny values to keep them viable.

paper : arXiv

Gradient Normalization

Norm Loss

paper : arXiv

Positive-Negative Momentum

code : github
paper : arXiv

Linear learning rate warmup

paper : arXiv

Stable weight decay

code : github
paper : arXiv

Explore-exploit learning rate schedule

code : github
paper : arXiv

Lookahead

k steps forward, 1 step back. Lookahead consisting of keeping an exponential moving average of the weights that is updated and substituted to the current weights every k lookahead steps (5 by default).

Chebyshev learning rate schedule

Acceleration via Fractal Learning Rate Schedules.

(Adaptive) Sharpness-Aware Minimization

Sharpness-Aware Minimization (SAM) simultaneously minimizes loss value and loss sharpness.
In particular, it seeks parameters that lie in neighborhoods having uniformly low loss.

On the Convergence of Adam and Beyond

Convergence issues can be fixed by endowing such algorithms with 'long-term memory' of past gradients.

Improved bias-correction in Adam

With the default bias-correction, Adam may actually make larger than requested gradient updates early in training.

Adaptive Gradient Norm Correction

Correcting the norm of a gradient in each iteration based on the adaptive training history of gradient norm.

Frequently asked questions

here

Visualization

here

Citation

Please cite the original authors of optimization algorithms. You can easily find it in the above table! If you use this software, please cite it below. Or you can get it from "cite this repository" button.

@software{Kim_pytorch_optimizer_optimizer_2021,
    author = {Kim, Hyeongchan},
    month = jan,
    title = {{pytorch_optimizer: optimizer &amp; lr scheduler &amp; loss function collections in PyTorch}},
    url = {https://github.com/kozistr/pytorch_optimizer},
    version = {3.1.0},
    year = {2021}
}

Maintainer

Hyeongchan Kim / @kozistr

FAQs

What is pytorch_optimizer?

optimizer & lr scheduler & objective function collections in PyTorch. Visit Snyk Advisor to see a full health score report for pytorch_optimizer, including popularity, security, maintenance & community analysis.

Is pytorch_optimizer popular?

The python package pytorch_optimizer receives a total of 20,069 weekly downloads. As such, pytorch_optimizer popularity was classified as a recognized. Visit the popularity section on Snyk Advisor to see the full health analysis.

Is pytorch_optimizer well maintained?

We found that pytorch_optimizer demonstrated a healthy version release cadence and project activity. It has a community of 9 open source contributors collaborating on the project. See the full package health analysis to learn more about the package maintenance status.

Is pytorch_optimizer safe to use?

The python package pytorch_optimizer was scanned for known vulnerabilities and missing license, and no issues were found. Thus the package was deemed as safe to use. See the full health analysis review.

Last updated on 23 November-2024, at 20:06 (UTC).

Build a secure application checklist

Select a recommended open source package

Minimize your risk by selecting secure & well maintained open source packages

DONE

Scan your app for vulnerabilities

Scan your application to find vulnerabilities in your: source code, open source dependencies, containers and configuration files

SCAN NOW

Fix identified vulnerabilities

Easily fix your code by leveraging automatically generated PRs

AUTO FIX

Monitor for new issues

New vulnerabilities are discovered every day. Get notified if your application is affected

MONITOR APP

Package Health Score

Direct Usage Popularity

Security and license risk for latest version

Commit Frequency

Embed Package Health Score Badge

Keep your project healthy

Check your requirements.txt

Snyk Vulnerability Scanner

Readme

pytorch-optimizer

Getting Started

Installation

Simple Usage

Supported Optimizers

Supported LR Scheduler

Supported Loss Function

Useful Resources

Adaptive Gradient Clipping

Gradient Centralization

Softplus Transformation

Gradient Normalization

Norm Loss

Positive-Negative Momentum

Linear learning rate warmup

Stable weight decay

Explore-exploit learning rate schedule

Lookahead

Chebyshev learning rate schedule

(Adaptive) Sharpness-Aware Minimization

On the Convergence of Adam and Beyond

Improved bias-correction in Adam

Adaptive Gradient Norm Correction

Frequently asked questions

Visualization

Citation

Maintainer

FAQs

What is pytorch_optimizer?

Is pytorch_optimizer popular?

Is pytorch_optimizer well maintained?

Is pytorch_optimizer safe to use?

Build a secure application checklist

Select a recommended open source package

Scan your app for vulnerabilities

Source Code

Open Source

Container

Config

Fix identified vulnerabilities

Monitor for new issues