vllm-rocm

v0.6.3

A high-throughput and memory-efficient inference and serving engine for LLMs with AMD GPU support For more information about how to use this package see README

Latest version published 28 days ago

License: Apache-2.0

PyPI

GitHub

Package Health Score

84 / 100

security
No known security issues
popularity
Small
maintenance
Healthy
community
Active

Popularity

Small

GitHub Stars: 29.93K
Forks: 4.52K
Contributors: 460

Direct Usage Popularity

Based on project statistics from the GitHub repository for the PyPI package vllm-rocm, we found that it has been starred 29,935 times.

Security

No known security issues

Security and license risk for latest version

Release Date: Oct 15, 2024
Direct Vulnerabilities: 0
C
0
H
0
M
0
L
Indirect Vulnerabilities: 0
C
0
H
0
M
0
L
License Risk: 0
H
0
M
0
L

All security vulnerabilities belong to production dependencies of direct and indirect packages.

License

Apache-2.0

Security Policy

Yes

You can connect your project's repository to Snyk to stay up to date on security alerts and receive automatic fix pull requests.

Keep your project free of vulnerabilities with Snyk

Maintenance

Healthy

Commit Frequency

Open Issues: 1.83K
Open PR: 403
Last Release: 28 days ago
Last Commit: 2 days ago

Further analysis of the maintenance status of vllm-rocm based on released PyPI versions cadence, the repository activity, and other data points determined that its maintenance is Healthy.

We found that vllm-rocm demonstrates a positive version release cadence with at least one new version released in the past 3 months.

As a healthy sign for on-going project maintenance, we found that the GitHub repository had at least 1 pull request or issue interacted with by the community.

Community

Active

Readme: Yes
Contributing.md: Yes
Code of Conduct: Yes
Contributors: 460
Funding: Yes

A good and healthy external contribution signal for vllm-rocm project, which invites more than one hundred open source maintainers to collaborate on the repository.

Embed Package Health Score Badge

Keep your project healthy

Check your requirements.txt

Ensure all the packages you're using are healthy and well-maintained

Snyk Vulnerability Scanner

Get health score & security insights directly in your IDE

Package

Python Versions Compatibility: >=3.8

Age: 28 days
Latest Release: 28 days ago
Dependencies: 18 Direct
Versions: 1
Maintainers: 1
Wheels: Linux

Readme

Easy, fast, and cheap LLM serving for everyone

Latest News 🔥

[2024/10] We have just created a developer slack (slack.vllm.ai) focusing on coordinating contributions and discussing features. Please feel free to join us there!
[2024/10] Ray Summit 2024 held a special track for vLLM! Please find the opening talk slides from the vLLM team here. Learn more from the talks from other vLLM contributors and users!
[2024/09] We hosted the sixth vLLM meetup with NVIDIA! Please find the meetup slides here.
[2024/07] We hosted the fifth vLLM meetup with AWS! Please find the meetup slides here.
[2024/07] In partnership with Meta, vLLM officially supports Llama 3.1 with FP8 quantization and pipeline parallelism! Please check out our blog post here.
[2024/06] We hosted the fourth vLLM meetup with Cloudflare and BentoML! Please find the meetup slides here.
[2024/04] We hosted the third vLLM meetup with Roblox! Please find the meetup slides here.
[2024/01] We hosted the second vLLM meetup with IBM! Please find the meetup slides here.
[2023/10] We hosted the first vLLM meetup with a16z! Please find the meetup slides here.
[2023/08] We would like to express our sincere gratitude to Andreessen Horowitz (a16z) for providing a generous grant to support the open-source development and research of vLLM.
[2023/06] We officially released vLLM! FastChat-vLLM integration has powered LMSYS Vicuna and Chatbot Arena since mid-April. Check out our blog post.

About

vLLM is a fast and easy-to-use library for LLM inference and serving.

vLLM is fast with:

State-of-the-art serving throughput
Efficient management of attention key and value memory with PagedAttention
Continuous batching of incoming requests
Fast model execution with CUDA/HIP graph
Quantizations: GPTQ, AWQ, INT4, INT8, and FP8.
Optimized CUDA kernels, including integration with FlashAttention and FlashInfer.
Speculative decoding
Chunked prefill

Performance benchmark: We include a performance benchmark at the end of our blog post. It compares the performance of vLLM against other LLM serving engines (TensorRT-LLM, SGLang and LMDeploy). The implementation is under nightly-benchmarks folder and you can reproduce this benchmark using our one-click runnable script.

vLLM is flexible and easy to use with:

Seamless integration with popular Hugging Face models
High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more
Tensor parallelism and pipeline parallelism support for distributed inference
Streaming outputs
OpenAI-compatible API server
Support NVIDIA GPUs, AMD CPUs and GPUs, Intel CPUs and GPUs, PowerPC CPUs, TPU, and AWS Neuron.
Prefix caching support
Multi-lora support

vLLM seamlessly supports most popular open-source models on HuggingFace, including:

Transformer-like LLMs (e.g., Llama)
Mixture-of-Expert LLMs (e.g., Mixtral)
Embedding Models (e.g. E5-Mistral)
Multi-modal LLMs (e.g., LLaVA)

Find the full list of supported models here.

Getting Started

Install vLLM with pip or from source:

pip install vllm

Visit our documentation to learn more.

Contributing

We welcome and value any contributions and collaborations. Please check out CONTRIBUTING.md for how to get involved.

Citation

If you use vLLM for your research, please cite our paper:

@inproceedings{kwon2023efficient,
  title={Efficient Memory Management for Large Language Model Serving with PagedAttention},
  author={Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica},
  booktitle={Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles},
  year={2023}
}

Contact Us

For technical questions and feature requests, please use Github issues or discussions.
For discussing with fellow users, please use Discord.
For security disclosures, please use Github's security advisory feature.
For collaborations and partnerships, please contact us at vllm-questions AT lists.berkeley.edu.

vllm-rocm dependencies

aiohttp awscli boto3 botocore einops importlib-metadata msgspec partial-json-parser peft pillow protobuf psutil py-cpuinfo pytest-asyncio pyyaml pyzmq sentencepiece tqdm

FAQs

What is vllm-rocm?

A high-throughput and memory-efficient inference and serving engine for LLMs with AMD GPU support. Visit Snyk Advisor to see a full health score report for vllm-rocm, including popularity, security, maintenance & community analysis.

Is vllm-rocm popular?

We’re still processing downloads data for the python package vllm-rocm. Visit the popularity section on Snyk Advisor to see the full popularity analysis.

Is vllm-rocm well maintained?

We found that vllm-rocm demonstrated a healthy version release cadence and project activity. It has a community of 460 open source contributors collaborating on the project. See the full package health analysis to learn more about the package maintenance status.

Is vllm-rocm safe to use?

The python package vllm-rocm was scanned for known vulnerabilities and missing license, and no issues were found. Thus the package was deemed as safe to use. See the full health analysis review.

Last updated on 12 November-2024, at 20:09 (UTC).

Build a secure application checklist

Select a recommended open source package

Minimize your risk by selecting secure & well maintained open source packages

DONE

Scan your app for vulnerabilities

Scan your application to find vulnerabilities in your: source code, open source dependencies, containers and configuration files

SCAN NOW

Fix identified vulnerabilities

Easily fix your code by leveraging automatically generated PRs

AUTO FIX

Monitor for new issues

New vulnerabilities are discovered every day. Get notified if your application is affected

MONITOR APP

vllm-rocm

A high-throughput and memory-efficient inference and serving engine for LLMs with AMD GPU support For more information about how to use this package see README

Ensure you're using the healthiest python packages

Package Health Score

Popularity

Direct Usage Popularity

Security

Security and license risk for latest version

Maintenance

Commit Frequency

Community

Embed Package Health Score Badge

Keep your project healthy

Check your requirements.txt

Snyk Vulnerability Scanner

Package

Readme

Easy, fast, and cheap LLM serving for everyone

About

Getting Started

Contributing

Sponsors

Citation

Contact Us

vllm-rocm dependencies

FAQs

What is vllm-rocm?

Is vllm-rocm popular?

Is vllm-rocm well maintained?

Is vllm-rocm safe to use?

Build a secure application checklist

Select a recommended open source package

Scan your app for vulnerabilities

Fix identified vulnerabilities

Monitor for new issues

Package Health Score

Direct Usage Popularity

Security and license risk for latest version

Commit Frequency

Embed Package Health Score Badge

Keep your project healthy

Check your requirements.txt

Snyk Vulnerability Scanner

Readme

Easy, fast, and cheap LLM serving for everyone

About

Getting Started

Contributing

Sponsors

Citation

Contact Us

vllm-rocm dependencies

FAQs

What is vllm-rocm?

Is vllm-rocm popular?

Is vllm-rocm well maintained?

Is vllm-rocm safe to use?

Build a secure application checklist

Select a recommended open source package

Scan your app for vulnerabilities

Source Code

Open Source

Container

Config

Fix identified vulnerabilities

Monitor for new issues