tokenizers

v0.13.3

![https://github.com/napi-rs/package-template/actions](https://github.com/napi-rs/package-template/workflows/CI/badge.svg) For more information about how to use this package see README

Latest version published 12 months ago

License: Apache-2.0

NPM

GitHub

Package Health Score

74 / 100

security
No known security issues
popularity
Small
maintenance
Sustainable
community
Sustainable

Explore Similar Packages

Security

No known security issues

All security vulnerabilities belong to production dependencies of direct and indirect packages.

Security and license risk for significant versions

All Versions

Version				Vulnerabilities	License Risk
0.13.3	\|	05/2023		0 C 0 H 0 M 0 L	0 H 0 M 0 L
0.12.0	\|	03/2022		0 C 0 H 0 M 0 L	0 H 0 M 0 L
0.8.2	\|	01/2022		0 C 0 H 0 M 0 L	0 H 0 M 0 L
0.7.0	\|	07/2020	Popular	0 C 0 H 0 M 0 L	0 H 0 M 0 L
0.6.2	\|	04/2020		0 C 0 H 0 M 0 L	0 H 0 M 0 L

License

Apache-2.0

Security Policy

Is your project affected by vulnerabilities?

Scan your projects for vulnerabilities. Fix quickly with automated fixes. Get started with Snyk for free.

Get started free

Popularity

Small

Weekly Downloads (328)

GitHub Stars: 8.39K
Forks: 707
Contributors: 90

Direct Usage Popularity

The npm package tokenizers receives a total of 328 downloads a week. As such, we scored tokenizers popularity level to be Small.

Based on project statistics from the GitHub repository for the npm package tokenizers, we found that it has been starred 8,387 times.

Downloads are calculated as moving averages for a period of the last 12 months, excluding weekends and known missing data points.

Community

Sustainable

Readme.md: Yes
Contributing.md: No
Code of Conduct: No
Contributors: 90
Funding: No

With more than 10 contributors for the tokenizers repository, this is possibly a sign for a growing and inviting community.

We found a way for you to contribute to the project! Looks like tokenizers is missing a Code of Conduct.

Embed Package Health Score Badge

Maintenance

Sustainable

Commit Frequency

Open Issues: 93
Open PR: 13
Last Release: 12 months ago
Last Commit: 18 days ago

Further analysis of the maintenance status of tokenizers based on released npm versions cadence, the repository activity, and other data points determined that its maintenance is Sustainable.

We found that tokenizers demonstrates a positive version release cadence with at least one new version released in the past 12 months.

As a healthy sign for on-going project maintenance, we found that the GitHub repository had at least 1 pull request or issue interacted with by the community.

Keep your project healthy

Check your package.json

Ensure all the packages you're using are healthy and well-maintained

Snyk Vulnerability Scanner

Get health score & security insights directly in your IDE

Package

Node.js Compatibility: >= 10

Age: 5 years
Dependencies: 13 Direct
Versions: 39
Install Size: 29.2 kB
Dist-tags: 2
# of Files: 5
Maintainers: 4
TS Typings: Yes

tokenizers has more than a single and default latest tag published for the npm package. This means, there may be other tags available for this package, such as next to indicate future releases, or stable to indicate stable releases.

Readme

NodeJS implementation of today's most used tokenizers, with a focus on performance and versatility. Bindings over the Rust implementation. If you are interested in the High-level design, you can go check it there.

Main features

Train new vocabularies and tokenize using 4 pre-made tokenizers (Bert WordPiece and the 3 most common BPE versions).
Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes less than 20 seconds to tokenize a GB of text on a server's CPU.
Easy to use, but also extremely versatile.
Designed for research and production.
Normalization comes with alignments tracking. It's always possible to get the part of the original sentence that corresponds to a given token.
Does all the pre-processing: Truncate, Pad, add the special tokens your model needs.

Installation

npm install tokenizers@latest

Basic example

import { Tokenizer } from "tokenizers";

const tokenizer = await Tokenizer.fromPretrained("gpt2");
const wpEncoded = await tokenizer.encode("Who is John?");

console.log(wpEncoded.getLength());
console.log(wpEncoded.getTokens());
console.log(wpEncoded.getIds());
console.log(wpEncoded.getAttentionMask());
console.log(wpEncoded.getOffsets());
console.log(wpEncoded.getOverflowing());
console.log(wpEncoded.getSpecialTokensMask());
console.log(wpEncoded.getTypeIds());
console.log(wpEncoded.getWordIds());

License

Apache License 2.0

tokenizers dependencies

tokenizers-win32-x64-msvc tokenizers-darwin-x64 tokenizers-linux-x64-gnu tokenizers-linux-x64-musl tokenizers-linux-arm64-gnu tokenizers-win32-ia32-msvc tokenizers-linux-arm-gnueabihf tokenizers-darwin-arm64 tokenizers-android-arm64 tokenizers-freebsd-x64 tokenizers-linux-arm64-musl tokenizers-win32-arm64-msvc tokenizers-android-arm-eabi

FAQs

What is tokenizers?

![https://github.com/napi-rs/package-template/actions](https://github.com/napi-rs/package-template/workflows/CI/badge.s... Visit Snyk Advisor to see a full health score report for tokenizers, including popularity, security, maintenance & community analysis.

Is tokenizers popular?

The npm package tokenizers receives a total of 328 weekly downloads. As such, tokenizers popularity was classified as small. Visit the popularity section on Snyk Advisor to see the full health analysis.

Is tokenizers well maintained?

We found indications that tokenizers maintenance is sustainable demonstrating some project activity. We saw a total of 90 open source contributors collaborating on the project. See the full package health analysis to learn more about the package maintenance status.

Is tokenizers safe to use?

The npm package tokenizers was scanned for known vulnerabilities and missing license, and no issues were found. Thus the package was deemed as safe to use. See the full health analysis review.

Last updated on 3 May-2024, at 13:26 (UTC).

Build a secure application checklist

Select a recommended open source package

Minimize your risk by selecting secure & well maintained open source packages

DONE

Scan your app for vulnerabilities

Scan your application to find vulnerabilities in your: source code, open source dependencies, containers and configuration files

SCAN NOW

Fix identified vulnerabilities

Easily fix your code by leveraging automatically generated PRs

AUTO FIX

Monitor for new issues

New vulnerabilities are discovered every day. Get notified if your application is affected

MONITOR APP

tokenizers

![https://github.com/napi-rs/package-template/actions](https://github.com/napi-rs/package-template/workflows/CI/badge.svg) For more information about how to use this package see README

Ensure you're using the healthiest npm packages

Package Health Score

Explore Similar Packages

Security

Security and license risk for significant versions

Is your project affected by vulnerabilities?

Popularity

Weekly Downloads (328)

Direct Usage Popularity

Community

Embed Package Health Score Badge

Maintenance

Commit Frequency

Keep your project healthy

Check your package.json

Snyk Vulnerability Scanner

Package

Readme

Main features

Installation

Basic example

License

tokenizers dependencies

tokenizers development dependencies

FAQs

What is tokenizers?

Is tokenizers popular?

Is tokenizers well maintained?

Is tokenizers safe to use?

Build a secure application checklist

Select a recommended open source package

Scan your app for vulnerabilities

Fix identified vulnerabilities

Monitor for new issues

Package Health Score

Explore Similar Packages

Security and license risk for significant versions

Is your project affected by vulnerabilities?

Weekly Downloads (328)

Direct Usage Popularity

Embed Package Health Score Badge

Commit Frequency

Keep your project healthy

Check your package.json

Snyk Vulnerability Scanner

Readme

Main features

Installation

Basic example

License

tokenizers dependencies

tokenizers development dependencies

FAQs

What is tokenizers?

Is tokenizers popular?

Is tokenizers well maintained?

Is tokenizers safe to use?

Build a secure application checklist

Select a recommended open source package

Scan your app for vulnerabilities

Source Code

Open Source

Container

Config

Fix identified vulnerabilities

Monitor for new issues