Poetry: Dependency Management for Python

Poetry helps you declare, manage and install dependencies of Python projects, ensuring you have the right stack everywhere.

It supports Python 2.7 and 3.4+.

Installation

Poetry provides a custom installer that will install poetry isolated from the rest of your system by vendorizing its dependencies. This is the recommended way of installing poetry.

curl -sSL https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py | python

Alternatively, you can download the get-poetry.py file and execute it separately.

If you want to install prerelease versions, you can do so by passing --preview to get-poetry.py:

python get-poetry.py --preview

Similarly, if you want to install a specific version, you can use --version:

python get-poetry.py --version 0.7.0

Using pip to install poetry is also possible.

pip install --user poetry

Be aware, however, that it will also install poetry's dependencies which might cause conflicts.

Updating poetry

Updating poetry to the latest stable version is as simple as calling the self:update command.

poetry self:update

If you want to install prerelease versions, you can use the --preview option.

poetry self:update --preview

And finally, if you want to install a specific version you can pass it as an argument to self:update.

poetry self:update 0.8.0

Enable tab completion for Bash, Fish, or Zsh

poetry supports generating completion scripts for Bash, Fish, and Zsh. See poetry help completions for full details, but the gist is as simple as using one of the following:

# Bash
poetry completions bash > /etc/bash_completion.d/poetry.bash-completion

# Bash (macOS/Homebrew)
poetry completions bash > $(brew --prefix)/etc/bash_completion.d/poetry.bash-completion

# Fish
poetry completions fish > ~/.config/fish/completions/poetry.fish

# Zsh
poetry completions zsh > ~/.zfunc/_poetry

# Zsh (macOS/Homebrew)
poetry completions zsh > $(brew --prefix)/share/zsh/site-functions/_poetry

Note: you may need to restart your shell in order for the changes to take effect.

For zsh, you must then add the following line in your ~/.zshrc before compinit (not for homebrew setup):

fpath+=~/.zfunc

Introduction

poetry is a tool to handle dependency installation as well as building and packaging of Python packages. It only needs one file to do all of that: the new, standardized pyproject.toml.

In other words, poetry uses pyproject.toml to replace setup.py, requirements.txt, setup.cfg, MANIFEST.in and the newly added Pipfile.

[tool.poetry]
name = "my-package"
version = "0.1.0"
description = "The description of the package"

license = "MIT"

authors = [
    "Sébastien Eustace "
]

readme = 'README.md'  # Markdown files are supported

repository = "https://github.com/sdispater/poetry"
homepage = "https://github.com/sdispater/poetry"

keywords = ['packaging', 'poetry']

[tool.poetry.dependencies]
python = "~2.7 || ^3.2"  # Compatible python versions must be declared here
toml = "^0.9"
# Dependencies with extras
requests = { version = "^2.13", extras = [ "security" ] }
# Python specific dependencies with prereleases allowed
pathlib2 = { version = "^2.2", python = "~2.7", allow-prereleases = true }
# Git dependencies
cleo = { git = "https://github.com/sdispater/cleo.git", branch = "master" }

# Optional dependencies (extras)
pendulum = { version = "^1.4", optional = true }

[tool.poetry.dev-dependencies]
pytest = "^3.0"
pytest-cov = "^2.4"

[tool.poetry.scripts]
my-script = 'my_package:main'

There are some things we can notice here:

• It will try to enforce semantic versioning as the best practice in version naming.
• You can specify the readme, included and excluded files: no more MANIFEST.in. poetry will also use VCS ignore files (like .gitignore) to populate the exclude section.
• Keywords (up to 5) can be specified and will act as tags on the packaging site.
• The dependencies sections support caret, tilde, wildcard, inequality and multiple requirements.
• You must specify the python versions for which your package is compatible.

poetry will also detect if you are inside a virtualenv and install the packages accordingly. So, poetry can be installed globally and used everywhere.

poetry also comes with a full fledged dependency resolution library.

Why?

Packaging systems and dependency management in Python are rather convoluted and hard to understand for newcomers. Even for seasoned developers it might be cumbersome at times to create all files needed in a Python project: setup.py, requirements.txt, setup.cfg, MANIFEST.in and the newly added Pipfile.

So I wanted a tool that would limit everything to a single configuration file to do: dependency management, packaging and publishing.

It takes inspiration in tools that exist in other languages, like composer (PHP) or cargo (Rust).

And, finally, there is no reliable tool to properly resolve dependencies in Python, so I started poetry to bring an exhaustive dependency resolver to the Python community.

What about Pipenv?

In short: I do not like the CLI it provides, or some of the decisions made, and I think we can make a better and more intuitive one. Here are a few things that I don't like.

Dependency resolution

The dependency resolution is erratic and will fail even if there is a solution. Let's take an example:

pipenv install oslo.utils==1.4.0

will fail with this error:

Could not find a version that matches pbr!=0.7,!=2.1.0,<1.0,>=0.6,>=2.0.0

while Poetry will get you the right set of packages:

poetry add oslo.utils=1.4.0

results in :

  - Installing pytz (2018.3)
  - Installing netifaces (0.10.6)
  - Installing netaddr (0.7.19)
  - Installing oslo.i18n (2.1.0)
  - Installing iso8601 (0.1.12)
  - Installing six (1.11.0)
  - Installing babel (2.5.3)
  - Installing pbr (0.11.1)
  - Installing oslo.utils (1.4.0)

This is possible thanks to the efficient dependency resolver at the heart of Poetry.

Here is a breakdown of what exactly happens here:

oslo.utils (1.4.0) depends on:

• pbr (>=0.6,!=0.7,<1.0)
• Babel (>=1.3)
• six (>=1.9.0)
• iso8601 (>=0.1.9)
• oslo.i18n (>=1.3.0)
• netaddr (>=0.7.12)
• netifaces (>=0.10.4)

What interests us is pbr (>=0.6,!=0.7,<1.0).

At this point, poetry will choose pbr==0.11.1 which is the latest version that matches the constraint.

Next it will try to select oslo.i18n==3.20.0 which is the latest version that matches oslo.i18n (>=1.3.0).

However this version requires pbr (!=2.1.0,>=2.0.0) which is incompatible with pbr==0.11.1, so poetry will try to find a version of oslo.i18n that satisfies pbr (>=0.6,!=0.7,<1.0).

By analyzing the releases of oslo.i18n, it will find oslo.i18n==2.1.0 which requires pbr (>=0.11,<2.0). At this point the rest of the resolution is straightforward since there is no more conflict.

Install command

When you specify a package to the install command it will add it as a wildcard dependency. This means that any version of this package can be installed which can lead to compatibility issues.

Also, you have to explicitly tell it to not update the locked packages when you install new ones. This should be the default.

Uninstall command

The uninstall command will only remove the package specified but not its dependencies if they are no longer needed.

You either have to use sync or clean to fix that.

Too limited in scope

Finally, the Pipfile is just a replacement from requirements.txt and, in the end, you will still need to populate your setup.py file (or setup.cfg) with the exact same dependencies you declared in your Pipfile. So, in the end, you will still need to manage a few configuration files to properly setup your project.

Commands

new

This command will help you kickstart your new Python project by creating a directory structure suitable for most projects.

poetry new my-package

will create a folder as follows:

my-package
├── pyproject.toml
├── README.rst
├── my_package
│   └── __init__.py
└── tests
    ├── __init__.py
    └── test_my_package

If you want to name your project differently than the folder, you can pass the --name option:

poetry new my-folder --name my-package

init

This command will help you create a pyproject.toml file interactively by prompting you to provide basic information about your package.

It will interactively ask you to fill in the fields, while using some smart defaults.

poetry init

Options

• --name: Name of the package.
• --description: Description of the package.
• --author: Author of the package.
• --dependency: Package to require with a version constraint. Should be in format foo:1.0.0.
• --dev-dependency: Development requirements, see --require.

install

The install command reads the pyproject.toml file from the current directory, resolves the dependencies, and installs them.

poetry install

If there is a poetry.lock file in the current directory, it will use the exact versions from there instead of resolving them. This ensures that everyone using the library will get the same versions of the dependencies.

If there is no poetry.lock file, Poetry will create one after dependency resolution.

You can specify to the command that you do not want the development dependencies installed by passing the --no-dev option.

poetry install --no-dev

You can also specify the extras you want to be installed by passing the -E|--extras option (See Extras for more info)

poetry install --extras "mysql pgsql"
poetry install -E mysql -E pgsql

By default poetry will install your project's package everytime you run install:

$ poetry install
Installing dependencies from lock file

No dependencies to install or update

  - Installing  (x.x.x)

If you want to skip this installation, use the --no-root option.

poetry install --no-root

Options

• --no-dev: Do not install dev dependencies.
• --no-root: Do not install the root package (your project).
• -E|--extras: Features to install (multiple values allowed).

update

In order to get the latest versions of the dependencies and to update the poetry.lock file, use the update command.

poetry update

This will resolve all dependencies of the project and write the exact versions into poetry.lock.

If you just want to update a few packages and not all, you can list them as such:

poetry update requests toml

Options

• --dry-run : Outputs the operations but will not execute anything (implicitly enables --verbose).
• --no-dev : Do not install dev dependencies.
• --lock : Do not perform install (only update the lockfile).

add

The add command adds required packages to your pyproject.toml and installs them.

If you do not specify a version constraint, poetry will choose a suitable one based on the available package versions.

poetry add requests pendulum

Options

• --D|dev: Add package as development dependency.
• --optional : Add as an optional dependency.
• --dry-run : Outputs the operations but will not execute anything (implicitly enables --verbose).

remove

The remove command removes a package from the current list of installed packages

poetry remove pendulum

Options

• --D|dev: Removes a package from the development dependencies.
• --dry-run : Outputs the operations but will not execute anything (implicitly enables --verbose).

show

To list all of the available packages, you can use the show command.

poetry show

If you want to see the details of a certain package, you can pass the package name.

poetry show pendulum

name        : pendulum
version     : 1.4.2
description : Python datetimes made easy

dependencies:
 - python-dateutil >=2.6.1
 - tzlocal >=1.4
 - pytzdata >=2017.2.2

Options

• --no-dev: Do not list the dev dependencies.
• --tree: List the dependencies as a tree.
• -l|--latest: Show the latest version.
• -o|--outdated: Show the latest version but only for packages that are outdated.

build

The build command builds the source and wheels archives.

poetry build

Note that, at the moment, only pure python wheels are supported.

Options

• -f|--format: Limit the format to either wheel or sdist.

publish

This command builds (if not already built) and publishes the package to the remote repository.

It will automatically register the package before uploading if this is the first time it is submitted.

poetry publish

Options

• -r|--repository: The repository to register the package to (default: pypi). Should match a repository name set by the config command.
• --username (-u): The username to access the repository.
• --password (-p): The password to access the repository.

config

The config command allows you to edit poetry config settings and repositories.

poetry config --list

Usage

poetry config [options] [setting-key] [setting-value1] ... [setting-valueN]

setting-key is a configuration option name and setting-value1 is a configuration value.

Modifying repositories

In addition to modifying the config section, the config command also supports making changes to the repositories section by using it the following way:

poetry config repositories.foo https://foo.bar/simple/

This will set the url for repository foo to https://foo.bar/simple/.

If you want to store your credentials for a specific repository, you can do so easily:

poetry config http-basic.foo username password

If you do not specify the password you will be prompted to write it.

Options

• --unset: Remove the configuration element named by setting-key.
• --list: Show the list of current config variables.

search

This command searches for packages on a remote index.

poetry search requests pendulum

Options

• -N|--only-name: Search only in name.

lock

This command locks (without installing) the dependencies specified in pyproject.toml.

poetry lock

export

This command exports the lock file to other formats.

poetry export -f requirements.txt > requirements.txt

Options

• --format (-f): the format to export to. Currently, only requirements.txt is supported.
• --output (-o): the name of the output file. If omitted, print to standard output.

The pyproject.toml file

The tool.poetry section of the pyproject.toml file is composed of multiple sections.

name

The name of the package. Required

version

The version of the package. Required

This should follow semantic versioning. However it will not be enforced and you remain free to follow another specification.

description

A short description of the package. Required

license

The license of the package.

The recommended notation for the most common licenses is (alphabetical):

• Apache-2.0
• BSD-2-Clause
• BSD-3-Clause
• BSD-4-Clause
• GPL-2.0-only
• GPL-2.0-or-later
• GPL-3.0-only
• GPL-3.0-or-later
• LGPL-2.1-only
• LGPL-2.1-or-later
• LGPL-3.0-only
• LGPL-3.0-or-later
• MIT

Optional, but it is highly recommended to supply this. More identifiers are listed at the SPDX Open Source License Registry.

authors

The authors of the package. Required

This is a list of authors and should contain at least one author. Authors must be in the form name .

readme

The readme file of the package. Required

The file can be either README.rst or README.md.

homepage

An URL to the website of the project. Optional

repository

An URL to the repository of the project. Optional

documentation

An URL to the documentation of the project. Optional

keywords

A list of keywords (max: 5) that the package is related to. Optional

include and exclude

A list of patterns that will be included in the final package.

You can explicitly specify to Poetry that a set of globs should be ignored or included for the purposes of packaging. The globs specified in the exclude field identify a set of files that are not included when a package is built.

If a VCS is being used for a package, the exclude field will be seeded with the VCS’ ignore settings (.gitignore for git for example).

[tool.poetry]
# ...
include = ["package/**/*.py", "package/**/.c"]

exclude = ["package/excluded.py"]

dependencies and dev-dependencies

Poetry is configured to look for dependencies on PyPi by default. Only the name and a version string are required in this case.

[tool.poetry.dependencies]
requests = "^2.13.0"

If you want to use a private repository, you can add it to your pyproject.toml file, like so:

[[tool.poetry.source]]
name = 'private'
url = 'http://example.com/simple'

Be aware that declaring the python version for which your package is compatible is mandatory:

[tool.poetry.dependencies]
python = "^3.6"

Caret requirement

Caret requirements allow SemVer compatible updates to a specified version. An update is allowed if the new version number does not modify the left-most non-zero digit in the major, minor, patch grouping. In this case, if we ran poetry update requests, poetry would update us to version 2.14.0 if it was available, but would not update us to 3.0.0. If instead we had specified the version string as ^0.1.13, poetry would update to 0.1.14 but not 0.2.0. 0.0.x is not considered compatible with any other version.

Here are some more examples of caret requirements and the versions that would be allowed with them:

^1.2.3 := >=1.2.3 <2.0.0
^1.2 := >=1.2.0 <2.0.0
^1 := >=1.0.0 <2.0.0
^0.2.3 := >=0.2.3 <0.3.0
^0.0.3 := >=0.0.3 <0.0.4
^0.0 := >=0.0.0 <0.1.0
^0 := >=0.0.0 <1.0.0

Tilde requirements

Tilde requirements specify a minimal version with some ability to update. If you specify a major, minor, and patch version or only a major and minor version, only patch-level changes are allowed. If you only specify a major version, then minor- and patch-level changes are allowed.

~1.2.3 is an example of a tilde requirement.

~1.2.3 := >=1.2.3 <1.3.0
~1.2 := >=1.2.0 <1.3.0
~1 := >=1.0.0 <2.0.0

Wildcard requirements

Wildcard requirements allow for any version where the wildcard is positioned.

*, 1.* and 1.2.* are examples of wildcard requirements.

* := >=0.0.0
1.* := >=1.0.0 <2.0.0
1.2.* := >=1.2.0 <1.3.0

Inequality requirements

Inequality requirements allow manually specifying a version range or an exact version to depend on.

Here are some examples of inequality requirements:

>= 1.2.0
> 1
< 2
!= 1.2.3

Multiple requirements

Multiple version requirements can also be separated with a comma, e.g. >= 1.2, < 1.5.

git dependencies

To depend on a library located in a git repository, the minimum information you need to specify is the location of the repository with the git key:

[tool.poetry.dependencies]
requests = { git = "https://github.com/requests/requests.git" }

Since we haven’t specified any other information, Poetry assumes that we intend to use the latest commit on the master branch to build our project. You can combine the git key with the rev, tag, or branch keys to specify something else. Here's an example of specifying that you want to use the latest commit on a branch named next:

[tool.poetry.dependencies]
requests = { git = "https://github.com/kennethreitz/requests.git", branch = "next" }

Python restricted dependencies

You can also specify that a dependency should be installed only for specific Python versions:

[tool.poetry.dependencies]
pathlib2 = { version = "^2.2", python = "~2.7" }

[tool.poetry.dependencies]
pathlib2 = { version = "^2.2", python = ["~2.7", "^3.2"] }

scripts

This section describe the scripts or executable that will be installed when installing the package

[tool.poetry.scripts]
poetry = 'poetry.console:run'

After installing a package with the above toml, poetry will be a global command available from the command line that will execute console.run in the poetry package.

extras

Poetry supports extras to allow expression of:

• optional dependencies, which enhance a package, but are not required; and
• clusters of optional dependencies.

[tool.poetry]
name = "awesome"

[tool.poetry.dependencies]
# These packages are mandatory and form the core of this package’s distribution.
mandatory = "^1.0"

# A list of all of the optional dependencies, some of which are included in the
# below `extras`. They can be opted into by apps.
psycopg2 = { version = "^2.7", optional = true }
mysqlclient = { version = "^1.3", optional = true }

[tool.poetry.extras]
mysql = ["mysqlclient"]
pgsql = ["psycopg2"]

When installing packages, you can specify extras by using the -E|--extras option:

poetry install --extras "mysql pgsql"
poetry install -E mysql -E pgsql

plugins

Poetry supports arbitrary plugins which work similarly to setuptools entry points. To match the example in the setuptools documentation, you would use the following:

[tool.poetry.plugins] # Optional super table

[tool.poetry.plugins."blogtool.parsers"]
".rst" = "some_module::SomeClass"

Resources

• Official Website
• Issue Tracker