Managing Version, Virtual Environments and Dependencies with Pyenv and Poetry

pyenv Installation

On Ubuntu/Debian/Mint, the installation of pyenv installation is done by its installer:

$ curl https://pyenv.run | bash

On MacOs, use brew to install pyenv:

$ brew update
$ brew install pyenv

For pyenv work correctly, you need to add the following lines to your shell configuration file. For Bash, this file is ~/.bashrc:

export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init --path)"

Next, it is necessary to start a new terminal session or run exec $SHELL to restart the current session.

Installing Python Versions

To see all Python versions available for installation, use the command pyenv install --list:

$ pyenv install --list | grep ' 3.9'
  3.9.0
  3.9-dev
  3.9.1
  3.9.2
  3.9.3
  3.9.4

To install versions 3.7.10 and 3.9.4, use:

$ pyenv install 3.7.10
$ pyenv install 3.9.4

The command pyenv versions lists all installed versions available:

$ pyenv versions
* system (set by $HOME/.pyenv/version)
  3.7.10
  3.9.4

How pyenv Works

pyenv inserts $HOME/.pyenv/shims at the begining of PATH to intercept calls to python and other related commands. This intercept leads to special executables called shims, which redirect the calls to a specific python version 1, depending on the first configuration level found, in the following order:

1. Shell. Version registered in the environment variable PYENV_VERSION. You can use the command pyenv shell <version> to set this variable in your current shell session, or use another equivalent command such as export PYENV_VERSION=<version>.
2. Local. Version registered in a file .python-version, which is searched recursively from the current directory until reaching the root directory. You can use the command pyenv local <version> to generate this file.
3. Global. Version registered in the file $(pyenv root)/version, can also be generated by the command pyenv global <version>.
4. System. If no configuration is found, the Python version installed on the operating system is used.

poetry Installation

There are a few options to install poetry. The most recommended one is using its installer, which prevents poetry dependencies from mixing with those of other libraries:

$ curl https://install.python-poetry.org | python -

On Linux, the installation is done in the directory $HOME/.local/bin. If this directory is not in your PATH, then you need to manually add it to your shell configuration file (~/.bashrc or ~/.profile) with the following lines:

if [ -d "$HOME/.local/bin" ] ; then
    PATH="$HOME/.local/bin:$PATH"
fi

# pyenv configuration goes here

To test the installation, run:

$ poetry --version

If the command does not return any error message, then the installation was completed successfully!

Poetry Initial Configuration

Poetry's configuration is kept in the $HOME/.config/pypoetry/config.toml. But instead of accessing this file directly, you should use the command poetry config and its subcommands.

To list the settings, run:

$ poetry config --list

After the installation, it is advisable to modify where poetry keeps virtual environments. The default configuration creates virtual environments inside the directory $HOME/.cache/pypoetry, but it is better that each virtual environment is created inside the project root, named after .venv, because editors such as VSCode can automatically find it and configure intellisense.

To change it, run:

$ poetry config virtualenvs.in-project true

Poetry Commands

To display available commands, run:

$ poetry

Let's focus on some related to the most common tasks:

1. Create a new project
2. Manage dependencies
3. Manage the virtual environment

Creating a New Project

The command poetry new creates a new project with a basic structure of directories and files. For example:

$ poetry new project-x --name app

results in the following structure:

project-x
├── app
│   └── __init__.py
├── pyproject.toml
├── README.rst
└── tests
    ├── __init__.py
    └── test_app.py

--name is optional and allows defining a different name for the project package directory. The poetry's default is to use the same project name.

The file pyproject.toml is the project's configuration file. Contains information such as name, version, dependencies, etc.

Managing Dependencies

The main commands related to dependency management are:

1. poetry add. Adds a dependency to the project.
2. poetry remove. Removes a dependency from the project.
3. poetry show. Displays project dependencies.
4. poetry update. Updates project dependencies.

Project dependencies can be divided into two groups: packages essential for project functioning and packages used only for project development, in activities such as testing and linting . When deploying the project in a staged or production environment, it is necessary to install packages from the first group, but those from the second group should be avoided to reduce the installation size.

Adding the main packages is done by the command poetry add. For example:

$ poetry add fastapi jinja2 aioredis[hiredis] databases loguru passlib[argon2]
Creating virtualenv project-x in /tmp/project-x/.venv
Using version ^0.70.0 for fastapi
Using version ^3.0.3 for Jinja2
Using version ^2.0.0 for aioredis
Using version ^0.5.3 for databases
Using version ^0.5.3 for loguru
Using version ^1.7.4 for passlib

Updating dependencies
Resolving dependencies... (8.8s)

Writing lock file

Package operations: 28 installs, 0 updates, 0 removals

• Installing idna (3.3)
• Installing pycparser (2.21)
• Installing sniffio (1.2.0)
• Installing anyio (3.4.0)
• Installing cffi (1.15.0)
• Installing greenlet (1.1.2)
...

Since versions have not been explicitly specified, the latest available versions of the packages and their dependencies are used.

To add development packages, use poetry add --dev:

$ poetry add --dev pytest alt-pytest-asyncio pytest-cov asgi-lifespan isort blue mypy httpx
The following packages are already present in the pyproject.toml and will be skipped:

• pytest

If you want to update it to the latest compatible version, you can use `poetry update package`.
If you prefer to upgrade it to the latest available version, you can use `poetry add package@latest`.

Using version ^0.6.0 for alt-pytest-asyncio
Using version ^3.0.0 for pytest-cov
Using version ^1.0.1 for asgi-lifespan
Using version ^5.10.1 for isort
Using version ^0.7.0 for blue
Using version ^0.910 for mypy
Using version ^0.21.1 for httpx

Updating dependencies
Resolving dependencies... (10.3s)

Writing lock file

Package operations: 25 installs, 0 updates, 0 removals

• Installing appdirs (1.4.4)
• Installing certifi (2021.10.8)
• Installing click (8.0.3)
• Installing h11 (0.12.0)
...

As packages are added, the file pyproject.toml is updated with information about dependencies. Note that development dependencies are in a separate section:

[tool.poetry.dependencies]
python = "^3.10"
fastapi = "^0.70.0"
Jinja2 = "^3.0.3"
aioredis = {extras = ["hiredis"], version = "^2.0.0"}
databases = "^0.5.3"
loguru = "^0.5.3"
passlib = {extras = ["argon2"], version = "^1.7.4"}

[tool.poetry.dev-dependencies]
pytest = "^5.2"
alt-pytest-asyncio = "^0.6.0"
pytest-cov = "^3.0.0"
asgi-lifespan = "^1.0.1"
isort = "^5.10.1"
blue = "^0.7.0"
mypy = "^0.910"
httpx = "^0.21.1"

Poetry accepte versions based on standard SemVer (Major.Minor.Patch) and uses some conventions to version update specification. The ^ notation indicates that a version upgrade is allowed if the new version number does not modify the left most non-zero digit in the grouping Major.Minor.Patch. For example, poetry update fastapi would accept any version >=0.70.0 and <0.71.0.

During the installation of the development dependencies, you might have noticed that pytest was not installed because it was already included by the command poetry new during project creation. However, the version 5.2 is not the latest version available. The first impulse is to try to update the version using the command poetry update pytest, but the specification ^5.2 does not allow a newer version such as 6 to be used. The solution is to explicitly add the latest pytest version as a development dependency:

$ poetry add --dev pytest@latest
Using version ^6.2.5 for pytest

Updating dependencies
Resolving dependencies... (0.7s)

Writing lock file

Package operations: 1 install, 1 update, 2 removals

• Removing more-itertools (8.12.0)
• Removing wcwidth (0.2.5)
• Installing iniconfig (1.1.1)
• Updating pytest (5.4.3 -> 6.2.5)

This installs the package and also updates the section [tool.poetry.dev-dependencies] in pyproject.toml with the new version.

Note that several other indirect dependencies are installed in addition to the specified packages. The exact relationship with all installed packages, their versions and other additional information is stored in the file poetry.lock. This file ensures that everyone in the project uses exactly the same package versions.

Rather than looking into poetry.lock directly, the best way to see the complete list of project dependencies is with the command poetry show --tree:

$ poetry show --tree
aioredis 2.0.0 asyncio (PEP 3156) Redis support
├── async-timeout *
│   └── typing-extensions >=3.6.5
├── hiredis >=1.0
└── typing-extensions *
alt-pytest-asyncio 0.6.0 Alternative pytest plugin to pytest-asyncio
└── pytest >=3.0.6
    ├── atomicwrites >=1.0
    ├── attrs >=19.2.0
    ├── colorama *
    ├── iniconfig *
    ├── packaging *
    │   └── pyparsing >=2.0.2,<3.0.5 || >3.0.5
    ├── pluggy >=0.12,<2.0
    ├── py >=1.8.2
    └── toml *
asgi-lifespan 1.0.1 Programmatic startup/shutdown of ASGI apps.
└── sniffio *
blue 0.7.0 Blue -- Some folks like black but I prefer blue.
├── black 21.7b0
│   ├── appdirs *
│   ├── click >=7.1.2
│   │   └── colorama *
│   ├── mypy-extensions >=0.4.3
│   ├── pathspec >=0.8.1,<1
│   ├── regex >=2020.1.8
│   └── tomli >=0.2.6,<2.0.0
└── flake8 3.8.4
    ├── mccabe >=0.6.0,<0.7.0
    ├── pycodestyle >=2.6.0a1,<2.7.0
    └── pyflakes >=2.2.0,<2.3.0
...

Enabling The Virtual Environment

If you just cloned the project, you should use it poetry install to create the virtual environment directory and install the dependencies.

Once installed in .venv, there are two options to enable the virtual environment through the terminal. The first is to activate the virtual environment with the command poetry shell:

$ poetry shell

(project-x) $

The command line prompt changes to show activation. To disable the virtual environment, you can run exit, press CTRL+D, or just open a new terminal.

The second option is to use poetry run <command>, that activates the virtual environment, runs the command, and then exits the virtual environment. It is particularly useful for running in scripts.

For example, be a project containing one Makefile with the following task:

test:
    pytest --cov-report term-missing --cov-report html --cov-branch \
        --cov app/

You can launch the virtual environment and then run make test to run the tests, or you can run poetry run make test in one step.