Python PEP 668 - working with "externally managed environment"

Python Linux users would have noticed that python is now an "externally managed environment" on newer releases of most OS. That is indeed the case for Debian 10/Bookworm and thus TurnKey v18.x. It certainly did for me - at least initially. Marcos, a long term friend of TurnKey recently reached out to me to ask about the best way to work around this when developing on our Odoo appliance.

The issue

Before I go on, for those of you who are not sure what I'm talking about, try installing or updating a python package via pip. It will fail with an error message:

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.11/README.venv for more information.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

This does fix a legitimate issue. The previous behavior (the same you get now with '--break-system-packages') could lead to some painful situations where apt and pip can fight for which version of a python library will installed. Meaning that a python library version might change unexpectedly. Regardless, this change is also a big PITA for developers used to the previous behavior, especially when working on a specific OS version when mostly only system packages are required. To read more detail about the rationale of this change, please see PEP 668.

Resolution options

As per the message, installing Python packages via apt is preferable. But what about if you need a python library not available in Debian? Or a newer version that what Debian provides? As noted by the error message, using a Python venv is the next best option. But that means duplication of any apt packages you may already have installed therefore bloat. It also means that you miss out on the automated security updates that TurnKey provides for Debian packages. The only remaining option is to "break system packages". That doesn't sound good! It will revert your system to the behavior before the application of PEP 668 - thus making life more complicated in the future...

Pros and Cons of each approach.

Assuming that you want/need versions and/or packages not available in Debian, what is the best path? Obviously each option has pros and cons, so which way should you go? In his message to me, Marcos nicely laid out the pros and cons of each of the 2 suggested approaches, so I'll share them here:

Virtual Environment

• Isolates application dependencies, avoiding conflicts with system packages.
• Allows for more flexible and up-to-date package management.

• Adds complexity to the setup and maintenance process.
• Increases the overall footprint and resource requirements of the deployment.

System-Wide Installation

• Simpler setup and integration with the system.
• Utilizes the standard Turnkey Linux deployment model.

• Potential conflicts with system-managed packages.
• Limited by the constraints imposed by Debian 12.

Another - perhaps better - option

Another option not noted in the pip error message is to create a virtual environment, with the system python libraries passed into the venv. Whilst it's still not perfect, in my option it is by far the best option for many scenarios - unless of course you can get by just using system packages alone. TBH, I'm a bit surprised that it's not noted in the error message. It's pretty easy to set up, just include adding the '--system-site-packages' switch when creating your virtual environment. I.e.:

python3 -m venv --system-site-packages /path/to/venv

What you get with this approach

Let's have a look at what you get when using '--system-site-packages'. First let's create an example venv. Note that all of this is running as root from root's home (/root). Although for any AWS Marketplace users (or non TurnKey users) most of these commands should work fine as a "sudo" user (for apt installs).

root@core ~# mkdir ~/test_venv
root@core ~# python3 -m venv --system-site-packages ~/test_venv
root@core ~# source ~/test_venv/bin/activate
(test_venv) root@core ~#

Now for a demonstration of what happens when you use it.

I'll use a couple of apt packages with my examples:

• python3-pygments (initially installed)
• python3-socks (initially not-installed)

Continuing on from creating the venv above, let's confirm the package versions and status:

(test_venv) root@core ~# apt list python3-pygments python3-socks
Listing... Done
python3-pygments/stable,now 2.14.0+dfsg-1 all [installed,automatic]
python3-socks/stable 1.7.1+dfsg-1 all

So we have python3-pygments installed and it's version 2.14.0. python3-socks is not installed, but the available version is 1.7.1. Now let's check that the installed package (pygments) is available in the venv and that it's the system version. For those not familiar with grep, the grep command used here does a case-insensitive search for lines that include socks or pygments.

(test_venv) root@core ~# pip list | grep -i 'socks\|pygments'
Pygments           2.14.0

Let's install python3-socks and check the status again:

(test_venv) root@core ~# apt install -y python3-socks
[...]
(test_venv) root@core ~# apt list python3-pygments python3-socks
Listing... Done
python3-pygments/stable,now 2.14.0+dfsg-1 all [installed,automatic]
python3-socks/stable,now 1.7.1+dfsg-1 all [installed]

Ok so python3-socks is installed now. And it's instantly available in our venv:

(test_venv) root@core ~# pip list | grep -i 'socks\|pygments'
Pygments           2.14.0
PySocks            1.7.1

Woohoo! :) And can we still install and/or update packages in our venv with pip?:

(test_venv2) root@core ~# pip install --upgrade pygments
Requirement already satisfied: pygments in /usr/lib/python3/dist-packages (2.14.0)
Collecting pygments
  Downloading pygments-2.18.0-py3-none-any.whl (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 6.5 MB/s eta 0:00:00
Installing collected packages: pygments
  Attempting uninstall: pygments
    Found existing installation: Pygments 2.14.0
    Not uninstalling pygments at /usr/lib/python3/dist-packages, outside environment /root/test_venv2
    Can't uninstall 'Pygments'. No files were found to uninstall.
Successfully installed pygments-2.18.0

Yes! When using pip install in our venv there are some extra lines related to the system package, but otherwise it's the same as using a standalone venv. Let's double check the new version:

(test_venv) root@core ~# pip list | grep -i 'socks\|pygments'
Pygments           2.18.0
PySocks            1.7.1

So we've updated from the system version of Pygments to 2.18.0, but the system version still exists - and is still 2.14.0:

(test_venv) root@core ~# apt list python3-pygments              
Listing... Done
python3-pygments/stable,now 2.14.0+dfsg-1 all [installed,automatic]

So what happens if we remove the pip installed version?:

(test_venv) root@core ~# pip uninstall pygments
Found existing installation: Pygments 2.18.0
Uninstalling Pygments-2.18.0:
  Would remove:
    /root/test_venv2/bin/pygmentize
    /root/test_venv2/lib/python3.11/site-packages/pygments-2.18.0.dist-info/*
    /root/test_venv2/lib/python3.11/site-packages/pygments/*
Proceed (Y/n)? y
  Successfully uninstalled Pygments-2.18.0

This time there is no mention of the system package. Let's double check the system and the venv:

(test_venv) root@core ~# apt list python3-pygments
Listing... Done
python3-pygments/stable,now 2.14.0+dfsg-1 all [installed,automatic]
(test_venv) root@core ~# pip list | grep -i 'pygments'
Pygments           2.14.0

Yep, the system package is still there and the venv has reverted to using the system version.

The best of both worlds?

In the context of TurnKey (and other Linux OS) usage, '--system-site-packages' essentially gives you the behavior of '--break-system-packages' - without having to actually break system packages. For those that used the previous behavior, I'd argue that it is the best of both worlds. It allows developers to develop in a manner similar to how it was done previously, but cleaner. Where possible you can use system packages via apt, but you still have many of the added advantages of a virtual environment.

However it should be noted that there are still some pitfalls. Such as accidental optional dependency fulfillment by system packages - e.g. a `try: import foo except ImportError:`. Or a system package update changing behavior of your dependencies - generally that shouldn't be an issue with a "stable" Linux distro like TurnKey/Debian; but occasionally can occur. It's also a pain if at some point you need to move your project somewhere else (different OS, different python version, etc) and/or you decide to move to a more isolated venv. Unless you've manually kept track of what dependencies you are using, there is no easy way to work out which libraries your project actually requires. A `pip list` will show all the system python libraries as well - which libraries are actually dependencies of your project? So for completeness, let's list the pros and cons for use of '--system-site-packages':

Virtual Environment With System Packages

• Allows for more flexible and up-to-date package management.
• Allows the benefits of turnkey (and therefor Debian) security updates.
• Simple integration with the system.
• Great if you only need a few pip packages or different package versions.

• Adds potential complexity to the setup and maintenance process.
• Limited by the constraints imposed by Debian 12.
• Simple integration with the system - even when you don't want it.
• Painful if you want to migrate your project to another OS.

So in summary, for python developers targeting a wider and more python version agnostic audience, this isn't for you. In a situation like that, use a default isolated venv. But for people like me that used to survive with the old pip behavior (aka '--break-system-packages' now) - IMO this is the best option by far!

What do you think? Feel free to share your thoughts and feedback below.