How to make Python version executables global across multiple pyenv-virtualenv virtual environments - pyenv

Question

A pyenv Python version (e.g. 3.10.4) has the "normal" expected Python executables associated with it (e.g., pip, 2to3, pydoc)
$ ls "${PYENV_ROOT}/versions/3.10.4/bin"
2to3 idle idle3.10 pip3 pydoc pydoc3.10 python-config python3-config python3.10-config
2to3-3.10 idle3 pip pip3.10 pydoc3 python python3 python3.10 python3.10-gdb.py
and a pyenv-virtualenv virtual environment has only the executables that one would a get inside the virtual environment directory structure
$ pyenv virtualenv 3.10.4 venv
$ ls "${PYENV_ROOT}/versions/venv"
bin include lib lib64 pyvenv.cfg
$ ls "${PYENV_ROOT}/versions/venv/bin/"
Activate.ps1 activate activate.csh activate.fish pip pip3 pip3.10 pydoc python python3 python3.10
by default after creation, the venv virtual environment doesn't know about executables of the Python version that it is associated to, like 2to3
$ pyenv activate venv
(venv) $ 2to3 --help
pyenv: 2to3: command not found
The `2to3' command exists in these Python versions:
3.10.4
Note: See 'pyenv help global' for tips on allowing both
python2 and python3 to be found.
so to allow for a virtual environment like venv to have access to these executables you add both it and the Python that created it to pyenv global so that pyenv will "fall back" to the Python version when the executable isn't found
(venv) $ pyenv deactivate
$ pyenv global venv 3.10.4
(venv) $ pyenv global
venv
3.10.4
(venv) $ 2to3 --help | head -n 3
Usage: 2to3 [options] file|dir ...
Options:
This pattern works for one virtual environment, but how do you maintain access to executables like 2to3 (or pipx as seen below`) when you have multiple virtual environments in play?
(venv) $ pyenv virtualenv 3.10.4 example && pyenv activate example
(example) $ 2to3
pyenv: 2to3: command not found
The `2to3' command exists in these Python versions:
3.10.4
Note: See 'pyenv help global' for tips on allowing both
python2 and python3 to be found.
Reproducible example
Using the following Dockerfile
FROM debian:bullseye
SHELL ["/bin/bash", "-c"]
USER root
RUN apt-get update -y && \
apt-get install --no-install-recommends -y \
make \
build-essential \
libssl-dev \
zlib1g-dev \
libbz2-dev \
libreadline-dev \
libsqlite3-dev \
wget \
curl \
llvm \
libncurses5-dev \
xz-utils \
tk-dev \
libxml2-dev \
libxmlsec1-dev \
libffi-dev \
liblzma-dev \
g++ && \
apt-get install -y \
git && \
apt-get -y clean && \
apt-get -y autoremove && \
rm -rf /var/lib/apt/lists/*
# Install pyenv and pyenv-virtualenv
ENV PYENV_RELEASE_VERSION=2.3.0
RUN git clone --depth 1 https://github.com/pyenv/pyenv.git \
--branch "v${PYENV_RELEASE_VERSION}" \
--single-branch \
~/.pyenv && \
pushd ~/.pyenv && \
src/configure && \
make -C src && \
echo 'export PYENV_ROOT="${HOME}/.pyenv"' >> ~/.bashrc && \
echo 'export PATH="${PYENV_ROOT}/bin:${PATH}"' >> ~/.bashrc && \
echo 'eval "$(pyenv init -)"' >> ~/.bashrc && \
. ~/.bashrc && \
git clone --depth 1 https://github.com/pyenv/pyenv-virtualenv.git $(pyenv root)/plugins/pyenv-virtualenv && \
echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bashrc
# Install CPython
ENV PYTHON_VERSION=3.10.4
RUN . ~/.bashrc && \
echo "Install Python ${PYTHON_VERSION}" && \
PYTHON_MAKE_OPTS="-j8" pyenv install "${PYTHON_VERSION}"
# Make 'base' virtual envirionment, add it and its Python version to global for
# executables like 2to3 or pipx to be findable
# c.f. https://github.com/pyenv/pyenv-virtualenv/issues/16#issuecomment-37640961
# and then install pipx into the 'base' virtual environment and use pipx to install
# pepotron
RUN . ~/.bashrc && \
pyenv virtualenv "${PYTHON_VERSION}" base && \
echo "" && echo "Python ${PYTHON_VERSION} has additional executables..." && \
ls -lh "${PYENV_ROOT}/versions/${PYTHON_VERSION}/bin" && \
echo "" && echo "...compared to 'base' virtualenv made with Python ${PYTHON_VERSION}" && \
ls -lh "${PYENV_ROOT}/versions/base/bin" && \
echo "" && echo "...because 'base' is actually a symlink" && \
ls -lh "${PYENV_ROOT}/versions/" && \
pyenv global base "${PYTHON_VERSION}" && \
python -m pip --quiet install --upgrade pip setuptools wheel && \
python -m pip --quiet install pipx && \
python -m pipx ensurepath && \
eval "$(register-python-argcomplete pipx)" && \
pipx install pepotron
WORKDIR /home/data
built with
docker build . --file Dockerfile --tag pyenv/multiple-virtualenvs:debug
it can be run with the following to demonstrate the problem
$ docker run --rm -ti pyenv/multiple-virtualenvs:debug
(base) root#26dfa530cd82:/home/data# pyenv global
base
3.10.4
(base) root#26dfa530cd82:/home/data# 2to3 --help | head -n 3
Usage: 2to3 [options] file|dir ...
Options:
(base) root#26dfa530cd82:/home/data# pipx list
venvs are in /root/.local/pipx/venvs
apps are exposed on your $PATH at /root/.local/bin
package pepotron 0.6.0, installed using Python 3.10.4
- bpo
- pep
(base) root#26dfa530cd82:/home/data# pep 3.11
https://peps.python.org/pep-0664/
(base) root#26dfa530cd82:/home/data# pyenv virtualenv 3.10.4 example
(base) root#26dfa530cd82:/home/data# pyenv activate example
pyenv-virtualenv: prompt changing will be removed from future release. configure `export PYENV_VIRTUALENV_DISABLE_PROMPT=1' to simulate the behavior.
(example) root#26dfa530cd82:/home/data# 2to3
pyenv: 2to3: command not found
The `2to3' command exists in these Python versions:
3.10.4
Note: See 'pyenv help global' for tips on allowing both
python2 and python3 to be found.
(example) root#26dfa530cd82:/home/data# pipx
pyenv: pipx: command not found
The `pipx' command exists in these Python versions:
3.10.4/envs/base
base
Note: See 'pyenv help global' for tips on allowing both
python2 and python3 to be found.
So how can one have something like pipx, that is designed to be installed globally, work globally if it is installed in a pyenv-virtualenv virtual environment so that you don't have to have anything installed with pip in the system Python?
It would seem that instead of ever using pyenv activate to activate a virtual environment you would need to deactivate any virtual environments and then only use pyenv global <virtual environment name> <virtual environment Python version> to effectively switch environments. I assume that can't be the only way to use Python version executables inside of a virtual environment, as that seems like that would remove the point of having a separate pyenv activate CLI API for pyenv-virtualenv.

Answer 1

You can directly execute the pipx binary in the pyenv prefix, it should work correctly.
Pyenv's shims mechanism isn't really designed for global binaries like this. I naively expected that the global environment would act as fallback when a local environment doesn't have an installed binary, but I think pyenv only looks at the system Python before falling back to $PATH.
So, if you don't install pipx into system (which, if you're not installing a system pip, I doubt you're doing), then the naive fallback doesn't work.
An alternative is to run pyenv with a temporary environment, i.e.
PYENV_VERSION=my-pipx-env pyenv exec pipx.
I'd want to make this an executable, so I'd suggest adding something like this into a special PATH directory that takes precedence from the pyenv paths:
#!/usr/bin/env bash
set -eu
export PYENV_VERSION="pipx"
exec "${PYENV_ROOT}/libexec/pyenv" exec pipx "$#"
Although, I'd be tempted to forgo the whole activation logic and just directly exec the pipx binary from the environment /bin to avoid running into any shell configuration errors.