python virtualenv failure after upgrade fedora from 31 to 33 - virtualenv

Question

When I upgraded fedora from 31 to 33, I found out that the base python package had been upgraded from 3.7.9 to 3.9, and that python references in virtual environment folders were now pointing to the new version of python.
There were no problems activating my python 3.7 virtual environment
[bou#bous-fed33 avguide]$ source ~/py37/bin/activate
(py37) [bou#bous-fed33 avguide]$ which python
~/py37/bin/python
However the python version was no longer 3.7.9 but 3.9, which came with fedora 33
(py37) [bou#bous-fed33 avguide]$ python -V
Python 3.9.0
Now when I tried running jupyter notebook get errors ModuleNotFoundError
(py37) [bou#bous-fed33 avguide]$ jupyter notebook --port 7777
[W 09:14:02.710 NotebookApp] Error loading server extension jupyterlab
ModuleNotFoundError: No module named 'jupyterlab'
Also get errors for other packages like pandas, numpy etc which had all been fine before.
(py37) [bou#bous-fed33 avguide]$ python
Python 3.9.0 (default, Oct 6 2020, 00:00:00)
[GCC 10.2.1 20200826 (Red Hat 10.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as ps
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pandas'
I had tried to reinstall all the packages from existing requirements.txt file and manual reinstalls as well - all failed with errors. There was also no point in reinstalling python 3.7 as it was still there.
(py37) [bou#bous-fed33 avguide]$ pip3 install --force-reinstall -r requirements.txt
[bou#bous-fed33 avguide]$ sudo dnf install python37
Package python3.7-3.7.9-2.fc33.x86_64 is already installed.
I found a relatively simple FIX after a while.
The way python versions is managed is by using symbolic links in virtual env folders. So all we have to do is find the location of existing python binary for 3.7.9 or whatever python version your virtual environment uses and update the symbolic links to point to the correct python base packages.
These are the python versions installed in my fedora OS/base.
[bou#bous-fed33 ~]$ ls -ltr /usr/bin/python3*
-rwxr-xr-x. 2 root root 15536 Sep 22 19:23 /usr/bin/python3.7
-rwxr-xr-x. 1 root root 15536 Sep 25 23:37 /usr/bin/python3.8
lrwxrwxrwx. 1 root root 9 Oct 7 00:19 /usr/bin/python3 -> python3.9 <<<
-rwxr-xr-x. 1 root root 15536 Oct 7 00:20 /usr/bin/python3.9 <<<
Note how /usr/bin/python3 points to python3.9
Locate the symbolic links in virtual environment ~/py37/bin/ folder
[bou#bous-fed33 avguide]$ cd ~/py37/bin/
[bou#bous-fed33 bin]$ ls -ltr python*
lrwxrwxrwx. 1 bou bou 16 Dec 29 2019 python3 -> /usr/bin/python3
lrwxrwxrwx. 1 bou bou 7 Dec 29 2019 python -> python3
Note how python points to python3 and python3 in turn points to operating system package /usr/bin/python3 - which after the fedora python upgrade no longer points to /usr/bin/python3.7 but to the new version of python /usr/bin/python3.9
So all we need to do is remove existing softlinks
[bou#bous-fed33 bin]$ rm python3 python
And then create new files or symbolic links python3 and python that point to python3.7 binary in /usr/bin/python3.7
[bou#bous-fed33 bin]$ ln -s /usr/bin/python3.7 python3
[bou#bous-fed33 bin]$ ln -s python3 python
Activate virtual environment and check python version is correct.
[bou#bous-fed33 bin]$ ls -ltr python*
lrwxrwxrwx. 1 bou bou 18 Dec 2 10:03 python3 -> /usr/bin/python3.7
lrwxrwxrwx. 1 bou bou 7 Dec 2 10:04 python -> python3
[bou#bous-fed33 avguide]$ source ~/py37/bin/activate
(py37) [bou#bous-fed33 avguide]$ python -V
Python 3.7.9
(py37) [bou#bous-fed33 avguide]$ which python
~/py37/bin/python
(py37) [bou#bous-fed33 avguide]$ python
Python 3.7.9 (default, Sep 22 2020, 09:19:36)
[GCC 10.2.1 20200826 (Red Hat 10.2.1-3)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> quit()
We got the correct python version back for our virtual env and JupyterLab runs all right as well now.
(py37) [bou#bous-fed33 avguide]$ jupyter notebook --port 7777
[I 10:07:54.408 NotebookApp] JupyterLab extension loaded from /home/bou/py37/lib64/python3.7/site-packages/jupyterlab
[I 10:07:54.408 NotebookApp] JupyterLab application directory is /home/bou/py37/share/jupyter/lab
[I 10:07:54.410 NotebookApp] The Jupyter Notebook is running at:
[I 10:07:54.410 NotebookApp] https://bous-fed33:7777/
Hope this helps out someone running into similar problems with using their python virtual environment after base OS and/or python upgrade to a new version.

Answer 1

A better solution is to use pyenv, as it is specifically designed for this situation.
Pyenv will be available in your package repo... so yum search pyenv, then install.
See all available Python versions that pyenv can install for you:
$ pyenv install --list
Then pick form the list, eg. you'll see your missing Python 3.7.9. Install it to your local pyenv store, which will allow you to use it in your virtual environment:
$ pyenv install 3.7.9
You can then activate it in the directory containing your virtual env:
$ cd avguide
$ python --version
Python 3.9.0
$ pyenv local 3.7.9
$ python --version
Python 3.7.9
and it will automatically take care of you python binary links. See this tutorial and search for similar ones to get all the details.