Use Python library in Slurm job - python-3.7

Question

I want to run a job on Slurm and my Python script needs the evaluate package which I have on my local machine. I don't know if I could change the Python path on the server to match the one on my local machine, and if I could I'm afraid I might break the system.
So I followed this answer, and included a requirements.txt file with just evaluate==0.1.2 in it, and I get even more errors:
load GCC/10.2.0 (PATH, MANPATH, INFOPATH, LIBRARY_PATH, LD_LIBRARY_PATH, STD COMP VARS)
load ROCM/5.1.1 (PATH, MANPATH, LD_LIBRARY_PATH, LIBRARY_PATH, C_INCLUDE_PATH)
Set INTEL compilers as MPI wrappers backend
load mkl/2018.4 (LD_LIBRARY_PATH)
load PYTHON/3.7.4 (PATH, MANPATH, LD_LIBRARY_PATH, LIBRARY_PATH, PKG_CONFIG_PATH, C_INCLUDE_PATH, CPLUS_INCLUDE_PATH, PYTHONHOME, PYTHONPATH)
/var/spool/slurmd/job216863/slurm_script: line 12: virtualenv: command not found
/var/spool/slurmd/job216863/slurm_script: line 16: /env/bin/activate: No such file or directory
ERROR: Could not find a version that satisfies the requirement evaluate==0.1.2 (from versions: none)
ERROR: No matching distribution found for evaluate==0.1.2
Traceback (most recent call last):
File "eval_comet.py", line 1, in <module>
from evaluate import load
ModuleNotFoundError: No module named 'evaluate'

Answer 1

Most of the time, the Python version on HPCs are old. My Uni's HPC cluster has Python 3.7. If you wish to create a Python virtual environment (not conda) with a newer version, then there is a trick.
Activate the Anaconda Module, some system uses Module load and some uses load depending on your organisation.
[s.1915438#sl2 ~]$ module load anaconda/2021.05
[s.1915438#sl2 ~]$ conda create -n surrogate python=3.8
Here I created a Conda environment named surrogate with Python 3.8. Here, you can choose any version of your choice. Now you can activate the Conda environment and check the Python version.
[s.1915438#sl2 ~]$ source activate surrogate
(modulus) [s.1915438#sl2 ~]$ which python
~/.conda/envs/surrogate/bin/python
(surrogate) [s.1915438#sl2 ~]$ python --version
Python 3.8.13
Now navigate to the directory where you want to install your Python virtual environment and install the virtual environment using the following command.
(surrogate) [s.1915438#sl2 s.1915438]$ mkdir modulus_pysdf
(surrogate) [s.1915438#sl2 s.1915438]$ cd modulus_pysdf/
(surrogate) [s.1915438#sl2 modulus_pysdf]$ python3 -m venv modulus_pysdf
Logout (ctrl + D) from the server to exit the Conda environment and then login again. Remember, in my case the path to the Python virtual environment was /scratch/s.1915438/modulus_pysdf.
This is how I will activate the Python virtual environment.
[s.1915438#sl2 ~]$ cd /scratch/s.1915438
[s.1915438#sl2 s.1915438]$ cd modulus_pysdf/
[s.1915438#sl2 modulus_pysdf]$ source modulus_pysdf/bin/activate
Now I can check the Python version and the path.
(modulus_pysdf) [s.1915438#sl2 modulus_pysdf]$ python --version
Python 3.8.13
(modulus_pysdf) [s.1915438#sl2 modulus_pysdf]$ which python
/scratch/s.1915438/modulus_pysdf/modulus_pysdf/bin/python
As usual, I can install any package using pip. For example, to install evaluate you can download it from PyPI
pip install evaluate
Or if you have a requirements.txt file then you can do this. See this for more details.
cat requirements.txt | grep -Eo '(^[^#]+)' | xargs -n 1 pip install