How to test on py27 and py37 in tox when the py37 changes aren't packaged to pypi
The py3.7 compatible changes exist in repo branches.
They can be run by hand through pip -e installing them and running pytest without tox.
I'd like to move to running them through tox, but I can't figure out the correct string to give the deps list, or perhaps this is done in another way.
Attempted solution:
tox.ini
[tox]
envlist = py27,py37
[testenv:py27]
deps =
pytest
pytest-cov
pytest-mock
pylint
; packages specified by the setup.py cover the other dependencies for py2.7
commands =
pytest -v
[testenv:py37]
deps =
pytest
pytest-cov
pytest-mock
pylint
git+ssh//repo_url/location1.git#branchname_that_supports_py37
git+ssh//repo_url/location2.git#branchname_that_supports_py37
git+ssh//repo_url/location3.git#branchname_that_supports_py37
git+ssh//repo_url/location4.git#branchname_that_supports_py37
git+ssh//repo_url/location5.git#branchname_that_supports_py37
git+ssh//repo_url/location6.git#branchname_that_supports_py37
git+ssh//repo_url/location7.git#branchname_that_supports_py37
git+ssh//repo_url/location8.git#branchname_that_supports_py37
commands =
pytest -v
For VCS URLs pip needs to know the name of the package that should be provided with #egg=name:
git+ssh//repo_url/location1.git#branchname_that_supports_py37#egg=package1
Otherwise your tox.ini looks good. I use the same approach, for example.
Related
There are plenty of perl packages missing in Centos 8 and Rocky Linux. So, I try to get the rpm spec by cpanspec and build rpm by myself. But, it seems like that rpmbuild could not find the rpm I built.
This is the script for me to build rpm.
cd /root/rpmbuild
cpanspec --packer 'Example <example#example.com>' <Perl-Package-Name>
mkdir SOURCES
cp <Perl-Package-Name>.tar.gz SOURCES
rpmbuild -ba perl-<Package-Name>.spec
Let's say we have two package A and B. A is needed by B.
I try to build both of the packages through the script above. I build A first, switch into /root/rpmbuild/RPMS/noarch and install A.rpm. Then, I try to build package B.
I got
error: Failed build dependencies:
perl(A) is needed by perl-<B>
I try to check the existence of package A.
yum list installed | grep A
and
perldoc -l A
Both of the commands show that A exists.
Did I miss something?
update 2022/06/07
I just gave up and commented the BuildRequires: A in B package. This is not a good approach but it works.
Last time I've followed recommended src layout (https://hynek.me/articles/testing-packaging/) with using tox with great success.
However VSCODE tests discovery fails because src package cannot be imported. That is expected as we want to test installed package.
But how to debug my tests in vscode?
(Q author here: I've done research on that before posting the question, so sharing what I found)
Not solution
You could modify your PYTHONPATH to point to your src directory, but it breaks the main benefit from having separate src directory (read the link from OP).
Solution
Use pip install -e path/to/your/package (usually pip install -e .) to enable development mode and test versus your codebase as it would be installed.
After that your tests should be discovered properly. Otherwise it is different issue - read vs code OUTPUT console.
Note: this requires setup.py as a build backend
workaround for poetry
pyproject.toml
[build-system]
requires = [
"poetry-core>=1.0.0",
"setuptools" # to support local installations
]
then
poetry build --format sdist && tar --wildcards -xvf dist/*.tar.gz -O '*/setup.py' > setup.py
pip install -e .
Source: https://github.com/python-poetry/poetry/issues/34
TLDR: proper solution is outside of poetry scope, links to python-list discussions: https://github.com/python-poetry/poetry/issues/34#issuecomment-732478605
my setup.py is like below:
from setuptools import find_packages, setup
tests_require = ['pytest-env', 'pytest-mock', 'pytest-cov', 'pytest-xdist', 'pytest', 'mock', 'moto<=1.3.10']
setup(
name='repo_name',
version='0.1.0',
description='repo_name',
keywords=['?'],
packages=find_packages('src', exclude=['tests', 'venv']),
package_dir={'': 'src'},
package_data={'': ['schema/*.yaml']},
install_requires=[
'boto3<=1.10.19',
'requests<=2.22.0',
'jsonschema<=3.0.1',
'objectpath<=0.6.1',
'pyyaml<=5.1.1',
'sqlalchemy<=1.3.5',
'psycopg2-binary<=2.8.3',
'auth-client<=1.0.23', # internal package
'policy_client<=1.0.9', # internal package
'audit-client<=1.1.20', # internal package
'flask<=1.1.1',
'click<=7.0',
'Werkzeug<=0.15.5',
'itsdangerous<=1.1.0',
'Jinja2<=2.10.1',
'MarkupSafe<=1.1.1',
'structlog<=19.2.0',
'python-rapidjson<=0.9.1'
],
setup_requires=['pytest-runner'],
tests_require=tests_require,
extras_require={'test': tests_require},
include_package_data=True,
zip_safe=False
)
When I run python setup.py test, it will always reinstall the most of the packages that I already installed into .eggs folder, which I understand. While it's installing the package with different versions that I provided in the install_requires section, which results in below error:
pkg_resources.ContextualVersionConflict: (urllib3 1.25.10 (/Users/***/Desktop/repo/.eggs/urllib3-1.25.10-py3.7.egg), Requirement.parse('urllib3<1.25,>=1.21.1'), {'requests'})
I have to pin the requests version to 2.22.0 which requires urllib3 version no larger than 1.25, which caused my issue. So, is there anyway I can pin the urllib3 version in somewhere so when it runs, it will only install the provided version? I have tried to add the urllib3 version in install_requires, tests_require and in requirements.txt, but no luck so far.
Probably not what you want to hear but... setup_requires and tests_require are deprecated. Although they (probably) still work (mostly) fine, you might want to look for alternative solutions. As far as I know the most common alternative solutions are to use Tox instead of tests_require and PEP 517 to replace setup_requires.
In your case, it seems that your usage of setup_requires is only a consequence of your usage of tests_require. So by switching to something like Tox you get rid of both tests_require and setup_requires.
I got it resolve to simply replace python setup.py test to be pip instll -e . [test], and then pytest directly.
It will still install all the tests_require packages and then run pytest directly. Instead of going through all the list packages and find the best match versions for all packages.
UPDATES:
The real problem is I did not remove the old .eggs/ and venv/ folder when I made the packages version change. So the solution is updating the requests version to be 2.21.0 in the setup.py file, then remove the .eggs/ and venv/ folder and rerun everything.
ruamel.yaml seems to require PIP to install, which is not the default buildroot solution to build and install a Python package.
Is is possible to -at least- install a pure Python version of ruamel.yaml into a buildroot image - and how to circumvent the pip limitation?
Is is possible to cross-build ruamel.yaml?
Forcing RUAMEL_NO_PIP_INSTALL_CHECK env. var. does not help:
test compiling test_ruamel_yaml
running install
Checking .pth file support in ...
Failed to import the site module
ModuleNotFoundError: No module named '_sysconfigdata_m_linux_arm-linux-gnueabihf'
error: command '.../output/host/bin/python' failed with exit status 1
package/pkg-generic.mk:310: recipe for target '.../output/build/python-ruamel-yaml-0.15.45/.stamp_target_installed' failed
ruamel.yaml indeed requires pip to install from PyPI (using the .tar.gz or a .whl appropriate for your platform), this is documented.
The reason for this is that the fixes necessary to enable nested package installs where only implemented for pip (and not for easy_install or python setup.py installs).
That however does not preclude you from using ruamel.yaml, especially if you don't need the C extension (which is checked for at load time).
You can either check out a tagged version from bitbucket or untar a .tar.gz from PyPI and move the result to your site-packages directory:
$ virtualenv /tmp/ruamel_yaml_no_pip
Using base prefix '/opt/python/3.7'
New python executable in /tmp/ruamel_yaml_no_pip/bin/python
Installing setuptools, pip, wheel...done.
$ cd /tmp/ruamel_yaml_no_pip/
$ source bin/activate
(ruamel_yaml_no_pip) $ mkdir lib/python3.7/site-packages/ruamel/
(ruamel_yaml_no_pip) $ touch lib/python3.7/site-packages/ruamel/__init__.py
(ruamel_yaml_no_pip) $ wget -q https://files.pythonhosted.org/packages/63/a5/dba37230d6cf51f4cc19a486faf0f06871d9e87d25df0171b3225d20fc68/ruamel.yaml-0.15.45.tar.gz
(ruamel_yaml_no_pip) $ python -m ruamel.yaml
/tmp/ruamel_yaml_no_pip/bin/python: Error while finding module specification for 'ruamel.yaml' (ModuleNotFoundError: No module named 'ruamel')
(ruamel_yaml_no_pip) $ tar xf ruamel.yaml-0.15.45.tar.gz
(ruamel_yaml_no_pip) $ mv ruamel.yaml-0.15.45 lib/python3.7/site-packages/ruamel/yaml
(ruamel_yaml_no_pip) $ python -c 'from ruamel.yaml import YAML; print(YAML().load("{hello: world}")["hello"])'
world
(ruamel_yaml_no_pip)
(ruamel_yaml_no_pip) $ python -c 'from ruamel.yaml import __with_libyaml__ as X; print(X)'
False
(The URL is copied from the 0.15.45 project download page)
For development I normally just make a soft link from a virtualenv's site-packages to my ruamel directory.
I don't know how and if that translates to a buildroot environment (if so please publish your result).
I overlooked the buildroot documentation.
There is a critical parameter to define: SETUP_TYPE = setuptools rather than SETUP_TYPE = distutils.
With the following snippet:
PYTHON_RUAMEL_YAML_VERSION = 0.15.45
PYTHON_RUAMEL_YAML_SOURCE = ruamel.yaml-$(PYTHON_RUAMEL_YAML_VERSION).tar.gz
PYTHON_RUAMEL_YAML_SITE = https://pypi.python.org/packages/63/a5/dba37230d6cf51f4cc19a486faf0f06871d9e87d25df0171b3225d20fc68
PYTHON_RUAMEL_YAML_SETUP_TYPE = setuptools
PYTHON_RUAMEL_YAML_LICENSE = MIT
PYTHON_RUAMEL_YAML_LICENSE_FILES = LICENSE
PYTHON_RUAMEL_YAML_ENV += RUAMEL_NO_PIP_INSTALL_CHECK=1
$(eval $(python-package))
ruamel.yaml installs perfectly on the target image.
You can cd to YOUR_ENV/lib/pythonxx/site-packages/ and have a look, but is there any convenient ways?
pip freeze list all the packages installed including the system environment's.
You can list only packages in the virtualenv by
pip freeze --local
or
pip list --local.
This option works irrespective of whether you have global site packages visible in the virtualenv.
Note that restricting the virtualenv to not use global site packages isn't the answer to the problem, because the question is on how to separate the two lists, not how to constrain our workflow to fit limitations of tools.
Credits to #gvalkov's comment here. Cf. also pip issue 85.
Calling pip command inside a virtualenv should list the packages visible/available in the isolated environment. Make sure to use a recent version of virtualenv that uses option --no-site-packages by default. This way the purpose of using virtualenv is to create a python environment without access to packages installed in system python.
Next, make sure you use pip command provided inside the virtualenv (YOUR_ENV/bin/pip). Or just activate the virtualenv (source YOUR_ENV/bin/activate) as a convenient way to call the proper commands for python interpreter or pip
~/Projects$ virtualenv --version
1.9.1
~/Projects$ virtualenv -p /usr/bin/python2.7 demoenv2.7
Running virtualenv with interpreter /usr/bin/python2.7
New python executable in demoenv2.7/bin/python2.7
Also creating executable in demoenv2.7/bin/python
Installing setuptools............................done.
Installing pip...............done.
~/Projects$ cd demoenv2.7/
~/Projects/demoenv2.7$ bin/pip freeze
wsgiref==0.1.2
~/Projects/demoenv2.7$ bin/pip install commandlineapp
Downloading/unpacking commandlineapp
Downloading CommandLineApp-3.0.7.tar.gz (142kB): 142kB downloaded
Running setup.py egg_info for package commandlineapp
Installing collected packages: commandlineapp
Running setup.py install for commandlineapp
Successfully installed commandlineapp
Cleaning up...
~/Projects/demoenv2.7$ bin/pip freeze
CommandLineApp==3.0.7
wsgiref==0.1.2
What's strange in my answer is that package 'wsgiref' is visible inside the virtualenv. Its from my system python. Currently I do not know why, but maybe it is different on your system.
In Python3
pip list
Empty venv is
Package Version
---------- -------
pip 19.2.3
setuptools 41.2.0
To create a new environment
python3 -m venv your_foldername_here
Activate
cd your_foldername_here
source bin/activate
Deactivate
deactivate
You can also stand in the folder and give the virtual environment a name/folder (python3 -m venv name_of_venv).
Venv is a subset of virtualenv that is shipped with Python after 3.3.
list out the installed packages in the virtualenv
step 1:
workon envname
step 2:
pip freeze
it will display the all installed packages and installed packages and versions
If you're still a bit confused about virtualenv you might not pick up how to combine the great tips from the answers by Ioannis and Sascha. I.e. this is the basic command you need:
/YOUR_ENV/bin/pip freeze --local
That can be easily used elsewhere. E.g. here is a convenient and complete answer, suited for getting all the local packages installed in all the environments you set up via virtualenvwrapper:
cd ${WORKON_HOME:-~/.virtualenvs}
for dir in *; do [ -d $dir ] && $dir/bin/pip freeze --local > /tmp/$dir.fl; done
more /tmp/*.fl
why don't you try pip list
Remember I'm using pip version 19.1 on python version 3.7.3
If you are using pip 19.0.3 and python 3.7.4. Then go for pip list command in your virtualenv. It will show all the installed packages with respective versions.
.venv/bin/pip freeze worked for me in bash.
In my case the flask version was only visible under so I had to go to
C:\Users\\AppData\Local\flask\venv\Scripts>pip freeze --local
Using python3 executable only, from:
Gitbash:
winpty my_venv_dir/bin/python -m pip freeze
Linux:
my_venv_dir/bin/python -m pip freeze