My pacakge has *.py files and *.c files, the *.py files use ctypes to import shared library
built from the c source.
Now I have problem how to write my setup.py.
The setup script needs to build my_c_file.c to my_c_file.so, and then copy it to python libpath.
I want to know the what is the 'should' way?
You should probably have a look at Building C and C++ Extensions with distutils.
If you build a setup.py file around the example below, setuptools should compile your c file into my_c_lib.so and automatically add it to your installed package (untested).
from distutils.core import setup, Extension
c_module = Extension('my_c_lib',
sources = ['my_c_file.c'])
setup (name = 'my_package',
version = '1.0',
description = 'This is a package in which I compile a C library.',
ext_modules = [c_module])
Related
I start a new project library with cabal. here is the .cabal file
cabal-version: 2.4
library
exposed-modules: MyLib
build-depends: base ^>=4.14.3.0
, vector
hs-source-dirs: src
default-language: Haskell2010
I then use some Data.Vector functions inside my code. Here is the MyLib source
module MyLib (someFunc) where
import qualified Data.Vector as V
func :: [a] -> V.Vector a
func a = V.fromList a
But Data.Vector module remains underlined in red and this error is shown inside Vcode:
Could not load module ‘Data.Vector’
It is a member of the hidden package ‘vector-0.13.0.0’.
the func function works when in cabal repl
when I try cabal install vector this error appears:
Wrote tarball sdist to
/home/. .. .tar.gz
Resolving dependencies...
cabal: Cannot build the executables in the package vector because it does not
contain any executables. Check the .cabal file for the package and make sure
that it properly declares the components that you expect.
I don't understand what I am doing wrong? I told I understood the workings of cabal: I import a module as a dependency in .cabal, and I should be able to use it? why VSC does tell me he can't find it?
I'm putting together a recipe that's supposed to add amqtt to my image (https://github.com/Yakifo/amqtt). The project only comes with a pyproject.toml but lacks a setup.py. Thus, bitbake is complaining that setup.py cannot be found I'm on branch dunfell and these are the most relevant parts of my recipe:
HOMEPAGE = "https://github.com/Yakifo/amqtt"
SRC_URI = "git://github.com/Yakifo/amqtt;protocol=https"
SRCREV = "4beb912c2a0d58d66140ce68b6a31991c2c48b30"
S = "${WORKDIR}/git"
inherit setuptools3 pypi distutils
Your input is highly appreciated.
As of Yocto Release 4.0 (Kirkstone), installation of Python packages using Poetry is supported via the python_poetry_core class - Release 4.0 Migration Guide - Python Changes
My particular scenario (using Gatesgarth 3.2.3) doesn't really justify the effort in updating to Kirkstone. Just changing the packaging system is not an option (it's not my package I'm trying to install).
My workaround: I've opted to simply use Poetry (1.2.1, the latest at time of writing) to manually build the .tar.gz. It turns out that Poetry generates a setup.py with the info contained in pyproject.toml. I commit that package archive to my Yocto project repo, and override SRC_URI in the appropriate recipe to use it instead of the version from PyPi.
Note that I did have some issues in getting poetry build to run due to TOML format difference between the version that it was written for (which I was unable to zero in on) and the latest. I had to modify the pyproject.toml a bit because of this.
If the pyproject.toml file is composed in a way that it can be used with setuptools then just creating simple setup.py
from setuptools import setup
setup()
by adding the following to the bitbake recipe fixes the problem:
do_configure:prepend() {
cat > ${S}/setup.py <<-EOF
from setuptools import setup
setup()
EOF
}
More details here
My full bitbake recipe python3-leapseconddata_2.0.0.bb:
HOMEPAGE = "https://pypi.org/project/leapseconddata/"
SUMMARY = "Python Leap Second List."
LICENSE = "GPLv3"
LIC_FILES_CHKSUM = "file://LICENSES/GPL-3.0-only.txt;md5=8da5784ab1c72e63ac74971f88658166"
PYPI_PACKAGE = "leapseconddata"
SRC_URI[sha256sum] = "c72d40f56bf7a1a98ee0c0c12ea2fca76a38a5cb7239f8e182f13e639436992d"
inherit pypi setuptools3
do_configure:prepend() {
cat > ${S}/setup.py <<-EOF
from setuptools import setup
setup()
EOF
}
I switched to the pypi installer in the end. The most important parts are:
PYPI_PACKAGE = "amqtt"
inherit pypi setuptools3
I am running Vscode with the following components:
Version: 1.51.1 (user setup)
Commit: e5a624b788d92b8d34d1392e4c4d9789406efe8f
Date: 2020-11-10T23:34:32.027Z
Electron: 9.3.3
Chrome: 83.0.4103.122
Node.js: 12.14.1
V8: 8.3.110.13-electron.0
OS: Windows_NT x64 10.0.20270
Pylance 2.6
I have the following directory structure:
src
m1.py
.vscode
settings.json
lib
m2.py
.vscode
settings.json
I use several linters with this environment when developing Python code. Mypy does not have a problem but pylance is unable to resolve imports.
I am trying to import the module m2.py from m1.py when pylance fails. My settings.json file under the src directory is:
{
"python.autoComplete.extraPaths": [
"*.lib"
]
}
Can anyone see how to resolve this problem?
Pylance uses python.analysis.extraPaths as opposed to python.autoComplete.extraPaths.
{
"python.analysis.extraPaths": [
"*.lib"
]
}
Have you tried that?
If your VSCode workspace folder is the parent of the src folder it is normal to have Pylance complain because by default the root of your project is your workspace folder. You can see that if I import src.lib.m2 Pylance doesn't complain but it does if I use lib.m2:
Since you don't have a runtime error when running your code I would say you are inside the src folder when you run m1.py.
If my assumptions are not true, you'll need to add more details (code sample, how do you run the m1.py file)
Set the sub-folders up as proper Python Packages
This method provides conformance with standard Python project packaging guidelines
I recommend a setup that makes the subfolders all proper python packages. To do that, add a blank __init__.py file to each sub-folder with Python modules (i.e. files) in it.
With your original setup, ignoring the .vscode folders:
src/
__init__.py
m1.py
lib/
__init__.py
m2.py
In this case, the imports would need to be from the src folder (it would be considered a package itself, since it has a __init__.py file in it):
import src.m1
import src.lib.m2
Make a proper scripts packages
However, I recommend putting your scripts into their own package, not directly in the src folder:
src/
scripts/
__init__.py
m1.py
lib/
__init__.py
m2.py
This allows all your packages to be referenced with a proper package name rather than src like import scripts.m1 and import lib.m2.
Side Notes
If you want these packages to be "sub-packages", you can keep an __init__.py in the src folder to make it the root folder for everything.
With that change, imports would be import src.scripts.m1 and import src.lib.m2.
Python will go up in the parent folders until it finds a folder without an __init__.py file and then start the import statements in a chain from any sub-folders that are packages (i.e. have an __init__.py file).
Any folders chained together as packages after this process can be accessed locally without being added to the System or Python path.
How to import modules from the packages
Under this scheme, the m1.py script should be able to import the m2.py with something like the following. Since src is not a package, it is the root from Python's perspective, and not included in import statements.
# In scripts.m1
import lib.m2 as m2
m2.function_1()
a = m2.function_2(m2.symbol_1)
or
from lib.m2 import function_1, function_2, symbol_1
function_1()
a = function_2(symbol_1)
If you add test files in this setup (say within a tests directory inside of scripts), then you can import the script functions as import scripts.m1 as m1 or from script.m1 import *.
This setup makes the package conform to the standard for python packages and so if you want to make it installable or upload it to PyPi (or otherwise distribute it privately with zip files or through a git repo), you can define and build the project using the setuptools package using a standard setup.py file. See Packaging Python Projects
Your file structure seems to be the problem, why PyLance can't resolve the imports.
The best way out:
create a python virtual env and activate it.
Linux
python -m venv env
source env/bin/activate
Windows powershell
py -3.6 -m venv env
.\env\Scripts\Activate
Final Step
Having activated your virtual environment,
Just hit ctrl+shift+p
search for "python" and hit "restart language server"
That should resolve all imports, thanks to the virtual environemnt.
Pip supports the pyproject.toml file but so far all practical usage of the new schema requires a 3rd party tool that auto-generates these files (e.g., poetry and pip). Unlike setup.py which is already human-writeable, pyproject.toml is not (yet).
From setuptools docs,
[build-system]
requires = [
"setuptools >= 40.9.0",
"wheel",
]
build-backend = "setuptools.build_meta"
However, this file does not include package dependencies (as outlined in PEP 621). Pip does support installing packages using pyproject.toml but nowhere does pep specify how to write package dependencies in pyproject.toml for the official build system setuptools.
How do I write package dependencies in pyproject.toml?
Related StackOverflow Questions:
How to init the pyproject.toml file
This question asks for a method to auto-generate pyproject.toml, my question differ because I ask for a human-written pyproject.toml.
my question differ because I ask for a human-written pyproject.toml
First, the pyproject.toml file is always "human-writable".
Then, it is important to know that in this context setuptools and Poetry take the role of what are called "PEP 517 build back-ends", and there are many such back-ends available today, setuptools and Poetry are just two examples of them.
As of today, it seems like most (if not all) of the build back-ends I know of expect their configuration (including dependencies) to be written in pyproject.toml.
PEP 621
There is a standard called PEP 621 that specifies how a project's metadata, including dependencies, should be laid out in the pyproject.toml file.
Here is a list of build back-ends I know of that have support for PEP 621:
enscons
flit_core (see flit)
hatchling (see hatch)
pdm-pep517 (see pdm)
setuptools (experimental support since version 61.0.0)
trampolim
whey
For all PEP 621 compatible build back-ends, the dependencies should be written in pyproject.toml file like in the following:
[project]
name = "Thing"
version = "1.2.3"
# ...
dependencies = [
"SomeLibrary ~= 2.2",
]
References:
https://peps.python.org/pep-0621/
https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html (experimental support for PEP 621 has been added to setuptools version 61.0.0, released 2022-03-24)
setuptools (before version 61.0.0)
In setuptools before version 61.0.0 there is no support for writing the configuration in pyproject.toml (in other words: no PEP 621 support). You have to either write a setup.cfg, or a setup.py, or a combination of both.
My recommendation is to write as much as possible in setup.cfg. Such a setup.cfg could look like this:
[metadata]
name = Thing
version = 1.2.3
[options]
install_requires =
SomeLibrary ~= 2.2
packages = find:
and in most cases the setup.py can be omitted completely or it can be as short as:
import setuptools
setuptools.setup()
References about the dependencies specifically:
https://setuptools.readthedocs.io/en/latest/userguide/dependency_management.html
https://www.python.org/dev/peps/pep-0508/
https://www.python.org/dev/peps/pep-0440/
Again, note that in most cases it is possible to omit the setup.py file entirely, one of the conditions is that the setup.cfg file and a pyproject.toml file are present and contain all the necessary information. Here is an example of pyproject.toml that works well for a setuptools build backend:
[build-system]
build-backend = 'setuptools.build_meta'
requires = [
'setuptools >= 43.0.0',
]
poetry
In poetry everything is defined in pyproject.toml, but it uses poetry-specific sections. In other words, Poetry does not currently use the PEP 621 standard, but there are some plans to move to this standard in the future.
This file can be hand-written. As far as I can tell, there is no strict need to ever explicitly install poetry itself (commands such as pip install and pip wheel can get you far enough).
The pyproject.toml file can be as simple as:
[tool.poetry]
name = 'Thing'
version = '1.2.3'
[tool.poetry.dependencies]
python = '^3.6'
SomeLibrary = '~2.2'
[build-system]
requires = ['poetry-core~=1.0']
build-backend = 'poetry.core.masonry.api'
References:
https://python-poetry.org/docs/pyproject/
https://python-poetry.org/docs/dependency-specification/
I have a mixed Python/C++ library with test files mixed in amongst source files in the same directories. The layout looks like
/home/irving/geode
geode
__init__.py
vector
__init__.py
test_vector.py
...
...
Unfortunately, the library is unusable in-place since it lacks .so extension modules. Question: Can I make py.test always use an installed version, even when run from /home/irving/geode or a subdirectory?
The test files have from __future__ import absolute_import, and run fine if executed directly as scripts. For example, if I do
cd geode/vector
./test_vector.py
which does import geode, it finds the installed version. However, if I run py.test in geode/vector, it finds the local copy of geode, and then dies.
I think you have two options:
run py.test --pyargs geode.vector.test_vector to make pytest interpretet the argument as an import path, deriving the file system path from it. This should run the test against the installed version.
move the tests out into a tests directory without an __init__.py file. This way you need to pip install -e . to work in-place or can do python setup.py install and the py.test tests to run tests against the installed version.