I'm putting together a recipe that's supposed to add amqtt to my image (https://github.com/Yakifo/amqtt). The project only comes with a pyproject.toml but lacks a setup.py. Thus, bitbake is complaining that setup.py cannot be found I'm on branch dunfell and these are the most relevant parts of my recipe:
HOMEPAGE = "https://github.com/Yakifo/amqtt"
SRC_URI = "git://github.com/Yakifo/amqtt;protocol=https"
SRCREV = "4beb912c2a0d58d66140ce68b6a31991c2c48b30"
S = "${WORKDIR}/git"
inherit setuptools3 pypi distutils
Your input is highly appreciated.
As of Yocto Release 4.0 (Kirkstone), installation of Python packages using Poetry is supported via the python_poetry_core class - Release 4.0 Migration Guide - Python Changes
My particular scenario (using Gatesgarth 3.2.3) doesn't really justify the effort in updating to Kirkstone. Just changing the packaging system is not an option (it's not my package I'm trying to install).
My workaround: I've opted to simply use Poetry (1.2.1, the latest at time of writing) to manually build the .tar.gz. It turns out that Poetry generates a setup.py with the info contained in pyproject.toml. I commit that package archive to my Yocto project repo, and override SRC_URI in the appropriate recipe to use it instead of the version from PyPi.
Note that I did have some issues in getting poetry build to run due to TOML format difference between the version that it was written for (which I was unable to zero in on) and the latest. I had to modify the pyproject.toml a bit because of this.
If the pyproject.toml file is composed in a way that it can be used with setuptools then just creating simple setup.py
from setuptools import setup
setup()
by adding the following to the bitbake recipe fixes the problem:
do_configure:prepend() {
cat > ${S}/setup.py <<-EOF
from setuptools import setup
setup()
EOF
}
More details here
My full bitbake recipe python3-leapseconddata_2.0.0.bb:
HOMEPAGE = "https://pypi.org/project/leapseconddata/"
SUMMARY = "Python Leap Second List."
LICENSE = "GPLv3"
LIC_FILES_CHKSUM = "file://LICENSES/GPL-3.0-only.txt;md5=8da5784ab1c72e63ac74971f88658166"
PYPI_PACKAGE = "leapseconddata"
SRC_URI[sha256sum] = "c72d40f56bf7a1a98ee0c0c12ea2fca76a38a5cb7239f8e182f13e639436992d"
inherit pypi setuptools3
do_configure:prepend() {
cat > ${S}/setup.py <<-EOF
from setuptools import setup
setup()
EOF
}
I switched to the pypi installer in the end. The most important parts are:
PYPI_PACKAGE = "amqtt"
inherit pypi setuptools3
Related
I have an external application that converted to an exe using pyinstaller. This needs to be added in yocto framework. But the application has a dependency on pylint & pyinstaller. pylint is already part of yocto framework. I used meson.build file in my application and have a recipe file in yocto that will source the entire application from git repo and builds using instructions in meson.build. In my application meson build file uses find_program() to find pylint program installation. It works currently on my laptop, but when I build my application recipe in yocto environment, it is unable to find pylint. Even though I added "python3-pylint" in IMAGE_INSTALL_append.
| Program python3 found: YES (.../usr/bin/python3-native/python3)
| Program python3 (pylint) found: NO
|
| ../git/src/meson.build:10:0: ERROR: python3 is missing modules: pylint
pyinst = find_program('pylint') // as per meson this is probably searching in /usr/local/bin. But this is not the case in yocto. How can I modify find_program() call to find this in the right location in yocto framework. Or is there any method to handle this?
meson.build under source file folder:
# get & run pylint to check for errors
prog = import('python').find_installation('python3', modules: ['pylint'])
if not prog.found()
message('pylint not found')
else
message(prog.path())
cmd = find_program('pylint', prog.path())
message('Running pylint on src files')
foreach each : src_files
run_command(cmd, '--confidence=HIGH', each)
endforeach
endif
project's meson.build
project('proj_name',
['cpp'],
version : '0.1',
license : 'MIT',
default_options: [
'cpp_std=c++11']
)
project_pretty_name = 'proj_name'
project_url = '<git repo of project>'
python = import('python')
python3 = python.find_installation('python3', required: false)
subdir('src')
subdir('tools') # cpp tools
subdir('build')
yocto recipe file:
inherit meson pkgconfig python3native
DEPENDS += "${PYTHON_PN}-distro-native"
DEPENDS += "zlib"
RDEPENDS:${PN} += "${PYTHON_PN}-requests \
${PYTHON_PN}-pylint"
SRC_URI += "<git branch>"
SRCREV = "<src-rev>"
S = "${WORKDIR}/git"
FILES:${PN}:append = " ${bindir}/mesonexe"
Your Meson build requires pylint to be available during compilation, therefore you need to add:
DEPENDS += "${PYTHON_PN}-pylint-native"
and then you will need to ensure the pylint recipe is available in one of your configured Yocto layers. It's in meta-openembedded/meta-python.
Finally, the python3-pylint recipe and one of its dependencies (python3-astroid) do not have native support enabled, so you'll need to create the followin .bbappends as well:
# python3-pylint_%.bbappend
BBCLASSEXTEND += "native"
# python3-astroid_%.bbappend
BBCLASSEXTEND += "native"
You should then see the following line in the Meson output:
Program pylint found: YES (.../recipe-sysroot-native/usr/bin/pylint)
As a suggestion - if this recipe is building your own source code, I'd suggest linting the code as it is pushed into the repository (via a CI job), rather than at compile-time. This will reduce the dependencies required to build your project.
Of course, there would still be an optional Meson target for developers to lint their code before they push new code. Perhaps something that is enabled for non-release builds by default.
I new to Yocto so there are probably some mistakes and misunderstanding that I've had, I appreciate if you can help.
So, I want to add a new package (tar file) to my custom image.
I have followed steps and steps in manual and some online instructions. While running: "bitbake mylayer", the layer is built fine but I got this error while building the image, here is the log file:
DEBUG: Executing python function rootfs_deb_bad_recommendations
DEBUG: Python function rootfs_deb_bad_recommendations finished
DEBUG: Executing python function extend_recipe_sysroot
NOTE: Installed into sysroot: []
NOTE: Skipping as already exists in sysroot: ['depmodwrapper-cross', 'apt-native', 'dpkg-native', 'pseudo-native', 'update-rc.d-native', 'prelink-native', 'makedevs-native', 'ldconfig-native', 'opkg-util$
DEBUG: Python function extend_recipe_sysroot finished
DEBUG: Executing python function do_rootfs
NOTE: ###### Generate rootfs #######
NOTE: Installing the following packages: apt busybox copy-uefiimg-to-sda coreutils dpkg e2fsprogs-resize2fs libfontconfig1 libfreetype6 libglib-2.0-0 gptfdisk libjemalloc2 kernel-module-axi-dma-sensor ku$
ERROR: Unable to install packages.
Reading package lists...
Building dependency tree...
Reading state information...
Package mypackage is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
E: Package 'mylayer' has no installation candidate
DEBUG: Python function do_rootfs finished
ERROR: Function failed: do_rootfs
And here is mylayer.bb:
SUMMARY = ""
LICENSE = "CLOSE"
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI += "file://mypackage.tar"
Also, I have included the package in conf/local.conf:
IMAGE_INSTALL_append += " mylayer"
So beside trying to figure out how to solve this problem, I also have some questions:
I have read some example of .bb, and they mentioned about LIC_FILES_CHKSUM. The mypackage.tar.gz is a package to install a platform for the device and I don't know much about the source code, so I don't know if it is necessary to include the license? Or how to know that this package need license to install?
In some answer I found online, there is one saying that I need to include PACKAGE_CLASSES ?= "package_deb" (they want to install the .deb file), so probably in my case I will need PACKAGE_CLASSES ?= "package.tar" right? I have tried to change variable, but it still not successful.
The mypackage.tar includes some deb files. If I could not install mypackage.tar, can I instead install these .deb files? Can I put it all in mylayer.bb?
Thank you in advanced, I have tried to study much documents as I could but I get so confused and there is huge amount of information to digest.
First, before answering your questions
Let me mention some best practices advice for you:
Rename the recipe to some significant name related to you compressed package.
Naming the recipe to mylayer confuses Yocto users, because there is the term layer also.
Regarding you recipe:
There is no need for FILESEXTRAPATHS because the recipe path is added automatically to Yocto paths.
FILESEXTRAPATHS it is required for .bbappend files.
You need to override the do_install task function, it does nothing by default.
do_install is the first essential task to make sure that your sources are included in the final image.
Beside that, when specifying a compressed source file into SRC_URI, yocto automatically decompresses it.
This is mentioned here.
So, here what your recipe should look like:
SUMMARY = ""
LICENSE = "CLOSED"
# Prevent Yocto from decompressing the file
SRC_URI = "file://mypackage.tar;unpack=0"
do_install(){
# Create the opt folder into the final image, ${D} is ${WORKDIR}/image
install -d ${D}/opt
# Copy the compressed file there; You can change permissions as you want
install -m 0755 ${WORKDIR}/mypackage.tar ${D}/opt
}
# Very important to specify what you installed in (do_install)
FILES_${PN} = "/opt/*"
Now, when you run IMAGE_INSTALL_append += " mylayer" your file will be installed.
Regarding your questions:
You mentioned that your compressed file contains .deb files, I assume that no license checksum is needed. Also, I understand that you may wanted to point to SRC_URI[md5sum] or other checksums for the full package. That is also not needed for local files, it is used to check for the integrity of online sources.
PACKAGE_CLASSES as mentioned here, is used by the system to know in what type the data should be packaged. By the data I mean the data that you installed with do_install. That data get packaged for according to your PACKAGE_CLASSES variable, for example, to deb file. And that is used, along side with all other recipes packages, to build the final rootfs.
Yes, if you are installing the tar file into the image and then unpack it to install all deb files, for example, with dpkg. You can use the bin_package class to do that, now the recipe must be changed for that reason:
Decompress the tar file and provide the deb files in the local files folder.
Add all deb files to SRC_URI
Inherit the bin_package class
Specify the files to be packaged.
Your recipe should look like this:
SUMMARY = ""
LICENSE = "CLOSED"
SRC_URI = "file://deb_file1.deb \
file://deb_file2.deb"
# No need to `do_install` , it is invoked by the (bin_package) class
FILES_${PN} = ""
Important:
About FILES_${PN}, you need to add all what the deb installed into the image folder
Example, if your deb file installs this:
/usr/bin/hello
/etc/hello.cfg
Specify them:
FILES_${PN} = "/usr/bin/*"
FILES_${PN} += "/etc/*"
Use * so if other deb files install files into the same folder as others it will include all.
Pip supports the pyproject.toml file but so far all practical usage of the new schema requires a 3rd party tool that auto-generates these files (e.g., poetry and pip). Unlike setup.py which is already human-writeable, pyproject.toml is not (yet).
From setuptools docs,
[build-system]
requires = [
"setuptools >= 40.9.0",
"wheel",
]
build-backend = "setuptools.build_meta"
However, this file does not include package dependencies (as outlined in PEP 621). Pip does support installing packages using pyproject.toml but nowhere does pep specify how to write package dependencies in pyproject.toml for the official build system setuptools.
How do I write package dependencies in pyproject.toml?
Related StackOverflow Questions:
How to init the pyproject.toml file
This question asks for a method to auto-generate pyproject.toml, my question differ because I ask for a human-written pyproject.toml.
my question differ because I ask for a human-written pyproject.toml
First, the pyproject.toml file is always "human-writable".
Then, it is important to know that in this context setuptools and Poetry take the role of what are called "PEP 517 build back-ends", and there are many such back-ends available today, setuptools and Poetry are just two examples of them.
As of today, it seems like most (if not all) of the build back-ends I know of expect their configuration (including dependencies) to be written in pyproject.toml.
PEP 621
There is a standard called PEP 621 that specifies how a project's metadata, including dependencies, should be laid out in the pyproject.toml file.
Here is a list of build back-ends I know of that have support for PEP 621:
enscons
flit_core (see flit)
hatchling (see hatch)
pdm-pep517 (see pdm)
setuptools (experimental support since version 61.0.0)
trampolim
whey
For all PEP 621 compatible build back-ends, the dependencies should be written in pyproject.toml file like in the following:
[project]
name = "Thing"
version = "1.2.3"
# ...
dependencies = [
"SomeLibrary ~= 2.2",
]
References:
https://peps.python.org/pep-0621/
https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html (experimental support for PEP 621 has been added to setuptools version 61.0.0, released 2022-03-24)
setuptools (before version 61.0.0)
In setuptools before version 61.0.0 there is no support for writing the configuration in pyproject.toml (in other words: no PEP 621 support). You have to either write a setup.cfg, or a setup.py, or a combination of both.
My recommendation is to write as much as possible in setup.cfg. Such a setup.cfg could look like this:
[metadata]
name = Thing
version = 1.2.3
[options]
install_requires =
SomeLibrary ~= 2.2
packages = find:
and in most cases the setup.py can be omitted completely or it can be as short as:
import setuptools
setuptools.setup()
References about the dependencies specifically:
https://setuptools.readthedocs.io/en/latest/userguide/dependency_management.html
https://www.python.org/dev/peps/pep-0508/
https://www.python.org/dev/peps/pep-0440/
Again, note that in most cases it is possible to omit the setup.py file entirely, one of the conditions is that the setup.cfg file and a pyproject.toml file are present and contain all the necessary information. Here is an example of pyproject.toml that works well for a setuptools build backend:
[build-system]
build-backend = 'setuptools.build_meta'
requires = [
'setuptools >= 43.0.0',
]
poetry
In poetry everything is defined in pyproject.toml, but it uses poetry-specific sections. In other words, Poetry does not currently use the PEP 621 standard, but there are some plans to move to this standard in the future.
This file can be hand-written. As far as I can tell, there is no strict need to ever explicitly install poetry itself (commands such as pip install and pip wheel can get you far enough).
The pyproject.toml file can be as simple as:
[tool.poetry]
name = 'Thing'
version = '1.2.3'
[tool.poetry.dependencies]
python = '^3.6'
SomeLibrary = '~2.2'
[build-system]
requires = ['poetry-core~=1.0']
build-backend = 'poetry.core.masonry.api'
References:
https://python-poetry.org/docs/pyproject/
https://python-poetry.org/docs/dependency-specification/
I am trying to install a set of CMake utilty functions under /usr/share/cmake-3.4/Modules/MYMODULES/useful.cmake within a Yocto build.
Here is my current (sanitized) recipe (call it my-useful-modules.bb)
SECTION = "devel"
LICENSE = "CLOSED"
inherit cmake
EXTERNALSRC := "path/to/source/code"
do_compile() {
:
}
FILES_${PN} += "${datadir}/cmake-3.4/Modules/MYMODULES/*"
BBCLASSEXTEND = "nativesdk"
The configure and install tasks work fine, and if I look under image in tmp/work/..., I see the full tree (including all my host directories as expected).
But I keep getting the following error
ERROR: nativesdk-my-useful-modules-1.0-r0 do_package: QA Issue:
nativesdk-my-useful-modules: Files/directories were installed but not
shipped in any package:
Followed by a long list of files which basically includes everything under image.
These modules need to be available both in the native sysroot during the build, and in the standard SDK built with populate_sdk.
Which package should I be specifying with FILES_${PN} to get them packaged?
I'd also appreciate knowing how to either avoid specifying the cmake version in the FILES statement, or get it from the build system.
Updating FILES_${PN} will resolve your error and also there is no need to provide cmake version in this case.
FILES_${PN} += "${datadir}/*"
I have been using Yocto to create Linux builds for an ARM board.
I had been cross compiling add on applications manually. Now we are in a place where we would like a nice integrated build so I started adding custom recipes to yocto.
I have been struggling with the ARM build (a x86 build with the same code seems fine).
Even a basic 'hello world' pretty much cut and paste from the development manual does not work (http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#new-recipe-writing-a-new-recipe)
Here is the recipe:
SUMMARY = "Simple helloworld application"
SECTION = "examples"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
RPROVIDES_${PN} = "helloworld"
FILES_${PN} += "${bindir}"
SRC_URI = "file://helloworld.c"
S = "${WORKDIR}"
do_compile() {
${CC} helloworld.c -o helloworld ${LDFLAGS}
}
do_install() {
install -d ${D}${bindir}
install -m 0755 helloworld ${D}${bindir}
}
Here is the error:
ERROR: helloworld not found in the base feeds (smarc_samx6i cortexa9t2hf-vfp-neon-mx6qdl cortexa9hf-vfp-neon-mx6qdl cortexa9t2hf-vfp-neon cortexa9t2hf-vfp cortexa9hf-vfp-neon cortexa9hf-vfp armv7at2hf-vfp-neon armv7ahf-vfp-neon armv7at2hf-vfp armv7ahf-vfp armv6thf-vfp armv6hf-vfp armv5tehf-vfp armv5ehf-vfp armv5thf-vfp armv5hf-vfp noarch any all).
ERROR: Function failed: do_rootfs
Any suggestions as to what would be causing this error?
The package does build properly; the problem seems to be isolated to finding it for the rootfs.
Thanks!
EDIT:
I have a solution that seems to work, although it is not ideal long term.
Changing the package name under IMAGE_INSTALL from helloworld to helloworld-0.0.1 resolves the issue. Obviously I would rather not hard code the version of each package in the top level recipe and other packages do not require this, so hopefully there is another solution.
EDIT 2:
Renaming the recipe and removing the version string also resolves the issue. Once again, this does not seem ideal long term.
OK, after some further testing I discovered this was a naming issue with the recipe.
It was named helloworld-0.0.1.bb (the same format with the other recipes I had put together driving me to try this simple test).
If anyone else encounters this simply replacing the '-' with a '_' resolves this.
1.Rename your recipe name e.g hello-0.1.bb to hello_0.1.bb
2.Add below line at last only:
FILES_${PN} = "${bindir}/*"
Abvoe line helps you copy your binary to rootfs.