Why doesn't my recipe re-build successfully when I'm offline? - yocto

I have a recipe that looks basically like this :
SUMMARY = "SomeLibrary"
LICENSE = "Apache-2.0"
LIC_FILES_CHKSUM = "file://LICENSE;md5=3b83ef96387f14655fc854ddc3c6bd57"
SRC_URI += "git://gitlab.com/some_library/some-library.git;protocol=https;nobranch=1"
SRCREV = "${PV}"
S = "${WORKDIR}/git"
inherit autotools pkgconfig
It builds successfully with bitbake some-library, and I can see there is a git2/gitlab.com.some_library.some-library.git/ directory and a git2/gitlab.com.some_library.some-library.git.done file in my downloads folder (the one DL_DIR point to).
My understanding is that if I then immediately run bitbake -c cleansstate some-library && bitbake some-library, given that there is no change in the recipe, bitbake shouldn't need to download anything (it already has everything it needs). In practice, if I turn off my network connection or add BB_NO_NETWORK="1" to my local.conf, I get the following error :
Initialising tasks: 100% |################################################################| Time: 0:00:01
Sstate summary: Wanted 12 Found 4 Missed 8 Current 251 (33% match, 96% complete)
NOTE: Executing Tasks
ERROR: some-library-v2.3.0-r0 do_fetch: Bitbake Fetcher Error: NetworkAccess('https://gitlab.com/some_library/some-library.git', 'git -c core.fsyncobjectfiles=0 ls-remote "https://gitlab.com/some_library/some-library.git" ')
ERROR: Logfile of failure stored in: /home/myusername/work/builddir/tmp/work/aarch64-poky-linux/some-library/v2.3.0-r0/temp/log.do_fetch.116252
ERROR: Task (/home/myusername/work/builddir/../../layers/meta-mymeta/recipes-core/some-library/some-library_v2.3.0.bb:do_fetch) failed with exit code '1'
NOTE: Tasks Summary: Attempted 806 tasks of which 804 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
/home/myusername/work/builddir/../../layers/meta-mymeta/recipes-core/some-library/some-library_v2.3.0.bb:do_fetch
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
Why is that ? How do other recipes avoid this pitfall ? (when I build my image, this recipe seems to be the only one trying to fetch things from the network, which suggests to me that I'm doing something wrong here)
EDIT :
What really puzzles me is that bitbakes seems to behave differently with recipes other than my own. For example, the recipe for can-utils located at meta-openembedded/meta-oe/recipes-extended/socketcan/can-utils_git.bb looks like this:
SUMMARY = "Linux CAN network development utilities"
LICENSE = "GPLv2 & BSD-3-Clause"
LIC_FILES_CHKSUM = "file://include/linux/can.h;endline=44;md5=a9e1169c6c9a114a61329e99f86fdd31"
DEPENDS = "libsocketcan"
SRC_URI = "git://github.com/linux-can/${BPN}.git;protocol=https;branch=master"
SRCREV = "da65fdfe0d1986625ee00af0b56ae17ec132e700"
PV = "2020.02.04"
S = "${WORKDIR}/git"
inherit autotools pkgconfig
which is very similar, but when I set BB_NO_NETWORK="1" in my local.conf and run bitbake -c cleansstate can-utils && bitbake can-utils I get Tasks Summary: Attempted 842 tasks of which 822 didn't need to be rerun and all succeeded.

This works for me:
After configuring the project, add the following lines to build/conf/site.conf file:
# Build offline
SOURCE_MIRROR_URL ?= "file:///path/to/oe-downloads"
INHERIT += " own-mirrors"
BB_GENERATE_MIRROR_TARBALLS = "1"
BB_NO_NETWORK = "1"
After that, it might be necessary to build project once when online.
After every re-configuration (different build options) the site.conf is overwritten, so I created a script to add these lines after re-configuration.

I believe I found the issue.
If I replace ${PV} (which was equal to v2.3.0 here) by the hash associated to that tag, then the issue stops happening.
If I interpret this correctly, it means that bitbake is able to tell if SRCREV is a hash or a tag, and that if it is a tag then do_fetch will always run git ls-remote to make sure that the tag has not been moved.

Related

No recipes available (meta-wandboard-ap)

I have already built an image for wandboard using fsl-community-bsp-platform manifest and dunfell branch. I use bitbake core-image-minimal to make the build.
The idea is that I want to turn Wandboard into a wireless access point using meta-wandboard-ap. I followed all the instructions and add the layer to BBLAYERS in build/conf/bblayers.conf.
When I attempt to run the bitbake command to create the image, I get the following error message:
sami#sami-deepevent:~/fsl-community-bsp-dunfell/wandboard$ bitbake core-image-minimal
WARNING: Layer wandboard-ap should set LAYERSERIES_COMPAT_wandboard-ap in its conf/layer.conf file to list the core layer names it is compatible with.
WARNING: Layer wandboard-ap should set LAYERSERIES_COMPAT_wandboard-ap in its conf/layer.conf file to list the core layer names it is compatible with.
Loading cache: 100% |##########################################################| Time: 0:00:00
Loaded 3269 entries from dependency cache.
Parsing recipes: 100% |########################################################| Time: 0:00:00
Parsing of 2177 .bb files complete (2176 cached, 1 parsed). 3269 targets, 267 skipped, 0 masked, 0 errors.
ERROR: No recipes available for:
/home/sami/fsl-community-bsp-dunfell/sources/meta-wandboard-ap/recipes-kernel/linux-firmware/linux-firmware_git.bbappend
Summary: There were 2 WARNING messages shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
That's the content of the linux-firmware_git.bbappend file:
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI_append = " \
file://fw_bcm4329_apsta.bin \
file://fw_bcm4330_apsta_bg.bin \
"
do_install_append () {
install -m 644 ${WORKDIR}/fw_bcm4329_apsta.bin ${D}/lib/firmware/brcm/brcmfmac4329-ap-sdio.bin
install -m 644 ${WORKDIR}/fw_bcm4330_apsta_bg.bin ${D}/lib/firmware/brcm/brcmfmac4330-ap-sdio.bin
}
FILES_${PN}-bcm4329 += " \
/lib/firmware/brcm/brcmfmac4329-ap-sdio.bin \
"
FILES_${PN}-bcm4330 += " \
/lib/firmware/brcm/brcmfmac4330-ap-sdio.bin \
"
ALTERNATIVE_TARGET_linux-firmware-bcm4329[brcmfmac4329-sdio.bin] = "${BRCM_FWDIR}/brcmfmac4329-ap-sdio.bin"
ALTERNATIVE_PRIORITY_linux-firmware-bcm4329[brcmfmac4329-sdio.bin] = "90"
ALTERNATIVE_TARGET_linux-firmware-bcm4330[brcmfmac4330-sdio.bin] = "${BRCM_FWDIR}/brcmfmac4330-ap-sdio.bin"
ALTERNATIVE_PRIORITY_linux-firmware-bcm4329[brcmfmac4330-sdio.bin] = "90"
What could be the reason for this error? How to make it find the files and recipes?
In Yocto, when creating a .bbappend file, the original .bb file must exist in one of the layers in bblayers.conf.
The problem is that Yocto parsing stage will try to append the content of linux-firmware_git.bbappend to linux-firmware_git.bb.
That failed, which means that you have no linux-firmware_git.bb recipe.
But the linux-firmware recipe exists in meta layer of poky. You can check in this link.
The recipe exists in dunfell under version: linux-firmware_20211027.bb
So, just change linux-firmware_git.bbappend to:
linux_firmware_%.bbappend
with that, it will match the highest version available for linux-firmware.
Also, make sure meta is present in your bblayers.conf.

Yocto rust recipe also produces -native output that needs packaging

I tried this approach on hardknott but I couldn't get it to work recipe also produces -native output that needs packaging
It is a rust recipe that generates an x86_64 app which I would like to package the right way in sdk, so that it can be used.
I can separate the main package to -native-bin, and I see it in the recipe-sysroot, but I can't get it to populate the recipe-sysroot of the workdir of the file when building the -native-helper recipe. And I suspect the reason is that I get an error that the main recipe for x86_64 can't be found?
ERROR: Manifest xxxxxx.populate_sysroot not found in vs_imx8mp cortexa53 armv8a-crc armv8a aarch64 allarch x86_64_x86_64-nativesdk (variant '')?
So any helpful information would be appreciated!
Hacked like this:
Recipe.bb:
do_install_append() {
# Set permision without run flag so that it doesn't fail on checks
chmod 644 ${D}/usr/bin/#RECIPE#-compiler
}
# #RECIPE# generates a compiler during the target generation step
#separate this to the -native-bin package, and skip the ARCH checks
#also in the image file for stations_sdk move the app to right dir and add execute flag
PACKAGES_prepend = "${PN}-native-bin "
PROVIDES_prepend = "${PN}-native-bin "
INSANE_SKIP_${PN}-native-bin = "arch"
FILES_${PN}-native-bin = "/usr/bin/#RECIPE#-compiler"
SYSROOT_DIRS += "/"
Image.bb:
# #RECIPE# produces a compiler that is produced as a part of the target generation
#so we use the recipe and hack it to supply the -compiler as part of the
#host binaries
TOOLCHAIN_TARGET_TASK_append = " #RECIPE#-native-bin"
do_fix_#RECIPE#() {
mv ${SDK_OUTPUT}/${SDKTARGETSYSROOT}/usr/bin/#RECIPE#-compiler ${SDK_OUTPUT}/${SDKPATHNATIVE}/usr/bin/#RECIPE#-compiler
chmod 755 ${SDK_OUTPUT}/${SDKPATHNATIVE}/usr/bin/#RECIPE#-compiler
}
SDK_POSTPROCESS_COMMAND_prepend = "do_fix_#RECIPE#; "
This produces at the end the binary in the right directory

How bitbake searches for recipe in build process?

I am trying to find out that how bitbake search for recipe in build process ?
For example,
I have a recipe something like below:
DESCRIPTION = "PetaLinux GSTREAMER supported packages"
inherit packagegroup
GSTREAMER_PACKAGES = " \
gstreamer1.0 \
gstreamer1.0-python \
gstreamer1.0-meta-base \
gstreamer1.0-plugins-base \
gstreamer1.0-plugins-good \
gstreamer1.0-plugins-bad \
gstreamer1.0-rtsp-server \
gst-shark \
gstd \
gst-perf \
gst-interpipes \
"
GSTREAMER_PACKAGES_append_zynqmp = " gstreamer1.0-omx"
RDEPENDS_${PN} = "${GSTREAMER_PACKAGES}"
When I searched gstreamer1.0 related recipe in yocto layers, I found two recipe, one of them is gstreamer1.0_1.16.1.bb in meta layer, and the other is gstreamer1.0_%.bbappend in meta-petalinux layer.
Both of these layers was added to the BBLAYERS in bblayers.conf file and the priorities that spesified with BBFILE_PRIORITY_* in related layer's layer.conf file is same.
So,
Which recipe will be used in build process in that case ?
What is the recipe lookup rules in yocto ?
I changed somethings to understand the behaviour:
For example,
I entered the invalid github URL that spesified in gstreamer1.0_%.bbappend recipe. When I tried to build the linux system, I encountered with an error. Thats fine.
Then I corrected the github URL in this recipe and entered invalid source code address that spesified in gstreamer1.0_1.16.1.bb recipe. When I tried to build linux system, process finished successfully.
Then I increased the priority of meta layer. I supposed to encounter with an error in this case but again build process finished successfully.
Could you please help me to understand this behaviour ?
Thanks.
You have two different files: a .bb and a .bbappend.
A .bb is the base recipe of one (or multiple) packages. It generally describe how to fetch, configure, compile, install files in a package for your target.
A .bbappend file is an 'append' file. It allows a meta (here meta-petalinux) to modify an existing recipe in another meta without copying it. A .bbappend can modify any steps of the bb file: source fetch, configure, compile, install...
You can for example create your own bbappend of Gstreamer, to enable pango (disbaled by default on my Yocto). The bbappend filename is gstreamer1.0-plugins-base_%.bbappend and only contains PACKAGECONFIG_append = "pango"
The Yocto Manual can give you more information on bbappend files here.

How can I get "HelloWorld - BitBake Style" working on a newer version of Yocto?

In the book "Embedded Linux Systems with the Yocto Project", Chapter 4 contains a sample called "HelloWorld - BitBake style". I encountered a bunch of problems trying to get the old example working against the "Sumo" release 2.5.
If you're like me, the first error you encountered following the book's instructions was that you copied across bitbake.conf and got:
ERROR: ParseError at /tmp/bbhello/conf/bitbake.conf:749: Could not include required file conf/abi_version.conf
And after copying over abi_version.conf as well, you kept finding more and more cross-connected files that needed to be moved, and then some relative-path errors after that... Is there a better way?
Here's a series of steps which can allow you to bitbake nano based on the book's instructions.
Unless otherwise specified, these samples and instructions are all based on the online copy of the book's code-samples. While convenient for copy-pasting, the online resource is not totally consistent with the printed copy, and contains at least one extra bug.
Initial workspace setup
This guide assumes that you're working with Yocto release 2.5 ("sumo"), installed into /tmp/poky, and that the build environment will go into /tmp/bbhello. If you don't the Poky tools+libraries already, the easiest way is to clone it with:
$ git clone -b sumo git://git.yoctoproject.org/poky.git /tmp/poky
Then you can initialize the workspace with:
$ source /tmp/poky/oe-init-build-env /tmp/bbhello/
If you start a new terminal window, you'll need to repeat the previous command which will get get your shell environment set up again, but it should not replace any of the files created inside the workspace from the first time.
Wiring up the defaults
The oe-init-build-env script should have just created these files for you:
bbhello/conf/local.conf
bbhello/conf/templateconf.cfg
bbhello/conf/bblayers.conf
Keep these, they supersede some of the book-instructions, meaning that you should not create or have the files:
bbhello/classes/base.bbclass
bbhello/conf/bitbake.conf
Similarly, do not overwrite bbhello/conf/bblayers.conf with the book's sample. Instead, edit it to add a single line pointing to your own meta-hello folder, ex:
BBLAYERS ?= " \
${TOPDIR}/meta-hello \
/tmp/poky/meta \
/tmp/poky/meta-poky \
/tmp/poky/meta-yocto-bsp \
"
Creating the layer and recipe
Go ahead and create the following files from the book-samples:
meta-hello/conf/layer.conf
meta-hello/recipes-editor/nano/nano.bb
We'll edit these files gradually as we hit errors.
Can't find recipe error
The error:
ERROR: BBFILE_PATTERN_hello not defined
It is caused by the book-website's bbhello/meta-hello/conf/layer.conf being internally inconsistent. It uses the collection-name "hello" but on the next two lines uses _test suffixes. Just change them to _hello to match:
# Set layer search pattern and priority
BBFILE_COLLECTIONS += "hello"
BBFILE_PATTERN_hello := "^${LAYERDIR}/"
BBFILE_PRIORITY_hello = "5"
Interestingly, this error is not present in the printed copy of the book.
No license error
The error:
ERROR: /tmp/bbhello/meta-hello/recipes-editor/nano/nano.bb: This recipe does not have the LICENSE field set (nano)
ERROR: Failed to parse recipe: /tmp/bbhello/meta-hello/recipes-editor/nano/nano.bb
Can be fixed by adding a license setting with one of the values that bitbake recognizes. In this case, add a line onto nano.bb of:
LICENSE="GPLv3"
Recipe parse error
ERROR: ExpansionError during parsing /tmp/bbhello/meta-hello/recipes-editor/nano/nano.bb
[...]
bb.data_smart.ExpansionError: Failure expanding variable PV_MAJOR, expression was ${#bb.data.getVar('PV',d,1).split('.')[0]} which triggered exception AttributeError: module 'bb.data' has no attribute 'getVar'
This is fixed by updating the special python commands being used in the recipe, because #bb.data was deprecated and is now removed. Instead, replace it with #d, ex:
PV_MAJOR = "${#d.getVar('PV',d,1).split('.')[0]}"
PV_MINOR = "${#d.getVar('PV',d,1).split('.')[1]}"
License checksum failure
ERROR: nano-2.2.6-r0 do_populate_lic: QA Issue: nano: Recipe file fetches files and does not have license file information (LIC_FILES_CHKSUM) [license-checksum]
This can be fixed by adding a directive to the recipe telling it what license-info-containing file to grab, and what checksum we expect it to have.
We can follow the way the recipe generates the SRC_URI, and modify it slightly to point at the COPYING file in the same web-directory. Add this line to nano.bb:
LIC_FILES_CHKSUM = "${SITE}/v${PV_MAJOR}.${PV_MINOR}/COPYING;md5=f27defe1e96c2e1ecd4e0c9be8967949"
The MD5 checksum in this case came from manually downloading and inspecting the matching file.
Done!
Now bitbake nano ought to work, and when it is complete you should see it built nano:
/tmp/bbhello $ find ./tmp/deploy/ -name "*nano*.rpm*"
./tmp/deploy/rpm/i586/nano-dbg-2.2.6-r0.i586.rpm
./tmp/deploy/rpm/i586/nano-dev-2.2.6-r0.i586.rpm
I have recently worked on that hands-on hello world project. As far as I am concerned, I think that the source code in the book contains some bugs. Below there is a list of suggested fixes:
Inheriting native class
In fact, when you build with bitbake that you got from poky, it builds only for the target, unless you mention in your recipe that you are building for the host machine (native). You can do the latter by adding this line at the end of your recipe:
inherit native
Adding license information
It is worth mentioning that the variable LICENSE is important to be set in any recipe, otherwise bitbake rises an error. In our case, we try to build the version 2.2.6 of the nano editor, its current license is GPLv3, hence it should be mentioned as follow:
LICENSE = "GPLv3"
Using os.system calls
As the book states, you cannot dereference metadata directly from a python function. Which means it is mandatory to access metadata through the d dictionary. Bellow, there is a suggestion for the do_unpack python function, you can use its concept to code the next tasks (do_configure, do_compile):
python do_unpack() {
workdir = d.getVar("WORKDIR", True)
dl_dir = d.getVar("DL_DIR", True)
p = d.getVar("P", True)
tarball_name = os.path.join(dl_dir, p+".tar.gz")
bb.plain("Unpacking tarball")
os.system("tar -x -C " + workdir + " -f " + tarball_name)
bb.plain("tarball unpacked successfully")
}
Launching the nano editor
After successfully building your nano editor package, you can find your nano executable in the following directory in case you are using Ubuntu (arch x86_64):
./tmp/work/x86_64-linux/nano/2.2.6-r0/src/nano
Should you have any comments or questions, Don't hesitate !

do_rootfs: Can't install packagegroup-core-x11-utils-1.0-r40#all: no package provides xserver-nodm-init

At first I run the command to see my layers:
$bitbake-layers show-layers
layer path priority
meta .../poky-krogoth-15.0.0/meta 5
meta-poky .../poky-krogoth-15.0.0/meta-poky 5
meta-yocto-bsp .../poky-krogoth-15.0.0/meta-yocto-bsp 5
meta-oe .../meta-openembedded/meta-oe 6
and build my image:
bitbake core-image-sato
everything is OK,I found the meta-oe layer is useless. So I delete it:
bitbake-layers remove-layer .../meta-openembedded/meta-oe
and build it again:
$bitbake core-image-sato
it show error:
ERROR: core-image-sato-1.0-r0 do_rootfs: Unable to install packages.
Command
'/buildarea/ddu/poky-krogoth-15.0.0/build-beaglebone/tmp/sysroots/x86_64-linux/usr/bin/smart
--log-level=warning --data-dir=/buildarea/ddu/poky-krogoth-15.0.0/build-beaglebone/tmp/work/beaglebone-poky-linux-gnueabi/core-image-sato/1.0-r0/rootfs/var/lib/smart
install -y packagegroup-core-ssh-dropbear#all
packagegroup-core-x11-sato-games#beaglebone
packagegroup-core-x11-base#all smartpm#cortexa8hf_neon
psplash#cortexa8hf_neon packagegroup-core-boot#beaglebone
packagegroup-base-extended#beaglebone rpm#cortexa8hf_neon
packagegroup-core-x11-sato#beaglebone
locale-base-en-us#cortexa8hf_neon locale-base-en-gb#cortexa8hf_neon'
returned 1:
Loading cache...
Updating cache...
################################## [100%]
Computing transaction...error: Can't install
packagegroup-core-x11-utils-1.0-r40#all: no package provides
xserver-nodm-init
ERROR: core-image-sato-1.0-r0 do_rootfs: Function failed: do_rootfs
ERROR: Logfile of failure stored in:
/buildarea/ddu/poky-krogoth-15.0.0/build-beaglebone/tmp/work/beaglebone-poky-linux-gnueabi/core-image-sato/1.0-r0/temp/log.do_rootfs.52174
ERROR: Task 9
(/buildarea/ddu/poky-krogoth-15.0.0/meta/recipes-sato/images/core-image-sato.bb,
do_rootfs) failed with exit code '1'
Why it said that no package provides xserver-nodm-init?
$find ../meta -name xserver-nodm-init*
../meta/recipes-graphics/x11-common/xserver-nodm-init.bb
../meta/recipes-graphics/x11-common/xserver-nodm-init
Because of the way meta-oe currently replaces some of the X11 startup components, you pretty much need to delete TMPDIR if you've build X11 with meta-oe in your bblayers.conf and then decide you want to remove meta-oe (as you have done). However, if you now just delete TMPDIR but not your sstate-cache and then try again then it'll restore everything back to where you were from sstate pretty quickly without needing to recompile everything, and the issue should be resolved.
What meta-oe is doing isn't desirable, there is a bug open to fix it:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5546
It's assigned to me and I'm sad to say I haven't had time to resolve it so far; it doesn't appear to be a priority for anyone else either.
First of all. Why do you think meta-oe is useless?
And why do you want to delete it? It does no harm?
The general rule is. If it does not break, Do not fix it.
Like you said. Put everything back and it will work.