Stripping base path off the unpacked source tree with bitbake SRC_URI file:// fetcher - yocto

The manual here says that there is a basepath option to SRC_URI that should "strip the specified directories from the source path when unpacking".
I'm trying to fetch the sources from a local directory, say /src/someproject.
For that purpose I configured my recipe as follows:
SRC_URI="file:///src/someproject;subdir=source;basepath=/src/someproject"
The intention was to have the sources taken from /src/someproject directory and put into build/tmp/work/target/someproject/1.0-r1/source/. Instead, I'm getting the sources under build/tmp/work/target/someproject/1.0-r1/source/src/someproject.
Is there a way to get rid of /src/someproject subdirectory inside source ?

The documentation you point to is for yocto 1.6, release April 2014. The basepath parameter appears to have been removed in later releases, with no fanfare and no apparent replacement.
Instead you can do something like:
SRCDIR = "/path/to/your/files"
SRC_URI = "file://${SRCDIR}/contents/;subdir=src"
S = "${WORKDIR}/src"
Then you can access your files under ${S}/${SRCDIR}.
If you find using the ${SRCDIR} part too cumbersome, you can hook onto the do_patch target, which need the prefunc mechanism and not _prepend if you want to use shell scripting, as do_patch is otherwise written in python:
relocate_source() {
mv ${S}/${SRCDIR}/* ${S}/
rmdir ${S}/${SRCDIR}
}
do_patch[prefunc] += "relocate_source"
This will reorganise your source before applying any patch you can add to SRC_URI.
Also note that a file:// URI does not get cached in ${DL_DIR}, so there is no name-conflict to handle (the way we would need to use downloadfilename= in an http:// URI).

Related

yocto-bitbake mirror keep branch

for my yocto project I'm mirroring to my repo to a private one, it's working fine for basic SRC_URI, but for url that have a branch defined it will erase it and bring the default one (master), making it fail since the branch referenced in the original uri was a tag.
DEBUG: Fetcher failure: Unable to find revision xxx_git_commit_hash_xxx in branch master even from upstream
In my .bb file:
SRC_URI = "git://github.com/google/protobuf.git;branch=3.11.x \
and in my local.conf (in the build folder) for the mirroring i have:
SOURCE_TOMYMIRROR="myUser#MyServerIdServer:/The/Path/to/directory"
PREMIRRORS_prepend ?= "\
git://.*/.* git://${SOURCE_TOMYMIRROR}/HOST/PATH;protocol=ssh\n \
"
(Yes i want to keep HOST and PATH as path to get the repo)
I can see in the logs that my repo is correctly fetch, but the "new" url is without branch and I can't find a way to keep the .bb one
I tried to understand and look at the documentation, but i don't see a way to keep a specific parameter (like the HOST and PATH variable)
So i tried to look at the init.py which define the rules, but i couldn't figure out if it's possible.
I think the right notation would be specifying the branch as well into the premirror URL.
SOURCE_MIRROR_URL = "git://${SOURCE_TOMYMIRROR};protocol=ssh;branch=3.11.x"
PREMIRRORS_prepend = " git://.*/.* ${SOURCE_MIRROR_URL} \
"
Yes it did work,i didn't need premirror_prepend, here's mine :
SOURCE_TOMYMIRROR="myUser#MyServerIdServer:/The/Path/to/directory"
SOURCE_MIRROR_URL="git://${SOURCE_TOMYMIRROR}/HOST/PATH;protocol=ssh"
INHERIT += "own-mirrors"
And if i want to add new url mirror i can use Mirror
MIRRORS += "\
git://.*/.* ${SOURCE_TOMYMIRROR_2}/HOST/PATH \n \
https?$://.*/.* ${SOURCE_TOMYMIRROR_2}/HOST/PATH \n \
"

How to copy whole directories containing subdirectories to /boot (i.e bootfs) in Yocto while inheriting core-image class?

I have a directory which again contains subdirectories, which are built has part of other recipe and moved to DEPLOY_DIR_IMAGE using deploy bb class. So now I want to copy it to main image boot partition.
If it was a single file then appending required filename to IMAGE_EFI_BOOT_FILES variable, then yocto copies it to /boot. But same doesn't work for directories containing subdirectories please provide style to include even the subdirectories. Thank you
PS: I have tried appending IMAGE_EFI_BOOT_FILES += "parent_dir/*" didnt work.
It is obvious that IMAGE_EFI_BOOT_FILES is acting like the well known IMAGE_BOOT_FILES and other variables that are responsible for having the files necessary to be shipped in the boot partition. And that needs files and not directories.
So, if you do not need to specify all the files by hand, but instead you want to pass the directory, I suggest you use a python method to collect the files for you and append them to the variable.
See the following example I developed and tested:
def get_files(d, dir):
import os
dir_path = dir
if not os.path.exists(os.path.dirname(dir)):
dir_path = d.getVar(dir)
return ' '.join(f for f in os.listdir(d.getVar(dir)) if os.path.isfile(f))
IMAGE_EFI_BOOT_FILES += "${#get_files(d, 'DEPLOY_DIR_IMAGE')}"
The method will test if the argument is a real path then it will directly check for files, if not it will assume that it is a bitbake variable and it will get its content, so if DEPLOY_DIR_IMAGE is, for example, /home/user/dir, passing DEPLOY_DIR_IMAGE or /home/usr/dir will give the same result.
IMPORTANT
It is obvious also that IMAGE_EFI_BOOT_FILES is used in a .conf file such as local.conf or a custom machine configuration file. So adding that python function in .conf file will not work. I suggest creating a class for it and inherit it globally in your .conf file:
meta-custom/classes/utils.bbclass
local.conf:
INHERIT += "utils"
IMAGE_EFI_BOOT_FILES += "${#get_files(d, 'DEPLOY_DIR_IMAGE')}"
Try this and let me know in the comments.
EDIT
I have just realized that bitbake already imports os within python expressions expansions, so you can do it in one line without any need for a separate python function:
PATH = "/path/to/directory/" or
PATH = "A variable containing the path"
IMAGE_EFI_BOOT_FILES += "${#' '.join('%s' % f for f in os.listdir('${PATH}') if os.path.isfile(f))}"
Note: I am looking for Yocto built-in which can achieve solution for above mentioned , would like to share other way to resolve the functionality for community's benefit.
Add following in bb file if you are using one or refer to talel-belhadjsalem answer to use utils.bbclass.
def add_directory_bootfs(d, dirname, bootfs_dir):
file_list = list()
boot_files_list = list()
deploy_dir_image = d.getVar('DEPLOY_DIR_IMAGE')
for (dirpath, dirnames, filenames) in os.walk(os.path.join(deploy_dir_image, dirname)):
file_list += [os.path.join(dirpath, file) for file in filenames]
for file in file_list:
file_rel_path = os.path.relpath(file, os.path.join(deploy_dir_image, dirname))
boot_file_entry = os.path.join(dirname, file_rel_path) + ';' + os.path.join(bootfs_dir, dirname, file_rel_path)
boot_files_list.append(boot_file_entry)
return ' '.join(boot_files_list)
IMAGE_EFI_BOOT_FILES += "${#add_directory_bootfs(d, 'relative_path_to_dir_in_deploy_dir_image', 'EFI/BOOT')}"

How can I get "HelloWorld - BitBake Style" working on a newer version of Yocto?

In the book "Embedded Linux Systems with the Yocto Project", Chapter 4 contains a sample called "HelloWorld - BitBake style". I encountered a bunch of problems trying to get the old example working against the "Sumo" release 2.5.
If you're like me, the first error you encountered following the book's instructions was that you copied across bitbake.conf and got:
ERROR: ParseError at /tmp/bbhello/conf/bitbake.conf:749: Could not include required file conf/abi_version.conf
And after copying over abi_version.conf as well, you kept finding more and more cross-connected files that needed to be moved, and then some relative-path errors after that... Is there a better way?
Here's a series of steps which can allow you to bitbake nano based on the book's instructions.
Unless otherwise specified, these samples and instructions are all based on the online copy of the book's code-samples. While convenient for copy-pasting, the online resource is not totally consistent with the printed copy, and contains at least one extra bug.
Initial workspace setup
This guide assumes that you're working with Yocto release 2.5 ("sumo"), installed into /tmp/poky, and that the build environment will go into /tmp/bbhello. If you don't the Poky tools+libraries already, the easiest way is to clone it with:
$ git clone -b sumo git://git.yoctoproject.org/poky.git /tmp/poky
Then you can initialize the workspace with:
$ source /tmp/poky/oe-init-build-env /tmp/bbhello/
If you start a new terminal window, you'll need to repeat the previous command which will get get your shell environment set up again, but it should not replace any of the files created inside the workspace from the first time.
Wiring up the defaults
The oe-init-build-env script should have just created these files for you:
bbhello/conf/local.conf
bbhello/conf/templateconf.cfg
bbhello/conf/bblayers.conf
Keep these, they supersede some of the book-instructions, meaning that you should not create or have the files:
bbhello/classes/base.bbclass
bbhello/conf/bitbake.conf
Similarly, do not overwrite bbhello/conf/bblayers.conf with the book's sample. Instead, edit it to add a single line pointing to your own meta-hello folder, ex:
BBLAYERS ?= " \
${TOPDIR}/meta-hello \
/tmp/poky/meta \
/tmp/poky/meta-poky \
/tmp/poky/meta-yocto-bsp \
"
Creating the layer and recipe
Go ahead and create the following files from the book-samples:
meta-hello/conf/layer.conf
meta-hello/recipes-editor/nano/nano.bb
We'll edit these files gradually as we hit errors.
Can't find recipe error
The error:
ERROR: BBFILE_PATTERN_hello not defined
It is caused by the book-website's bbhello/meta-hello/conf/layer.conf being internally inconsistent. It uses the collection-name "hello" but on the next two lines uses _test suffixes. Just change them to _hello to match:
# Set layer search pattern and priority
BBFILE_COLLECTIONS += "hello"
BBFILE_PATTERN_hello := "^${LAYERDIR}/"
BBFILE_PRIORITY_hello = "5"
Interestingly, this error is not present in the printed copy of the book.
No license error
The error:
ERROR: /tmp/bbhello/meta-hello/recipes-editor/nano/nano.bb: This recipe does not have the LICENSE field set (nano)
ERROR: Failed to parse recipe: /tmp/bbhello/meta-hello/recipes-editor/nano/nano.bb
Can be fixed by adding a license setting with one of the values that bitbake recognizes. In this case, add a line onto nano.bb of:
LICENSE="GPLv3"
Recipe parse error
ERROR: ExpansionError during parsing /tmp/bbhello/meta-hello/recipes-editor/nano/nano.bb
[...]
bb.data_smart.ExpansionError: Failure expanding variable PV_MAJOR, expression was ${#bb.data.getVar('PV',d,1).split('.')[0]} which triggered exception AttributeError: module 'bb.data' has no attribute 'getVar'
This is fixed by updating the special python commands being used in the recipe, because #bb.data was deprecated and is now removed. Instead, replace it with #d, ex:
PV_MAJOR = "${#d.getVar('PV',d,1).split('.')[0]}"
PV_MINOR = "${#d.getVar('PV',d,1).split('.')[1]}"
License checksum failure
ERROR: nano-2.2.6-r0 do_populate_lic: QA Issue: nano: Recipe file fetches files and does not have license file information (LIC_FILES_CHKSUM) [license-checksum]
This can be fixed by adding a directive to the recipe telling it what license-info-containing file to grab, and what checksum we expect it to have.
We can follow the way the recipe generates the SRC_URI, and modify it slightly to point at the COPYING file in the same web-directory. Add this line to nano.bb:
LIC_FILES_CHKSUM = "${SITE}/v${PV_MAJOR}.${PV_MINOR}/COPYING;md5=f27defe1e96c2e1ecd4e0c9be8967949"
The MD5 checksum in this case came from manually downloading and inspecting the matching file.
Done!
Now bitbake nano ought to work, and when it is complete you should see it built nano:
/tmp/bbhello $ find ./tmp/deploy/ -name "*nano*.rpm*"
./tmp/deploy/rpm/i586/nano-dbg-2.2.6-r0.i586.rpm
./tmp/deploy/rpm/i586/nano-dev-2.2.6-r0.i586.rpm
I have recently worked on that hands-on hello world project. As far as I am concerned, I think that the source code in the book contains some bugs. Below there is a list of suggested fixes:
Inheriting native class
In fact, when you build with bitbake that you got from poky, it builds only for the target, unless you mention in your recipe that you are building for the host machine (native). You can do the latter by adding this line at the end of your recipe:
inherit native
Adding license information
It is worth mentioning that the variable LICENSE is important to be set in any recipe, otherwise bitbake rises an error. In our case, we try to build the version 2.2.6 of the nano editor, its current license is GPLv3, hence it should be mentioned as follow:
LICENSE = "GPLv3"
Using os.system calls
As the book states, you cannot dereference metadata directly from a python function. Which means it is mandatory to access metadata through the d dictionary. Bellow, there is a suggestion for the do_unpack python function, you can use its concept to code the next tasks (do_configure, do_compile):
python do_unpack() {
workdir = d.getVar("WORKDIR", True)
dl_dir = d.getVar("DL_DIR", True)
p = d.getVar("P", True)
tarball_name = os.path.join(dl_dir, p+".tar.gz")
bb.plain("Unpacking tarball")
os.system("tar -x -C " + workdir + " -f " + tarball_name)
bb.plain("tarball unpacked successfully")
}
Launching the nano editor
After successfully building your nano editor package, you can find your nano executable in the following directory in case you are using Ubuntu (arch x86_64):
./tmp/work/x86_64-linux/nano/2.2.6-r0/src/nano
Should you have any comments or questions, Don't hesitate !

bitbake recipe - doing a simple copy of the image

I am attempting to write a recipe that would simple copy two files (MyfileA , MyfileB) to a specific directory when the overall image is built. This is what my directory structure looks like:
MyDir/MyRecipe.bb
MyDir/files/MyfileA
MyDir/files/MyfileB
I would like the two files to be copied to a folder in home (which would not exist initially hence the directories should be created)The folder lets say is called "Testfolder"
This is what my bitbake file looks like
DESCRIPTION = "Testing Bitbake file"
PR = "r0"
SRC_URI = "file://MyfileA \
file://MyfileB "
do_install() {
install -d MyfileA ~/TestFolder/
}
Kindly let me know if I am doing something wrong here?
When i run bitbake on this I get the following
The BBPATH variable is not set and bitbake did not find a conf/bblayers.conf file in the expected location.
Maybe you accidentally invoked bitbake from the wrong directory?
DEBUG: Removed the following variables from the environment: LANG, LS_COLORS, LESSCLOSE, XDG_RUNTIME_DIR, SHLVL, SSH_TTY, OLDPWD, LESSOPEN, SSH_CLIENT, MAIL, SSH_CONNECTION, XDG_SESSION_ID, _, BUILDDIR
Any help in this regard would be appreciated.
First of all, to create your own meta-layer, you should run command yocto-layer create MyRecipe in your Yocto Environment. This is to make sure that you have all the necessary element in your meta layer. Make sure to put the new meta-layer into conf/bblayers.conf
Creating HelloWorld Recipe Video can be found here
Second, to copy a file from one to another directories.
DESCRIPTION = "Testing Bitbake file"
SECTION = "TESTING"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
PR = "r0"
SRC_URI = "file://MyfileA \
file://MyfileB "
#specify where to get the files
S = "${WORKDIR}"
inherit allarch
#create the folder in target machine
#${D} is the directory of the target machine
#move the file from working directory to the target machine
do_install() {
install -d ${D}/TestFolder
install -m ${WORKDIR}/MyfileA ${D}/TestFolder
}
To get more in details, this is my understanding of how the files move around in Yocto.
You have a directory that stored metadata in /sourced/meta-mylayer/recipes-myRecipe/. In that directory, there would be a folder with the same name as the recipe. I.E. myRecipe/ myRecipe_001.bb.
You would store the files that are related to myRecipe.bb (usually it is a patch) in myRecipe/ so that SRC_URI will get into that myRecipe/ directory to grab files. I.E. myFileA, myFileB
Then, you specify the S. This is the location in the Build Directory where unpacked recipe source code resides. By that mean, myFileA and myFileB are moved/copied to there when myRecipe builds.
Usually, S is equal to ${WORKDIR}, this is equivalent to ${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}
The actual directory depends on several things:
TMPDIR: The top-level build output directory
MULTIMACH_TARGET_SYS: The target system identifier
PN: The recipe name
EXTENDPE: The epoch - (if PE is not specified, which is usually the case for most recipes, then EXTENDPE is blank)
PV: The recipe version
PR: The recipe revision
After that, we inherit allarch. This class is used for architecture independent recipes/data files (usually scripts).
Then, the last thing we have to do is copy the files.
${D} is the location in the Build Directory where components are installed by do_install task. This location defaults to ${WORKDIR}/image
${WORKDIR}/image can also be described as the / directory in the target system.
Go to ${D} directory and create a folder call TestFolder
Then, copy myFileA from ${WORKDIR} to the ${D}/TestFolder
P.S. Please add comment to fix. There might be mistaken information here, cause I learned all this by myself.

Automake, generated source files and VPATH builds

I'm doing VPATH builds with automake. I'm now also using generated source, with SWIG. I've got rules in Makefile.am like:
dist_noinst_DATA = whatever.swig
whatever.cpp: whatever.swig
swig -c++ -php $^
Then the file gets used later:
myprogram_SOURCES = ... whatever.cpp
It works fine when $builddir == $srcdir. But when doing VPATH builds (e.g. mkdir build; cd build; ../configure; make), I get error messages about missing whatever.cpp.
Should generated source files go to $builddir or $srcdir? (I reckon probably $builddir.)
How should dependencies and rules be specified to put generated files in the right place?
Simple answer
You should assume that $srcdir is a read-only, so you must not write anything there.
So, your generated source-code will end up in $(builddir).
By default, autotool-generated Makefiles will only look for source-files in $srcdir, so you have to tell it to check $builddir as well. Adding the following to your Makefile.am should help:
VPATH = $(srcdir) $(builddir)
After that you might end up with a no rule to make target ... error, which you should be able to fix by updating your source-generating rule as in:
$(builddir)/whatever.cpp: whatever.swig
# ...
A better solution
You might notice that in your current setup, the release tarball (as created by make dist) will contain the whatever.cpp file as part of your sources, since you added this file to the myprogram_SOURCES.
If you don't want this (e.g. because it might mean that the build-process will really take the pregenerated file rather than generating it again), you might want to use something like the following.
It uses a wrapper source-file (whatever_includer.cpp) that simply includes the generated file, and it uses -I$(builddir) to then find the generated file.
Makefile.am:
dist_noinst_DATA = whatever.swig
whatever.cpp: whatever.swig
swig -c++ -php $^
whatever_includer.cpp: whatever.cpp
myprogram_SOURCES = ... whatever_includer.cpp
myprogram_CPPFLAGS = ... -I$(builddir)
clean-local::
rm -f $(builddir)/whatever.cpp
whatever_includer.cpp:
#include "whatever.cpp"
Usually, you want to keep $srcdir readonly, so that if for instance the source is distributed unpacked on a CDROM, you can still run /.../configure from some other part of the file-system.
However if you are using SWIG to generate source code for a wrapper library, you probably want to distribute that SWIG-generated code as well so that your users do not need to install SWIG to compile your code. Then you have indeed a choice: you can decide that the SWIG-generated code should end in $builddir (it's OK: make dist will collect it there and include it in the tarball), or you could decide to output SWIG-generated code in $srcdir since it is really a source from the point of view of the distributed package. An advantage of keeping it in $srcdir is that when make distcheck attempts to build your package from a read-only source directory, it will fail on any attempt to call SWIG to regenerate the wrapper source. If you have your wrapper source in $builddir, you might not notice you have some broken rule that cause SWIG to be run on the user's host; by generating in $srcdir you ensure that SWIG is not needed by your users.
So my preference is to output SWIG wrapper sources in $srcdir. My setup for Python wrappers looks as follows:
EXTRA_DIST = spot.i
python_PYTHON = $(srcdir)/spot.py # _PYTHON is distributed by default
pyexec_LTLIBRARIES = _spot.la
MAINTAINERCLEANFILES = $(srcdir)/spot_wrap.cxx $(srcdir)/spot.py
_spot_la_SOURCES = $(srcdir)/spot_wrap.cxx $(srcdir)/spot_wrap.h
_spot_la_LDFLAGS = -avoid-version -module
_spot_la_LIBADD = $(top_builddir)/src/libspot.la
$(srcdir)/spot_wrap.cxx: $(srcdir)/spot.i
$(SWIG) -c++ -python -I$(srcdir) -I$(top_srcdir)/src $(srcdir)/spot.i
# Handle the multi-file output of SWIG.
$(srcdir)/spot.py: $(srcdir)/spot.i
$(MAKE) $(AM_MAKEFLAGS) spot_wrap.cxx
Note that I use $(srcdir) for all targets, because of limitations of the VPATH feature on various flavors of make. My setup to deal with the multiple files output by SWIG could be improved, but as these rules are not run by users and it has never caused me any problem, I do not bother.