I'm attempting to replace a file from another layer with a .bbappend file. My goal is to overwrite a specific configuration file with a customized one during the unpack stage.
In my .bbappend I'm attempting to append the do_unpack to copy a file from the same directory as the .bbappend file into the working directory ${WORKDIR} The problem is: When inside do_unpack_append, ${THISDIR} is returning the directory of the original .bb recipe, rather than the directory of .bbappend
Here's an example:
The original recipe resides in: meta-origLayer/recipe.bb
My *.bbappend resides in: meta-newLayer/recipe.bbappend
recipe.bbappend:
`FILESEXTRAPATHS_prepend := "${THISDIR}:"`
do_unpack_append(){
bb.build.exec_func('replace_file', d)
}
replace_file(){
cp -f ${THISDIR}/fileToBeReplaced ${WORKDIR}/fileToBeReplaced
echo ${THISDIR} > ${WORKDIR}/shouldContain_meta-newLayer
}
There are two issues with recipe.bbappend:
I would expect the file shouldContain_meta-newLayer to contain meta-newLayer, but instead it contains meta-origLayer.
I'd primarily like to understand why ${THISDIR} behaves differently when placed inside do_unpack_append() from when it is used for prepending FILESEXTRAPATHS
When running bitbake, the recipe fails, producing the following error:
cp: cannot stat '/fileToBeReplaced': No such file or directory
This error occurs because fileToBeReplaced resides in a subdirectory of meta-origLayer (i.e. meta-origLayer/machine1/fileToBeReplaced) and the .bbappend expects to find the file in /fileToBeReplaced
My Question. . .
I have assumed ${THISDIR} would behave consistently within the same .bbappend, but it doesn't appear to. What is the best way to reference meta-newLayer/fileToBeReplaced from within do_unpack_append()?
This *.bbappend correctly overwrites fileToBeReplaced in the working directory during the unpack task:
FILESEXTRAPATHS_prepend := "${THISDIR}:"
SRC_URI_append += " file://fileToBeReplaced "
SAVED_DIR := "${THISDIR}"
do_unpack_append(){
bb.build.exec_func('replace_file', d)
}
replace_file(){
cp -f ${SAVED_DIR}/fileToBeReplaced ${WORKDIR}/fileToBeReplaced
}
Thanks for the explanation between bbappend parsing and execution johannes-schaub-ltb
Related
I have a directory which again contains subdirectories, which are built has part of other recipe and moved to DEPLOY_DIR_IMAGE using deploy bb class. So now I want to copy it to main image boot partition.
If it was a single file then appending required filename to IMAGE_EFI_BOOT_FILES variable, then yocto copies it to /boot. But same doesn't work for directories containing subdirectories please provide style to include even the subdirectories. Thank you
PS: I have tried appending IMAGE_EFI_BOOT_FILES += "parent_dir/*" didnt work.
It is obvious that IMAGE_EFI_BOOT_FILES is acting like the well known IMAGE_BOOT_FILES and other variables that are responsible for having the files necessary to be shipped in the boot partition. And that needs files and not directories.
So, if you do not need to specify all the files by hand, but instead you want to pass the directory, I suggest you use a python method to collect the files for you and append them to the variable.
See the following example I developed and tested:
def get_files(d, dir):
import os
dir_path = dir
if not os.path.exists(os.path.dirname(dir)):
dir_path = d.getVar(dir)
return ' '.join(f for f in os.listdir(d.getVar(dir)) if os.path.isfile(f))
IMAGE_EFI_BOOT_FILES += "${#get_files(d, 'DEPLOY_DIR_IMAGE')}"
The method will test if the argument is a real path then it will directly check for files, if not it will assume that it is a bitbake variable and it will get its content, so if DEPLOY_DIR_IMAGE is, for example, /home/user/dir, passing DEPLOY_DIR_IMAGE or /home/usr/dir will give the same result.
IMPORTANT
It is obvious also that IMAGE_EFI_BOOT_FILES is used in a .conf file such as local.conf or a custom machine configuration file. So adding that python function in .conf file will not work. I suggest creating a class for it and inherit it globally in your .conf file:
meta-custom/classes/utils.bbclass
local.conf:
INHERIT += "utils"
IMAGE_EFI_BOOT_FILES += "${#get_files(d, 'DEPLOY_DIR_IMAGE')}"
Try this and let me know in the comments.
EDIT
I have just realized that bitbake already imports os within python expressions expansions, so you can do it in one line without any need for a separate python function:
PATH = "/path/to/directory/" or
PATH = "A variable containing the path"
IMAGE_EFI_BOOT_FILES += "${#' '.join('%s' % f for f in os.listdir('${PATH}') if os.path.isfile(f))}"
Note: I am looking for Yocto built-in which can achieve solution for above mentioned , would like to share other way to resolve the functionality for community's benefit.
Add following in bb file if you are using one or refer to talel-belhadjsalem answer to use utils.bbclass.
def add_directory_bootfs(d, dirname, bootfs_dir):
file_list = list()
boot_files_list = list()
deploy_dir_image = d.getVar('DEPLOY_DIR_IMAGE')
for (dirpath, dirnames, filenames) in os.walk(os.path.join(deploy_dir_image, dirname)):
file_list += [os.path.join(dirpath, file) for file in filenames]
for file in file_list:
file_rel_path = os.path.relpath(file, os.path.join(deploy_dir_image, dirname))
boot_file_entry = os.path.join(dirname, file_rel_path) + ';' + os.path.join(bootfs_dir, dirname, file_rel_path)
boot_files_list.append(boot_file_entry)
return ' '.join(boot_files_list)
IMAGE_EFI_BOOT_FILES += "${#add_directory_bootfs(d, 'relative_path_to_dir_in_deploy_dir_image', 'EFI/BOOT')}"
I have added yaml files to add new dbus objects and I added PHOSPHOR_MAPPER_SERVICE_append = " com/newCoName"
(newCoName is the name of my company)
But when I run bitbake, do_configure for phosphor_mapper bails when it passes the option -Ddata_com_newCoName to meson. The following readme says I need to run ./regenerate_meson from the gen directory when I add new YAML files. But how do I do that from a recipe file?
https://github.com/openbmc/phosphor-dbus-interfaces
One option is to generate these files outside ot yocto environment (i.e. not involving bitbake). Thus
clone that git repo
place your yaml file where you cloned repo
do what readme tells, i.e. go to gen directory and execute meson-regenerate script
collect changes that are done by script and create patch
add patch to your layer and reference it in .bbappend file (meta-/recipes-phosphor/dbus/phosphor-dbus-interfaces_git.bbappend)
Another option would be to add to .bbappend file additional task that runs before do_configure - and call that script from there:
do_configure_prepend() {
cd ${S}/gen && ./meson-regenerate
}
Along this .bbappend you should add your yaml so that it lands inside gen folder in patch or directly in your layer (check FILESEXTRAPATHS).
In both cases you'll need to patch meson_options.txt: add option
option('data_com_newCoName', type: 'boolean', value: true)
Why do we need to give path of files in SRC_URI even though we are including the files path in FILESEXTRAPATHS_prepend variable? For example:
SUMMARY = "Simple Hello application"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI = "file://Hello_1.c \
file://Hello_2.c \
"
do_compile() {
oe_runmake
}
do_install() {
install -d ${D}${bindir}
install -m 0755 Hello ${D}${bindir}
}
In the "files" folder I have two files: hello1.c and hello2.c. When I remove SRC_URI it outputs the following error,
ERROR: Hello-1.0-r0 do_compile: oe_runmake failed
But if I remove
FILESEXTRAPATHS_prepend it is working fine.
What is the purpose of the variable FILESEXTRAPATHS_prepend?
Why error occurs when I remove SRC_URI even though I'm including my files path in FILESEXTRAPATHS_prepend?
Simple way let assume meta-layer/recipes-core/example
In above path created hello and hello.bb
Here hello is a directory having your source and other data and hello.bb is recipe.
Now
SRC_URI : The SRC_URI variable always checks the data in hello dir only.
FILESEXTRAPATHS_prepend := "${THISDIR}:" : if you add this line in your recipe, then the SRC_URI variable checks the data in present directory where the hello.bb file present.
In your case
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
The SRC_URI variable checks the data in files dir where the hello.bb is present.
Note: Most of the time people will use this FILESEXTRAPATHS variable in .bbappend files to apply patches and other files to the recipe.
For every .bb file, the SRC_URI variable is used to specify which files to fetch the source from - either from
an online repository or a local one, and the FILESEXTRAPATHS specifies where these files are looked for, and depends on your source
path.
BitBake uses the SRC_URI variable to point to source files
regardless of their location. Each recipe must have a SRC_URI
variable that points to the source.
SRC_URI = file:// Fetches files, which are usually files shipped
with the Metadata, from the local machine. The path is relative to the
FILESPATH variable. Thus, the build system searches, in order, from
the following directories, which are assumed to be a subdirectories of
the directory in which the recipe file (.bb) or append file
(.bbappend) resides:
FILESPATH: The default set of directories the OpenEmbedded build
system uses when searching for patches and files. During the build
process, BitBake searches each directory in FILESPATH in the specified
order when looking for files and patches specified by each file:// URI
in a recipe.
The default value for the FILESPATH variable is defined in the
base.bbclass class found in meta/classes in the Source Directory:
FILESPATH = "${#base_set_filespath(["${FILE_DIRNAME}/${BP}", \
"${FILE_DIRNAME}/${BPN}", "${FILE_DIRNAME}/files"], d)}"
Do not hand-edit the FILESPATH variable; The
default directories BitBake uses when it processes recipes are
initially defined by the FILESPATH variable. You can extend FILESPATH
variable by using FILESEXTRAPATHS.
> Best practices dictate that you accomplish this by using
FILESEXTRAPATHS from within a .bbappend file
FILESEXTRAPATHS: Extends the search path the OpenEmbedded build system
uses when looking for files and patches as it processes recipes and
append files. The default directories BitBake uses when it processes
recipes are initially defined by the FILESPATH variable.
If you want the build system to pick up files specified through a
SRC_URI statement from your append file, you need to be sure to extend
the FILESPATH variable by also using the FILESEXTRAPATHS variable from
within your append file.
http://www.yoctoproject.org/docs/2.1/ref-manual/ref-manual.html#var-FILESPATH
Back to your error, since each recipe MUST have a SRC_URI; it will not work if you delete it;
Since your recipe is not an .bbappend, adding FILESEXTRAPATHS is not appropriate and not necessary.
I am attempting to write a recipe that would simple copy two files (MyfileA , MyfileB) to a specific directory when the overall image is built. This is what my directory structure looks like:
MyDir/MyRecipe.bb
MyDir/files/MyfileA
MyDir/files/MyfileB
I would like the two files to be copied to a folder in home (which would not exist initially hence the directories should be created)The folder lets say is called "Testfolder"
This is what my bitbake file looks like
DESCRIPTION = "Testing Bitbake file"
PR = "r0"
SRC_URI = "file://MyfileA \
file://MyfileB "
do_install() {
install -d MyfileA ~/TestFolder/
}
Kindly let me know if I am doing something wrong here?
When i run bitbake on this I get the following
The BBPATH variable is not set and bitbake did not find a conf/bblayers.conf file in the expected location.
Maybe you accidentally invoked bitbake from the wrong directory?
DEBUG: Removed the following variables from the environment: LANG, LS_COLORS, LESSCLOSE, XDG_RUNTIME_DIR, SHLVL, SSH_TTY, OLDPWD, LESSOPEN, SSH_CLIENT, MAIL, SSH_CONNECTION, XDG_SESSION_ID, _, BUILDDIR
Any help in this regard would be appreciated.
First of all, to create your own meta-layer, you should run command yocto-layer create MyRecipe in your Yocto Environment. This is to make sure that you have all the necessary element in your meta layer. Make sure to put the new meta-layer into conf/bblayers.conf
Creating HelloWorld Recipe Video can be found here
Second, to copy a file from one to another directories.
DESCRIPTION = "Testing Bitbake file"
SECTION = "TESTING"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
PR = "r0"
SRC_URI = "file://MyfileA \
file://MyfileB "
#specify where to get the files
S = "${WORKDIR}"
inherit allarch
#create the folder in target machine
#${D} is the directory of the target machine
#move the file from working directory to the target machine
do_install() {
install -d ${D}/TestFolder
install -m ${WORKDIR}/MyfileA ${D}/TestFolder
}
To get more in details, this is my understanding of how the files move around in Yocto.
You have a directory that stored metadata in /sourced/meta-mylayer/recipes-myRecipe/. In that directory, there would be a folder with the same name as the recipe. I.E. myRecipe/ myRecipe_001.bb.
You would store the files that are related to myRecipe.bb (usually it is a patch) in myRecipe/ so that SRC_URI will get into that myRecipe/ directory to grab files. I.E. myFileA, myFileB
Then, you specify the S. This is the location in the Build Directory where unpacked recipe source code resides. By that mean, myFileA and myFileB are moved/copied to there when myRecipe builds.
Usually, S is equal to ${WORKDIR}, this is equivalent to ${TMPDIR}/work/${MULTIMACH_TARGET_SYS}/${PN}/${EXTENDPE}${PV}-${PR}
The actual directory depends on several things:
TMPDIR: The top-level build output directory
MULTIMACH_TARGET_SYS: The target system identifier
PN: The recipe name
EXTENDPE: The epoch - (if PE is not specified, which is usually the case for most recipes, then EXTENDPE is blank)
PV: The recipe version
PR: The recipe revision
After that, we inherit allarch. This class is used for architecture independent recipes/data files (usually scripts).
Then, the last thing we have to do is copy the files.
${D} is the location in the Build Directory where components are installed by do_install task. This location defaults to ${WORKDIR}/image
${WORKDIR}/image can also be described as the / directory in the target system.
Go to ${D} directory and create a folder call TestFolder
Then, copy myFileA from ${WORKDIR} to the ${D}/TestFolder
P.S. Please add comment to fix. There might be mistaken information here, cause I learned all this by myself.
I've seen that I can use this command in order to copy a directory using cmake:
file(COPY "myDir" DESTINATION "myDestination")
(from this post)
My problem is that I don't want to copy all of myDir, but only the .h files that are in there. I've tried with
file(COPY "myDir/*.h" DESTINATION "myDestination")
but I obtain the following error:
CMake Error at CMakeLists.txt:23 (file):
file COPY cannot find
"/full/path/to/myDIR/*.h".
How can I filter the files that I want to copy to a destination folder?
I've found the solution by myself:
file(GLOB MY_PUBLIC_HEADERS
"myDir/*.h"
)
file(COPY ${MY_PUBLIC_HEADERS} DESTINATION myDestination)
this also works for me:
install(DIRECTORY "myDir/"
DESTINATION "myDestination"
FILES_MATCHING PATTERN "*.h" )
The alternative approach provided by jepessen does not take into account the fact that sometimes the number of files to be copied is too high. I encountered the issue when doing such thing (more than 110 files)
Due to a limitation on Windows on the number of characters (2047 or 8191) in a single command line, this approach may randomly fail depending on the number of headers that are in the folder. More info here https://support.microsoft.com/en-gb/help/830473/command-prompt-cmd-exe-command-line-string-limitation
Here is my solution:
file(GLOB MY_HEADERS myDir/*.h)
foreach(CurrentHeaderFile IN LISTS MY_HEADERS)
add_custom_command(
TARGET MyTarget PRE_BUILD
COMMAND ${CMAKE_COMMAND} -E copy_if_different ${CurrentHeaderFile} ${myDestination}
COMMENT "Copying header: ${CurrentHeaderFile}")
endforeach()
This works like a charm on MacOS. However, if you have another target that depends on MyTarget and needs to use these headers, you may have some compile errors due to not found includes on Windows. Therefore you may want to prefer the following option that defines an intermediate target.
function (CopyFile ORIGINAL_TARGET FILE_PATH COPY_OUTPUT_DIRECTORY)
# Copy to the disk at build time so that when the header file changes, it is detected by the build system.
set(input ${FILE_PATH})
get_filename_component(file_name ${FILE_PATH} NAME)
set(output ${COPY_OUTPUT_DIRECTORY}/${file_name})
set(copyTarget ${ORIGINAL_TARGET}-${file_name})
add_custom_target(${copyTarget} DEPENDS ${output})
add_dependencies(${ORIGINAL_TARGET} ${copyTarget})
add_custom_command(
DEPENDS ${input}
OUTPUT ${output}
COMMAND ${CMAKE_COMMAND} -E copy_if_different ${input} ${output}
COMMENT "Copying file to ${output}."
)
endfunction ()
foreach(HeaderFile IN LISTS MY_HEADERS)
CopyFile(MyTarget ${HeaderFile} ${myDestination})
endforeach()
The downside indeed is that you end up with multiple target (one per copied file) but they should all end up together (alphabetically) since they start with the same prefix ORIGINAL_TARGET -> "MyTarget"