How to dynamical create a file with git info and include it in the the image and save it on build system - yocto

We have several developers working on a project. The areas we are concerned about (and we regularly modify) are kernel, our custom code, and the yocto space itself.
We'd like to create a file at some point in the process (do_fetch, or do_install?) that contains info about what's being built. Such as the git branch name and hash for each of the repos above. We would then install that file (or files if need be) onto the image as well as archive it away on a centralized server.
I know that some of this info is available in the buildhistory, but I'm not sure if it is there when we'd like to install and package.
Getting the branch and hash should be easy to get via shell commands in the recipe functions.
Before I go off and hack something out, I thought I'd ask if there is a standard way to do something similar to this.
Thanks!

In case you need to include custom information. A nice way consists in creating a custom layer bbclass, defined as follow :
DEPENDS += "git-native"
do_rootfs_save_versions() {
#Do custom tasks here like getting layer names and linked SHA numbers
#Store these information in a file and deploy it in ${DEPLOY_DIR_IMAGE}
}
ROOTFS_POSTPROCESS_COMMAND += "do_rootfs_save_versions;"
Then, include the bbclass in your image file
IMAGE_CLASSES += "<bbclass_name>"
It is very useful when you want to determine the layer version/image name/.. running on target.

OK, Here is what I did.
Added appends to the do_install functions I wanted to keep track of and put them in the top of the build dir:
do_install_append () {
echo ${SRCPV} > ${TOPDIR}/kernel_manifest.txt
git rev-parse --abbrev-ref HEAD >> ${TOPDIR}/kernel_manifest.txt
}
Added a new bbclass in our meta- dir:
DEPENDS += "git-native"
do_rootfs_save_manifests[nostamp] = "1"
do_rootfs_save_manifests() {
date > ${TOPDIR}/buildinfo.txt
hostname >> ${TOPDIR}/buildinfo.txt
git config user.name >> ${TOPDIR}/buildinfo.txt
cp ${TOPDIR}/buildinfo.txt ${IMAGE_ROOTFS}/usr/custom_space/
if [ ! -f ${TOPDIR}/kernel_manifest.txt ]; then
echo "kernel_manifest empty: Rebuild or run cleanall on it's recipe" > ${TOPDIR}/error_kernel_manifest.txt
cp ${TOPDIR}/error_kernel_manifest.txt ${IMAGE_ROOTFS}/usr/custom_space/
else
cp ${TOPDIR}/kernel_manifest.txt ${IMAGE_ROOTFS}/usr/custom_space/
if [ -f ${TOPDIR}/error_kernel_manifest.txt ]; then
rm ${TOPDIR}/error_kernel_manifest.txt
fi
fi
if [ ! -f ${TOPDIR}/buildhistory/metadata-revs ]; then
echo " metadata_revs empty: Make sure INHERIT += \"buildhistory\" and" > ${TOPDIR}/error_yocto_manifest.txt
echo " BUILDHISTORY_COMMIT = "1" are in your local.conf " >> ${TOPDIR}/error_yocto_manifest.txt
cp ${TOPDIR}/error_yocto_manifest.txt ${IMAGE_ROOTFS}/usr/custom_space/
else
if [ -f ${TOPDIR}/error_yocto_manifest.txt ]; then
rm ${TOPDIR}/error_yocto_manifest.txt
fi
cp ${TOPDIR}/buildhistory/metadata-revs ${TOPDIR}/yocto_manifest.txt
cp ${TOPDIR}/buildhistory/metadata-revs ${IMAGE_ROOTFS}/usr/custom_space/yocto_manifest.txt
fi
}
ROOTFS_POSTPROCESS_COMMAND += "do_rootfs_save_manifests;"
Added the following lines to the image recipes that we wanted to use the process:
IMAGE_CLASSES += "manifest"
inherit ${IMAGE_CLASSES}
Thanks for the help!

Related

Yocto & wpa_supplicant - installing libwpa_client.so

I'm working with this Yocto/ Bitbake recipe for wpa_supplicant (v2.9) which is a bbappend for this recipe. I had significant difficulties getting the associated library (libwpa_client.so) included in the install process during my Yocto/ Poky build. Eventually I created my own bbappend which contained:
# Force build of library libwpa_client.so
do_configure_append() {
echo 'CONFIG_BUILD_WPA_CLIENT_SO=y' >> wpa_supplicant/.config
}
# Avoid QA errors:
INSANE_SKIP_${PN} += " ldflags"
INHIBIT_PACKAGE_STRIP = "1"
INHIBIT_SYSROOT_STRIP = "1"
SOLIBS = ".so"
FILES_SOLIBSDEV = ""
# Force install of libwpa_client.so
do_install_append(){
install -d ${D}${libdir}
install -m 0644 ${S}/wpa_supplicant/libwpa_client.so ${D}${libdir}
}
Using the config option CONFIG_BUILD_WPA_CLIENT_SO=y failed to install the library so thats why I went with the do_install_append() method. I find it hard to believe this hasn't come up before for others so I think I'm missing something, I can't find similar hacks/ patches.
Can't someone point out if I'm missing something obvious?? There's a patch file for v2.10 which may be associated with this issue, but I think it's more an issue with the Makefile as the library file was successfully built for v2.9, just not populated to the final image.
If this isn't a waste of half a day perhaps this will help others.
Regards,

How to update modules.conf for SELINUX in BUILDROOT?

looking to disable some SELinux modules (set to off) and create others in modules.conf. I don't see an obvious way of updating modules.conf as I tried adding my changes as a modules.conf patch but it failed given that the modules.conf file gets built and is not just downloaded by BR so it is not available for patching like other things under the refpolicy directory:
Build window output:
refpolicy 2.20190609 PatchingApplying 0001-refpolicy-update-modules-conf.patch using patch:
can't find file to patch at input line 3
I did see in the log that there is a support/sedoctool.py that autogenerates the policy/modules.conf file so that the file is NOT patchable like most other things in the ref policy.
The relevant section of the buildroot/output/build/refpolicy-2.20190609/Makefile:
# policy building support tools
support := support
genxml := $(PYTHON) $(support)/segenxml.py
gendoc := $(PYTHON) $(support)/sedoctool.py
<...snip...>
########################################
#
# Create config files
#
conf: $(mod_conf) $(booleans) generate$(booleans) $(mod_conf): conf.intermediate.INTERMEDIATE: conf.intermediate
conf.intermediate: $(polxml)
#echo "Updating $(booleans) and $(mod_conf)"
$(verbose) $(gendoc) -b $(booleans) -m $(mod_conf) -x $(polxml)
Part of the hsmlinux build.log showing the sedoctool.py (gendoc) being run:
Updating policy/booleans.conf and policy/modules.conf
.../build-buildroot-sawshark/buildroot/output/host/usr/bin/python3 support/sedoctool.py -b policy/booleans.conf -m policy/modules.conf -x doc/policy.xml
I'm sure there is a standard way of doing this, just doesn't seem to be documented anywhere I can find.
Thanks.
Turns out that the sedoctool.py script is reading the doc/policy.xml. Looking at sedoctool.py:
#modules enabled and disabled values
MOD_BASE = "base"
MOD_ENABLED = "module"
MOD_DISABLED = "off"
<...snip...>
def gen_module_conf(doc, file_name, namevalue_list):
"""
Generates the module configuration file using the XML provided and the
previous module configuration.
"""
# If file exists, preserve settings and modify if needed.
# Otherwise, create it.
<...snip...>
mod_name = node.getAttribute("name")
mod_layer = node.parentNode.getAttribute("name")
<...snip...>
if mod_name and mod_layer:
file_name.write("# Layer: %s\n# Module: %s\n" % (mod_layer,mod_name))
if required:
file_name.write("# Required in base\n")
file_name.write("#\n")
if [mod_name, MOD_DISABLED] in namevalue_list:
file_name.write("%s = %s\n\n" % (mod_name, MOD_DISABLED))
# If the module is set as enabled.
elif [mod_name, MOD_ENABLED] in namevalue_list:
file_name.write("%s = %s\n\n" % (mod_name, MOD_ENABLED))
# If the module is set as base.
elif [mod_name, MOD_BASE] in namevalue_list:
file_name.write("%s = %s\n\n" % (mod_name, MOD_BASE))
So sedoctool.py has the nice feature of: "# If file exists, preserve settings and modify if needed." and modules.conf can just be added whole here via a complete file patch and the modules that are not desired set as "off" : refpolicy-2.20190609/policy/modules.conf and the script will update as needed based on desired policy.
One more detail is that in the next stage of the refpolicy Makefile (Building) the modules.conf with the updates is deleted in the beginning which kind of clashes with the ability of sedoctool to preserve the patched version of modules.conf...so patched the removal in the Building stage of the Makefile.
[7m>>> refpolicy 2.20190609 Building^[
<...snip...>
rm -f policy/modules.conf
The Makefile in refpolicy-2.20190609 has this line that I patched out because we are patching in our own modules.conf:
bare: clean
<...snip...>
$(verbose) rm -f $(mod_conf)
That patch looks like:
--- BUILDROOT/Makefile 2020-08-17 13:25:06.963804709 -0400
+++ FIX/Makefile 2020-08-17 19:25:29.540607763 -0400
## -636,7 +636,6 ##
$(verbose) rm -f $(modxml)
$(verbose) rm -f $(tunxml)
$(verbose) rm -f $(boolxml)
- $(verbose) rm -f $(mod_conf)
$(verbose) rm -f $(booleans)
$(verbose) rm -fR $(htmldir)
$(verbose) rm -f $(tags)
BTW,
Creating a patch with a complete new file in pp1:q!:
diff -crB --new-file pp0 pp1 > pp0.patch

Yocto: Install different config files based on MACHINE type or target image

I've got a couple of HW platforms (same cpu, etc.) that require different asound.conf files.
The way that I'm controlling the target platform is via the MACHINE variable and target image (i.e., MACHINE=machine_1 nice bitbake machine-1-bringup-image)
Normally, if just replacing the conf file I'd just create an alsa-state.bbappend and create a do_install_append function to replace it.
However since the different HW platforms require differ conf files I'm unsure how to handle it.
I've tried putting some logic into the append file do_install_append function but it's not working out. It's not always picking up the correct file (like it thinks that nothing has changed so that it uses the previous cached conf?)
Here's an example of one of the append files that I've tried:
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
SRC_URI += " \ file://asound_MACHINE1.conf \
file://asound_MACHINE2.conf \ "
do_install_append() {
echo " alsa-state.bbappend MACHINE: ${MACHINE}"
if [ "${MACHINE}" = "machine_1" ]; then
echo " machine_1"
echo " installing ${WORKDIR}/asound_MACHINE1.conf to ${D}${sysconfdir}/asound.conf"
install -m 644 ${WORKDIR}/asound_MACHINE1.conf {D}${sysconfdir}/asound.conf
else
echo " installing ${WORKDIR}/asound_MACHINE2.conf to ${D}${sysconfdir}/asound.conf"
install -m 644 ${WORKDIR}/asound_MACHINE2.conf ${D}${sysconfdir}/asound.conf
fi
}
I can see the correct echoes in the logs per the logic.
At any rate I don't think that the path I'm going down is the best way to deal with this.
Is there a 'standard' way to have different files installed based on either the target image or MACHINE variable?
do_install_append () {
// install common things here
}
do_install_append_machine-1 () {
// install machine-1 specific things here
}
do_install_append_machine-2 () {
// install machine-2 specific things here
}
The value of MACHINE is automatically added to OVERRIDES, which can be used at the end of a function append to have a MACHINE-specific addition to a function.
Maybe useful: https://www.yoctoproject.org/docs/2.4/mega-manual/mega-manual.html#var-OVERRIDES
You can have configuration files in machine specific directories in your particular case (just a specific configuration file for each machine). OpenEmbedded will fetch the most specific one. The directory structure in your recipe directory will look like:
files/<machine1>/asound.conf
files/<machine2>/asound.conf
And your alsa-state.bbappend will contain just one line (you don't need to change do_install because alsa-state.bb already installs asound.conf):
FILESEXTRAPATHS_prepend := "${THISDIR}/files:"
BTW: We are using that setup to have specific asound.state file per machine in our project.
Moreover, OpenEmbedded will detect that SRC_URI contains machine specific file and change the PACKAGE_ARCH accordingly, see: https://www.yoctoproject.org/docs/2.5/mega-manual/mega-manual.html#var-SRC_URI_OVERRIDES_PACKAGE_ARCH
Few more words on machine, distro or arch specific files: OE is trying to fetch the most specific file in file:// fetcher. It searches also in the directories named by distro (e.g files/<distro>/asound.conf) and architecture (e.g. armv7a, arm). It might be useful if you want to have file specific for some set of devices. More information: https://www.yoctoproject.org/docs/2.5/mega-manual/mega-manual.html#var-FILESOVERRIDES and also https://www.yoctoproject.org/docs/2.5/mega-manual/mega-manual.html#best-practices-to-follow-when-creating-layers (section "Place Machine-Specific Files in Machine-Specific Locations")
The above answer by clsulliv worked better than advertised. For future reference below is the append file I used:
FILESEXTRAPATHS_prepend:= "${THISDIR}/${PN}:"
SRC_URI += " \
file://machine1_asound.conf \
file://machine2_asound.conf \
"
do_install_append_machine1() {
echo " machine1"
echo " installing ${WORKDIR}/machine1_asound.conf to ${D}${sysconfdir}/asound.conf"
install -m 644 ${WORKDIR}/machine1_asound.conf ${D}${sysconfdir}/asound.conf
}
do_install_append_machine2() {
echo " machine2"
echo " installing ${WORKDIR}/machine2_asound.conf to ${D}${sysconfdir}/asound.conf"
install -m 644 ${WORKDIR}/machine2_asound.conf ${D}${sysconfdir}/asound.conf
}
Thanks for the help!

Configuring the compile sources of a build phase

I have a project with many targets.
Every time I create a new target, I am adding a specific class to the Compile Sources and I am setting a compiler flag for that class. How can I apply this setting automatically?
I think this needs to be done by modifying the project file directly. In other words, write a script that will make the changes.
Here's a simple example to add arc flags to a file for all targets.
#!/bin/sh
set -e
USAGE="$0 <filename> ..."
PROJECTFILE="project.pbxproj"
FILES="$*"
COMPILER_FLAGS="\"-fobjc-arc\"";
if [ "${FILES}" = "" ] ; then
echo ${USAGE};
exit 1 ;
fi
if [ ! -f "${PROJECTFILE}" ] ; then
echo "Cannot find ${PROJECTFILE} - Please run from within the .xcodeproj directory";
exit 1 ;
fi
cp ${PROJECTFILE} ${PROJECTFILE}.bak
for f in ${FILES}; do
echo "Adding flags for $f" ;
sed -ie "s/\(${f}.*{ *isa *= *PBXBuildFile.*\)};$/\\1 settings = {COMPILER_FLAGS = ${COMPILER_FLAGS}; }; };/g" "${PROJECTFILE}"
done

How to change the order of rsync in symfony deployment task

I want to deploy a part of my symfony application, say, it's like a module.
I want to exclude all files first, and then include only the files of
my new module.
For deployment I use the following symfony task
php symfony project:deploy production -t
The parameter -t prints all files to the output that are included in this dry run of rsync.
Content of config/rsync_exclude.txt is only *, since I like to exclude everthing:
*
In config/rsync_include.txt I list all the files and folders for the inclusion:
config/
config/mysupermodule.yml
lib/model/doctrine/
lib/model/doctrine/MySuperclass.php
lib/model/doctrine/MySuperclassTable.php
lib/
lib/MySuperLibrary/
lib/MySuperLibrary/*
The symfony task builds the following rsync command:
rsync --dry-run -azC --force --delete --progress --exclude-from=config/rsync_exclude.txt --include-from=config/rsync_include.txt -e "ssh -p22" ./ user#www.server.com:/test_deployment/
Problem 1: The the task doesn't sync any files.
Solution to 1: Change order: Include first, then exclude.
I figured out, that if I change my need to this one:
I want to include all files of my new module and exclude then all
other.
This means using the following command:
rsync --dry-run -azC --force --delete --progress --include-from=config/rsync_include.txt --exclude-from=config/rsync_exclude.txt -e "ssh -p22" ./ user#www.server.com:/test_deployment/
The rsync works.
Problem 2: How can I change the order of the rsync when using the symfony task?
The symfony task first excludes than includes.
Solution 2: ?
It is NOT possible.
But you can edit the deployment task in lib/task/project/sfProjectDeployTask.class.php.
Replace this (line 145 to 154 in SF 1.4):
if (file_exists($options['rsync-dir'].'/rsync_exclude.txt'))
{
$parameters .= sprintf(' --exclude-from=%s/rsync_exclude.txt', $options['rsync-dir']);
}
if (file_exists($options['rsync-dir'].'/rsync_include.txt'))
{
$parameters .= sprintf(' --include-from=%s/rsync_include.txt', $options['rsync-dir']);
}
with this:
if (file_exists($options['rsync-dir'].'/rsync_include.txt'))
{
$parameters .= sprintf(' --include-from=%s/rsync_include.txt', $options['rsync-dir']);
}
if (file_exists($options['rsync-dir'].'/rsync_exclude.txt'))
{
$parameters .= sprintf(' --exclude-from=%s/rsync_exclude.txt', $options['rsync-dir']);
}
In short: switch this two IF statements.
Let's change the way you want to do.
You should use only the exclude file. Exclude only directories that changed but you don't want to sync.
Because anyway if you modules/, app/, ... directories haven't change, you don't have to put them in the exclude file because they will remain the same on both server.