Where is the symlink created for a generated Yocto sdcard image - yocto

I modified an existing Yocto class (sdcard_image-rpi.bbclass) which generates an SD card image. The new class has a different name as well. Everything works as expected with one exception. When bitbake is finished, symlinks are created, which point to the last built file. For the .sdcard file this symlink is not created.
It is only not clear to me where (or in which stage) this symlink should be created.
Any suggestions what might causing this ?
class file content added below. (Removed the partitioning and copy opererations for clarity). This file is derived from meta-raspberrypi
and on some locations the name "rpi-sdimg" is replaced by "dual-sdimg".
The original file does not contain a command to create a symlink to the generated sdcard image, which implies it should be done outside this class. Most likely the changed name influenced this.
inherit image_types
# This image depends on the rootfs image
IMAGE_TYPEDEP_dual-sdimg = "${SDIMG_ROOTFS_TYPE}" # Name changed
# Set kernel and boot loader
IMAGE_BOOTLOADER ?= "bootfiles"
# Kernel image name
SDIMG_KERNELIMAGE_raspberrypi ?= "kernel.img"
SDIMG_KERNELIMAGE_raspberrypi2 ?= "kernel7.img"
SDIMG_KERNELIMAGE_raspberrypi3-64 ?= "kernel8.img"
# Boot partition volume id
BOOTDD_VOLUME_ID ?= "${MACHINE}"
# Use an uncompressed ext3 by default as rootfs
SDIMG_ROOTFS_TYPE ?= "ext3"
SDIMG_ROOTFS = "${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.${SDIMG_ROOTFS_TYPE}"
# For the names of kernel artifacts
inherit kernel-artifact-names
RPI_SDIMG_EXTRA_DEPENDS ?= ""
do_image_dual_sdimg[depends] = " \ # Name changed
parted-native:do_populate_sysroot \
mtools-native:do_populate_sysroot \
dosfstools-native:do_populate_sysroot \
virtual/kernel:do_deploy \
${IMAGE_BOOTLOADER}:do_deploy \
rpi-config:do_deploy \
${#bb.utils.contains('MACHINE_FEATURES', 'armstub', 'armstubs:do_deploy', '' ,d)} \
${#bb.utils.contains('RPI_USE_U_BOOT', '1', 'u-boot:do_deploy', '',d)} \
${#bb.utils.contains('RPI_USE_U_BOOT', '1', 'u-boot-default-script:do_deploy', '',d)} \
${RPI_SDIMG_EXTRA_DEPENDS} \
"
do_image_rpi_sdimg[recrdeps] = "do_build"
# SD card image name
SDIMG = "${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.rpi-sdimg"
# Additional files and/or directories to be copied into the vfat partition from the IMAGE_ROOTFS.
FATPAYLOAD ?= ""
# SD card vfat partition image name
SDIMG_VFAT_DEPLOY ?= "${RPI_USE_U_BOOT}"
SDIMG_VFAT = "${IMAGE_NAME}.vfat"
SDIMG_LINK_VFAT = "${IMGDEPLOYDIR}/${IMAGE_LINK_NAME}.vfat"
IMAGE_CMD_dual-sdimg () {
echo "Creating SD card image of ${SDIMG_SIZE} KiB with a boot partition of ${PT_BOOT_SIZE_ALIGNED} KiB and a rootfs of ${PT_ROOTFS_SIZE_ALIGNED} KiB"
# Check if we are building with device tree support
DTS="${KERNEL_DEVICETREE}"
# Initialize sdcard image file
dd if=/dev/zero of=${SDIMG} bs=1024 count=0 seek=${SDIMG_SIZE}
# Partitioning here
# Create a vfat image with boot files
# Bootloader copying here
# Add files (eg. hypervisor binaries) from the deploy dir
# Add stamp file
# Deploy vfat partition
# Write partitions to image file
}
ROOTFS_POSTPROCESS_COMMAND += " rpi_generate_sysctl_config ; "
rpi_generate_sysctl_config() {
# systemd sysctl config
test -d ${IMAGE_ROOTFS}${sysconfdir}/sysctl.d && \
echo "vm.min_free_kbytes = 8192" > ${IMAGE_ROOTFS}${sysconfdir}/sysctl.d/rpi-vm.conf
# sysv sysctl config
IMAGE_SYSCTL_CONF="${IMAGE_ROOTFS}${sysconfdir}/sysctl.conf"
test -e ${IMAGE_ROOTFS}${sysconfdir}/sysctl.conf && \
sed -e "/vm.min_free_kbytes/d" -i ${IMAGE_SYSCTL_CONF}
echo "" >> ${IMAGE_SYSCTL_CONF} && echo "vm.min_free_kbytes = 8192" >> ${IMAGE_SYSCTL_CONF}
}

Related

Is there a way to automatically create a container when starting Azurite?

For test purposes I create and run an Azurite docker image, in a test pipeline.
I would like to have the blob container automatically created though after Azurite is started, as it would simplify things.
Is there any good way to achieve this?
For the Postgres image we use, we can specify an init.sql which is run on startup. If something similar is available for Azurite, that would be awesome.
You can use the following Dockerfile to install the azure-storage-blob Python package on the Alpine based azurite image. The resulting image size is ~400MB compared to the ~1.2GB azure-cli image.
ARG AZURITE_VERSION="3.17.0"
FROM mcr.microsoft.com/azure-storage/azurite:${AZURITE_VERSION}
# Install azure-storage-blob python package
RUN apk update && \
apk --no-cache add py3-pip && \
apk add --virtual=build gcc libffi-dev musl-dev python3-dev && \
pip3 install --upgrade pip && \
pip3 install azure-storage-blob==12.12.0
# Copy init_azurite.py script
COPY ./init_azurite.py init_azurite.py
# Copy local blobs to azurite
COPY ./init_containers init_containers
# Run the blob emulator and initialize the blob containers
CMD python3 init_azurite.py --directory=init_containers & \
azurite-blob --blobHost 0.0.0.0 --blobPort 10000
The init_azurite.py script is a local Python script that uses the azure-storage-blob package to batch upload files and directories to the azurite blob storage emulator.
import argparse
import os
from time import sleep
from azure.core.exceptions import ResourceExistsError
from azure.storage.blob import BlobServiceClient, ContainerClient
def upload_file(container_client: ContainerClient, source: str, dest: str) -> None:
"""
Upload a single file to a path inside the container.
"""
print(f"Uploading {source} to {dest}")
with open(source, "rb") as data:
try:
container_client.upload_blob(name=dest, data=data)
except ResourceExistsError:
pass
def upload_dir(container_client: ContainerClient, source: str, dest: str) -> None:
"""
Upload a directory to a path inside the container.
"""
prefix = "" if dest == "" else dest + "/"
prefix += os.path.basename(source) + "/"
for root, dirs, files in os.walk(source):
for name in files:
dir_part = os.path.relpath(root, source)
dir_part = "" if dir_part == "." else dir_part + "/"
file_path = os.path.join(root, name)
blob_path = prefix + dir_part + name
upload_file(container_client, file_path, blob_path)
def init_containers(
service_client: BlobServiceClient, containers_directory: str
) -> None:
"""
Iterate on the containers directory and do the following:
1- create the container.
2- upload all folders and files to the container.
"""
for container_name in os.listdir(containers_directory):
container_path = os.path.join(containers_directory, container_name)
if os.path.isdir(container_path):
container_client = service_client.get_container_client(container_name)
try:
container_client.create_container()
except ResourceExistsError:
pass
for blob in os.listdir(container_path):
blob_path = os.path.join(container_path, blob)
if os.path.isdir(blob_path):
upload_dir(container_client, blob_path, "")
else:
upload_file(container_client, blob_path, blob)
if __name__ == "__main__":
parser = argparse.ArgumentParser(
description="Initialize azurite emulator containers."
)
parser.add_argument(
"--directory",
required=True,
help="""
Directory that contains subdirectories named after the
containers that we should create. Each subdirectory will contain the files
and directories of its container.
"""
)
args = parser.parse_args()
# Connect to the localhost emulator (after 5 secs to make sure it's up).
sleep(5)
blob_service_client = BlobServiceClient(
account_url="http://localhost:10000/devstoreaccount1",
credential={
"account_name": "devstoreaccount1",
"account_key": (
"Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq"
"/K1SZFPTOtr/KBHBeksoGMGw=="
)
}
)
# Only initialize if not already initialized.
if next(blob_service_client.list_containers(), None):
print("Emulator already has containers, will skip initialization.")
else:
init_containers(blob_service_client, args.directory)
This script will be copied to the azurite container and will populate the initial blob containers every time the azurite container is started unless some containers were already persisted using docker volumes. In that case, nothing will happen.
Following is an example docker-compose.yml file:
azurite:
build:
context: ./
dockerfile: Dockerfile
args:
AZURITE_VERSION: 3.17.0
restart: on-failure
ports:
- 10000:10000
volumes:
- azurite-data:/opt/azurite
volumes:
azurite-data:
Using such volumes will persist the emulator data until you destroy them (e.g. by using docker-compose down -v).
Finally, init_containers is a local directory that contains the containers and their folders/files. It will be copied to the azurite container when the image is built.
For example:
init_containers:
container-name-1:
dir-1:
file.txt
img.png
dir-2:
file.txt
container-name-2:
dir-1:
file.txt
img.png
I've solved the issue by creating a custom docker image and executing azure-cli tools from a health check. There could certainly be better solutions, and I will update the accepted answer if someone posts a better solution.
In more details
A solution to create the required data on startup is to run my own script. I chose to trigger the script from a health check I defined in docker-compose. What it does is use azure cli tools to create a container and then verify that it exists.
The script:
AZURE_STORAGE_CONNECTION_STRING="UseDevelopmentStorage=true"
export AZURE_STORAGE_CONNECTION_STRING
az storage container create -n images
az storage container show -n images
exit $?
However, the azurite image is based on alpine, which doesn't have apt, so installing azure cli was a bit tricky. So I did it the other way around, and based my image on mcr.microsoft.com/azure-cli:latest. With that done I installed Azurite like this:
RUN apk add npm
RUN npm install -g azurite --silent
All that's left is to actually run azurite, see the official azurite dockerfile for details.
It is possible to do this without azure-cli and use curl instead (and with that, not having to use the azure-cli docker image). However this was a bit complicated to get the authentication header working properly, so using azure-cli was easier.

Create a recipe for a QT5 application

I have been struggling for too long, so I need help :)
I made a big QT5.8 application and usually when I want to compile it with my PC I just have to run the following command: qmake -qt=5.9 -spec linux-arm-gnueabihf-g++ -config configuration_name.
With this command, I can cross compile my source code for the armhf architecture using linux-arm-gnueabihf-g++ toolchain.
But now, as can easily create a yocto image for my target (Raspberry pi), I want to make a recipe in order to compile my qt software and put it into my image.
For now, I achieved to perform these following task in my recipe without errors:
do_fetch -> Yocto fetch the source from git repo (OK)
do_unpack -> OK
After that I want to perform a qmake command in order to generate my makefile, but here is my problem :/
First, I included the qmake5 class in my recipe using
require recipes-qt/qt5/qt5.inc
Then I tried a lot of things..
writing "qmake" into the do_configure task doesn't work. Last thing I tried was: '${OE_QMAKE_QMAKE} ${S}/my_software.pro -config my_config' but still the same error:
Could not find qmake spec 'linux-oe-g++'
I don't know what to do and I can't find any recipe exemple doing the things that I want to do.
If somebody already experienced this issue or have experience compiling qt5 software with a yocto recipe I would like your help :)
my recipe:
LICENSE = "CLOSED"
LIC_FILES_CHKSUM = " "
USERNAME = "****"
PASSWORD = "*****"
S = "${WORKDIR}/git"
require recipes-qt/qt5/qt5.inc
do_fetch(){
git clone http://${USERNAME}:${PASSWORD}#gitlab.....
}
do_configure () {
${OE_QMAKE_QMAKE} ${S}/my_software.pro -config my_config
}
Thanks
Short answer
Add "inherit qmake5" and let Yocto take care of it :)
Long answer
Here's an example of how I added a QT project. It is not using git, it is using local files. However for getting one step further I suggest:
Use my way as a test. Copy the QT project to /yocto/local_sources/Myproject/
and make this known to Yocto by using FILESEXTRAPATHS_prepend_ (as shown below).
If this works on your environment, adapt it to your needs (e.g. build from git instead of local_source, which is better off course.)
This way is tested and works also well for later remote-debugging with qt-creator and the yocto SDK by the way. Stick to Yocto, it's worth it in the end.
Here's my .bb file:
#
# Build QT xyz application
#
SUMMARY = "..."
SECTION = "Mysection"
LICENSE = "CLOSED"
#Add whatever you need here
DEPENDS += "qtbase qtmultimedia qtsvg"
#Add here your .pro and all other relevant files (if you use git later this step will be less tedious ...)
SRC_URI += "file://Myproject.pro"
SRC_URI += "file://*.h"
SRC_URI += "file://*.cpp"
SRC_URI += "file://subdir1/*.h"
SRC_URI += "file://subdir1/*.cpp"
SRC_URI += "file://subdir2/*.h"
SRC_URI += "file://subdir2/*.cpp"
SRC_URI += "file://subdir2/subdir3/*.h"
SRC_URI += "file://subdir2/subdir3/*.cpp"
#If you need autostart:
#SRC_URI += "file://myproject.service"
#Register for root file system aggregation
FILES_${PN} += "${bindir}/Myproject"
#RBE todo: both needed ?
FILESEXTRAPATHS_prepend_${PN} := "/yocto/local_sources/Myproject/Src:"
FILESEXTRAPATHS_prepend := "/yocto/local_sources/Myproject/Src:"
S = "${WORKDIR}"
#If you want to auto-start this add:
#SYSTEMD_SERVICE_${PN} = "Myproject.service"
FILES_${PN} = "${datadir} ${bindir} ${systemd_unitdir}"
FILES_${PN}-dbg = "${datadir}/${PN}/.debug"
FILES_${PN}-dev = "/usr/src"
#Check what's needed in your case ...
RDEPENDS_${PN} += "\
qtmultimedia-qmlplugins \
qtvirtualkeyboard \
qtquickcontrols2-qmlplugins \
gstreamer1.0-libav \
gstreamer1.0-plugins-base-audioconvert \
gstreamer1.0-plugins-base-audioresample \
gstreamer1.0-plugins-base-playback \
gstreamer1.0-plugins-base-typefindfunctions \
gstreamer1.0-plugins-base-videoconvert \
gstreamer1.0-plugins-base-videoscale \
gstreamer1.0-plugins-base-volume \
gstreamer1.0-plugins-base-vorbis \
gstreamer1.0-plugins-good-autodetect \
gstreamer1.0-plugins-good-matroska \
gstreamer1.0-plugins-good-ossaudio \
gstreamer1.0-plugins-good-videofilter \
"
do_install_append () {
# Install application
install -d ${D}${bindir}
install -m 0755 Myproject ${D}${bindir}/
# Uncomment if you want to autostart this application as a service
#install -Dm 0644 ${WORKDIR}/myproject.service ${D}${systemd_system_unitdir}/myproject.service
# Install resource files (example)
#install -d ${D}${datadir}/${PN}/Images
#for f in ${S}/Images/*; do \
# install -Dm 0644 $f ${D}${datadir}/${PN}/Images/
#done
}
#Also inherit "systemd" if you need autostart
inherit qmake5

u-boot bbappend causes file system corruption

I need to configure u-boot to boot immediately (disabling the press key to interrupt prompt) for a yocto build.
I've added the following bbappend to do so:
# u-boot_%.bbappend
FILESEXTRAPATHS_prepend := "${THISDIR}/u-boot:"
SRC_URI_append_edge = " \
file://disable-boot-delay.cfg \
"
where disable-boot-delay.cfg looks like this:
CONFIG_AUTOBOOT_KEYED=y
CONFIG_AUTOBOOT_KEYED_CTRLC=n
CONFIG_BOOTDELAY=-2
This successfully disables the boot prompt, but also causes the FAT partition that u-boot is installed on to become corrupt:
$ # without above bbappend
$ dosfsck /dev/mapper/loop0p1
fsck.fat 4.1 (2017-01-24)
/dev/mapper/loop0p1: 49 files, 6510/20431 clusters
$ # with above bbappend
$ dosfsck /dev/mapper/loop0p1
fsck.fat 4.1 (2017-01-24)
/OVERLAYS/.
Bad short file name (.).
1) Drop file
2) Rename file
3) Auto-rename
4) Keep it
? 4
/OVERLAYS/..
Bad short file name (..).
1) Drop file
2) Rename file
3) Auto-rename
4) Keep it
? 4
/dev/mapper/loop0p1: 49 files, 6510/20431 clusters
I'm massively, massively confused as to how this change could be causing file system corruption.
Environment:
Yocto Warrior (2.7)
meta-raspberrypi (warrior branch)
RPI_USE_U_BOOT = "1"
Diffing workdir/build/.config with and without the above append:
229c229
< CONFIG_BOOTDELAY=2
---
> CONFIG_BOOTDELAY=-2
306c306,311
< # CONFIG_AUTOBOOT_KEYED is not set
---
> CONFIG_AUTOBOOT_KEYED=y
> CONFIG_AUTOBOOT_PROMPT="Autoboot in %d seconds\n"
> # CONFIG_AUTOBOOT_ENCRYPTION is not set
> CONFIG_AUTOBOOT_DELAY_STR=""
> CONFIG_AUTOBOOT_STOP_STR=""
> # CONFIG_AUTOBOOT_KEYED_CTRLC is not set

bitbake ERROR: Nothing PROVIDES Raspberry Pi

I am having an issue with building Raspberry Pi image with Mender.io and Yocto.
Here is my bblayers.conf
# POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
POKY_BBLAYERS_CONF_VERSION = "2"
BBPATH = "${TOPDIR}"
BBFILES ?= ""
BBLAYERS ?= " \
/root/poky/meta \
/root/poky/meta-poky \
/root/poky/meta-yocto-bsp \
/root/poky/meta-mender/meta-mender-core \
/root/poky/meta-mender/meta-mender-raspberrypi \
/root/poky/meta-raspberrypi \
/root/poky/meta-openembedded/meta-oe \
/root/poky/meta-openembedded/meta-multimedia \
/root/poky/meta-openembedded/meta-python \
/root/poky/meta-openembedded/meta-networking \
"
And here my local.conf
PACKAGECONFIG_append_pn-qemu-native = " sdl"
PACKAGECONFIG_append_pn-nativesdk-qemu = " sdl"
#ASSUME_PROVIDED += "libsdl-native"
# CONF_VERSION is increased each time build/conf/ changes incompatibly and is used to
# track the version of this file when it was generated. This can safely be ignored if
# this doesn't mean anything to you.
CONF_VERSION = "2"
# The name of the disk image or Artifact that will be built.
# This is what the device will report that it is running, and different updates must have different names
# because Mender will skip installation of an artifact if it is already installed.
MENDER_ARTIFACT_NAME = "release-1"
INHERIT += "mender-full"
# A MACHINE integrated with Mender.
# vexpress-qemu or beaglebone can be used for testing.
MACHINE = "raspberrypi0"
DISTRO_FEATURES_append = " systemd"
VIRTUAL-RUNTIME_init_manager = "systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
VIRTUAL-RUNTIME_initscripts = ""
IMAGE_FSTYPES = "ext4"
KERNEL_IMAGETYPE = "uImage"
MENDER_PARTITION_ALIGNMENT_KB = "4096"
MENDER_BOOT_PART_SIZE_MB = "40"
do_image_sdimg[depends] += " bcm2835-bootfiles:do_populate_sysroot"
# raspberrypi files aligned with mender layout requirements
IMAGE_BOOT_FILES_append = " boot.scr u-boot.bin;${SDIMG_KERNELIMAGE}"
IMAGE_INSTALL_append = " kernel-image kernel-devicetree"
IMAGE_FSTYPES_remove += " rpi-sdimg"
When I run the following command
bitbake image1
I get the following error:
Loading cache: 100% |################################################################################################################################| Time: 0:00:00
Loaded 160 entries from dependency cache.
Parsing recipes: 100% |##############################################################################################################################| Time: 0:01:22
Parsing of 1982 .bb files complete (97 cached, 1885 parsed). 2743 targets, 184 skipped, 0 masked, 0 errors.
ERROR: Nothing PROVIDES 'image1'
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
What could I be missing?
When you run "bitbake " the has to be a recipe or target (at minimum). The recipe/target corresponds to a .*.bb file (ie. recipe file) in one of the layers included in bblayers.conf (there are cases where this relationship isn't exactly true but let's ignore those for now). Most .bb files will be recipes for packages but there are a few which define images. Since the image recipes are 'special' they are usually contained in directories called 'images' within the layers. So perform a
'find . -type d -name images'
from within the various layers directories, then inspect contents of these images directories to find the image recipe you are interested in (you are most likely interested in the images found in meta-raspberrypi). Determine the image you want then run the bitbake command with that image name,
ex. 'bitbake rpi-basic-image'

Buildroot Config Option for applying custom patch

I am new to buildroot and working to build Linaro with buildroot ..I have multiple fragment kernel config files and specified that in buildroot defconfig.
I have specified a custom kernel patches directory with BR2_LINUX_PATCH_DIR .
I dont have some of the config flags not set which are supposed to be there in the .config files..so i suspect that the Patches are applied successfully..so i tried giving a non existing location as Linux Patch dir and it does not give any error..
Is there anything other than giving value to BR2_LINUX_PATCH_DIR and what should be the format of the dir structure...in buildroot manual it says it should be
Package_name/patch name..For linux what should be the package name? It should be the same with which linux dir is created.for example for me it is linux-custom
Plz suggest and guide me in this.
Thanks in Advance
The option is named BR2_LINUX_KERNEL_PATCH, there is nothing named BR2_LINUX_PATCH_DIR. It applies all patches listed in this option (if those are files), or all files named *.patch if what's given in this option is a directory. See the code in linux/linux.mk:
define LINUX_APPLY_LOCAL_PATCHES
for p in $(filter-out ftp://% http://% https://%,$(LINUX_PATCHES)) ; do \
if test -d $$p ; then \
$(APPLY_PATCHES) $(#D) $$p \*.patch || exit 1 ; \
else \
$(APPLY_PATCHES) $(#D) `dirname $$p` `basename $$p` || exit 1; \
fi \
done
endef
Also, I would recommend that you watch the output of Buildroot: it shows everything it is doing, especially it lists the patches it applied. Look at the line >>> linux .... Patching, which is the marker for the beginning of the patching step of the linux package.