Protobuf Build Protoc select all .proto files in a directory - sh

I have the following .sh file that I run to build my protos.
mkdir -p ./src/gen
protoc -I=../protos/ \
../protos/common/*.proto \
../protos/service/protos/*.proto \
../protos/ui/*.proto \
../protos/*.proto \
--js_out=import_style=commonjs,binary:./src/gen \
--grpc-web_out=import_style=commonjs+dts,mode=grpcwebtext:./src/gen
It works great, however, my team might add other directories to the protos directory that this file doesn't see.
Is there a way for me to update this script to get all .proto files in the protos directory?
I've tried the following but it didn't work:
mkdir -p ./src/gen
protoc -I=../protos/**/*.proto \
--js_out=import_style=commonjs,binary:./src/gen \
--grpc-web_out=import_style=commonjs+dts,mode=grpcwebtext:./src/gen
I'm sure it's just some syntax issue, but I'm not sure how to do it.
EDIT:
I've implemented something to make my generators more flexible, but it's still messy:
mkdir -p ./src/gen
protoc -I=../protos/ \
../protos/*.proto \
../protos/**/*.proto \
../protos/**/**/*.proto \
../protos/**/**/**/*.proto \
../protos/**/**/**/**/*.proto \
--js_out=import_style=commonjs,binary:./src/gen \
--grpc-web_out=import_style=commonjs+dts,mode=grpcwebtext:./src/gen

Since your script will pass the complete path for every file, it should be enough to add the protos/ folder with -I=../protos. The code below works fine for me, with multiple proto files in different subfolders.
$ tree ../protos
../protos
├── a
│   └── a.proto
└── b
├── b.proto
└── c
└── c.proto
$ protoc \
-I=../protos \
../protos/**/*.proto \
--js_out=import_style=commonjs,binary:./src/gen
$ tree src/gen
src/gen
├── a
│   └── a_pb.js
└── b
├── b_pb.js
└── c
└── c_pb.js
3 directories, 3 files
In your original question, you may just have mixed up ../protos/*/**.proto with ../protos/**/*.proto.

Related

wget issue with certain html files

I wrote a little wget command to download a page + all links that belong to the same directory, but only one level deep.
wget -Ekpx -np -l 1 -D <domain> <page>
The flags:
E: adjust-extension
k: convert-links
p: page-requisites
r: recursive
np: no-parent
l: level
D: domains
This works great for a lot of sites, but I had issues with this site in particular.
wget -Ekpr -np -l 1 -D www.eurocanadians.ca https://www.eurocanadians.ca/2022/02/the-origins-of-the-personal-computer-and-what-you-can-do-about-it.html
It treats the html file as a directory:
└── www.eurocanadians.ca
├── 2022
│   └── 02
│   └── the-origins-of-the-personal-computer-and-what-you-can-do-about-it.html
│   └── feed
When I replace the -r flag with -x (--force-directories), I get this result:
└── www.eurocanadians.ca
├── 2022
│   └── 02
│   └── the-origins-of-the-personal-computer-and-what-you-can-do-about-it.html
It treats it as a html file (as it should), but it's a single page download and doesn't download all the first level links.
How to let the page be treated as html, but still use -r? -F (--force-html) didn't work btw.

What is the difference between image and package directory in Yocto

I wrote a basic hello world recipe
DESCRIPTION = "Simple helloworld C application"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
SRC_URI = "file://userprog.c file://ReadMe.txt"
S = "${WORKDIR}"
do_compile() {
${CC} -DUSE_SYSCALL userprog.c ${LDFLAGS} -o userprog
}
do_install() {
install -d ${D}${bindir}
install -m 0755 userprog ${D}${bindir}
install -d ${D}${docdir}
install -m 0644 ReadMe.txt ${D}${docdir}
}
After, looking at the WORKDIR, the contents of 'package' and 'image' folder are same.
$ tree image/
image/
└── usr
├── bin
│   └── userprog
└── share
└── doc
└── ReadMe.txt
$ tree package
package
└── usr
├── bin
│   └── userprog
└── share
└── doc
└── ReadMe.txt
What is the difference between both folders, I know image folder is controlled in do_install task, what about in package folder?
images/ is for staging install dir, something like make install DESTDIR=<..>
package is for do_package task which is then further split into individual output packages in packages-split,
even though the content looks similar context is different, since it operates on PACKAGES and FILES variables, and if the files are not mentioned in these variables it won't copy those from image/ into package/ it is described in detail in Yocto project manual

Yocto: cp can't stat file: no such file or directory

I am trying to copy two folders(containing some scripts) in my target rootfs. I have created a custom layer and a custom recipe inside it.
My directory structure is like this:
../sources/meta-company/recipes-bla_2.06/
└── bla
├── bla
│   ├── dir1
│   │   ├── dir
│   │   │   └── files.sh
│   └── dir2
│   ├── dir
│   │   ├── files.sql
│   ├── test.sh
└── bla_2.06.bb
My .bb file is as follows:
DESCRIPTION = " bla "
LICENSE = "CLOSED"
SRC_URI = "file://dir1/ \
file://dir2/ "
do_install() {
install -d ${D}/root/dir1
install -d ${D}/root/dir2
cp -r --no-dereference --preserve=mode,links -v ${S}/dir1/ ${D}/root/dir1
cp -r --no-dereference --preserve=mode,links -v ${S}/dir2/ ${D}/root/dir2/
}
FILE_$PN = "/root/"
The error I am getting:
> Log data follows: | DEBUG: Executing shell function do_install | cp:
> cannot stat
> '/home/amol/test/fsl-arm-yocto-bsp/build-cl-som-imx7-fsl-imx-x11/tmp/work/cortexa7hf-neon-poky-linux-gnueabi/bla/1.0-r0/bla-1.0/dir1':
> No such file or directory | WARNING: exit code 1 from a shell command.
> | ERROR: Function failed: do_install (log file is located at
> /home/amol/test/fsl-arm-yocto-bsp/build-cl-som-imx7-fsl-imx-x11/tmp/work/cortexa7hf-neon-poky-linux-gnueabi/seriald/1.0-r0/temp/log.do_install.49808)
> NOTE: recipe bla-1.0-r0: task do_install: Failed NOTE: Tasks Summary:
> Attempted 334 tasks of which 333 didn't need to be rerun and 1 failed.
I am new to yocto, is my .bb file correct?.Thanks in advance.
There are two problems in your do_install section,
${S} points to source directory, but SRC_URI copies your content in ${WORKDIR}. So you should be using ${WORKSIR} in your install section
You are trying to copy ${S}/dir1/ inside ${D}/root/dir1, this means your final structure is /root/dir1/dir1/. You may not want this.
So the modified version would look like,
do_install() {
install -d ${D}/root/dir1
install -d ${D}/root/dir2
cp -r --no-dereference --preserve=mode,links -v ${WORKDIR}/dir1/* ${D}/root/dir1/
cp -r --no-dereference --preserve=mode,links -v ${WORKDIR}/dir2/* ${D}/root/dir2/
}

Enable systemd services using yocto

Hi this is my layer tree
├── conf
│   └── layer.conf
├── COPYING.MIT
├── README
└── recipes-hello
├── helloworld
│   ├── helloworld-0.1
│   │   ├── helloworld.c
│   │   ├── helloworld.patch
│   │   └── newhelloworld.c
│   └── helloworld_0.1.bb
├── message
│   ├── message-0.1
│   │   └── message.txt
│   └── message_0.1.bb
└── service
├── service-0.1
│   ├── test_systemd.service
│   └── test_systemd.sh
└── service_0.1.bb
Here test_systemd.service is the service file which have to invoke test_systemd.sh, which I am trying to achieve using service_0.1.bb
# This recipe performs the following tasks
# 1) Install .sh file in /home/root/ and .sh script creates a random text file
# 2) Install the .service file in systemd directory
# 3) Invoke the .sh script via .service file
inherit systemd
SUMMARY = "Install and start a systemd service"
SECTION = "examples"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
#here we specify the source we want to build
SRC_URI = "file://test_systemd.sh"
SRC_URI += "file://test_systemd.service"
#here we specify the source directory, where we can do all the building and expect sources to be placed
S = "${WORKDIR}"
SYSTEMD_SERVICE_${PN} = "test_systemd.service"
#bitbake task
#created a directory /home/root for target install the script
do_install() {
install -d ${D}/home/root
install -m 0755 ${WORKDIR}/test_systemd.sh ${D}/home/root
install -d ${D}{systemd_system_unitdir}
install -m 0644 ${WORKDIR}/test_systemd.service ${D}{systemd_system_unitdir}
}
#Pack the path
FILES_${PN} += "/home/root"
FILES_${PN} += "/lib/systemd/system"
REQUIRED_DISTRO_FEATURES= "systemd"
The problem is when I try to bitbake system recipe, bitbake throws an error saying test_systemd.service not found.
I managed to install both the files in RFS with a previous attempt but when I include the systemd concept. I get the no such file error. What could be the reason ?
Error message
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
ERROR: service-0.1-r0 do_package: SYSTEMD_SERVICE_service value test_systemd.service does not exist
ERROR: service-0.1-r0 do_package: Function failed: systemd_populate_packages
ERROR: Logfile of failure stored in: /home/guest/yocto_practice/poky/build-beaglebone/tmp/work/cortexa8hf-neon-poky-linux-gnueabi/service/0.1-r0/temp/log.do_package.2860
ERROR: Task (/home/guest/yocto_practice/meta-testlayer/recipes-hello/service/service_0.1.bb:do_package) failed with exit code '1'
NOTE: Tasks Summary: Attempted 514 tasks of which 506 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
/home/guest/yocto_practice/meta-testlayer/recipes-hello/service/service_0.1.bb:do_package
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.
Also is this the correct way to write bb recipe for systemd and what is the significance of writing this
#Pack the path
FILES_${PN} += "/home/root"
FILES_${PN} += "/lib/systemd/system"
without this bitbake throws error.
SYSTEMD_SERVICE_${PN} += "file://test_systemd.service"
This should be:
SYSTEMD_SERVICE_${PN} = "test_systemd.service"
Other notes (unrelated to the error):
Installing things into /home is probably not a great idea (you could use e.g. ${libexecdir} for scripts that other scripts need.
there's no reason for having a do_install_append() in a bb file: you can just put everything in do_install()
If your Yocto is recent, using ${systemd_system_unitdir} instead of /lib/systemd/system is a good idea (in older releases ${systemd_unitdir}/system/ works)
In order to resolve this packaging error, I used the following install step
do_install() {
install -d ${D}${bindir}
install -m 0755 ${WORKDIR}/test_systemd.sh ${D}${bindir}
install -d ${D}${systemd_unitdir}/system
install -m 0644 ${WORKDIR}/test_systemd.service ${D}${systemd_unitdir}/system
}

Bad exit status from /var/tmp/rpm-tmp.ajKra4 (%prep)

I am having a weird RPM issue, I am new to it so bear with me... I have the spec file created and when I run to do the build I get an error:
/var/tmp/rpm-tmp.ajKra4: line 36: cd: hero-01: No such file or directory
error: Bad exit status from /var/tmp/rpm-tmp.ajKra4 (%prep)
Then I check that temp file and it is trying to CD to a directory that does not exist.. Should it be creating this in the spec file? if so where?
Here is my spec file:
Summary: Install Hero
Name: hero
Version: 01
Release: 1
Group: Billing reporting
Source: %{name}-%{version}.tar.gz
License: SLA
%description
Hero billing reports system
%prep
rm -rf %{_topdir}/BUILD/*
%setup
%install
mkdir -p /opt/%{name}
cp -r * /opt/%{name}
%post
find /opt/%{name} -type d -exec chmod 755 {} \;
find /opt/%{name} -type f -exec chmod 644 {} \;
chmod -R 755 /opt/%{name}/bin
%files
/opt/%{name}
%defattr(-,root,root,0755)
%clean
rm -rf $RPM_BUILD_ROOT
%postun
rm -rf /opt/%{name}
Perhaps I am missing something? Would not be the first lol, thanks
Here is also what that tmp file is outputting:
#!/bin/sh
RPM_SOURCE_DIR="/root/rpmbuild/SOURCES"
RPM_BUILD_DIR="/root/rpmbuild/BUILD"
RPM_OPT_FLAGS="-O2 -g"
RPM_ARCH="x86_64"
RPM_OS="linux"
export RPM_SOURCE_DIR RPM_BUILD_DIR RPM_OPT_FLAGS RPM_ARCH RPM_OS
RPM_DOC_DIR="/usr/share/doc"
export RPM_DOC_DIR
RPM_PACKAGE_NAME="hero"
RPM_PACKAGE_VERSION="01"
RPM_PACKAGE_RELEASE="1"
export RPM_PACKAGE_NAME RPM_PACKAGE_VERSION RPM_PACKAGE_RELEASE
LANG=C
export LANG
unset CDPATH DISPLAY ||:
RPM_BUILD_ROOT="/root/rpmbuild/BUILDROOT/hero-01-1.x86_64"
export RPM_BUILD_ROOT
PKG_CONFIG_PATH="/usr/lib64/pkgconfig:/usr/share/pkgconfig"
export PKG_CONFIG_PATH
set -x
umask 022
cd "/root/rpmbuild/BUILD"
rm -rf /root/rpmbuild/BUILD/*
cd '/root/rpmbuild/BUILD'
rm -rf 'hero-01'
/usr/bin/gzip -dc '/root/rpmbuild/SOURCES/hero-01.tar.gz' | /bin/tar -xvvf -
STATUS=$?
if [ $STATUS -ne 0 ]; then
exit $STATUS
fi
cd 'hero-01'
/bin/chmod -Rf a+rX,u+w,g-w,o-w .
exit 0
Check out http://www.rpm.org/max-rpm/s1-rpm-inside-macros.html, specifically the "-n — Set Name of Build Directory" section.
The %setup macro is expecting that after untaring the tar.gz, there will be a hero-01 directory available, but your hero-01.tar.gz probably creates some other directory name, probably one without the version included in the name.
So, for example, if there's a 'hero' directory instead of a 'hero-01' directory in /root/rpmbuild/BUILD after the untarring, then update the spec file to use '%setup -n hero' instead of just '%setup'.
Also noteworthy is that some tarballs will not create themselves as a parent directory to the install paths. I.e., my tarball has the tree:
usr
├── bin
│   ├── check_for_incorrect_quotes
│   └── check_for_incorrect_quotes.py
└── lib
└── python2.6
└── site-packages
├── incorrectquotes
│   ├── check_for_incorrect_quotes.py
│   ├── check_for_incorrect_quotes.pyc
│   ├── __init__.py
│   ├── __init__.pyc
│   └── test
│   ├── __init__.py
│   └── __init__.pyc
└── IncorrectQuotes-0.2.0-py2.6.egg-info
├── dependency_links.txt
├── entry_points.txt
├── PKG-INFO
├── requires.txt
├── SOURCES.txt
└── top_level.txt
Because this is where it wants to install these packages
To make this work, you can just change setup -n to setup -c to create and move to that directory before untarring (You'll want to ctrl+f for "create directory (and change to it)")
TL;DR: setup -n -> setup -c might help
In your rpmbuild folder, go to SOURCES and rename your source folder this way:
mypackage-1.0
then create the tarball:
mypackage-1.0.tar.gz
And it should work.
What happens is that after untarring the archive, rpmbuild expects a folder named mypackage-1.0 and not mypackage or mypackage-something else.
Respect naming conventions. Check Guidelines