for my yocto project I'm mirroring to my repo to a private one, it's working fine for basic SRC_URI, but for url that have a branch defined it will erase it and bring the default one (master), making it fail since the branch referenced in the original uri was a tag.
DEBUG: Fetcher failure: Unable to find revision xxx_git_commit_hash_xxx in branch master even from upstream
In my .bb file:
SRC_URI = "git://github.com/google/protobuf.git;branch=3.11.x \
and in my local.conf (in the build folder) for the mirroring i have:
SOURCE_TOMYMIRROR="myUser#MyServerIdServer:/The/Path/to/directory"
PREMIRRORS_prepend ?= "\
git://.*/.* git://${SOURCE_TOMYMIRROR}/HOST/PATH;protocol=ssh\n \
"
(Yes i want to keep HOST and PATH as path to get the repo)
I can see in the logs that my repo is correctly fetch, but the "new" url is without branch and I can't find a way to keep the .bb one
I tried to understand and look at the documentation, but i don't see a way to keep a specific parameter (like the HOST and PATH variable)
So i tried to look at the init.py which define the rules, but i couldn't figure out if it's possible.
I think the right notation would be specifying the branch as well into the premirror URL.
SOURCE_MIRROR_URL = "git://${SOURCE_TOMYMIRROR};protocol=ssh;branch=3.11.x"
PREMIRRORS_prepend = " git://.*/.* ${SOURCE_MIRROR_URL} \
"
Yes it did work,i didn't need premirror_prepend, here's mine :
SOURCE_TOMYMIRROR="myUser#MyServerIdServer:/The/Path/to/directory"
SOURCE_MIRROR_URL="git://${SOURCE_TOMYMIRROR}/HOST/PATH;protocol=ssh"
INHERIT += "own-mirrors"
And if i want to add new url mirror i can use Mirror
MIRRORS += "\
git://.*/.* ${SOURCE_TOMYMIRROR_2}/HOST/PATH \n \
https?$://.*/.* ${SOURCE_TOMYMIRROR_2}/HOST/PATH \n \
"
Related
I am trying to find out that how bitbake search for recipe in build process ?
For example,
I have a recipe something like below:
DESCRIPTION = "PetaLinux GSTREAMER supported packages"
inherit packagegroup
GSTREAMER_PACKAGES = " \
gstreamer1.0 \
gstreamer1.0-python \
gstreamer1.0-meta-base \
gstreamer1.0-plugins-base \
gstreamer1.0-plugins-good \
gstreamer1.0-plugins-bad \
gstreamer1.0-rtsp-server \
gst-shark \
gstd \
gst-perf \
gst-interpipes \
"
GSTREAMER_PACKAGES_append_zynqmp = " gstreamer1.0-omx"
RDEPENDS_${PN} = "${GSTREAMER_PACKAGES}"
When I searched gstreamer1.0 related recipe in yocto layers, I found two recipe, one of them is gstreamer1.0_1.16.1.bb in meta layer, and the other is gstreamer1.0_%.bbappend in meta-petalinux layer.
Both of these layers was added to the BBLAYERS in bblayers.conf file and the priorities that spesified with BBFILE_PRIORITY_* in related layer's layer.conf file is same.
So,
Which recipe will be used in build process in that case ?
What is the recipe lookup rules in yocto ?
I changed somethings to understand the behaviour:
For example,
I entered the invalid github URL that spesified in gstreamer1.0_%.bbappend recipe. When I tried to build the linux system, I encountered with an error. Thats fine.
Then I corrected the github URL in this recipe and entered invalid source code address that spesified in gstreamer1.0_1.16.1.bb recipe. When I tried to build linux system, process finished successfully.
Then I increased the priority of meta layer. I supposed to encounter with an error in this case but again build process finished successfully.
Could you please help me to understand this behaviour ?
Thanks.
You have two different files: a .bb and a .bbappend.
A .bb is the base recipe of one (or multiple) packages. It generally describe how to fetch, configure, compile, install files in a package for your target.
A .bbappend file is an 'append' file. It allows a meta (here meta-petalinux) to modify an existing recipe in another meta without copying it. A .bbappend can modify any steps of the bb file: source fetch, configure, compile, install...
You can for example create your own bbappend of Gstreamer, to enable pango (disbaled by default on my Yocto). The bbappend filename is gstreamer1.0-plugins-base_%.bbappend and only contains PACKAGECONFIG_append = "pango"
The Yocto Manual can give you more information on bbappend files here.
I'm a new hand and try to build a xilinx by pocto.
As the guide, I cloned repositories (branch thud), source oe-... and change MACHINE="zedborad-zynq7", then bitbake petalinux-image-minimal, but I get following error:
ERROR: tcf-agent-1.7.0+gitAUTOINC+dad3a6f568-r0 do_fetch: Fetcher
failure: Fetch command ...
https://git.eclipse.org/gitroot/tcf/org.eclipse.tcf.agent.git
refs/:refs/ failed with exit code 128, output: fatal: repository
'https://git.eclipse.org/gitroot/tcf/org.eclipse.tcf.agent.git/' not
found ... ERROR: Task
(~/poky/meta/recipes-devtools/tcf-agent/tcf-agent_git.bb:do_fetch)
failed with exit code '1'
The issue is that the statement in tcf-agent_git.bb:
SRC_URI = "git://git.eclipse.org/gitroot/tcf/org.eclipse.tcf.agent \
It is NOT the wrong address, In fact, I can clone successfully with this address. On the other hand, any my modify on this variable will NOT work either.
I already grep -rn "eclipse.org", but only find this file.
Any recommendation will be welcome.
Thanks lot.
——————————————————————————
I can't resolve this issue finally.
I find that the builder does NOT fetch from the address the SRC_URL offer at all, instead, it fetches from a mirror given somewhere.
As a test, I edited the .bb file, add PREMIRRORS="" and MIRROS="", and add protocal=git statement for the SRC_URI. The statements are effective realy, the builder fetches from the SRC_URL address, but the protocol is still HTTPS, the function still fails.
My solution is cloning the source manually, and putting it to corresponding directory, in order to let the builder know this, I also touch a package_name.done and chmod 777 in the same directory, then I can continue.
I've run into the exact same issue using Xilinx Yocto stack (rel-v2018.3 branch). For me, the problem wasn't in the tcf-agent_git.bb recipe in core/meta/recipes-devtools/tcf-agent, but in the tcf-agent_%.bbappend file in meta-petalinux/recipes-devtools/tcf-agent. In there, I replaced
SRC_URI = " \
git://git.eclipse.org/gitroot/tcf/org.eclipse.tcf.agent.git;branch=master;protocol=https \
file://fix_ranlib.patch;striplevel=2 \
file://ldflags.patch \
file://tcf-agent.init \
file://tcf-agent.service \
"
with
SRC_URI = " \
git://git.eclipse.org/gitroot/tcf/org.eclipse.tcf.agent.git;branch=master \
file://fix_ranlib.patch;striplevel=2 \
file://ldflags.patch \
file://tcf-agent.init \
file://tcf-agent.service \
"
and it finishes building correctly.
The former used to work fine last time I built the image (a few months ago) but for some reason the protocol=https option makes it fail now.
Your SRC_URI seems wrong.
it should be
SRC_URI = "git://git.eclipse.org/gitroot/tcf/org.eclipse.tcf.agent.git \
This one is working perfect for me.
Note : The back slash () at the end means you have multiple line SRC_URI. correct it if you have only single line.
In December 2021, using branch rel-v2020.1, I needed to change the line into :
SRC_URI = "git://git.eclipse.org/r/tcf/org.eclipse.tcf.agent.git;protocol=https \
I'm learning Yocto and to try it I'm following a very detailed example.
When I try to build the image (qt5.image) it returns a couple of errors after parsing recipes:
$bitbake qt5-image
...
ERROR: No recipes available for:
/home/dev/yocto/poky/meta-raspberrypi/recipes-bsp/u-boot/u-boot_2019.07.bbappend
/home/dev/yocto/poky/meta-raspberrypi/recipes-multimedia/gstreamer/gstreamer1.0-omx_1.16%.bbappend
The error is pretty clear, but because those meta- are not mine, but downloaded from the official repository git://git.yoctoproject.org/meta-raspberrypi I'm not sure why this error may happen.
Is there something I can fix by myself?
UPDATE:
Here the content of bblayer.conf I've adjusted the paths but it's the same file of the example linked above:
$ cat conf/bblayers.conf
# POKY_BBLAYERS_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
POKY_BBLAYERS_CONF_VERSION = "2"
BBPATH = "${TOPDIR}"
BBFILES ?= ""
BBLAYERS ?= " \
${HOME}/yocto/poky/meta \
${HOME}/yocto/poky/meta-poky \
${HOME}/yocto/poky/meta-openembedded/meta-oe \
${HOME}/yocto/poky/meta-openembedded/meta-multimedia \
${HOME}/yocto/poky/meta-openembedded/meta-networking \
${HOME}/yocto/poky/meta-openembedded/meta-perl \
${HOME}/yocto/poky/meta-openembedded/meta-python \
${HOME}/yocto/poky/meta-qt5 \
${HOME}/yocto/poky/meta-raspberrypi \
${HOME}/rpi/meta-rpi \
"
What the error message is telling you is that meta-raspberrypi layer is trying to modify recipes that it expects to exist in another layer. In this case the u-boot and gstreamer1.0-omx recipes come from "meta" (so the yocto/openembedded main layer).
You should check that your layers are all using the same branches (and are all up-to-date). I'm guessing that your meta layer (typically from poky git) is on some release branch and your meta-raspberrypi is using master. In that case the build would fail like yours does (e.g. gstreamer1.0-omx would be version 1.14.4 and the bbappend would look for another version).
I had the same issue, what I did is to checkout meta-raspberrypi with branch corresponding to the poky branch:
git clone -b warrior git://git.yoctoproject.org/poky
cd poky
git clone -b warrior git://git.yoctoproject.org/meta-raspberrypi
source oe-init-build-env
Add layer to conf/bblayers.conf
Change MACHINE in conf/local.conf
bitbake rpi-basic-image
Can you make sure 'meta-raspberrypi' is included in bblayers.conf file. You can check this layer included in
build-*/conf/bblayers.conf
If it is not included, add this layer into bbpath in bblayers.conf and rebuild it.
If it's still available could you please send the complete logs for the error?
How do I use PREMIRRORS in Bitbake local configurations or recipes?
I want to provide my own download locations for some slow or inaccessible third-party URLs, but the official PREMIRRORS documentation is vague and lacks examples.
Note: These results are based on experimentation with Yocto 2.3, but probably apply to 2.5 as well.
A simple example
Suppose that your recipe file contains this target URL:
SRC_URI = "http://download.example.com:8080/foo/bar/baz-1.0.tar.gz"
Then in your local.conf, you can define your custom download location as:
PREMIRRORS_prepend = "http://download\.example\.com:8080/.* http://my-mirror.example.com/copies/\n"
In this default case (with no special placeholders) Bitbake does not include the additional /foo/bar path elements, and instead tries to download just the filename from http://my-mirror.example.com/copies/baz-1.0.tar.gz
Advanced examples
These samples use special predefined placeholders, which are detailed in the next section.
HTTP/HTTPS with same file structure
Recipe: SRC_URI = "https://example.com:1234/foo/bar.zip"
Setting: PREMIRRORS_prepend = "http(s)?://example\.com(:\d+)?/.* http://mirror.local/PATH\n"
Attempts: http://mirror.local/foo/bar.zip
HTTP/HTTPS with flat structure
Recipe: SRC_URI = "https://example.com:1234/foo/bar.zip"
Setting: PREMIRRORS_prepend = "http(s)?://example\.com(:\d+)?/.* http://mirror.local/MIRRORNAME\n"
Attempts: http://mirror.local/example.com.1234.foo.bar.zip
Just switch the hostname
Recipe: SRC_URI = "ftp://example.com:1234/foo/bar.zip"
Setting: PREMIRRORS_prepend = "(\w+)://example\.com(:\d+)?/.* TYPE://mirror.local/PATH\n"
Attempts: ftp://mirrors.local/foo/bar.zip
Placeholders in replacement URI
PREMIRRORS parses all matched URIs and provides five special placeholder values in the target URI. Supposing the matched URI is http://host.example.com:1234/foo/bar/baz.txt:
TYPE https
HOST host.example.com%3A1234
PATH foo/bar/baz.txt
BASENAME baz.txt
MIRRORNAME host.example.com.1234.foo.bar.baz.txt
Altering the PREMIRRORS variable
The PREMIRRORS variable consists of series of lines (separated by \n) each with a regular expression to match a URI, and then a replacement string, with both portions separated by a space.
Bitbake tries them in order of appearance, and you generally want your private mirrors to take priority, so prepend onto PREMIRRORS, ex:
PREMIRRORS_prepend = "http://original/location/.* http://alternate/location/\n"
What file should I edit?
You can add entries to PREMIRRORS inside your bitbake recipes, but it is not recommended, since a major use of PREMIRRORS is for people reusing your recipe in some other context or location.
Instead, you can put it inside your local.conf file in an existing build directory. Alternately, edit the source template which Poky script use when creating a new local.conf in a fresh build-directory.
Other questions
What about SOURCE_MIRROR_URL?
The SOURCE_MIRROR_URL is a quick way to add a series of PREMIRROR entries for all supported protocols. For example, this setting:
INHERIT += "own-mirrors"
SOURCE_MIRROR_URL = "TYPE://mirror.local/PATH"
is the same as writing:
PREMIRRORS_prepend = "\
cvs://.*/.* TYPE://mirror.local/PATH \
svn://.*/.* TYPE://mirror.local/PATH \
git://.*/.* TYPE://mirror.local/PATH \
gitsm://.*/.* TYPE://mirror.local/PATH \
hg://.*/.* TYPE://mirror.local/PATH \
bzr://.*/.* TYPE://mirror.local/PATH \
p4://.*/.* TYPE://mirror.local/PATH \
osc://.*/.* TYPE://mirror.local/PATH \
https?$://.*/.* TYPE://mirror.local/PATH \
ftp://.*/.* TYPE://mirror.local/PATH \
npm://.*/?.* TYPE://mirror.local/PATH \
"
It seems the INHERIT+SOURCE_MIRROR_URL directives will still work if used in your local.conf (as opposed to a particular recipe.) However, Bitbake will emit warnings, so it may not be the intended use-case. Ex:
WARNING: Invalid protocol in PREMIRRORS: ('cvs://.*/.*', 'TYPE://mirror.local/PATH')
How can I check and debug my settings?
The -D debug flag will cause bitbake to emit information about what URLs it attempts to download from. You can also use -C do_fetch, which will force it to try the fetch step and re-download anything needed for the given recipe.
bitbake -D -C do_fetch software-recipe-name-here
Here's some example debug output, showing the PREMIRROR URL it attempts to access:
DEBUG: some-software-1.0 do_fetch: Trying PREMIRRORS
DEBUG: some-software-1.0 do_fetch: Fetcher accessed the network with the command /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /home/user/build_foo/DL_DIR 'http://mirror.local/path/to/the/filename.ext
If you need to experiment and run bitbake many times, it will be faster to temporarily put your new PREMIRRORS_prepend directive into a particular test-recipe, as opposed to modifying the local.conf. This is because Bitbake won't need to re-parse all the other recipes whenever you change it.
What if I want to isolate a port-number, e.g. http://host:123/foo?
Apparently there's no easy way to get the 123 on its own. While PREMIRRORS allows you to match with regular expressions, it does not seem to support using captured text from the match inside the replacement URI.
The port number is present inside HOST and MIRRORNAME, but there's no standard mechanism to split those values apart.
Fllow up. What if I want to isolate a port-number, e.g. http://host:123/foo?
Maybe you can use captured.
like:
org: "http://somewhere.org:1234/somedir1/somedir2/somefile_1.2.3.tar.gz"
reg: "http://somewhere.org(:\d+)?/.*"
sub: "http://somewhere2.org\1/somedir3"
result: "http://somewhere2.org:1234/somedir3/somefile_1.2.3.tar.gz"
The manual here says that there is a basepath option to SRC_URI that should "strip the specified directories from the source path when unpacking".
I'm trying to fetch the sources from a local directory, say /src/someproject.
For that purpose I configured my recipe as follows:
SRC_URI="file:///src/someproject;subdir=source;basepath=/src/someproject"
The intention was to have the sources taken from /src/someproject directory and put into build/tmp/work/target/someproject/1.0-r1/source/. Instead, I'm getting the sources under build/tmp/work/target/someproject/1.0-r1/source/src/someproject.
Is there a way to get rid of /src/someproject subdirectory inside source ?
The documentation you point to is for yocto 1.6, release April 2014. The basepath parameter appears to have been removed in later releases, with no fanfare and no apparent replacement.
Instead you can do something like:
SRCDIR = "/path/to/your/files"
SRC_URI = "file://${SRCDIR}/contents/;subdir=src"
S = "${WORKDIR}/src"
Then you can access your files under ${S}/${SRCDIR}.
If you find using the ${SRCDIR} part too cumbersome, you can hook onto the do_patch target, which need the prefunc mechanism and not _prepend if you want to use shell scripting, as do_patch is otherwise written in python:
relocate_source() {
mv ${S}/${SRCDIR}/* ${S}/
rmdir ${S}/${SRCDIR}
}
do_patch[prefunc] += "relocate_source"
This will reorganise your source before applying any patch you can add to SRC_URI.
Also note that a file:// URI does not get cached in ${DL_DIR}, so there is no name-conflict to handle (the way we would need to use downloadfilename= in an http:// URI).