I would like to list all files that bitbake will fetch when I bake an image.
Currently, I am able to get the SRC_URI of all files needed to bake a Yocto image by executing bitbake core-image-minimal -c fetchall and then parse log files.
Is there is a simpler way to get the same result without the need to download files ?
I am not sure bitbake supports such feature. Ideally, I am looking for a command that prints out the package name and lists all files with corresponding URL
> bitbake core-image-minimal -c fetchall --print-only
Generally bitbake doesn't provides such functionality.
But I was able to create a simple solution witch creating simple .bbclass file which is inherited in all recipes, by adding it into local.conf file, please see my steps in order to archive that:
Steps:
let's create a class print-src.bbclass file used to get and print SRC_URI variable (remember to store this class file in layer which is available in conf/bblayers.conf):
$ cat print-src.bbclass
python do_print_src () {
# should probably be indented
srcuri = d.getVar('SRC_URI', True).split()
bb.warn("SRC_URI look like: %s" % srcuri)
}
addtask do_print_src before do_fetch
Add INHERIT += "print-src" into Your conf/local.conf file
Edit: it is important to use bitbake --runonly option, that allows to run specific task of the taskgraph for the specified target (with --runonly option do_print_src needs to be used as print_src),
Edit: Please note that --runall=RUNALL and --runonly=RUNONLY was introduced with Yocto Sumo release 2.5,
$ bitbake core-image-minimal --runonly print_src
Loaded 1236 entries from dependency cache.
NOTE: Resolving any missing task queue dependencies
Build Configuration:
BB_VERSION = "1.37.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "universal-4.8"
TARGET_SYS = "i586-poky-linux"
MACHINE = "qemux86"
DISTRO = "poky"
DISTRO_VERSION = "2.5"
TUNE_FEATURES = "m32 i586"
TARGET_FPU = ""
meta
meta-poky
meta-yocto-bsp = "master:13cc30cd7de4841990b600e83e1249c81a5171dd"
Initialising tasks: 100% |##########################################################################################################################################################################| Time: 0:00:00
NOTE: Executing RunQueue Tasks
WARNING: ptest-runner-2.2+gitAUTOINC+49956f65bb-r0 do_print_src: SRC_URI look like: ['git://git.yoctoproject.org/ptest-runner2']
WARNING: grep-3.1-r0 do_print_src: SRC_URI look like: ['http://ftp.gnu.org/gnu/grep/grep-3.1.tar.xz', 'file://0001-Unset-need_charset_alias-when-building-for-musl.patch']
...
...
NOTE: Tasks Summary: Attempted 201 tasks of which 0 didn't need to be rerun and all succeeded.
Summary: There were 202 WARNING messages shown.
Please see sample warning output log line:
WARNING: ptest-runner-2.2+gitAUTOINC+49956f65bb-r0 do_print_src: SRC_URI look like: ['git://git.yoctoproject.org/ptest-runner2'].
I patched poky to create *.src files in downloads that contains the effective fetch URL of the package.
bitbake/lib/bb/fetch2/__init__.py | 19 ++++++++++++++-----
1 file changed, 14 insertions(+), 5 deletions(-)
diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index b853da30bd..03e84e0919 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
## -1257,16 +1257,16 ## class FetchData(object):
# Note: .done and .lock files should always be in DL_DIR whereas localpath may not be.
if self.localpath and self.localpath.startswith(dldir):
- basepath = self.localpath
+ self.basepath = self.localpath
elif self.localpath:
- basepath = dldir + os.sep + os.path.basename(self.localpath)
+ self.basepath = dldir + os.sep + os.path.basename(self.localpath)
elif self.basepath or self.basename:
- basepath = dldir + os.sep + (self.basepath or self.basename)
+ self.basepath = dldir + os.sep + (self.basepath or self.basename)
else:
bb.fatal("Can't determine lock path for url %s" % url)
- self.donestamp = basepath + '.done'
- self.lockfile = basepath + '.lock'
+ self.donestamp = self.basepath + '.done'
+ self.lockfile = self.basepath + '.lock'
def setup_revisions(self, d):
self.revisions = {}
## -1607,6 +1607,15 ## class Fetch(object):
m = ud.method
localpath = ""
+ p = "%s.src"%ud.basepath
+ d = os.path.dirname(p)
+ if d != '':
+ bb.utils.mkdirhier(d)
+ with open(p, 'wb') as f:
+ data = "%s" % ud.url
+ f.write(bytes(data, 'ASCII'))
+ return True
+
if ud.lockfile:
lf = bb.utils.lockfile(ud.lockfile)
Running bitbake core-image-minimal -c fetchall results:
$> find downloads/ -name '*.src' | head -n 5
downloads/lzop-1.03.tar.gz.src
downloads/libtheora-1.1.1.tar.bz2.src
downloads/mpfr-3.1.5.tar.xz.src
downloads/makedevs.c.src
downloads/expat-2.2.0.tar.bz2.src
This is not an optimal solution but I hope that such feature gets its way to mainline stream.
I needed something like this and got part way there before now.
I can generate a messy list of URIs used by executing the following command:
bitbake -g zlib && cat recipe-depends.dot | \
grep -v -e '-native' | grep -v digraph | \
grep -v -e '-image' | awk '{print $1}' | \
sort | uniq | xargs -I {} -t bitbake -e {} | grep SRC_URI=
This gives you all the URIs and files used in a recipe and some comments.
Not a perfect solution but I will see if I can improve on it.
A bit manual and straight forward way is to build your image from scratch, and copy all the fetch logs, and go thru it.
$ mkdir fetch_logs
$ find tmp/work/ -type f -name log.do_fetch* | cpio -pdm fetch_logs
Now grep "Fetch(ing/er)" in this fetch_logs and it will provide info on which URL your recipe used to fetch the source. This will be helpful to see the final URI used to fetch in PREMIRROR based builds.
NOTE: if you have sstate-cache enabled and functional, you may not see do_fetch at all.
I wrote scripts based on the solution in the question.
Basically I need to make a cscope project of all the sources, and hence I need to list all URI and clone all repos.
I wrote 2 scripts 1) List all SRC_URI and Branches
2) Install(clone) all the code in a single directory
listURI.sh
#!/bin/bash
TMP_FILE="___x__2242230p.txt"
SRCH_FILE="log.do_fetch"
TIME=$(date +%Y-%m-%d-%H-%M)
OUT_FILE="Project-$TIME.txt"
DEF_BRANCH="master"
BRANCH=""
SRC_URI=""
function usage {
echo "Usage : $0 < -f | -g > <Component-List-file>"
echo "Options :"
echo " -f : Run bitbake fetch and collect URI"
echo " -g : Get URI from Already fetched Project"
}
if [ $# -eq 0 ]
then
usage
exit
fi
if [ "$1" == "-f" ] || [ "$1" == "-g" ];then
if [ "$1" == "-f" ] && [ -z "$2" ];then
echo "Should pass Project-Name after -f Option"
exit
fi
else
echo "Unknown Option"
usage
exit
fi
POKYROOTDIR=`basename "$PWD"`
#echo $POKYROOTDIR
if [[ $POKYROOTDIR != "build" ]];then
echo "You are not on poky/build (Sourced Poky) Directory"
exit 0
fi
POKYROOTDIR=$PWD
if [ "$1" == "-f" ];then
echo "Running === bitbake -c fetchall $2 -f --no-setscene =="
bitbake -c fetchall $2 -f --no-setscene
fi
if [ "$1" == "-g" ];then
if [ ! -d tmp/work ];then
echo " Please run == bitbake -c fetchall <image> -f --no-setscene == before this"
usage
exit
fi
fi
echo "Looking for URIs --- It may take couple of minuites!"
rm -rf $OUT_FILE
cd tmp/work
find . -name $SRCH_FILE -exec grep -i 'DEBUG: For url git' {} \; -print > $POKYROOTDIR/$TMP_FILE
cd $POKYROOTDIR
while IFS= read -r line
do
#echo "$line"
BRANCH=$(echo $line | awk -F"branch=" '/branch=/{print $2}' | sed -e 's/;/ /' | awk '{print $1}')
SRC_URI=$(echo $line | cut -d';' -f1-1 | awk -F"url" '/url/{print $2}' | awk '{print $1}' | sed -e 's/git:/ssh:/')
if [ ! -z "$BRANCH" ] && [ ! -z "$SRC_URI" ];then
echo "$BRANCH $SRC_URI" >> $OUT_FILE
elif [ ! -z "$SRC_URI" ];then
echo "$DEF_BRANCH $SRC_URI" >> $OUT_FILE
fi
if [ ! -z "$BRANCH" ];then
echo "Found URI : BRANCH:$BRANCH URI:$SRC_URI"
elif [ ! -z "$SRC_URI" ];then
echo "Found URI : BRANCH:Default URI:$SRC_URI"
fi
#echo "$BRANCH $SRC_URI" >> $OUT_FILE
done < "$TMP_FILE"
#echo "=================== OUT PUT ==========================="
#cat $OUT_FILE
rm -rf $TMP_FILE
echo "----------------INFO-----------------"
echo "Project Creation: Success"
echo "Project Path : $POKYROOTDIR"
echo "Project file : $OUT_FILE"
echo "-------------------------------------"
installURI.sh
#!/bin/bash
function usage {
echo "Usage : $0 <List-file>"
}
if [ $# -eq 0 ]
then
usage
exit
fi
if [ -z "$1" ];then
usage
exit
fi
if [ ! -z "$(ls -A $PWD)" ]; then
echo "Directory ($PWD) must be Empty.... else it will mess up project creation"
exit
fi
if [ ! -f $1 ];then
echo "list file ($1) not found"
usage
exit
fi
filename=$1
while read -r line; do
name="$line"
echo "Fetching Projects"
git clone -b $line
done < "$filename"
echo "----------------INFO-----------------"
echo "Project Creation: Success"
echo "Project Path : $PWD"
echo "-------------------------------------"
I have a directory with projects and I create a function to cd to this directory.
projects_folder="/opt/work/projects/"
function pr () {
cd $projects_folder
if [ ! -z $1 ]; then
cd $1
fi
}
I want to create zsh autocompletion for this function
compdef '_path_files -W /opt/work/projects -/ && return 0 || return 1' pr
Also I create custom complete function
_pr(){
local -a list
for dir in $(ls $projects_folder); do
inside_dir=$projects_folder$dir
if [ -d $inside_dir ]; then
list=( $list $dir )
for d in $(ls $inside_dir); do
double_dir=$inside_dir/$d
if [ -d $double_dir ]; then
list=( $list $dir/$d )
fi
done
fi
done
compadd -a $list
}
compdef _pr pr
But both of these variants don`t work. Can somebody explain me that is wrong? My .zshrc here
UPD:
Solution is delete antigen apply from .zshrc, regarding this issue
I have a very large project where we have made changes to lots of files. Now we need to exactly know which line numbers we have changed/added so that we can run some other tool on that particular line only.
While searching for the answer i found this as an answer to some other question..
echo ${f}:
for n in $(git --no-pager blame --line-porcelain $1 |
awk '/author Not Committed Yet/{if (a && a !~ /author Not Committed Yet/) print a} {a=$0}' |
awk '{print $3}') ; do
if (( prev_line > -1 )) ; then
if (( "$n" > (prev_line + 1) )) ; then
if (( (prev_line - range_start) > 1 )) ; then
echo -n "$range_start-$prev_line,"
else
echo -n "$range_start,$prev_line,"
fi
range_start=$n
fi
else
range_start=$n
fi
prev_line=$n
done
if (( "$range_start" != "$prev_line" )) ; then
echo "$range_start-$prev_line"
else
echo "$range_start"
fi
And it ends up looking like this:
views.py:
403,404,533-538,546-548,550-552,554-559,565-567,580-582
It does great.. i need the exact same output using cvs.. Is there any way to get similar output..
In my ZSH prompt I want to be able to output the current history event number (%! or %h) -1
If current history event number is !256, I want to subtract it by one and output the result in my prompt (i.e. !255).
Here's the way it looks now and how I'd like it to be:
Below is my current ZSH theme (and the code pertaining to this question lies in the previous_history_event_number () function, which is triggered from the return_code_enabled= declaration:
# ------------------------------------------------------------------------------
# FILE: hced.zsh-theme
# DESCRIPTION: oh-my-zsh theme file.
# (Credits to authors of blinks.zsh-theme and dieter.zsh-theme
# from which themes I've taken useful bits.)
# AUTHOR: hced
# VERSION: 0.0.1
# SCREENSHOT:
# ------------------------------------------------------------------------------
function _prompt_char() {
if $(git rev-parse --is-inside-work-tree >/dev/null 2>&1); then
echo "%{%F{blue}%}±%{%f%k%b%}"
else
echo ' '
fi
}
ZSH_THEME_GIT_PROMPT_PREFIX=" [%{%B%F{blue}%}"
ZSH_THEME_GIT_PROMPT_SUFFIX="%{%f%k%b%K{black}%B%F{green}%}]"
ZSH_THEME_GIT_PROMPT_DIRTY=" %{%F{red}%}*%{%f%k%b%}"
ZSH_THEME_GIT_PROMPT_CLEAN=""
PROMPT='%{%f%k%b%}
%{%K{black}%B%F{green}%}%n%{%B%F{blue}%}#%{%B%F{cyan}%}%m%{%B%F{green}%} %{%b%F{yellow}%K{black}%}%~%{%B%F{green}%}$(git_prompt_info)%E%{%f%k%b%}
%{%K{black}%}$(_prompt_char)%{%K{black}%} %#%{%f%k%b%} '
#RPROMPT='!%{%B%F{cyan}%}%!%{%f%k%b%}'
function previous_history_event_number () {
prev_hist_num=("%!"-1)
declare -i prev_hist_num
echo "$prev_hist_num gave exit code: "
}
# elaborate exitcode on the right when >0
return_code_enabled="%(?..%{$fg[red]%}$(previous_history_event_number)%?%{$reset_color%})"
return_code_disabled=
return_code=$return_code_enabled
RPS1='${return_code} !%{%B%F{cyan}%}%!%{%f%k%b%}'
function accept-line-or-clear-warning () {
if [[ -z $BUFFER ]]; then
time=$time_disabled
return_code=$return_code_disabled
else
time=$time_enabled
return_code=$return_code_enabled
fi
zle accept-line
}
zle -N accept-line-or-clear-warning
bindkey '^M' accept-line-or-clear-warning
Disclaimer: I'm not a programmer so the code within the previous_history_event_number () function is pretty (vastly) clueless right now.
If current history event number is shown when you add %! atom in a prompt then you can get it into a variable by using (%) modifier:
V='%!'
integer HISTORY_EVENT_NUMBER=${(%)V}-1
I need to execute a script on a remote server over ssh, I can't locate the script as a file on the remote server nor create files during the script process.
The script checks for a non existent or zero byte file, and if exists, checks if is outdated.
I've followed a thread here on SO and tried this:
myvar=$(ssh user#server <<EOF
myfile=/mnt/file.csv
if [ -s $myfile ]; then
filedate=$(stat -c %x $myfile|grep '[0-9\-]*' --max-count=1 -o);
yesterday=$(date --date 'now -1 day' --iso-8601);
if [ $filedate < $yesterday ]; then
echo '1 '$yesterday;
else
echo '0 ok';
fi
else
echo $(date --iso-8601);
fi
EOF
)
sadly, the pipe appears to be truncating the string or something, because the script returns
stat: too few arguments
maybe just cannot use "myfile" var declaration. Any suggestions?
Thanks in advance.
---- Edit: Clarifying answer:
keber-laptop:~ keberflores$ echo $myvar
keber-laptop:~ keberflores$ myvar=$(ssh user#server <<EOF
> myfile=/mnt/file.csv
> if [ -s \$myfile ]; then
> filedate=\$(stat -c %x \$myfile|grep '[0-9\-]*' --max-count=1 -o);
> yesterday=\$(date --date 'now -1 day' --iso-8601);
> if [ \$filedate < \$yesterday ]; then
> echo '1 '\$yesterday;
> else
> echo '0 ok';
> fi
> else
> echo '1 '\$(date --iso-8601);
> fi
> EOF
> )
Pseudo-terminal will not be allocated because stdin is not a terminal.
user#server's password:
keber-laptop:~ keberflores$ echo $?
0
keber-laptop:~ keberflores$ echo $myvar
1 2011-10-22
---- Edit: calling inside perl:
my $myvar = qx'ssh user#server <<\'EOF\'
myfile=/mnt/file.csv
if [ -s $myfile ]; then
filedate=$(stat -c %x $myfile|grep \'[0-9\-]*\' --max-count=1 -o);
yesterday=$(date --date \'now -1 day\' --iso-8601);
if [ $filedate < $yesterday ]; then
echo \'1 \'$yesterday;
else
echo \'0 ok\';
fi
else
echo \'1 \'$(date --iso-8601);
fi
EOF
';
print $myvar;
You need more escaping to get this to work. bash is going to evaluate the variables, etc in the here-file locally, then the result of that will be evaluated on the remote server.
In particular, your $myfile is probably evaluating to an empty string when the here-file is being evaluated, causing the stat to not have a file argument.