Completion for filenames without root, only extension (hidden files) - fish

I'm trying to write a completion for a command that takes filenames that have no root/base part, only extension (such as "foo/.ext"). Hidden files. (Specifically, ".envrc" for direnv.)
There are several examples to follow in fish's installation that use
__fish_complete_suffix for files that have a root and extension, such as fish itself:
complete -c fish -k -x -a "(__fish_complete_suffix .fish)"
The same thing for hidden files, files without a root, doesn't work.
# completions/foo.fish
complete -c foo -k -x -a "(__fish_complete_suffix .bar)"
# functions/foo.fish
function foo
echo $argv
end
> mkdir -p a/b; touch a/1.bar a/2.bar a/.bar
> foo a/<TAB>
a/1.bar a/2.bar a/b/
Is there a way to accomplish this with this or some other function(s)?

fish doesn't add hidden files to the list of completions unless the token being completed already starts with a '.'.
You should be able to write a custom completion that does what you want, but unfortunately a long-standing bug makes that impossible.

I spent way too much time on this, it may be overly complex, and probably needs to be optimized and simplified, but it does work in almost every case. The one case that doesn't is the outstanding bug #Zanchey mentioned.
# completions/foo.fish
complete -c foo -k -x -a "(__fish_complete_dotfile_or_subdirectories .bar)"
function __fish_complete_dotfile_or_subdirectories -d "Complete dotfile or directories"
set -l token
set -l suffix
switch (count $argv)
case 1
set token (commandline -ct)
set suffix $argv
case 2
set token $argv[1]
set suffix $argv[2]
end
set -l suffix_len (string length $suffix)
# trim leading . from suffix if present
if test (string sub -l 1 $suffix) = "."
set suffix_len (math $suffix_len - 1)
set suffix (string sub -l $suffix_len -s "-"$suffix_len $suffix)
end
set -l root_name (string split -rm 1 "/" $token)
set -l root $root_name[1..-2]"/"
set -l name $root_name[-1]
# build regex
# $suffix = foo -> '^(?:|\.|\.f|\.fo|\.foo)$'
# $suffix = bar -> '^(?:|\.|\.b|\.ba|\.bar)$'
set -l re (
echo -n "^(?:|\."
for i in (seq $suffix_len)
echo -n "|\."(string sub -l $i $suffix)
end
echo ')$'
)
if string match -qr $re "$name"
set -l i
if test -n "$root"
set i (printf "%s.%s" $root $suffix)
else
set i (printf ".%s" $suffix)
end
if test -f "$i"
printf "%s\n" $i
end
end
# all subdirs except . and ..
for i in (complete -C "nonexistentcommand $token" |
string match -er '[^\.]/$')
printf "%s\n" $i
end
end
* (complete -C ... line inspiration)
> mkdir -p a/{.a1,.a2,b1,b2,.b3,.b4} c
> touch .bar .baz a/.bar a/.baz a/foo.bar c/.bar
> tree -aF
.
├── a/
│   ├── .a1/
│   ├── .a2/
│   ├── b1/
│   ├── b2/
│   ├── .b3/
│   ├── .b4/
│   ├── .bar
│   ├── .baz
│   └── foo.bar
├── .bar
├── .baz
└── c/
└── .bar
> foo <TAB>
> __fish_complete_dotfile_or_subdirectories "" bar
.bar # .bar is included ...
a/
c/
> foo <TAB>
a/ c/ # ... but is not here because of the long-standing bug
> foo .<TAB> # completes to `foo .bar`
> foo ./<TAB>
./.bar ./a/ ./c/ # now it's included
> foo ./.<TAB> # completes to `foo ./.bar`
> foo a<TAB> # completes to `foo a/`
> foo a/<TAB>
a/.bar a/b1/ a/b2/
> foo a/.<TAB>
a/.bar a/.a1/ a/.a2/ a/.b3/ a/.b4/
> foo a/.b<TAB>
a/.bar a/.b3/ a/.b4/
> foo a/.ba<TAB> # completes to `foo a/.bar`
> foo c<TAB> # completes to `foo c/`
> foo c/<TAB> # completes to `foo c/.bar`

Related

Perl abs_path undefined return using File::Find::dir value from relative directory

A Perl script running on macOS takes a directory argument and then calls the File::Find &wanted() function to process the contents.
If I supply the directory as an absolute path (e.g. "/Users/sw/Downloads/FGT") I have no problems.
But if I supply a relative path (i.e. "." when run from above directory) I get an error.
The error happens when processing a file in a subdirectory: it is always the first file in the particular subdirectory (the first second level subdirectory) that generates the error (if I create or touch a new file with an alphabetically earlier name than any existing one in the directory, it is this new file that causes the error). Many earlier files are processed OK.
Complete simplified code below, with output.
So why does the below "abs_path" call give an undefined result for this one particular file (first file in first second level subdirectory)?
COMPLETE CODE:
#!/usr/local/bin/perl
use warnings;
use English;
use vars qw ( $VERSION );
use Cwd 'abs_path';
use File::Find;
find({ wanted => \&processFile, follow => 0 }, $ARGV[0]); #we do NOT follow symbolic links
exit();
sub processFile
{
if (substr($_,0,1) eq ".") { return; } #ignore entry starting with dot
print "\nPROCESSING WITH:\n";
print ">> \$File::Find::dir = <<$File::Find::dir>>\n";
print ">> \$_ = <<$_>>\n";
print ">> \$File::Find::name=<<$File::Find::name>>\n";
my $dirAbsPath=abs_path($File::Find::dir);
die "ERROR: undef abs_path() using Find::File::dir value!\n" if ! defined $dirAbsPath;
print "Dir Abs Path is <<$dirAbsPath>>\n";
}
OUTPUT:
PROCESSING WITH:
>> $File::Find::dir = <<.>>
>> $_ = <<L1DIR>>
>> $File::Find::name=<<./L1DIR>>
Dir Abs Path is <</Users/sw/Downloads/FGT>>
PROCESSING WITH:
>> $File::Find::dir = <<./L1DIR>>
>> $_ = <<LPT1.csv>>
>> $File::Find::name=<<./L1DIR/LPT1.csv>>
Dir Abs Path is <</Users/sw/Downloads/FGT/L1DIR/L1DIR>>
PROCESSING WITH:
>> $File::Find::dir = <<./L1DIR>>
>> $_ = <<L2DIR2>>
>> $File::Find::name=<<./L1DIR/L2DIR2>>
Dir Abs Path is <</Users/sw/Downloads/FGT/L1DIR/L1DIR>>
PROCESSING WITH:
>> $File::Find::dir = <<./L1DIR/L2DIR2>>
>> $_ = <<abc>>
>> $File::Find::name=<<./L1DIR/L2DIR2/abc>>
ERROR: undef abs_path() using Find::File::dir value!
DIRECTORY TREE OF ".":
sw#Max FGT % tree .
.
└── L1DIR
├── L2DIR1
│   └── LD.csv
├── L2DIR2
│   ├── abc
│   └── x.csv
├── L2DIR3
│   └── xsym2 -> T2/x.csv
└── LPT1.csv
$File::Find::dir is relative to the original CWD. But File::Find changes the CWD unless you use no_chdir => 1.
When processing ./L1DIR/L2DIR2/abc with no_chdir => 0 (default),
The CWD is $orig_cwd/./L1DIR/L2DIR2.
$File::Find::name is ./L1DIR/L2DIR2/abc.
$File::Find::dir is ./L1DIR/L2DIR2.
$_ is abc.
(Where $orig_cwd is the absolute path of . before you call find.)
So that means that abs_path( $File::Find::dir ) is the absolute path of $orig_cwd/./L1DIR/L2DIR2/./L1DIR/L2DIR2, which obviously fails.
Using abs_path(".") would work.
When processing ./L1DIR/L2DIR2/abc with no_chdir => 1,
The CWD is $orig_cwd.
$File::Find::name is ./L1DIR/L2DIR2/abc.
$File::Find::dir is ./L1DIR/L2DIR2.
$_ is ./L1DIR/L2DIR2/abc.
So that means that abs_path( $File::Find::dir ) is the absolute path of $orig_cwd/./L1DIR/L2DIR2, which is what you want.
I prefer to avoid all those needless chdir, I prefer using no_chdir => 1 at all times.

How do I make the autosuggestions of a zsh function use syntax highlighting

I use zsh and wrote a function to replace cd function. With some help I got it to work like I want it to (mostly). This is a followup to one of my other question.
The function almost works like I want it to, but I still have some problems with syntax highlighting and autocompletion.
For the examples, lets say your directories look like this:
/
a/
b/
c/
d/
some_dir/
I am also assuming the following code has been sourced:
cl () {
local first=$( echo $1 | cut -d/ -f1 )
if [ -d $first ]; then
pushd $1 >/dev/null # If the first argument is an existing normal directory, move there
else
pushd ${PWD%/$first/*}/$1 >/dev/null # Otherwise, move to a parent directory or a child of that parent directory
fi
}
_cl() {
_cd
pth=${words[2]}
opts=""
new=${pth##*/}
local expl
# Generate the visual formatting and store it in `$expl`
_description -V ancestor-directories expl 'ancestor directories'
[[ "$pth" != *"/"*"/"* ]] && middle="" || middle="${${pth%/*}#*/}/"
if [[ "$pth" != *"/"* ]]; then
# If this is the start of the path
# In this case we should also show the parent directories
local ancestor=$PWD:h
while (( $#ancestor > 1 )); do
# -f: Treat this as a file (incl. dirs), so you get proper highlighting.
# -Q: Don't quote (escape) any of the characters.
# -W: Specify the parent of the dir we're adding.
# ${ancestor:h}: The parent ("head") of $ancestor.
# ${ancestor:t}: The short name ("tail") of $ancestor.
compadd "$expl[#]" -fQ -W "${ancestor:h}/" - "${ancestor:t}"
# Move on to the next parent.
ancestor=$ancestor:h
done
else
# $first is the first part of the path the user typed in.
# it it is part of the current direoctory, we know the user is trying to go back to a directory
first=${pth%%/*}
# $middle is the rest of the provided path
if [ ! -d $first ]; then
# path starts with parent directory
dir=${PWD%/$first/*}/$first
first=$first/
# List all sub directories of the $dir/$middle directory
if [ -d "$dir/$middle" ]; then
for d in $(ls -a $dir/$middle); do
if [ -d $dir/$middle/$d ] && [[ "$d" != "." ]] && [[ "$d" != ".." ]]; then
compadd "$expl[#]" -fQ -W $dir/ - $first$middle$d
fi
done
fi
fi
fi
}
compdef _cl cl
The problem:
I use syntax-highlighting, but the path i type is just white (when going to a parent directory. the normal cd functions are colored).
Example:
$ cd /a
$ cl c # 'c' is colored
$ pwd
/a/c
$ cl a/b # 'a/b' is not colored
$ cl a/[tab] # 'a/b', 'a/c' and 'a/some_dir' are not colored
How do I get these paths to be colored?
That's not possible out of the box with the zsh-syntax-highlighting plugin. It checks only whether what you've typed is
A) a valid absolute path or
B) a valid path relative to the present working dir.
This is not specific to your command. Highlighting also fails when specifying path arguments to other commands that are otherwise valid, but are not absolute paths or valid paths relative to the present working dir.
For example:
% cd /a/b
% cd b c # perfectly valid args, but will not get highlighted as valid paths
% pwd
/a/c

How do I make a zsh function autocomplet from the middle of a word?

I use zsh and wrote a function to replace cd function. With some help I got it to work like I want it to (mostly). This is a followup to one of my other question.
The function almost works like I want it to, but I still have some problems with syntax highlighting and autocompletion.
For the examples, lets say your directories look like this:
/
a/
b/
c/
d/
some_dir/
I am also assuming the following code has been sourced:
cl () {
local first=$( echo $1 | cut -d/ -f1 )
if [ -d $first ]; then
pushd $1 >/dev/null # If the first argument is an existing normal directory, move there
else
pushd ${PWD%/$first/*}/$1 >/dev/null # Otherwise, move to a parent directory or a child of that parent directory
fi
}
_cl() {
_cd
pth=${words[2]}
opts=""
new=${pth##*/}
local expl
# Generate the visual formatting and store it in `$expl`
_description -V ancestor-directories expl 'ancestor directories'
[[ "$pth" != *"/"*"/"* ]] && middle="" || middle="${${pth%/*}#*/}/"
if [[ "$pth" != *"/"* ]]; then
# If this is the start of the path
# In this case we should also show the parent directories
local ancestor=$PWD:h
while (( $#ancestor > 1 )); do
# -f: Treat this as a file (incl. dirs), so you get proper highlighting.
# -Q: Don't quote (escape) any of the characters.
# -W: Specify the parent of the dir we're adding.
# ${ancestor:h}: The parent ("head") of $ancestor.
# ${ancestor:t}: The short name ("tail") of $ancestor.
compadd "$expl[#]" -fQ -W "${ancestor:h}/" - "${ancestor:t}"
# Move on to the next parent.
ancestor=$ancestor:h
done
else
# $first is the first part of the path the user typed in.
# it it is part of the current direoctory, we know the user is trying to go back to a directory
first=${pth%%/*}
# $middle is the rest of the provided path
if [ ! -d $first ]; then
# path starts with parent directory
dir=${PWD%/$first/*}/$first
first=$first/
# List all sub directories of the $dir/$middle directory
if [ -d "$dir/$middle" ]; then
for d in $(ls -a $dir/$middle); do
if [ -d $dir/$middle/$d ] && [[ "$d" != "." ]] && [[ "$d" != ".." ]]; then
compadd "$expl[#]" -fQ -W $dir/ - $first$middle$d
fi
done
fi
fi
fi
}
compdef _cl cl
The problem:
In my zshrc, I have a line:
zstyle ':completion:*' matcher-list 'm:{a-z}={A-Za-z}' '+l:|=* r:|=*'
This should make autocompletions case insensitive and make sure I can start typing the last part of a directory name, and in will still finnish the full name
Example:
$ cd /a
$ cd di[tab] # replaces 'di' with 'some_dir/'
$ cl di[tab] # this does not do anything. I would like it to replace 'di' with 'some_dir'
How do get it to suggest 'some_dir' when I type 'di'?
The second matcher in your matcher-list never gets called, because _cl() returns "true" (exit status 0, actually) even when it has not added any matches. Returning "true" causes _main_complete() to assume that we're done completing and it will thus not try the next matcher in the list.
To fix this, add the following to the start of _cl():
local -i nmatches=$compstate[nmatches]
and this to the end of _cl():
(( compstate[nmatches] > nmatches ))
That way, _cl() will return "true" only when it has managed to actually add completions.

can't use '~' in zsh autocompletion

I use zsh and I want to use a function I wrote to replace cd.
This function gives you the ability to move to a parent directory:
$ pwd
/a/b/c/d
$ cl b
$ pwd
/a/b
You can also move into a subdirectory of a parent directory:
$ pwd
/a/b/c/d
$ cl b/e
$ pwd
/a/b/e
If the first part of the path is not a parent directory, it will just function as normal cd would. I hope that makes sense.
In summary, when in /a/b/c/d, I want to be able to move to /a, /a/b, /a/b/c, all subdirectories of /a/b/c/d and any absolute path starting with /, ~/ or ../ (or ./).
I hope that makes sense.
This is the function I wrote:
cl () {
local first=$( echo $1 | cut -d/ -f1 )
if [ $# -eq 0 ]; then
# cl without any arguments moves back to the previous directory
cd - > /dev/null
elif [ -d $first ]; then
# If the first argument is an existing normal directory, move there
cd $1
else
# Otherwise, move to a parent directory
cd ${PWD%/$first/*}/$1
fi
}
There is probably a better way to this (tips are welcome), but I haven't had any problems with this so far.
Now I want to add autocompletion. This is what I have so far:
_cl() {
pth=${words[2]}
opts=""
new=${pth##*/}
[[ "$pth" != *"/"*"/"* ]] && middle="" || middle="${${pth%/*}#*/}/"
if [[ "$pth" != *"/"* ]]; then
# If this is the start of the path
# In this case we should also show the parent directories
opts+=" "
first=""
d="${${PWD#/}%/*}/"
opts+="${d//\/// }"
dir=$PWD
else
first=${pth%%/*}
if [[ "$first" == "" ]]; then
# path starts with "/"
dir="/$middle"
elif [[ "$first" == "~" ]]; then
# path starts with "~/"
dir="$HOME/$middle"
elif [ -d $first ]; then
# path starts with a directory in the current directory
dir="$PWD/$first/$middle"
else
# path starts with parent directory
dir=${PWD%/$first/*}/$first/$middle
fi
first=$first/
fi
# List al sub directories of the $dir directory
if [ -d "$dir" ]; then
for d in $(ls -a $dir); do
if [ -d $dir/$d ] && [[ "$d" != "." ]] && [[ "$d" != ".." ]]; then
opts+="$first$middle$d/ "
fi
done
fi
_multi_parts / "(${opts})"
return 0
}
compdef _cl cl
Again, probably not the best way to do this, but it works... kinda.
One of the problems is that what I type cl ~/, it replaces it with cl ~/ and does not suggest any directories in my home folder. Is there a way to get this to work?
EDIT
cl () {
local first=$( echo $1 | cut -d/ -f1 )
if [ $# -eq 0 ]; then
# cl without any arguments moves back to the previous directory
local pwd_bu=$PWD
[[ $(dirs) == "~" ]] && return 1
while [[ $PWD == $pwd_bu ]]; do
popd >/dev/null
done
local pwd_nw=$PWD
[[ $(dirs) != "~" ]] && popd >/dev/null
pushd $pwd_bu >/dev/null
pushd $pwd_nw >/dev/null
elif [ -d $first ]; then
pushd $1 >/dev/null # If the first argument is an existing normal directory, move there
else
pushd ${PWD%/$first/*}/$1 >/dev/null # Otherwise, move to a parent directory or a child of that parent directory
fi
}
_cl() {
_cd
pth=${words[2]}
opts=""
new=${pth##*/}
local expl
# Generate the visual formatting and store it in `$expl`
_description -V ancestor-directories expl 'ancestor directories'
[[ "$pth" != *"/"*"/"* ]] && middle="" || middle="${${pth%/*}#*/}/"
if [[ "$pth" != *"/"* ]]; then
# If this is the start of the path
# In this case we should also show the parent directories
local ancestor=$PWD:h
while (( $#ancestor > 1 )); do
# -f: Treat this as a file (incl. dirs), so you get proper highlighting.
# -Q: Don't quote (escape) any of the characters.
# -W: Specify the parent of the dir we're adding.
# ${ancestor:h}: The parent ("head") of $ancestor.
# ${ancestor:t}: The short name ("tail") of $ancestor.
compadd "$expl[#]" -fQ -W "${ancestor:h}/" - "${ancestor:t}"
# Move on to the next parent.
ancestor=$ancestor:h
done
else
# $first is the first part of the path the user typed in.
# it it is part of the current direoctory, we know the user is trying to go back to a directory
first=${pth%%/*}
# $middle is the rest of the provided path
if [ ! -d $first ]; then
# path starts with parent directory
dir=${PWD%/$first/*}/$first
first=$first/
# List all sub directories of the $dir/$middle directory
if [ -d "$dir/$middle" ]; then
for d in $(ls -a $dir/$middle); do
if [ -d $dir/$middle/$d ] && [[ "$d" != "." ]] && [[ "$d" != ".." ]]; then
compadd "$expl[#]" -fQ -W $dir/ - $first$middle$d
fi
done
fi
fi
fi
}
compdef _cl cl
This is as far as I got on my own. It does works (kinda) but has a couple of problems:
When going back to a parent directory, completion mostly works. But when you go to a child of the paretn directory, the suggestions are wrong (they display the full path you have typed, not just the child directory). The result does work
I use syntax-hightlighting, but the path I type is just white (when using going to a parent directory. the normal cd functions are colored)
In my zshrc, I have the line:
zstyle ':completion:*' matcher-list 'm:{a-z}={A-Za-z}' '+l:|=* r:|=*'
Whith cd this means I can type "load" and it will complete to "Downloads". With cl, this does not work. Not event when using the normal cd functionality.
Is there a way to fix (some of these) problems?
I hope you guys understand my questions. I find it hard to explain the problem.
Thanks for your help!
This should do it:
_cl() {
# Store the number of matches generated so far.
local -i nmatches=$compstate[nmatches]
# Call the built-in completion for `cd`. No need to reinvent the wheel.
_cd
# ${PWD:h}: The parent ("head") of the present working dir.
local ancestor=$PWD:h expl
# Generate the visual formatting and store it in `$expl`
# -V: Don't sort these items; show them in the order we add them.
_description -V ancestor-directories expl 'ancestor directories'
while (( $#ancestor > 1 )); do
# -f: Treat this as a file (incl. dirs), so you get proper highlighting.
# -W: Specify the parent of the dir we're adding.
# ${ancestor:h}: The parent ("head") of $ancestor.
# ${ancestor:t}: The short name ("tail") of $ancestor.
compadd "$expl[#]" -f -W ${ancestor:h}/ - $ancestor:t
# Move on to the next parent.
ancestor=$ancestor:h
done
# Return true if we've added any matches.
(( compstate[nmatches] > nmatches ))
}
# Define the function above as generating completions for `cl`.
compdef _cl cl
# Alternatively, instead of the line above:
# 1. Create a file `_cl` inside a dir that's in your `$fpath`.
# 2. Paste the _contents_ of the function `_cl` into this file.
# 3. Add `#compdef cl` add the top of the file.
# `_cl` will now get loaded automatically when you run `compinit`.
Also, I would rewrite your cl function like this, so it no longer depends on cut or other external commands:
cl() {
if (( $# == 0 )); then
# `cl` without any arguments moves back to the previous directory.
cd -
elif [[ -d $1 || -d $PWD/$1 ]]; then
# If the argument is an existing absolute path or direct child, move there.
cd $1
else
# Get the longest prefix that ends with the argument.
local ancestor=${(M)${PWD:h}##*$1}
if [[ -d $ancestor ]]; then
# Move there, if it's an existing dir.
cd $ancestor
else
# Otherwise, print to stderr and return false.
print -u2 "$0: no such ancestor '$1'"
return 1
fi
fi
}
Alternative Solution
There is an easier way to do all of this, without the need to write a cd replacement or any completion code:
cdpath() {
# `$PWD` is always equal to the present working directory.
local dir=$PWD
# In addition to searching all children of `$PWD`, `cd` will also search all
# children of all of the dirs in the array `$cdpath`.
cdpath=()
# Add all ancestors of `$PWD` to `$cdpath`.
while (( $#dir > 1 )); do
# `:h` is the direct parent.
dir=$dir:h
cdpath+=( $dir )
done
}
# Run the function above whenever we change directory.
add-zsh-hook chpwd cdpath
Zsh's completion code for cd automatically takes $cdpath into account. No need to even configure that. :)
As an example of how this works, let's say you're in /Users/marlon/.zsh/prezto/modules/history-substring-search/external/.
You can now type cd pre and press Tab, and Zsh will complete it to cd prezto. After that, pressing Enter will take you directly to /Users/marlon/.zsh/prezto/.
Or let's say that there also exists /Users/marlon/.zsh/prezto/modules/prompt/external/agnoster/. When you're in the former dir, you can do cd prompt/external/agnoster to go directly to the latter, and Zsh will complete this path for you every step of the way.

Listing SRC_URI of all packages/files needed to build Yocto image

I would like to list all files that bitbake will fetch when I bake an image.
Currently, I am able to get the SRC_URI of all files needed to bake a Yocto image by executing bitbake core-image-minimal -c fetchall and then parse log files.
Is there is a simpler way to get the same result without the need to download files ?
I am not sure bitbake supports such feature. Ideally, I am looking for a command that prints out the package name and lists all files with corresponding URL
> bitbake core-image-minimal -c fetchall --print-only
Generally bitbake doesn't provides such functionality.
But I was able to create a simple solution witch creating simple .bbclass file which is inherited in all recipes, by adding it into local.conf file, please see my steps in order to archive that:
Steps:
let's create a class print-src.bbclass file used to get and print SRC_URI variable (remember to store this class file in layer which is available in conf/bblayers.conf):
$ cat print-src.bbclass
python do_print_src () {
# should probably be indented
srcuri = d.getVar('SRC_URI', True).split()
bb.warn("SRC_URI look like: %s" % srcuri)
}
addtask do_print_src before do_fetch
Add INHERIT += "print-src" into Your conf/local.conf file
Edit: it is important to use bitbake --runonly option, that allows to run specific task of the taskgraph for the specified target (with --runonly option do_print_src needs to be used as print_src),
Edit: Please note that --runall=RUNALL and --runonly=RUNONLY was introduced with Yocto Sumo release 2.5,
$ bitbake core-image-minimal --runonly print_src
Loaded 1236 entries from dependency cache.
NOTE: Resolving any missing task queue dependencies
Build Configuration:
BB_VERSION = "1.37.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "universal-4.8"
TARGET_SYS = "i586-poky-linux"
MACHINE = "qemux86"
DISTRO = "poky"
DISTRO_VERSION = "2.5"
TUNE_FEATURES = "m32 i586"
TARGET_FPU = ""
meta
meta-poky
meta-yocto-bsp = "master:13cc30cd7de4841990b600e83e1249c81a5171dd"
Initialising tasks: 100% |##########################################################################################################################################################################| Time: 0:00:00
NOTE: Executing RunQueue Tasks
WARNING: ptest-runner-2.2+gitAUTOINC+49956f65bb-r0 do_print_src: SRC_URI look like: ['git://git.yoctoproject.org/ptest-runner2']
WARNING: grep-3.1-r0 do_print_src: SRC_URI look like: ['http://ftp.gnu.org/gnu/grep/grep-3.1.tar.xz', 'file://0001-Unset-need_charset_alias-when-building-for-musl.patch']
...
...
NOTE: Tasks Summary: Attempted 201 tasks of which 0 didn't need to be rerun and all succeeded.
Summary: There were 202 WARNING messages shown.
Please see sample warning output log line:
WARNING: ptest-runner-2.2+gitAUTOINC+49956f65bb-r0 do_print_src: SRC_URI look like: ['git://git.yoctoproject.org/ptest-runner2'].
I patched poky to create *.src files in downloads that contains the effective fetch URL of the package.
bitbake/lib/bb/fetch2/__init__.py | 19 ++++++++++++++-----
1 file changed, 14 insertions(+), 5 deletions(-)
diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index b853da30bd..03e84e0919 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
## -1257,16 +1257,16 ## class FetchData(object):
# Note: .done and .lock files should always be in DL_DIR whereas localpath may not be.
if self.localpath and self.localpath.startswith(dldir):
- basepath = self.localpath
+ self.basepath = self.localpath
elif self.localpath:
- basepath = dldir + os.sep + os.path.basename(self.localpath)
+ self.basepath = dldir + os.sep + os.path.basename(self.localpath)
elif self.basepath or self.basename:
- basepath = dldir + os.sep + (self.basepath or self.basename)
+ self.basepath = dldir + os.sep + (self.basepath or self.basename)
else:
bb.fatal("Can't determine lock path for url %s" % url)
- self.donestamp = basepath + '.done'
- self.lockfile = basepath + '.lock'
+ self.donestamp = self.basepath + '.done'
+ self.lockfile = self.basepath + '.lock'
def setup_revisions(self, d):
self.revisions = {}
## -1607,6 +1607,15 ## class Fetch(object):
m = ud.method
localpath = ""
+ p = "%s.src"%ud.basepath
+ d = os.path.dirname(p)
+ if d != '':
+ bb.utils.mkdirhier(d)
+ with open(p, 'wb') as f:
+ data = "%s" % ud.url
+ f.write(bytes(data, 'ASCII'))
+ return True
+
if ud.lockfile:
lf = bb.utils.lockfile(ud.lockfile)
Running bitbake core-image-minimal -c fetchall results:
$> find downloads/ -name '*.src' | head -n 5
downloads/lzop-1.03.tar.gz.src
downloads/libtheora-1.1.1.tar.bz2.src
downloads/mpfr-3.1.5.tar.xz.src
downloads/makedevs.c.src
downloads/expat-2.2.0.tar.bz2.src
This is not an optimal solution but I hope that such feature gets its way to mainline stream.
I needed something like this and got part way there before now.
I can generate a messy list of URIs used by executing the following command:
bitbake -g zlib && cat recipe-depends.dot | \
grep -v -e '-native' | grep -v digraph | \
grep -v -e '-image' | awk '{print $1}' | \
sort | uniq | xargs -I {} -t bitbake -e {} | grep SRC_URI=
This gives you all the URIs and files used in a recipe and some comments.
Not a perfect solution but I will see if I can improve on it.
A bit manual and straight forward way is to build your image from scratch, and copy all the fetch logs, and go thru it.
$ mkdir fetch_logs
$ find tmp/work/ -type f -name log.do_fetch* | cpio -pdm fetch_logs
Now grep "Fetch(ing/er)" in this fetch_logs and it will provide info on which URL your recipe used to fetch the source. This will be helpful to see the final URI used to fetch in PREMIRROR based builds.
NOTE: if you have sstate-cache enabled and functional, you may not see do_fetch at all.
I wrote scripts based on the solution in the question.
Basically I need to make a cscope project of all the sources, and hence I need to list all URI and clone all repos.
I wrote 2 scripts 1) List all SRC_URI and Branches
2) Install(clone) all the code in a single directory
listURI.sh
#!/bin/bash
TMP_FILE="___x__2242230p.txt"
SRCH_FILE="log.do_fetch"
TIME=$(date +%Y-%m-%d-%H-%M)
OUT_FILE="Project-$TIME.txt"
DEF_BRANCH="master"
BRANCH=""
SRC_URI=""
function usage {
echo "Usage : $0 < -f | -g > <Component-List-file>"
echo "Options :"
echo " -f : Run bitbake fetch and collect URI"
echo " -g : Get URI from Already fetched Project"
}
if [ $# -eq 0 ]
then
usage
exit
fi
if [ "$1" == "-f" ] || [ "$1" == "-g" ];then
if [ "$1" == "-f" ] && [ -z "$2" ];then
echo "Should pass Project-Name after -f Option"
exit
fi
else
echo "Unknown Option"
usage
exit
fi
POKYROOTDIR=`basename "$PWD"`
#echo $POKYROOTDIR
if [[ $POKYROOTDIR != "build" ]];then
echo "You are not on poky/build (Sourced Poky) Directory"
exit 0
fi
POKYROOTDIR=$PWD
if [ "$1" == "-f" ];then
echo "Running === bitbake -c fetchall $2 -f --no-setscene =="
bitbake -c fetchall $2 -f --no-setscene
fi
if [ "$1" == "-g" ];then
if [ ! -d tmp/work ];then
echo " Please run == bitbake -c fetchall <image> -f --no-setscene == before this"
usage
exit
fi
fi
echo "Looking for URIs --- It may take couple of minuites!"
rm -rf $OUT_FILE
cd tmp/work
find . -name $SRCH_FILE -exec grep -i 'DEBUG: For url git' {} \; -print > $POKYROOTDIR/$TMP_FILE
cd $POKYROOTDIR
while IFS= read -r line
do
#echo "$line"
BRANCH=$(echo $line | awk -F"branch=" '/branch=/{print $2}' | sed -e 's/;/ /' | awk '{print $1}')
SRC_URI=$(echo $line | cut -d';' -f1-1 | awk -F"url" '/url/{print $2}' | awk '{print $1}' | sed -e 's/git:/ssh:/')
if [ ! -z "$BRANCH" ] && [ ! -z "$SRC_URI" ];then
echo "$BRANCH $SRC_URI" >> $OUT_FILE
elif [ ! -z "$SRC_URI" ];then
echo "$DEF_BRANCH $SRC_URI" >> $OUT_FILE
fi
if [ ! -z "$BRANCH" ];then
echo "Found URI : BRANCH:$BRANCH URI:$SRC_URI"
elif [ ! -z "$SRC_URI" ];then
echo "Found URI : BRANCH:Default URI:$SRC_URI"
fi
#echo "$BRANCH $SRC_URI" >> $OUT_FILE
done < "$TMP_FILE"
#echo "=================== OUT PUT ==========================="
#cat $OUT_FILE
rm -rf $TMP_FILE
echo "----------------INFO-----------------"
echo "Project Creation: Success"
echo "Project Path : $POKYROOTDIR"
echo "Project file : $OUT_FILE"
echo "-------------------------------------"
installURI.sh
#!/bin/bash
function usage {
echo "Usage : $0 <List-file>"
}
if [ $# -eq 0 ]
then
usage
exit
fi
if [ -z "$1" ];then
usage
exit
fi
if [ ! -z "$(ls -A $PWD)" ]; then
echo "Directory ($PWD) must be Empty.... else it will mess up project creation"
exit
fi
if [ ! -f $1 ];then
echo "list file ($1) not found"
usage
exit
fi
filename=$1
while read -r line; do
name="$line"
echo "Fetching Projects"
git clone -b $line
done < "$filename"
echo "----------------INFO-----------------"
echo "Project Creation: Success"
echo "Project Path : $PWD"
echo "-------------------------------------"