I would like to be able to do this:
bat misc | rg -v -e 'TXT|txt' | map path_explode
where
function map
while read line
command $argv $line
end
end
and
function path_explode --description 'Return filename, ext, and directory from the path'
echo $argv[1] | sed 's/\(.*\)\/\(.*\)\.\(.*\)$/\2\n\3\n\1/'
end
does that make sense? I get this error:
fish: Unknown command: path_explode
fish:
command $argv $line
^
in function 'map' with arguments 'path_explode'
In fish 3 or later, variables can be used directly as commands, so you don't need eval or command in your map function.
ohh I just need to use eval instead of command
cool :)
Related
Is there any convenient way to strip an arbitrary extension from a file name, something à la bash ${i%%.*}? Do I stick to my friend sed?
If you know the extension (eg _bak, a common usecase) this is possibly more convenient:
for f in (ls *_bak)
mv $f (basename $f _bak)
end
Nope. fish has a much smaller feature set than bash, relying on external commands:
$ set filename foo.bar.baz
$ set rootname (echo $filename | sed 's/\.[^.]*$//')
$ echo $rootname
foo.bar
You can strip off the extension from a filename using the string command:
echo (string split -r -m1 . $filename)[1]
This will split filename at the right-most dot and print the first element of the resulting list. If there is no dot, that list will contain a single element with filename.
If you also need to strip off leading directories, combine it with basename:
echo (basename $filename | string split -r -m1 .)[1]
In this example, string reads its input from stdin rather than being passed the filename as a command line argument.
--- Update 2022-08-02 ---
As of fish 3.5+, there is a path command (docs) which was designed to handle stripping extensions:
$ touch test.txt.bak
$ path change-extension '' ./test.txt.bak
test.txt
You can also strip a set number of extensions:
set --local file ./test.txt.1.2.3
for i in (seq 3)
set file (path change-extension '' $file)
end
echo $file
# ./test.txt
Or strip all extensions:
set --local file ./test.txt.1.2.3
while path extension $file
set file (path change-extension '' $file)
end
echo $file
# ./test
--- Original answer ---
The fish string command is still the canonical way to handle this. It has some really nice sub commands that haven't been shown in other answers yet.
split lets you split from the right with a max of 1, so that you just get the last extension.
for f in *
echo (string split -m1 -r '.' "$f")[1]
end
replace lets you use a regex to lop off the extension, defined as the final dot to the end of the string
for f in *
string replace -r '\.[^\.]*$' '' "$f"
end
man string for more info and some great examples.
Update:
If your system has proper basename and dirname utilities, you can use something like this:
function stripext \
--description "strip file extension"
for arg in $argv
echo (dirname $arg)/(string replace -r '\.[^\.]+$' '' (basename $arg))
end
end
With the string match function built into fish you can do
set rootname (string match -r "(.*)\.[^\.]*\$" $filename)[2]
The string match returns a list of 2 items. The first is the whole string, and the second one is the first regexp match (the stuff inside the parentheses in the regex). So, we grab the second one with the [2].
I too need a function to split random files root and extension. Rather than re-implementing naively the feature at the risk of meeting caveats (ex: dot before separator), I am forwarding the task to Python's built-in POSIX path libraries and inherit from their expertise.
Here is an humble example of what one may prefer:
function splitext --description "Print filepath(s) root, stem or extension"
argparse 'e/ext' 's/stem' -- $argv
for arg in $argv
if set -q _flag_ext
set cmd 'import os' \
"_, ext = os.path.splitext('$arg')" \
'print(ext)'
else if set -q _flag_stem
set cmd 'from pathlib import Path' \
"p = Path('$arg')" \
'print(p.stem)'
else
set cmd 'import os' \
"root, _ = os.path.splitext('$arg')" \
'print(root)'
end
python3 -c (string join ';' $cmd)
end
end
Examples:
$ splitext /this/is.a/test.path
/this/is.a/test
$ splitext --ext /this/is.a/test.path
.path
$ splitext --stem /this/is.a/test.path
test
$ splitext /this/is.another/test
/this/is.another/test
Let's say I declare two variables key and value like this in shell script
$ key=key1
$ value=value1
now I want to create a variable of name key1 and assign it value value1. What I tried is
$ export ${key}
$ export ${value}
$ $($key=$value)
output: key1=value1: command not found
I don't know how to do this.
Use eval $key=$value instead of $($key=$value): The shell substitutes variable values first, and then the $(...) substitution. As there is not command with that name the shell shows command not found in STDERR. Tell the shell to evaluate the substitution result as a regular shell command again by using eval. Later you can export the result with export $key. Using the -x flag for the shell provides a good insight of what happens:
$ key=key1
$ value=value1
$ set -x
$ $key=$value
+ key1=value1
sh: key1=value1: not found [No such file or directory]
$ $($key=$value)
+ key1=value1
sh: key1=value1: not found [No such file or directory]
$ eval $key=$value
+ eval key1=value1
+ key1=value1
$ echo $key1
+ echo value1
value1
$ export $key
+ export key1
Also, be careful when dealing with variables this way: Whitespaces in shell variables can have unexpected results in this kind of constructions.
If multiple arguments are passed to perl's system function then the shell expansion will not work:
# COMMAND
$ perl -e 'my $s="*"; system("echo", "$s" )'
# RESULT
*
If the command is passed as an one argument then the expansion will work:
# COMMAND
$ perl -e 'my $s="echo *"; system("$s")'
# RESULT
Desktop Documents Downloads
The system function also allows to using multiple commands and connect them using pipes. This only works when argument is passed as an one command:
# COMMAND
$ perl -e 'my $s="echo * | cat -n"; system("$s")'
# RESULT
1 Desktop Documents Downloads
How can I combine mentioned commands and use both pipes and prevent shell expansion?
I have tried:
# COMMAND
$ perl -e 'my $s="echo"; system("$s", "* | cat -n")'
# RESULT
* | cat -n
but this did not work because of reasons that I've described above (multiple arguments are not expanded). The result that I want is:
1 *
EDIT:
The problem that I'm actually facing is that when I use following command:
system("echo \"$email_message\" | mailx -s \"$email_subject\" $recipient");
Then the $email_message is expanded and it will break mailx if it contains some characters that are further expanded by shell.
system has three calling conventions:
system($SHELL_CMD)
system($PROG, #ARGS) # #ARGS>0
system( { $PROG } $NAME, #ARGS ) # #ARGS>=0
The first passes a command to the shell. It's equivalent to
system('/bin/sh', '-c', $SHELL_CMD)
The other two execute the program $PROG. system never prevents shell expansion or performs any escaping. There's simply no shell involved.
So your question is about building a shell command. If you were at the prompt, you might use
echo \* | cat -n
or
echo '*' | cat -n
to pass *. You need a function that performs the job of escaping * before interpolating it. Fortunately, one already exists: String::ShellQuote's shell_quote.
$ perl -e'
use String::ShellQuote qw( shell_quote );
my $s = "*";
my $cmd1 = shell_quote("printf", q{%s\n}, $s);
my $cmd2 = "cat -n";
my $cmd = "$cmd1 | $cmd2";
print("Executing <<$cmd>>\n");
system($cmd);
'
Executing <<printf '%s\n' '*' | cat -n>>
1 *
I used printf instead of echo since it's very hard to handle arguments starting with - in echo. Most programs accept -- to separate options from non-options, but not my echo.
All these complications beg the question: Why are you shelling out to send an email? It's usually much harder to handle errors from external programs than from libraries.
You can use open to pipe directly to mailx, without your content being interpreted by the shell:
open( my $mail, "|-", "mailx", "-s", $email_subject, $recipient );
say $mail $email_message;
close $mail;
More details can be found in open section of perlipc.
I am trying to use following command in perl but it giving me error
system("zcat myfile.gz | wc > abc.txt");
But when i run this I am getting error
syntax error near unexpected token `|'
Even if I remove >abc.txt I am still getting error.
Can we use pipe with system command?
Here are error details:
sh: -c: line 1: syntax error near unexpected token `|'
sh: -c: line 1: ` | wc '
Next time, test your demo program to make sure it actually exhibits the behaviour you said it does. You actually ran something closer to
while (my $file_name = <>) {
system("zcat $file_name | wc > abc.txt");
}
There are two errors in that:
You didn't remove the trailing newline, so the shell was trying to execute
zcat def.gz
| wc >abc.txt
instead of
zcat def.gz | wc >abc.txt
You didn't transform the file name into a shell literal before emdedding it your command.
Consider what would happen if the file name contained a space. You would be executing
zcat def ghi.gz | wc >abc.txt
instead of
zcat 'def ghi.gz' | wc >abc.txt
Solution:
use String::ShellQuote qw( shell_quote );
while (my $file_name = <>) {
chomp($file_name);
system("zcat -- ".shell_quote($file_name)." | wc > abc.txt");
}
It is working as expected:
perl -lne 'system("cat *.java|wc");'
Something odd with your filename, maybe.
You could check the interpolation of your shell like this:
my #file = `ls -1 myfile*.gz`;chomp(#files);
print join("\n",#files);
There are other possibilites to execute in perl, like backtick, open with |, qx.
If you are trouble with filenames, you could get the filenames by yourself and call the system in a specific way to avoid executing shell: http://docstore.mik.ua/orelly/perl/cookbook/ch19_07.htm
If there is only one scalar argument, the argument is checked for shell metacharacters, and if there are any, the entire argument is passed to the system's command shell for parsing (this is /bin/sh -c on Unix platforms, but varies on other platforms). If there are no shell metacharacters in the argument, it is split into words and passed directly to execvp , which is more efficient.
I got this error message when trying to use backticks to execute a pipeline that included a cut -d \| command. Turns out I had to double escape the pipe character eg cut -d \\|
Is there an idiomatic way to simulate Perl's diamond operator in bash? With the diamond operator,
script.sh | ...
reads stdin for its input and
script.sh file1 file2 | ...
reads file1 and file2 for its input.
One other constraint is that I want to use the stdin in script.sh for something else other than input to my own script. The below code does what I want for the file1 file2 ... case above, but not for data provided on stdin.
command - $# <<EOF
some_code_for_first_argument_of_command_here
EOF
I'd prefer a Bash solution but any Unix shell is OK.
Edit: for clarification, here is the content of script.sh:
#!/bin/bash
command - $# <<EOF
some_code_for_first_argument_of_command_here
EOF
I want this to work the way the diamond operator would work in Perl, but it only handles filenames-as-arguments right now.
Edit 2: I can't do anything that goes
cat XXX | command
because the stdin for command is not the user's data. The stdin for command is my data in the here-doc. I would like the user data to come in on the stdin of my script, but it can't be the stdin of the call to command inside my script.
Sure, this is totally doable:
#!/bin/bash
cat $# | some_command_goes_here
Users can then call your script with no arguments (or '-') to read from stdin, or multiple files, all of which will be read.
If you want to process the contents of those files (say, line-by-line), you could do something like this:
for line in $(cat $#); do
echo "I read: $line"
done
Edit: Changed $* to $# to handle spaces in filenames, thanks to a helpful comment.
Kind of cheezy, but how about
cat file1 file2 | script.sh
I am (like everyone else, it seems) a bit confused about exactly what the goal is here, so I'll give three possible answers that may cover what you actually want. First, the relatively simple goal of getting the script to read from either a list of files (supplied on the command line) or from its regular stdin:
if [ $# -gt 0 ]; then
exec < <(cat "$#")
fi
# From this point on, the script's stdin is redirected from the files
# (if any) supplied on the command line
Note: the double-quoted use of $# is the best way to avoid problems with funny characters (e.g. spaces) in filenames -- $* and unquoted $# both mess this up. The <() trick I'm using here is a bash-only feature; it fires off cat in the background to feed data from files supplied on the command line, and then we use exec to replace the script's stdin with the output from cat.
...but that doesn't seem to be what you actually want. What you seem to really want is to pass the supplied filenames or the script's stdin as arguments to a command inside the script. This requires sort of the opposite process: converting the script's stdin into a file (actually a named pipe) whose name can be passed to the command. Like this:
if [[ $# -gt 0 ]]; then
command "$#" <<EOF
here-doc goes here
EOF
else
command <(cat) <<EOF
here-doc goes here
EOF
fi
This uses <() to launder the script's stdin through cat to a named pipe, which is then passed to command as an argument. Meanwhile, command's stdin is taken from the here-doc.
Now, I think that's what you want to do, but it's not quite what you've asked for, which is to both redirect the script's stdin from the supplied files and pass stdin to the command inside the script. This can be done by combining the above techniques:
if [ $# -gt 0 ]; then
exec < <(cat "$#")
fi
command <(cat) <<EOF
here-doc goes here
EOF
...although I can't think why you'd actually want to do this.
The Perl diamond operator essentially loops across all the command line arguments, treating each as a filename. It opens each file and reads them line-by-line. Here's some bash code that will do approximately the same.
for f in "$#"
do
# Do something with $f, such as...
cat $f | command1 | command2
-or-
command1 < $f
-or-
# Read $f line-by-line
cat $f | while read line_from_f
do
# Do stuff with $line_from_f
done
done
You want to take the first argument and do something with it, and then either read from any files specified or stdin if no files?
Personally, I'd suggest using getopt to indicate arguments using the "-a value" syntax to help disambiguate, but that's just me. Here's how I'd do it in bash without getopts:
firstarg=${1?:usage: $0 arg [file1 .. fileN]}
shift
typeset -a files
if [[ ${##} -gt 0 ]]
then
files=( "$#" )
else
files=( "/dev/stdin" )
fi
for file in "${files[#]}"
do
whatever_you_want < "$file"
done
The ?: operator will die if there are no args specified, since you seem to want at least one arg either way. After grabbing that, shift the args over by one, and then either use the remaining args as your file list, or the bash special filehandle "/dev/stdin" if there were no other args.
I think that the "if no files are specified, use /dev/stdin - otherwise use the files on the command line" piece is probably what you're looking for, but the rest of the code is at least useful for context.
Also a little cheezy, but how about this:
if [[ $# -eq 0 ]]
then
# read from stdin
else
# read from $* (args)
fi
If you need to read and process line-by-line (which is likely) and don't want to copy/paste the same code twice (which is likely), define a function in your script and just pass the lines one-by-one to this function, and process them in said function.
Why not use ``cat #* in the script? For example:
x=`cat $*`
echo $x