I want to create generic SH script which will be called with a specified function to be used in the middle of JQ query.
1. How can I call a function by name specified as command-line argument?
The straightforward way do so generates string with function name instead of invoking such a function:
jq --arg query_function "foo_function" '
include "foo_function";
. |
($query_function)' \
input.txt
2. Is it possible to also specify a file to include with a command-line argument to JQ?
I also tried to do it in a straightforward manner, but it didn't work.
Neither jq function names nor module names can be specified as jq variables, and so to achieve what you seem to want, you would have to resort to shell string-interpolation, or lookup expressions.
It is worth noting, however, that jq supports array-based path expressions, and so certain kinds of lookups can be supported using jq variables.
Related
I'm having a problem moving forward through a path with PowerShell. I am able to move up the directory but not down. Here's the situation:
I open PowerShell and type in the "pwd" command and it shows that I am currently in PS C:\Users\Robert Inspiron14>
I type the command "cd.." and now I move to PS C:\Users>
I then attempt to change directories by typing: "cd C:\Users\Robert Inspiron14" and I am unable to. Unfortunately, I can't post a picture yet due to lack of reputation.
I'm able to perform the change in CMD but not PowerShell. Also, I don't know how to change the User from "Robert Inspiron14" to just "Robert". Any help is appreciated!
Before PowerShell can execute your cd command, it needs to parse it, and PowerShell's parser interprets your command like this:
cd C:\Users\Robert Inspiron14
\/ \_____________/ \________/
Command Name | |
argument 1 |
argument 2
In other words, C:\Users\Robert and Inspiron14 are interpreted as separate arguments.
Neither argument is a path to a valid directory, so cd (or rather Set-Location for which cd is an alias) throws an error.
You can force PowerShell to recognize C:\Users\Robert Inspiron14 as a single string argument by qualifying its boundaries using quotation marks (both " and ' will work):
cd 'C:\Users\Robert Inspiron14'
You can read more about how PowerShell parses command expressions in the about_Parsing help topic
To complement Mathias R. Jessen's helpful answer with more background information:
Quoting an argument that contains spaces is a general syntactic necessity, in all shells, because unquoted spaces are used to separate multiple arguments.
It isn't only spaces that require quoting, but any of PowerShell's so-called metacharacters (characters that, when used unquoted, have syntactic function); for instance, passing the path to a directory literally named a;b requires quoting as well, given that ; would otherwise be interpreted as a statement separator.
There are multiple quoting styles:
Since your path value is a literal - it contains no variable references or expressions - a verbatim (single-quoted) string ('...') is the best choice.
cd 'C:\Users\Robert Inspiron14'
If your path contains variables or subexpressions, you must use an expandable (double-quoted) string ("...")[1]
cd "$HOME\Documents"
Another, less common solution is to individually escape the space characters with `, the so-called backtick, PowerShell's escape character:
cd C:\Users\Robert` Inspiron14
Also note:
PowerShell's tab-completion automatically applies quoting as necessary.
cd.. is the name of a built-in function in PowerShell, whose sole purpose is to emulate cmd.exe's (questionably permissive) behavior (see below); the function performs a syntactically correct Set-Location .. call (verify by executing ${function:cd..}), with a space separating the command name from its argument.
Contrast with cmd.exe:
Unfortunately, cmd.exe's built-in cd command decided not to enforce its usual syntax rules, and enabled calls such as cd C:\Program Files.
It should never have done that: While convenient at first glance, it constitutes a problematic exception from the usual rules that invites confusion.
Note that cmd.exe's tab completion properly quotes arguments that contain spaces.
Similarly, cd.. was unfortunately allowed as as syntactically exceptional alternative to the correct cd .. - see the comments on this answer for details.
[1] Note "..."-quoting isn't strictly necessary if you use variable references in a path, as long as any literal components do not require quoting; e.g., $HOME\foo is fine without quoting, whereas the " around "$HOME\foo bar" are required. With subexpressions ($(...)), the rules get more complicated, so the simplest approach is to always use "..."-quoting with them.
I am trying to use 'sed' to replace a list of paths in a file with another path.
An example string to process is:
/path/to/file/block
I want to replace /path/to/file with something else.
I have Tried
sed -r '/\s(\S+)\/block/s/\1/new_path/'
I know it's finding the matching string but I'm getting an invalid back reference error.
How can I do this?
This may do:
echo "/path/to/file/block" | sed -r 's|/\S*/(block)|/newpath/\1|'
/newpath/block
Test
echo "test=/path/file test2=/path/to/file/block test3=/home/root/file" | sed -r 's|/\S*/(block)|/newpath/\1|'
test=/path/file test2=/newpath/block test3=/home/root/file
Back-references always refer to the pattern of the s command, not to any address (before the command).
However, in this case, there's no need for addressing: we can apply the substitution to all lines (and it will change only lines where it matches), so we can write:
s,\s(\S+)/block/, \1/new_path,
(I added a space to the RHS, as I'm guessing you didn't mean to overwrite that; also used a different separator to reduce the need for backslashes.)
There are 84 PATTERN need to be check, i store them in file name pattern.txt.
Is silver-searcher (also named Ag) able to obtain these patterns from pattern.txt?
grep has -f options to read pattern from file, but the man page of silver-searcher mention nothing about it.
No, there isn't a similar -f option in ag. The simple approach is to use loop to pass the patterns to ag; for instance you could use a while loop to read the patterns like this:
while read pattern; do ag "$pattern" -G '.*.txt' ; done < patterns.txt
I suggest the faster approach of using GNU parallel with ag. Parallel and ag work very well together:
< patterns.txt | parallel 'ag --filename --parallel --color "{}" '
Here, I'm passing each pattern to parallel which in turn spawns a number of ag processes which search for their own pattern matches. Parallel is somewhat smart about how many processes to start, but you can tweak it to your heart's content (https://www.gnu.org/software/parallel/man.html). In short, you'll rip through your 84 patterns far faster with parallelization.
Joining the lines in the pattern file to create a regex group:
ag "($(paste -sd "|" pattern.txt))" .
I'm trying to access a series of webpages in perl and write them to a series of files. The code I have looks like this:
open IN , "AbsoluteFinalData.txt"; #Each line has a number and ID name separated by a tab.
while(my $line = <IN>){
chop $line; #removes newline at the end
my #first_split = split(/\t/, $line);
my $IDnum = $first_split[0];
my $Uniprot = $first_split[1];
system('Uniprot=$Uniprot; curl -o "$Uniprot.html" http://pfam.xfam.org/\protein/.$Uniprot'); #More stuff after
The program, however, is giving me fits when I try to call $Uniprot in system(). Is there any way to call a variable defined in the perl script using system()?
system('Uniprot=$Uniprot; curl -o "$Uniprot.html" http://pfam.xfam.org/\protein/.$Uniprot');
You use single quotes, which doesn't interpolate. The literal command:
Uniprot=$Uniprot; curl -o "$Uniprot.html" http://pfam.xfam.org/\protein/.$Uniprot
Is being executed.
You want to interpolate your variables, which means using double quotes (and escaping contained ones:)
system("Uniprot=$Uniprot; curl -o \"$Uniprot.html\" http://pfam.xfam.org/\protein/.$Uniprot");
Or the qq quote-like operator which functions like the double quote but avoids needing to escape contained double quotes:
system(qq(Uniprot=$Uniprot; curl -o "$Uniprot.html" http://pfam.xfam.org/\protein/.$Uniprot"));
Given that you're not relying on the system command to perform any shell interpretation or file I/O redirection, you can achieve everything you want safely like this:
system 'curl', '-o', "$Uniprot.html", "http://pfam.xfam.org/protein/.$Uniprot";
The "list" version of system is safer to use than the single string version because it prevents shell command injection attacks.
Note also the use of double quotes to enable Perl's own variable interpolation, and also that there's no need to create the shell local variable Uniprot=$Uniprot since it's not used by Curl and is only being used by you to attempt to perform variable interpolation yourself.
Perl only interpolates variables within double quotes ("..."), not single quotes ('...').
system("Uniprot=$Uniprot; curl -o \"$Uniprot.html\" http://pfam.xfam.org/\protein/.$Uniprot");
Will do the substitution you're looking for.
I am developing a tomcat application and would like to be able to search for specific things and highlight it when viewing the log. I want something like an alias that takes a parameter (regex) as input and highlight the matching string.
So far, I've figured this works, but its not practical enough to have to change a small part of it for every time I want something new:
tail -n 100 -f /opt/apache-tomcat-6.0.26/logs/catalina.out | perl -pe 's/null/\e[1;31m$&\e[0m/g'
This is what I thought would work:
logColor(){
x="'s/"
y="/\e[1;31m$&\e[0m/g'"
tail -n 100 -f /opt/apache-tomcat-6.0.26/logs/catalina.out | perl -pe $x$1$y
}
alias logC=logColor
I've tested that this prints out the two same lines:
logColorTest(){
x="'s/"
y="/\e[1;31m$&\e[0m/g'"
echo $x$1$y
echo "'s/null/\e[1;31m$&\e[0m/g'"
}
alias logCT=logColorTest
logCT null
So I am lost on why this does not work and would appreciate input from someone who knows how this works :)
Problem with grep is that, you get only matching lines & other lines are filtered out. (That's what is grep supposed to do anyway.) Many times however, we need all the output, but with some particular strings highlighted.
I have this small bash function in my .bashrc for such requirement:
mark ()
{
local searchExpr=${1/\//\\\/};
sed "s/$searchExpr/"`echo -n -e "\e[91;1m"`'&'`echo -n -e "\e[0m"`'/gi' $2
}
Usage:
command | mark some_string # OR
mark some_string some_file
Rename to suitable function name if required.
NOTE: There is a great command called highlight. Hence I could not use that as my function name.
As #fedorqui pointed out, you can use grep to do this:
grep --colour 'null\|$'
This will match and highlight null or the end of a line, meaning all lines are shown.
Using the GREP_COLORS environment variable you can control how different parts are highlighted, e.g mark matched text in yellow:
export GREP_COLORS='ms=1;33'