I want to create scipt to faciliate producing local text file extracts from Hive.
This is to basically execute commands like below:
hive -e "SET hive.cli.print.header=true;SELECT * FROM dropme"|perl -pe 's/(?:\t|^)\KNULL(?=\t|$)//g'>extract/outbound/dropme.txt
While the above works like a charm I find it quite problematic to implement through the parametrized following script (much simplified):
#!/bin/sh
TNAME=dropme
SQL="SELECT * FROM $TNAME"
echo $SQL
echo "SQL: $SQL"
EXTRACMD="hive -e \"SET hive.cli.print.header=true;$SQL\"|perl -pe 'BEGIN{if(defined(\$_=<ARGV>)){s/\b\w+\.//g;print}}s/(?:\t|^)\KNULL(?=\t|$)//g'>extract/outbound/$TNAME.txt"
echo "CMD: $EXTRACMD";
${EXTRACMD}
When run I get: Exception in thread "main" java.lang.NumberFormatException: For input string: "e"
I know there may be many flavours you can print the text or execute command. For instance the line echo $SQL prints me list of files in the directory instead:
SELECT file1.txt file2.txt file3.txt file4.txt FROM dropme
while the next one: echo "SQL: $SQL" gives just what I want: SQL: SELECT * FROM dropme
echo "CMD: $EXTRACMD" prints the (almost) the command to be executed. Almost, as I see \t in perl code being expanded:
CMD: hive -e "SET hive.cli.print.header=true;SELECT * FROM dropme"|perl -pe 'BEGIN{if(defined($_=<ARGV>)){s\w+\.//g;print}}s/(?: |^)\KNULL(?= |$)//g'>extract/outbound/dropme.txt
Maybe that's still ok, but what I want is to be able to copy&paste this command into (other) terminal and execute as the command I put at the top. Ideally I would like that command to be exactly the same (so with \t there)
Biggest problem I have comes when I try to execute it (${EXTRACMD} line). I'm getting the error:
Exception in thread "main" java.lang.NumberFormatException: For input string: "e" …and so on, irrelevant as bash treats every 'word' as single command here. I assume as I don't even know what is really tries to run (prior print attempt obviously doesn't help)
I'm aware that I have multiple options, like:
escaping special characters in the command definition string (like I did with doublequotes)
experimenting with echo and $VAR, '$VAR' or "$VAR"
experimenting with "${EXTRACMD}" or evaluating through eval "${EXTRACMD}"
experimenting with shopt -s extglob or set -f
but as number of combinations is quite large and with my little bash experience I feel it's better to ask for good practice here so my question is:
Is there a way to print a (complex/compound shell) command first and subsequently be able to execute it (exactly as per printed output)? In this case it would be printing the exact command from the top, then executing it the same way as by manually copying that output into terminal prompt and pressing Enter.
Do not construct commands as strings. See http://mywiki.wooledge.org/BashFAQ/050 for details.
That page also talks about a built-in way of getting the shell to tell you what it is running (section 6).
If that doesn't do what you want you can also, with bash, try using printf %q\\n "${arr[*]}".
Related
I have a perl script that prints a message. This script is being called by GNU make. In my GNU make, I want to display the message printed out by the script AND store it in a variable also.
I'm doing it this way.
result=`$(PERL) parse.pl report.log` #parse the report
echo $(result) #echo the message here
ifneq ($(strip $$(result)),) #check if message is empty
#if not empty, search for filepath string pattern and exit
echo filepath
exit 1
endif
But it is not displaying the string message from parse.pl.
You are capturing into a shell variable, but then trying to echo a makefile variable (and even if you tried to echo the shell variable, that wouldn't work because make runs each line in a separate shell process).
Changing it to echo the shell varible and all to run in one shell should work:
foo:
result=`$(PERL) parse.pl report.log`; \
echo $$result
but whatever you later need to do to use the captured result would also need to be in the same shell execution.
Apparently you can capture into a makefile variable too, which may be more convenient:
foo:
$(eval result := $(shell $(PERL) parse.pl report.log))
echo $(result)
The critical thing to keep in mind with make is first, that the entire makefile is parsed before any rules are run, and second a makefile has two completely distinct syntaxes in it: makefile syntax for most of it, and shell syntax for the recipes. The shell syntax is run by the shell, not by make: make just starts a shell, hands over the recipe, and waits for the shell to exit to see if it worked or not.
As a result of this you CANNOT combine make constructs like ifeq with shell commands and their results: it cannot work because all the make constructs are parsed first, while the makefile is being read in, and the shell commands are not run until much later, when the target is to be built.
In your case you need to write the entire thing in shell syntax, because you want things to depend on the shell invocation.
So, like this:
foo:
result=`$(PERL) parse.pl report.log`; \
echo $$result; \
if [ "$$result" = "" ]; then \
echo filepath; \
exit 1; \
fi
Note how each line ends with a backslash, so it's appended to the previous line instead of being a separate line: make runs each separate line in a different shell.
Alternatively if you have a new-enough GNU make you can use the one shell feature:
.ONESHELL:
foo:
result=`$(PERL) parse.pl report.log`
echo $$result
if [ "$$result" = "" ]; then
echo filepath
exit 1
fi
Basically, I want to add the filter to the output of my perl script. This filter would chop the redundant lines matching the pattern , 'Jobs found shutdown' and would result in the clean output. Now what and where should I use grep or sed to implement this approach?? And this filter should be the part of the script which would help in getting the clean output.
I am planning to use below sed command to match the lines and remove them from the console output. But need help in implementation
have the script which has some redundant lines from the server in its output.At the end of the script , I would be running the following sed command to clear the output of the script at the console.
"sed -i '/No Job found./d' ";
I think grep is your friend in this case. Something along the lines
perl ... | grep -v "No Job found"
Will result in only lines not containing No Job found being printed
I often see this command in node.js programs: NODE_ENV=test node app.js which sets the NODE_ENV variable to test and works. I also read here https://en.wikipedia.org/wiki/Environment_variable that this should work for any shell command, but running some tests on my own, here is what I see
$ HELLO="WORLD"
$ HELLO="MARS" echo "$HELLO"
WORLD
$
I would expect this to print MARS. Is there something I am missing here?
The syntax VAR=value command means that the command will be invoked with the environment variable VAR set to VALUE, and this will apply only for the scope of that command.
However, when you are using the command line:
HELLO="MARS" echo "$HELLO"
The shell first interprets the "$HELLO" parameter, determines that it is WORLD, and then what it actually does is run:
HELLO="MARS" echo "WORLD"
So the echo may have the HELLO variable set, but it doesn't affect what it prints - it has already been interpreted before.
Doing
HELLO="MARS"; echo "$HELLO"
does something else entirely. First it sets HELLO to MARS in the current shell, and then it goes on to interpret the echo command. By this time HELLO contains MARS, not WORLD. But this is an entirely different effect - the variable HELLO stays with the value MARS, which is not the case in the command without the ;.
Your problem is that echo is just a poor choice for a demonstartion of this. You can do other demonstrations to prove that HELLO is changed properly:
HELLO="MARS" eval 'echo $HELLO'
In this case, the shell will not interpret the $HELLO because it is within a string in single quotes. It will first put MARS in HELLO, and then call the eval 'echo $HELLO' with that variable set. The eval command with then run echo $HELLO, and you'll get the output you were expecting.
This syntax is best used for things that don't use the given variable as part of the command line, but rather use it internally.
Other answers are correct, but here a refinement :
There are 2 cases in fact when defining a list of variable separated by spaces in bash whether it ends or not with a command.
VAR1=value1 VAR2=value2 ... VARn=valuen command arg1 arg2 ... argn
and
VAR1=value1 VAR2=value2 ... VARn=valuen
don't export VAR1 ... VARn the same way.
In first case VAR1 ... VARn will be set only for command and will then not be exported to current shell.
In second case VAR1 ... VARn will alter current shell.
then ( remark that ';' is very same of using a new line )
HELLO=WORLD
HELLO=MARS echo "i don't export HELLO."
echo "HELLO=$HELLO"
will display
i don't export HELLO.
HELLO=WORLD
and
HELLO=WORLD
HELLO=MARS ; echo "i did export HELLO."
echo "HELLO=$HELLO"
will display
i did export HELLO.
HELLO=MARS
I'm confronted with a rather strange problem an echo command causes in a script.
It's supposed to be really REALLY basic stuff, but still, there's something "off".
Suppose, I have this script:
#!/bin/bash
# SERVERPID='cat lite_server_pid.txt'
# kill -9 $SERVERPID
nohup java -Xmx3G -Xms2G -jar tekkit_lite_065.jar nogui > output.txt &
echo $! > lite_server_pid.txt
Yes, this starts my own little Minecraft/Tekkit-Server. ;-)
The Problem is, the file thats created is (for some reason) named
lite_server_pid.txt?
and YES, this includes the "?"! Doing the same command in shell, a file without ? is correctly created! Also, the content of the file is the desired processID.
Still, the ? following the filename is a major problem...
What am I doing wrong?
Check your file for DOS line endings. I suspect that ? is actually your terminal's attempt to display a carriage return (\r). Since bash expects UNIX-style newlines, the carriage return part of the DOS newline (\r\n) is treated as a legal character for the file name.
Run your script through dos2unix.
It seems to me that there are two ways to run Matlab in batch mode:
the first one:
unset DISPLAY
matlab > matlab.out 2>&1 << EOF
plot(1:10)
print file
exit
EOF
The second one uses option "-r MATLAB_command":
matlab -nojvm -nosplash -r MyCommand
Are these two equivalent?
What does "<< EOF" and the last "EOF" mean in the first method?
Thanks and regards!
The first method simply redirects the standard output > matlab.out and the standard error 2>&1 to the file matlab.out.
Then it uses the heredoc way of passing input to MATLAB (this is not specific to MATLAB, it is a method of passing multiple lines as input to command line programs in general).
The syntax is << followed by an unique identifier, then your text, finally the unique id to finish.
You can try this on the shell:
cat << END
some
text
multiple lines
END
The second method of using the -r option starts MATLAB and execute the statement passed immediately. It could be some commands or the name of a script or function found on the path.
It is equivalent to doing something like:
python -c "print 'hello world'"
Refer to this page for a list of the other start options.