Can anyone explain this behaviour of the Windows Command processor:
Open cmd.exe
Type for example:
echo Hello 0>my.txt
Result:
An empty file named "my.txt"
Repeat this but replace '0' by the digits '1' to '9'
Observation: Just the same, an empty file.
Then replace the '0' by '10'
Observation: File 'my.txt' now contains the line "Hello 10"
Then rework the redirection statement like:
echo >my.txt Hello 0
Observation: File 'my.txt' now contains the line "Hello 0"
Question: WHY do the single digits before the '>' let the line get swallowed?
(I did this on Win7, but quite optimistic that its not version dependent)
Related
This question already has answers here:
I just assigned a variable, but echo $variable shows something else
(7 answers)
Closed 2 years ago.
The file myfile.sh looks like this:
echo "hello"
The file I run looks like this:
a=$(cat myfile.sh)
echo $a
When I run the file, I only get the output:
echo "hello"
And not what the actual file content is. What's going on here?
http://www.gnu.org/software/bash/manual/html_node/Command-Substitution.html
In a=$(cat myfile.sh) your variable gets assigned
the standard output of the command, with any trailing newlines deleted
And that is where your extra lines went.
From the man page for dash (/bin/sh) on my system,
The shell expands the command substitution by executing command in a subshell environment and replacing the command substitution with the standard output of the command, removing sequences of one or more ⟨newline⟩s at the end of the substitution. (Embedded ⟨newline⟩s before the end of the output are not removed; however, during field splitting, they may be translated into ⟨space⟩s, depending on the value of IFS and quoting that is in effect.)
(Emphasis mine.)
You could use the following:
#!/bin/sh
LF='
'
a=
while read -r line; do
a="$a$line$LF"
done
printf -- '--\n%s--' "$a"
Test:
$ printf 'a b c\nd\n\n\ne\n\n\n' | ./a.sh
--
a b c
d
e
--
I am making a API for Batch. One of its functions is a colortext, which uses PowerShell (and findstr if PowerShell is not supported on the machine) to do its work. (It simply converts the batch color given, like 3C, converts it to PowerShell-friendly names, like DarkCyan and Red, and prints the text given in color using more things.)
However, when I try to set the foreground as F (White), for some reason it tries to execute it 2 or 4 times (depending if I have Delayed Expansion enabled or not).
It sometimes fails on the last run/try showing the same thing if the supposed %fcn% and %bcn% never were applied. (this also depends if I have Delayed Expansion enabled or not).
I only want this to execute one time (and not fail for whatever reason), but it's executing 2 or 4 times for some reason. How would I do/fix this program?
Here's the minimal code needed to reproduce this problem, assuming %bcn%, %fcn% and %text% are already set (let's say Cyan as %bcn% and White as %fcn%, and Should not happen as %text%):
#echo off
set bcn=Cyan
set fcn=White
set text="This should not work"
setlocal EnableDelayedExpansion
set script=write-host %text% -ForegroundColor %fcn% -BackgroundColor %bcn%
if EXIST colortest.ps1 ( del colortest.ps1 )
echo %script% >> colortest.ps1
#echo on
#powershell -executionpolicy remotesigned -file colortest.ps1
#echo off
The output of this becomes:
If I do a color that works (like CD), with the same text:
EDIT: Since no one seems to say "Hey it's working perfectly", I meant about the second bogus execution, I know this works fine, I'm just trying to get rid of the bogus second execution.
Here is the entire code of the batch file (save it as bapi.bat and try to run "bapi colortext "This should not work" BF") in a pastebin:
http://pastebin.com/WyayVUVS
The problem can't be reproduced with the posted batch code and the values for the environment variables.
However some suggestions as partly also posted by aschipfl in his comment.
1. Deletion of a single file
The deletion of a single file can be done with either
if exist colortest.ps1 del colortest.ps1
or with
del colortest.ps1 2>nul
The first command line checks first if the file to delete exists and runs command DEL only if the file really exists.
The second command line runs the command DEL and redirects an error message written to handle STDERR to device NUL to suppress it. Command DEL outputs an error message if the file to delete does not exist at all.
On running just a single command after an IF or in a FOR loop it is really not necessary to write this command line in round brackets which define a command block. FOR and IF are designed for execution of a single command and using ( ... ) which defines a command block is just an extension of Windows command processor to be able to run multiple commands where a single command is expected by design.
2. Redirection to file with overwriting existing file
It is possible here to use just > instead of >> to create or overwrite colortest.ps1. This avoids the need to separately delete the file before.
See also the Microsoft article Using command redirection operators.
3. Space character between output text and redirection operator
On the command line
echo %script% >> colortest.ps1
there is a space character between environment variable reference %script% and redirection operator >>. This space character is also output by command ECHO and therefore also written into the file as trailing space.
This does not matter here, but often trailing spaces are not wanted in produced file. One solution is removing the space character and use:
echo %script%>> colortest.ps1
But this can result in an unexpected behavior if the string of environment variable script ends with a space and a single digit number in range of 1 to 9 as this results after preprocessing this command line before execution in 1>> colortest.ps1 or 2>> colortest.ps1, ... which is a redirection of handle 1 to 9 to file colortest.ps1 instead of printing 1 to 9 to handle STDOUT and finally to the file.
The solution is writing the redirection first and next the command
>>colortest.ps1 echo %script%
or use delayed expansion which would be here even better in case of script PowerShell script line contains special characters.
echo !script!>>colortest.ps1
The space character between >> and file name colortest.ps1 is ignored by Windows command interpreter.
4. Quoting parameter strings with special characters
The environment variable text could hold a text which requires double quoting this parameter string for correct processing by Windows command interpreter and by PowerShell. A space character (delimiter on command line) and the characters &()[]{}^=;!'+,`~<>| require often enclosing the entire parameter string in double quotes for being interpreted completely as literal text.
set "script=write-host "!text!" -ForegroundColor %fcn% -BackgroundColor %bcn%"
See also answer on Why is no string output with 'echo %var%' after using 'set var = text' on command line? why set "script=script line" is used instead of just set script=script line and why it does not matter how many double quotes are specified in the script line on assigning it to the environment variable.
5. Double quotes to output in text in PowerShell script file
To output also text containing 1 or more double quotes by the PowerShell script it is necessary to escape each double quote character with one more " before writing the text string into the script file colortest.ps1.
#echo off
setlocal EnableExtensions EnableDelayedExpansion
set "bcn=Cyan"
set "fcn=White"
set "text="This should not happen^^!""
set "text=!text:"=""!"
set "script=write-host "!text!" -ForegroundColor %fcn% -BackgroundColor %bcn%"
echo !script!>colortest.ps1
echo on
#powershell -executionpolicy remotesigned -file colortest.ps1
#echo off
endlocal
It can be seen here on this example that with delayed expansion already enabled on definition of text with a string containing also an exclamation mark to output as character additionally to the also to output two double quotes that the exclamation mark must be escaped with two caret characters ^^ for assigning the exclamation mark as literal character to environment variable text. That would not be necessary on definition of text before enabling delayed expansion.
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "text="This should not happen!""
setlocal EnableDelayedExpansion
set "bcn=Cyan"
set "fcn=White"
set "text=!text:"=""!"
set "script=write-host "!text!" -ForegroundColor %fcn% -BackgroundColor %bcn%"
echo !script!>colortest.ps1
endlocal
echo on
#powershell -executionpolicy remotesigned -file colortest.ps1
#echo off
endlocal
For details about the commands SETLOCAL and ENDLOCAL see this answer which explains in detail how these two commands work.
This is because, for some random reason which didn't happen to the rest, the call command executed 2 times, the second without setting any %fcn% and %bcn% parameters, when something like call :label was used.
However, this does not happen with goto label.
I often see this command in node.js programs: NODE_ENV=test node app.js which sets the NODE_ENV variable to test and works. I also read here https://en.wikipedia.org/wiki/Environment_variable that this should work for any shell command, but running some tests on my own, here is what I see
$ HELLO="WORLD"
$ HELLO="MARS" echo "$HELLO"
WORLD
$
I would expect this to print MARS. Is there something I am missing here?
The syntax VAR=value command means that the command will be invoked with the environment variable VAR set to VALUE, and this will apply only for the scope of that command.
However, when you are using the command line:
HELLO="MARS" echo "$HELLO"
The shell first interprets the "$HELLO" parameter, determines that it is WORLD, and then what it actually does is run:
HELLO="MARS" echo "WORLD"
So the echo may have the HELLO variable set, but it doesn't affect what it prints - it has already been interpreted before.
Doing
HELLO="MARS"; echo "$HELLO"
does something else entirely. First it sets HELLO to MARS in the current shell, and then it goes on to interpret the echo command. By this time HELLO contains MARS, not WORLD. But this is an entirely different effect - the variable HELLO stays with the value MARS, which is not the case in the command without the ;.
Your problem is that echo is just a poor choice for a demonstartion of this. You can do other demonstrations to prove that HELLO is changed properly:
HELLO="MARS" eval 'echo $HELLO'
In this case, the shell will not interpret the $HELLO because it is within a string in single quotes. It will first put MARS in HELLO, and then call the eval 'echo $HELLO' with that variable set. The eval command with then run echo $HELLO, and you'll get the output you were expecting.
This syntax is best used for things that don't use the given variable as part of the command line, but rather use it internally.
Other answers are correct, but here a refinement :
There are 2 cases in fact when defining a list of variable separated by spaces in bash whether it ends or not with a command.
VAR1=value1 VAR2=value2 ... VARn=valuen command arg1 arg2 ... argn
and
VAR1=value1 VAR2=value2 ... VARn=valuen
don't export VAR1 ... VARn the same way.
In first case VAR1 ... VARn will be set only for command and will then not be exported to current shell.
In second case VAR1 ... VARn will alter current shell.
then ( remark that ';' is very same of using a new line )
HELLO=WORLD
HELLO=MARS echo "i don't export HELLO."
echo "HELLO=$HELLO"
will display
i don't export HELLO.
HELLO=WORLD
and
HELLO=WORLD
HELLO=MARS ; echo "i did export HELLO."
echo "HELLO=$HELLO"
will display
i did export HELLO.
HELLO=MARS
I want to create scipt to faciliate producing local text file extracts from Hive.
This is to basically execute commands like below:
hive -e "SET hive.cli.print.header=true;SELECT * FROM dropme"|perl -pe 's/(?:\t|^)\KNULL(?=\t|$)//g'>extract/outbound/dropme.txt
While the above works like a charm I find it quite problematic to implement through the parametrized following script (much simplified):
#!/bin/sh
TNAME=dropme
SQL="SELECT * FROM $TNAME"
echo $SQL
echo "SQL: $SQL"
EXTRACMD="hive -e \"SET hive.cli.print.header=true;$SQL\"|perl -pe 'BEGIN{if(defined(\$_=<ARGV>)){s/\b\w+\.//g;print}}s/(?:\t|^)\KNULL(?=\t|$)//g'>extract/outbound/$TNAME.txt"
echo "CMD: $EXTRACMD";
${EXTRACMD}
When run I get: Exception in thread "main" java.lang.NumberFormatException: For input string: "e"
I know there may be many flavours you can print the text or execute command. For instance the line echo $SQL prints me list of files in the directory instead:
SELECT file1.txt file2.txt file3.txt file4.txt FROM dropme
while the next one: echo "SQL: $SQL" gives just what I want: SQL: SELECT * FROM dropme
echo "CMD: $EXTRACMD" prints the (almost) the command to be executed. Almost, as I see \t in perl code being expanded:
CMD: hive -e "SET hive.cli.print.header=true;SELECT * FROM dropme"|perl -pe 'BEGIN{if(defined($_=<ARGV>)){s\w+\.//g;print}}s/(?: |^)\KNULL(?= |$)//g'>extract/outbound/dropme.txt
Maybe that's still ok, but what I want is to be able to copy&paste this command into (other) terminal and execute as the command I put at the top. Ideally I would like that command to be exactly the same (so with \t there)
Biggest problem I have comes when I try to execute it (${EXTRACMD} line). I'm getting the error:
Exception in thread "main" java.lang.NumberFormatException: For input string: "e" …and so on, irrelevant as bash treats every 'word' as single command here. I assume as I don't even know what is really tries to run (prior print attempt obviously doesn't help)
I'm aware that I have multiple options, like:
escaping special characters in the command definition string (like I did with doublequotes)
experimenting with echo and $VAR, '$VAR' or "$VAR"
experimenting with "${EXTRACMD}" or evaluating through eval "${EXTRACMD}"
experimenting with shopt -s extglob or set -f
but as number of combinations is quite large and with my little bash experience I feel it's better to ask for good practice here so my question is:
Is there a way to print a (complex/compound shell) command first and subsequently be able to execute it (exactly as per printed output)? In this case it would be printing the exact command from the top, then executing it the same way as by manually copying that output into terminal prompt and pressing Enter.
Do not construct commands as strings. See http://mywiki.wooledge.org/BashFAQ/050 for details.
That page also talks about a built-in way of getting the shell to tell you what it is running (section 6).
If that doesn't do what you want you can also, with bash, try using printf %q\\n "${arr[*]}".
I'm confronted with a rather strange problem an echo command causes in a script.
It's supposed to be really REALLY basic stuff, but still, there's something "off".
Suppose, I have this script:
#!/bin/bash
# SERVERPID='cat lite_server_pid.txt'
# kill -9 $SERVERPID
nohup java -Xmx3G -Xms2G -jar tekkit_lite_065.jar nogui > output.txt &
echo $! > lite_server_pid.txt
Yes, this starts my own little Minecraft/Tekkit-Server. ;-)
The Problem is, the file thats created is (for some reason) named
lite_server_pid.txt?
and YES, this includes the "?"! Doing the same command in shell, a file without ? is correctly created! Also, the content of the file is the desired processID.
Still, the ? following the filename is a major problem...
What am I doing wrong?
Check your file for DOS line endings. I suspect that ? is actually your terminal's attempt to display a carriage return (\r). Since bash expects UNIX-style newlines, the carriage return part of the DOS newline (\r\n) is treated as a legal character for the file name.
Run your script through dos2unix.