I can use the command to devices id from /var/log/20140522.device.log.
Here is my command.
for i in `cat /var/log/20140522.device.log`; do echo ---$i; done
Now I want to make it by daily cron job, but it is not working.
for i in `cat /var/log/`/bin/date +%Y%m%d`.device.log`; do echo ---$i; done
Is it possible to put cat and date in one line?
Yes; you can use cat and date on the same line using command substution.
for file in $(cat /var/log/$(date +'%Y%m%d').device.log); do echo ----$file; done
Using the $(…) notation makes that nested operation much simpler than the back-tick notation:
for file in `cat /var/log/\`date +'%Y%m%d'\`.device.log`; do echo ----$file; done
You don't have to think as hard about where backslash escapes are needed.
Also, in Bash, you can avoid the use of cat with:
for file in $(</var/log/$(date +'%Y%m%d').device.log); do echo ----$file; done
A $ character need to be added as follows:
for i in `cat "/var/log/$(date +"%Y%m%d").device.log"`
Related
I have a bash script in which I have a few qsubs. Each of them are waiting for a preivous qsub to be done before starting.
My first qsub consist of sending files in a certain directory to a perl program and having the outfiles printed in a new directory. At the end, I echo the array with all my jobs names. This script works as intented.
mkdir -p /perl_files_dir
for ID_FILES in `ls Infiles_dir/*.txt`;
do
JOB_ID=`echo "perl perl_scirpt.pl $ID_FILES" | qsub -j oe `
JOB_ID_ARRAY="${JOB_ID_ARRAY}:$JOB_ID"
done
echo $JOB_ID_ARRAY
My second qsub is meant to sort all my previous files made with my perl script in a new outfile and to start after all these jobs are done (about 100 jobs) with depend=afterany. Again, this part is working fine.
SORT_JOB=`echo "sort -m -n perl_files_dir/*.txt >>sorted_file.txt" | qsub -j oe -W depend=afterany$JOB_ID_ARRAY`
SORT_ARRAY="${SORT_ARRAY}:$SORT_JOB"
My issue is that in my sorted file, I have a few columns I wish to remove (2 to 6), so I came up with this last line using awk piped to sed with another depend=afterany
SED=`echo "awk '{\$2="";\$3="";\$4="";\$5="";\$6=""; print \$0}' sorted_file.txt \
| sed 's/ //g' >final_file.txt" | qsub -j oe -W depend=afterany$SORT_ARRAY`
This last step creates final_file.txt, but leaves it empty. I added SED= before my echo because it would otherwise give me Command not found.
I tried without the pipe so it would just print everything. Unfortunately it prints nothing.
I assume it is not opening my sorted file and this is why my final file is empty after my sed. If it's the case, then why won't awk read it?
In my script, I am using variables to define my directories and files (with the correct path). I know my issue is not about find my files or directories since they are perfectly defined at the beginning and used throughout the script. I tried to write the whole path instead of a variable and I get the same results.
for ID_FILES in `ls Infiles_dir/*.txt`
Simplify this to
for ID_FILES in Infiles_dir/*.txt
ls lists the files you pass it (except when you pass it directories, then it lists their content). Rather than telling it to display a list of files and parse the output, use the list of files you already have! This is more reliable (parsing the output of ls will fail if the file names contain whitespace or wildcard characters), clearer and faster. Don't parse the output of ls.
SORT_JOB=`echo "sort -m -n perl_files_dir/*.txt >>sorted_file.txt" | qsub -j oe -W depend=afterany$JOB_ID_ARRAY`
You'd make your life simpler if you used the right form of quoting in the right place. Don't use backquotes, because it's difficult to know how to quote things inside. Use $(…) instead, it's exactly equivalent except that it is parsed in a sane way.
I recommend using a here document for the shell snippet that you're feeding to qsub. You have fewer quoting issues to worry about, and it's more readable.
While we're at it, always put double quotes around variable substitutions and command substitutions: "$some_variable", "$(some_command)". Annoyingly, $var in shell syntax doesn't mean “take the value of the variable var”, it means “take the value of the variable var, parse it as a list of wildcard patterns, and replace each pattern by the list of matching files if there are matching files”. This extra stuff is turned off if the substitution happens inside double quotes (or in a here document, by the way): "$var" means “take the value of the variable var”.
SORT_JOB=$(qsub -j oe -W depend="afterany$JOB_ID_ARRAY" <<'EOF'
sort -m -n perl_files_dir/*.txt >>sorted_file.txt
EOF
)
We now get to the snippet where the quoting was actually causing a problem.
SED=`echo "awk '{\$2="";\$3="";\$4="";\$5="";\$6=""; print \$0}' sorted_file.txt \
| sed 's/ //g' >final_file.txt" | qsub -j oe -W depend=afterany$SORT_ARRAY`
The string that becomes the argument to the echo command is:
awk '{$2=;$3=;$4=;$5=;$6=; print $0}' sorted_file.txt | sed 's/ //g' >final_file.txt
This is syntactically incorrect, and that's why you're not getting any output.
You didn't escape the double quotes inside what was meant to be the awk snippet. It's a lot clearer if you use a here document. Also, you don't need the SED= part. You added it because you had a command substitution (a command between …), which substitutes the output of a command. But since you aren't interested in the output of the qsub command, don't take its output, just execute it.
qsub -j oe -W depend="afterany$SORT_ARRAY" <<'EOF'
awk '{$2="";$3="";$4="";$5="";$6=""; print $0}' sorted_file.txt |
sed 's/ //g' >final_file.txt
EOF
I'm not familiar with qsub, but presumably there's a way to get the error output and the return status of the commands it runs. Inspect that error output, you should have seen the errors from awk.
The version of awk that I am using, does not like the character escapes
awk --version
GNU Awk 3.1.7
spuder#cent64$ awk '{\$2="";\$3="";\$4=""; print \$0}' foo.txt
awk: {\$2="";\$3="";\$4=""; print \$0}
awk: ^ backslash not last character on line
Try the following syntax
awk '{for(i=2;i<=7;i++) $i="";print}' foo.txt
As a side note, if you are using Torque 4.x you may not be able to use a comma separated list of jobs with -W depend=, instead you may need to create a new PBS declarative (-W) for each job.
eg...
#Invalid syntax in newer versions of torque
qsub -W depend=foo,bar
Resources
backslash in gawk fields
Print all but the first three columns
http://docs.adaptivecomputing.com/torque/help.htm#topics/commands/qsub.htm#-W
I have to convert filenames having a forward slash to filenames with a back slash in a makefile using mingw32. I used the following sed command to store the result in a variable:
ORIGINAL=./a/b/main1.c ./a/b/main2.c ./a/b/main3.c
sed command:
RESULT=$(shell echo $(ORIGINAL) | sed 's/\//\\/g')
And the resulting output is:
.\a\b\main1.c .abmain2.c .abmain3.c
It works fine if I run it directly on bash. Can anyone tell me whats wrong?
Thanks!
I'm trying to duplicate your test in a makefile, but I don't have your environment.
But I would not use '/' as a command separator if I am searching for the same character. Use another character, like ':'
sed 's:/:\\:g'
RESULT=$(sed 's/\//\\/g' <<< "$ORIGINAL")
$ ORIGINAL='./a/b/main1.c ./a/b/main2.c ./a/b/main3.c'
$ echo "$ORIGINAL"
./a/b/main1.c ./a/b/main2.c ./a/b/main3.c
$ RESULT=$(sed 's/\//\\/g' <<< "$ORIGINAL")
$ echo $RESULT
.\a\b\main1.c .\a\b\main2.c .\a\b\main3.c
Most likely the shell that make is invoking is not bash. However, I find the behavior you're seeing very strange for any shell (it replaces in the first word but not the rest?!?!)
Have you considered using GNU make's $(subst ...) function, instead of the shell?
i want to trim an output of wipe command with sed.
i try to use this one:
wipe -vx7 /dev/sdb 2>&1 | sed -u 's/.*\ \([0-9]\+\).*/\1/g'
but it don't work for some reason.
when i use echo & sed to print the output of wipe command it works!
echo "/dev/sdb: 10%" | sed -u 's/.*\ \([0-9]\+\).*/\1/g'
what i'm doing wrong?
Thanks!
That looks like a progress indicator. They are often output directly to the tty instead of to stdout or stderr. You may be able to use the expect script called unbuffer (source) or some other method to create a pseudo tty. Be aware that there will probably be more junk such as \r, etc., that you may need to filter out.
Demonstration:
$ cat foo
#!/bin/sh
echo hello > /dev/tty
$ a=$(./foo)
hello
$ echo $a
$ a=$(unbuffer ./foo)
$ echo $a
hello
I'd like to use Sed to expand variables inside a file.
Suppose I exported a variable VARIABLE=something, and have a "test" file with the following:
I'd like to expand this: "${VARIABLE}"
I've been trying commands like the following, but to no avail:
cat test | sed -e "s/\(\${[A-Z]*}\)/`eval "echo '\1'"`/" > outputfile
The result is the "outputfile" with the variable still not expanded:
I'd like to expand this: "${VARIABLE}"
Still, running eval "echo '${VARIABLE}' in bash console results in the value "something" being echoed. Also, I tested and that pattern is trully being matched.
The desired output would be
I'd like to expand this: "something"
Can anyone shed a light on this?
Consider your trial version:
cat test | sed -e "s/\(\${[A-Z]*}\)/`eval "echo '\1'"`/" > outputfile
The reason this doesn't work is because it requires prescience on the part of the shell. The sed script is generated before any pattern is matched by sed, so the shell cannot do that job for you.
I've done this a couple of ways in the past. Normally, I've had a list of known variables and their values, and I've done the substitution from that list:
for var in PATH VARIABLE USERNAME
do
echo 's%${'"$var"'}%'$(eval echo "\$$var")'%g'
done > sed.script
cat test | sed -f sed.script > outputfile
If you want to map variables arbitrarily, then you either need to deal with the whole environment (instead of the fixed list of variable names, use the output from env, appropriately edited), or use Perl or Python instead.
Note that if the value of an environment variable contains a slash in your version, you'd run into problems using the slash as the field separator in the s/// notation. I used the '%' since relatively few environment variables use that - but there are some found on some machines that do contain '%' characters and so a complete solution is trickier. You also need to worry about backslashes in the value. You probably have to use something like '$(eval echo "\$$var" | sed 's/[\%]/\\&/g')' to escape the backslashes and percent symbols in the value of the environment variable. Final wrinkle: some versions of sed have (or had) a limited capacity for the script size - older versions of HP-UX had a limit of about 100. I'm not sure whether that is still an issue, but it was as recently as 5 years ago.
The simple-minded adaptation of the original script reads:
env |
sed 's/=.*//' |
while read var
do
echo 's%${'"$var"'}%'$(eval echo "\$$var" | sed 's/[\%]/\\&/g')'%g'
done > sed.script
cat test | sed -f sed.script > outputfile
However, a better solution uses the fact that you already have the values in the output from env, so we can write:
env |
sed 's/[\%]/\\&/g;s/\([^=]*\)=\(.*\)/s%${\1}%\2%/' > sed.script
cat test | sed -f sed.script > outputfile
This is altogether safer because the shell never evaluates anything that should not be evaluated - you have to be so careful with shell metacharacters in variable values. This version can only possibly run into any trouble if some output from env is malformed, I think.
Beware - writing sed scripts with sed is an esoteric occupation, but one that illustrates the power of good tools.
All these examples are remiss in not cleaning up the temporary file(s).
Maybe you can get by without using sed:
$ echo $VARIABLE
something
$ cat test
I'd like to expand this: ${VARIABLE}
$ eval "echo \"`cat test`\"" > outputfile
$ cat outputfile
I'd like to expand this: something
Let shell variable interpolation do the work.
I have a file that contains this kind of paths:
C:\bad\foo.c
C:\good\foo.c
C:\good\bar\foo.c
C:\good\bar\[variable subdir count]\foo.c
And I would like to get the following file:
C:\bad\foo.c
C:/good/foo.c
C:/good/bar/foo.c
C:/good/bar/[variable subdir count]/foo.c
Note that the non matching path should not be modified.
I know how to do this with sed for a fixed number of subdir, but a variable number is giving me trouble. Actually, I would have to use many s/x/y/ expressions (as many as the max depth... not very elegant).
May be with awk, but this kind of magic is beyond my skills.
FYI, I need this trick to correct some gcov binary files on a cygwin platform.
I am dealing with binary files; therefore, I might have the following kind of data:
bindata\bindata%bindataC:\good\foo.c
which should be translated as:
bindata\bindata%bindataC:/good/foo.c
The first \ must not be translated, despite that it is on the same line.
However, I have just checked my .gcno files while editing this text and it looks like all the paths are flanked with zeros, so most of the answers below should fit.
sed -e '/^C:\\good/ s/\\/\//g' input_file.txt
I would recommend you look into the cygpath utility, which converts path names from one format to another. For instance on my machine:
$ cygpath `pwd`
/home/jericson
$ cygpath -w `pwd`
D:\root\home\jericson
$ cygpath -m `pwd`
D:/root/home/jericson
Here's a Perl implementation of what you asked for:
$ echo 'C:\bad\foo.c
C:\good\foo.c
C:\good\bar\foo.c
C:\good\bar\[variable subdir count]\foo.c' | perl -pe 's|\\|/|g if /good/'
C:\bad\foo.c
C:/good/foo.c
C:/good/bar/foo.c
C:/good/bar/[variable subdir count]/foo.c
It works directly with the string, so it will work anywhere. You could combine it with cygpath, but it only works on machines that have that path:
perl -pe '$_ = `cygpath -m $_` if /good/'
(Since I don't have C:\good on my machine, I get output like C:goodfoo.c. If you use a real path on your machine, it ought to work correctly.)
You want to substitute '/' for all '\' but only on the lines that match the good directory path. Both sed and awk will let you do this by having a LHS (matching) expression that only picks the lines with the right path.
A trivial sed script to do this would look like:
/[Cc]:\\good/ s/\\/\//g
For a file:
c:\bad\foo
c:\bad\foo\bar
c:\good\foo
c:\good\foo\bar
You will get the output below:
c:\bad\foo
c:\bad\foo\bar
c:/good/foo
c:/good/foo/bar
Here's how I would do it in awk:
# fixpaths.awk
/C:\\good/ {
gsub(/\\/,"/",$1);
print $1 >> outfile;
}
Then run it using the command:
awk -f fixpaths.awk paths.txt; mv outfile paths.txt
Or with some help from good ol' Bash:
#!/bin/bash
cat file | while read LINE
do
if <bad_condition>
then
echo "$LINE" >> newfile
else
echo "$LINE" | sed -e "s/\\/\//g" >> newfile
fi
done
try this
sed -re '/\\good\\/ s/\\/\//g' temp.txt
or this
awk -F"\\" '{if($2=="good"){OFS="\/"; $1=$1;} print $0}' temp.txt