I have a problem i could not figure out if it's even possible. I am parsing a file with filenames in it, and want to check if those filenames represent an existing file within the system.
i figured out a possibility to to check if a file exists:
[ -f FILENAME ] && echo "File exists" || echo "File does not exists"
now my problem is: How can i pipe into to the conditional that it tests for all the filenames?
i was trying like tihs, but it did not work:
cat myfilenames.txt | xargs command from above without FILENAME
does anybody know if it is possible?
thanks, dmeu!
while read file; dp
[ -e "$file" ] && echo "$file exists";
done <filelist.txt
I believe what you want is a for loop. This worked for me in bash (I put it in a shell script, but you could probably do it on the command line):
for i in `cat $1` ; do
[ -f $i ] && echo File $i exists || echo File $i does not exist
done
the backticks around the cat execute the command and substitute the output into the loop.
Related
Given is a directory with a large number of files.
Also given is a Perl script that I want to run on each file of the directory. But this Perl script has options.
FILES=absolutepathtomyfiles/*
PROGRAMME=absolutepathtoperlscript/script.pl;
for f in $FILES
do
if [[ $f == *.txt ]]; then
absolutepathtoperlscript/script.pl -infile=$f -replace #both necessary options
fi
done
If there are spaces or other strange characters in file names, you may have problems using the for construct. You could do this:
for file in /absolute/path/to/myfiles/*.txt
do
[[ -f "$file" ]] || continue
/absolute/path/to/perl/script/script.pl -infile="$file" -replace
done
Note the [[ -f "$file" ]] || continue. This says that if $file is not a file, skip that file. It's similar to this:
if [[ -f "$file" ]]
then
continue;
fi
If this doesn't work, try this:
export PS4="\$LINENO: "
for file in /absolute/path/to/myfiles/*.txt
do
[[ -f "$file" ]] || continue
set -xv # Turn on debugging
/absolute/path/to/perl/script/script.pl -infile="$file" -replace
set +xv # Turn off debugging
done
This will print out your exact command line you're passing to your Perl script and may help you figure out what your issue could be.
I have a text file and need to remove all lines that DO NOT contain http in them. Alternatively, it could just output all the files that DO contain http in them to the new file.
The name of my original file is list.txt and I need to generate a new file with a name like new.txt
I know that there are several ways to do this via command line, but what I'm really looking for is the quickest way since I need to do this with several files and each of them are a few gigs in size...
The quickest, shortest solution,
fgrep -v "http"
Of course, grep, egrep, awk, perl, etc make this more fungible.
Here is a short shell script. Edit "delhttp.sh" containing,
#!/bin/bash
if [ $# -eq 0 ] ; then
fgrep -v "http"
elif [ $# -eq 1 ] ; then
f1=${1:-"null"}
if [ ! -f $f1 ]; then echo "file $f1 dne"; exit 1; fi
fgrep -v "http" $f1 #> $f2
elif [ $# -eq 2 ]; then
f1=${1:-"null"}
if [ ! -f $f1 ]; then echo "file $f1 dne"; exit 1; fi
f2=${2:-"null"}
fgrep -v "http" $f1 > $f2
fi
Then make this file executable using,
chmod +x delhttp.sh
Here is a perl script (if you prefer), Edit "delhttp.pl" containing,
#!/bin/env perl
use strict;
use warnings;
my $f1=$ARGV[0]||"-";
my $f2=$ARGV[1]||"-";
my ($fh, $ofh);
open($fh,"<$f1") or die "file $f1 failed";
open($ofh,">$f2") or die "file $f2 failed";
while(<$fh>) { if( !($_ =~ /http/) ) { print $ofh "$_"; } }
Again, make this file executable using,
chmod +x delhttp.pl
perl -i -lne 'print if(/http/)' your_file
This above command will delete all the lines from the file if they do not have http.
If you insist on keeping the original file backup, the you can anyhow give and option of ".bak" like mentioned below:
perl -i.bak -lne 'print if(/http/)' your_file
By this your_file.bak will be generated which is nothing but a copy of the original file and original file will be modified according to your need.
Also you can use awk:
awk '/http/' your_file
This will out put to the console. You can anyhow use '>' to store the output in a new file.
You could use grep. Using -v inverts the sense of matching, to select non-matching lines.
grep -v 'http' list.txt
Using Perl one-liner:
perl -ne '/^(?:(?!http).)*$/ and print' list.txt > new.txt
I sometimes want to merge multiple pairs of files, suppose I want to merge fileA.old and fileA.new, as well as fileB.old and fileB.new..and so on.Currently I have to open emacs. Do M-x ediff-merge-files and enter name of first file, return key, name of second file, return key..and im in merge mode...is there a way to launch emacs with both file names as arguments and land in merge mode?
You can pass Lisp code to Emacs through the command line:
emacs --eval '(ediff-merge-files "path/to/file1" "path/to/file2")'
Of course this could be wrapped in a script to make it more convenient to call. For instance, in a bourne shell, you could do a simple version like this:
#!/bin/sh
# check correct invocation
if [ $# != 2 ]; then
echo "USAGE: $(basename "${0}") <file1> <file2>"
exit 1
fi
# check that file1 exists and is readable
if [ -f "${1}" ]; then
if [ ! -r "${1}" ]; then
echo "Cannot open '${1}', access denied."
exit 3
fi
else
echo "File not found: '${1}'"
exit 2
fi
# check that file2 exists and is readable
if [ -f "${2}" ]; then
if [ ! -r "${2}" ]; then
echo "Cannot open '${2}', access denied."
exit 5
fi
else
echo "File not found: '${2}'"
exit 4
fi
# invoke emacs
emacs --eval "(ediff-merge-files \"${1}\" \"${2}\")"
If you save this in a file ediff on your $PATH, you can then simply write:
ediff file1 file2
on the command line and Emacs will pop up with the two given files in ediff-mode.
i want to tail log file with grep and sent it via mail
like:
tail -f /var/log/foo.log | grep error | mail -s subject name#example.com
how can i do this?
You want to send an email when emailing errors occur? That might fail ;)
You can however try something like this:
tail -f $log |
grep --line-buffered error |
while read line
do
echo "$line" | mail -s subject "$email"
done
Which for every line in the grep output sends an email.
Run above shell script with
nohup ./monitor.sh &
so it will keep running in the background.
I'll have a go at this. Perhaps I'll learn something if my icky bash code gets scrutinised. There is a chance there are already a gazillion solutions to do this, but I am not going to find out, as I am sure you have trawled the depths and widths of the cyberocean. It sounds like what you want can be separated into two bits: 1) at regular intervals obtain the 'latest tail' of the file, 2) if the latest tail actually exists, send it by e-mail. For the regular intervals in 1), use cron. For obtaining the latest tail in 2), you'll have to keep track of the file size. The bash script below does that - it's a solution to 2) that can be invoked by cron. It uses the cached file size to compute the chunk of the file it needs to mail. Note that for a file myfile another file .offset.myfile is created. Also, the script does not allow path components in the file name. Rewrite, or fix it in the invocation [e.g. (cd /foo/bar && segtail.sh zut), assuming it is called segtail.sh ].
#!/usr/local/bin/bash
file=$1
size=0
offset=0
if [[ $file =~ / ]]; then
echo "$0 does not accept path components in the file name" 2>&1
exit 1
fi
if [[ -e .offset.$file ]]; then
offset=$(<".offset.$file")
fi
if [[ -e $file ]]; then
size=$(stat -c "%s" "$file") # this assumes GNU stat, possibly present as gstat. CHECK!
# (gstat can also be Ganglias Status tool - careful).
fi
if (( $size < $offset )); then # file might have been reduced in size
echo "reset offset to zero" 2>&1
offset=0
fi
echo $size > ".offset.$file"
if [[ -e $file && $size -gt $offset ]]; then
tail -c +$(($offset+1)) "$file" | head -c $(($size - $offset)) | mail -s "tail $file" foo#bar
fi
How about:
mail -s "catalina.out errors" blah#myaddress.com < grep ERROR catalina.out
Is there an idiomatic way to simulate Perl's diamond operator in bash? With the diamond operator,
script.sh | ...
reads stdin for its input and
script.sh file1 file2 | ...
reads file1 and file2 for its input.
One other constraint is that I want to use the stdin in script.sh for something else other than input to my own script. The below code does what I want for the file1 file2 ... case above, but not for data provided on stdin.
command - $# <<EOF
some_code_for_first_argument_of_command_here
EOF
I'd prefer a Bash solution but any Unix shell is OK.
Edit: for clarification, here is the content of script.sh:
#!/bin/bash
command - $# <<EOF
some_code_for_first_argument_of_command_here
EOF
I want this to work the way the diamond operator would work in Perl, but it only handles filenames-as-arguments right now.
Edit 2: I can't do anything that goes
cat XXX | command
because the stdin for command is not the user's data. The stdin for command is my data in the here-doc. I would like the user data to come in on the stdin of my script, but it can't be the stdin of the call to command inside my script.
Sure, this is totally doable:
#!/bin/bash
cat $# | some_command_goes_here
Users can then call your script with no arguments (or '-') to read from stdin, or multiple files, all of which will be read.
If you want to process the contents of those files (say, line-by-line), you could do something like this:
for line in $(cat $#); do
echo "I read: $line"
done
Edit: Changed $* to $# to handle spaces in filenames, thanks to a helpful comment.
Kind of cheezy, but how about
cat file1 file2 | script.sh
I am (like everyone else, it seems) a bit confused about exactly what the goal is here, so I'll give three possible answers that may cover what you actually want. First, the relatively simple goal of getting the script to read from either a list of files (supplied on the command line) or from its regular stdin:
if [ $# -gt 0 ]; then
exec < <(cat "$#")
fi
# From this point on, the script's stdin is redirected from the files
# (if any) supplied on the command line
Note: the double-quoted use of $# is the best way to avoid problems with funny characters (e.g. spaces) in filenames -- $* and unquoted $# both mess this up. The <() trick I'm using here is a bash-only feature; it fires off cat in the background to feed data from files supplied on the command line, and then we use exec to replace the script's stdin with the output from cat.
...but that doesn't seem to be what you actually want. What you seem to really want is to pass the supplied filenames or the script's stdin as arguments to a command inside the script. This requires sort of the opposite process: converting the script's stdin into a file (actually a named pipe) whose name can be passed to the command. Like this:
if [[ $# -gt 0 ]]; then
command "$#" <<EOF
here-doc goes here
EOF
else
command <(cat) <<EOF
here-doc goes here
EOF
fi
This uses <() to launder the script's stdin through cat to a named pipe, which is then passed to command as an argument. Meanwhile, command's stdin is taken from the here-doc.
Now, I think that's what you want to do, but it's not quite what you've asked for, which is to both redirect the script's stdin from the supplied files and pass stdin to the command inside the script. This can be done by combining the above techniques:
if [ $# -gt 0 ]; then
exec < <(cat "$#")
fi
command <(cat) <<EOF
here-doc goes here
EOF
...although I can't think why you'd actually want to do this.
The Perl diamond operator essentially loops across all the command line arguments, treating each as a filename. It opens each file and reads them line-by-line. Here's some bash code that will do approximately the same.
for f in "$#"
do
# Do something with $f, such as...
cat $f | command1 | command2
-or-
command1 < $f
-or-
# Read $f line-by-line
cat $f | while read line_from_f
do
# Do stuff with $line_from_f
done
done
You want to take the first argument and do something with it, and then either read from any files specified or stdin if no files?
Personally, I'd suggest using getopt to indicate arguments using the "-a value" syntax to help disambiguate, but that's just me. Here's how I'd do it in bash without getopts:
firstarg=${1?:usage: $0 arg [file1 .. fileN]}
shift
typeset -a files
if [[ ${##} -gt 0 ]]
then
files=( "$#" )
else
files=( "/dev/stdin" )
fi
for file in "${files[#]}"
do
whatever_you_want < "$file"
done
The ?: operator will die if there are no args specified, since you seem to want at least one arg either way. After grabbing that, shift the args over by one, and then either use the remaining args as your file list, or the bash special filehandle "/dev/stdin" if there were no other args.
I think that the "if no files are specified, use /dev/stdin - otherwise use the files on the command line" piece is probably what you're looking for, but the rest of the code is at least useful for context.
Also a little cheezy, but how about this:
if [[ $# -eq 0 ]]
then
# read from stdin
else
# read from $* (args)
fi
If you need to read and process line-by-line (which is likely) and don't want to copy/paste the same code twice (which is likely), define a function in your script and just pass the lines one-by-one to this function, and process them in said function.
Why not use ``cat #* in the script? For example:
x=`cat $*`
echo $x