III am writing a Perl script that will need to SSH out to numerous remote servers to perform some gzipping of log files. In the following line, I keep receiving this error and am struggling to determine what's causing this. The error I'm getting is;
bash: -c: line 0: syntax error near unexpected token `('
bash: -c: line 0: `cd /appdata/log/cdmbl/logs/; echo cd /appdata/log/cdmbl/logs/; find . -type f ( -iname '*' ! -iname '*.gz' ) -mmin +1440 ;; exit 0'
And of course, as you can tell by the error, the line I am trying to write is;
my $id = qx{ssh -q $cur_host "cd $log_path; echo cd $log_path; find . -type f \( -iname '*' ! -iname '*.gz' \) -mmin +1440 \;; exit 0"};
Am I overlooking something here that is causing the unexpected token '(' issue I am
receiving?
NOTE: I removed the -exec from find just so I could see if I can get past this issue first.
Thanks.
You need to backslash the parentheses for the shell. Using single backslash in double quotes is not enough, Perl removes the backslash. Use double backslash \\(.
This is probably not going to answer your question, but it's a nice alternative that I would like to propose.
You said you cannot install additional modules on the production servers. You need to run a bunch of stuff where you are looking for files and zipping them. That can all be done in Perl, and you may have more controll over it than through the "doing command line stuff from a Perl script" approach.
Take a look at Object::Remote, which was written for exactly that purpose. It lets you ssh into machines and run Perl stuff there that you have installed on your local machine. That way, you do not need to add modules or install anything on the remote. All it needs is any kind of more or less recent Perl, which fortunately almost every Linux comes with.
There is a very good lightning talk about it by the author Matt Trout that is well worth watching.
If the command you built results in a syntax error, wouldn't the first step be to see what command you built?
print qq{ssh -q $cur_host "cd $log_path; echo cd $log_path; find . -type f \( -iname '*' ! -iname '*.gz' \) -mmin +1440 \;; exit 0"}, "\n";
Related
I am trying to find executable files. Trying to use bourne shell /bin/sh for greater portability. Below script echos everything with find: at beginning of string.
#!/bin/sh
DIRS=`find / -perm -4000`
for DIR in "$DIRS"
do
case "$DIR" in
find:*);;
esac
done
QUESTION) Why is it echoing for find:*) when no commands are given?
If i add *) echo "$DIR";; clause to the case statement, it will echo the files that are executable for current user, this is all i really want, but isn't happening (i haven't scripted for /bin/sh really, but this has bewildered me)
Yeah sed, awk, cut can help immensely, but some of these commands most likely will not be available (why aren't they available. because they might not be!) so i thought a bourne shell version is more portable. Maybe there is a better way for /bin/sh substring matching, any ideas?
The lines that you are trying to get rid of presumably look like this:
find: `/root': Permission denied
That's an error message. The command substitution
`find ...`
only captures output, not errors. You need to add a redirection to include the errors:
`find ... 2>&1`
Also, -perm 4000 is the setuid bit, not an executable bit.
You can put find directly in the for loop
for DIR in `find / -perm -4000`
I'm trying to create a symbolic link (soft link) from the results of a find command. I'm using sed to remove the ./ that precedes the file name. I'm doing this so I can paste the file name to the end of the path where the link will be saved. I'm working on this with Ubuntu Server 8.04.
I learned from this post, which is kind of the solution to my problem but not quite-
How do I selectively create symbolic links to specific files in another directory in LINUX?
The resulting file name didn't work, though, so I started trying to learn awk and then decided on sed.
I'm using a one-line loop to accomplish this. The problem is that the structure of the loop is separating the filename, creating a link for each word in the filename. There are quite a few files and I would like to automate the process with each link taking the filename of the file it's linked to.
I'm comfortable with basic bash commands but I'm far from being a command line expert. I started this with ls and awk and moved to find and sed. My sed syntax could probably be better but I've learned this in two days and I'm kind of stuck now.
for t in find -type f -name "*txt*" | sed -e 's/.//' -e 's$/$$'; do echo ln -s $t ../folder2/$t; done
Any help or tips would be greatly appreciated. Thanks.
Easier:
Go to the folder where you want to have the files in and do:
find /path/with/files -type f -name "*txt*" -exec ln -s {} . ';'
Execute your for loop like this:
(IFS=$'\n'; for t in `find -type f -name "*txt*" | sed 's|.*/||'`; do ln -s $t ../folder2/$t; done)
By setting the IFS to only a newline, you should be able to read the entire filename without getting splitted at space.
The brackets are to make sure the loop is executed in a sub-shell and the IFS of the current shell does not get changed.
I am calling many Perl scripts in my Bash script (sometimes from csh also).
At the start of the Bash script I want to put a test which checks if all the Perl scripts are devoid of any compilation errors.
One way of doing this would be to actually call the Perl script from the Bash script and grep for "compilation error" in the piped log file, but this becomes messy as different Perl scripts are called at different points in the code, so I want to do this at the very start of the Bash script.
Is there a way to check if the Perl script has no compilation error?
Beware!!
Using the below command to check compilation errors in your Perl program can be dangerous.
$ perl -c yourperlprogram
Randal has written a very nice article on this topic which you should check out
Sanity-checking your Perl code (Linux Magazine Column 91, Mar 2007)
Quoting from his article:
Probably the simplest thing we can tell is "is it valid?". For this,
we invoke perl itself, passing the compile-only switch:
perl -c ourprogram
For this operation, perl compiles the program,
but stops just short of the execution phase. This means that every
part of the program text is translated into the internal data
structure that represents the working program, but we haven't actually
executed any code. If there are any syntax errors, we're informed, and
the compilation aborts.
Actually, that's a bit of a lie. Thanks to BEGIN blocks (including
their layered-on cousin, the use directive), some Perl code may have
been executed during this theoretically safe "syntax check". For
example, if your code contains:
BEGIN { warn "Hello, world!\n" }
then you will see that message,
even during perl -c! This is somewhat surprising to people who
consider "compile only" to mean "executes no code". Consider the
code that contains:
BEGIN { system "rm", "-rf", "/" }
and you'll see the problem with
that argument. Oops.
Apart from perl -c program.pl, it's also better to find warnings using the command:
perl -w program.pl
For details see: http://www.perl.com/pub/2004/08/09/commandline.html
I use the following part of a bash func for larger perl projects :
# foreach perl app in the src/perl dir
while read -r dir ; do
echo -e "\n"
echo "start compiling $dir ..." ;
cd $product_instance_dir/src/perl/$dir ;
# run the autoloader utility
find . -name '*.pm' -exec perl -MAutoSplit -e 'autosplit($ARGV[0], $ARGV[1], 0, 1, 1)' {} \;
# foreach perl file check the syntax by setting the correct INC dirs
while read -r file ; do
perl -MCarp::Always -I `pwd` -I `pwd`/lib -wc "$file"
# run the perltidy inline
# perltidy -b "$file"
# sleep 3
ret=$? ;
test $ret -ne 0 && break 2 ;
done < <(find "." -type f \( -name "*.pl" -or -name "*.pm" \))
test $ret -ne 0 && break ;
echo "stop compiling $dir ..." ;
echo -e "\n\n"
cd $product_instance_dir ;
done < <(ls -1 "src/perl")
When you need to check errors/warnings before running but your file depends on mutliple other files you can add option -I:
perl -I /path/to/dependency/lib -c /path/to/file/to/check
Edit: from man perlrun
Directories specified by -I are prepended to the search path for modules (#INC).
I'm looking for help with a one-liner that I can run from the Mac OS X terminal. I use MAMP for web development on my Mac. I have a lot of CakePHP projects in my "/Applications/MAMP/htdocs" directory. For the sake of simplicity, let's just say that I had two CakePHP projects and that this was the output of the find /Applications/MAMP/htdocs -type d -iname Controller* command:
/Applications/MAMP/htdocs/my_cake1.3_project/app/controllers
/Applications/MAMP/htdocs/my_cake1.3_project/app/tests/cases/controllers
/Applications/MAMP/htdocs/my_cake1.3_project/cake/console/templates/skel/controllers
/Applications/MAMP/htdocs/my_cake1.3_project/cake/console/templates/skel/tests/cases/controllers
/Applications/MAMP/htdocs/my_cake1.3_project/cake/libs/controller
/Applications/MAMP/htdocs/my_cake1.3_project/cake/tests/cases/libs/controller
/Applications/MAMP/htdocs/my_cake1.3_project/cake/tests/test_app/controllers
/Applications/MAMP/htdocs/my_cake1.3_project/cake/tests/test_app/plugins/test_plugin/controllers
/Applications/MAMP/htdocs/my_cake2_project/app/Controller
/Applications/MAMP/htdocs/my_cake2_project/app/Test/Case/Controller
/Applications/MAMP/htdocs/my_cake2_project/lib/Cake/Console/Templates/skel/Controller
/Applications/MAMP/htdocs/my_cake2_project/lib/Cake/Console/Templates/skel/Test/Case/Controller
/Applications/MAMP/htdocs/my_cake2_project/lib/Cake/Controller
/Applications/MAMP/htdocs/my_cake2_project/lib/Cake/Test/Case/Controller
/Applications/MAMP/htdocs/my_cake2_project/lib/Cake/Test/test_app/Controller
/Applications/MAMP/htdocs/my_cake2_project/lib/Cake/Test/test_app/Plugin/TestPlugin/Controller
Now, sometimes I want to find a piece of code that I know I used in one of my CakePHP projects' controllers, but I can't remember which project it was, so I want to search all of them. I don't want to waste time searching in the "app/tests/cases/controllers" folder or any of the ones within "cake/", though. The find /Applications/MAMP/htdocs -type d -iname Controller* | grep -i /app/Controller command gives me the list of folders I want to search in:
/Applications/MAMP/htdocs/my_cake1.3_project/app/controllers
/Applications/MAMP/htdocs/my_cake2_project/app/Controller
I just need to find a way to take that output, add a slash and asterisk (/*) to the end of each line, and pipe each line to the grep -il "string to search for" command. Any help would be appreciated. Thanks!
solution 1
maybe you want to check two options of find command: (i)path and regex
with them you could narrow your find result and pass found files to your grep -il "searchString" for example by |xargs . it looks like:
find /Applications/MAMP/htdocs -type f -ipath "*/app/Controller/*.php"
| xargs grep -il 'foo'
with -regex would be more flexiable.
solution 2
however if you really really want to :
find a way to take that output, add a slash and asterisk (/*) to the
end of each line, and pipe each line to the grep -il "string to search
for" command.
(btw, here "pipe" won't work.)
you could do this:
find .(your original find).. |grep -i "/app/Controller"
|sed -r 's#^(.*)$#grep -il "foo" \1/*#g'|sh
the trick was done by the sed....|sh. the sed line will pick the result of your previous grep, add grep command and options :(grep -il "foo") and append "/*" in order to construct a complete grep command. finally pipe to sh, to execute it.
Have you tried this?
find /Applications/MAMP/htdocs -type d -iname Controller*
-exec grep -il "string to search for" {} /;
I have a perl script which is used to process some data files from a given directory. I have written below bash script to look for the last updated file in the given directory and process that file.
cd $data_dir
find \( -type f -mtime -1 \) -exec ./script.pl {} \;
Sometimes, user copied multiple files to the data dir and hence the previous one skipped. The perl script execute only the last updated file. Can you please suggest me how to fix this using bash script.
Try
cd $data_dir
find \( -type f -mtime -1 \) -exec ./script.pl {} +
Note the termination of -exec with a + vs your \;
From the man page
-exec command {} +
This variant of the -exec action runs the specified command on the selected files, but the command line is built by appending each selected file name at the end;
Now that you'll have one or more file names passed into your perl script, you can alter your perl script to iterate over each passed in file name.
If I understood the question correctly, you need to process any files that were created or modified in a directory since the last time your script was run.
In my opinion find is not the right tool to determine those files, because it has no notion of which files it has already seen.
Using any of the -atime/-ctime/-mtime options will either produce duplicates if you run your script twice in the specified period, or miss some files if it is not executed at the right time. The timing intricacies of using these options for something like this are not easy to deal with.
I can propose a few alternatives:
a) Use three directories instead of one: incoming/ processing/ done/. Your users should only be allowed to put files in incoming/. You move any files in there to processing/ with a simple mv incoming/* processing/ before running your perl script. Then you move them from processing/ to done/ when its over.
In my opinion this is the simplest and best solution, and the one used by mail servers etc when dealing with this issue. If I were you and there were not any special circumstances preventing you from doing this, I'd stop reading here.
b) Have your finder script touch a special file (e.g. .timestamp, perhaps in a different directory, so that your users will not tamper with it) when it's done. This will allow your script to remember the last time it was run. Then use
find \( -cnewer .timestamp -o -newer .timestamp \) -type f -exec ./script.pl '{}' ';'
to run your perl script for each file. You should modify your perl script so that it can run repeatedly with a different file name each time. If you can modify it to accept multiple files in one go, you can also run it with
find \( -cnewer .timestamp -o -newer .timestamp \) -type f -exec ./script.pl '{}' +
which will minimise the number of ./script.pl processes. Take care to handle the first run of the find script, when the .timestamp file is missing. A good solution would be to simply ignore it by not using the -*newer options at all in that case. Also keep in mind that there is a race condition where files added after find was started but before touching the timestamp file will not be processed.
c) As a variation of (b), have your script update the timestamp with the time of the processed file that was created/modified most recently. This is tricky, because find cannot order its output on its own. You could use a wrapper around your perl script to handle this:
#!/bin/bash
for i in "$#"; do
find "$i" \( -cnewer .timestamp -o -newer .timestamp \) -exec touch -r '{}' .timestamp ';'
done
./script.pl "$#"
This will update the timestamp if it is called to process a file with a newer mtime or ctime, minimising (but not eliminating) the race condition. It is however somewhat awkward - unavoidable since bash's [[ -nt option seems to only check the mtime. It might be better if your perl script handled that on its own.
d) Have your script store each processed filename and its timestamps somewhere and then skip duplicates. That would allow you to just pass all files in the directory to it and let it sort out the mess. Kinda tricky though...
e) Since your are using Linux, you might want to have a look at inotify and the inotify-tools package - specifically the inotifywait tool. With a bit of scripting it would allow you to process files as they are added in the directory:
inotifywait -e MOVED_TO -e CLOSE_WRITE -m -r testd/ | grep --line-buffered -e MOVED_TO -e CLOSE_WRITE | while read d e f; do ./script.pl "$f"; done
This has no race conditions, as long as your users do not create/copy/move any directories rather than just files.
The perl script will only execute against the file which find gives it. Perhaps you should remove the -mtime -1 option from the find command so that it picks up all the files in the directory?