Watching multiple files with inotify in perl - perl

I need to watch multiple files in Perl, and I am using Linux::Inotify2. However I am encountering an issue in that the first file being watched needs to be modified and hit, then the second, then the first etc etc
For example if the second file is changed before the first, it will not trigger out, or if the first is triggered twice in a row without the second being triggered in between.
This is the section of code I am using which is having this issue.
my $inotify = new Linux::Inotify2;
my $inotify2 = new Linux::Inotify2;
$inotify->watch ("/tmp/rules.txt", IN_MODIFY);
$inotify2->watch ("/tmp/csvrules.out", IN_MODIFY);
while () {
my #events = $inotify->read;
unless (#events > 0){
print "read error: $!";
last ;
}
foreach $mask (#events) {
printf "mask\t%d\n", $mask;
open (WWWRULES, "/tmp/rules.txt");
my #lines = <WWWRULES>;
foreach $line (#lines) {
#things = split(/,/, $line);
addrule(#things[0], #things[1], #things[2], #things[3], trim(#things[4]));
print "PRINTING: #things[0], #things[1], #things[2], #things[3], #things[4]";
close (WWWRULES);
open (WWWRULES, ">/tmp/rules.txt");
close (WWWRULES);
}
}
my #events2 = $inotify2->read;
unless (#events2 > 0){
print "read error: $!";
last ;
}
foreach $mask (#events) {
printf "mask\t%d\n", $mask;
open (SNORTRULES, "/tmp/csvrules.out");
my #lines2 = <SNORTRULES>;
foreach $line2 (#lines2) {
#things2 = split(/,/, $line2);
addrule("INPUT", #things2[0], #things2[1], #things2[2], trim(#things2[3]));
print "PRINTING: INPUT, #things2[0], #things2[1], #things2[2], #things2[3]";
close (SNORTRULES);
open (SNORTRULES, ">/tmp/csvrules.out");
close (SNORTRULES);
}
}
}
Ideally I would like to be watching 3 files but as I cannot get 2 working it seems a little pointless at this stage.
Thanks for any help!

A single inotify object can handle any number of watches. That's one of the advantages of inotify over the older and now obsolete dnotify. So you should be saying:
my $inotify = Linux::Inotify2->new;
$inotify->watch("/tmp/rules.txt", IN_MODIFY);
$inotify->watch("/tmp/csvrules.out", IN_MODIFY);
Then you can see which watch was triggered by checking the fullname property of the event object:
while () {
my #events = $inotify->read;
unless (#events > 0){
print "read error: $!";
last ;
}
foreach my $event (#events) {
print $event->fullname . " was modified\n" if $event->IN_MODIFY;
}
}
The big problem is that your code is modifying the same files that you're watching for modifications. When /tmp/rules.txt is modified, you open it, read it, and then truncate it, which triggers another modification notice, starting the whole process over again. In general, this is hard to solve without race conditions, but in your case, you should be able to just check for an empty file (next if -z $event->fullname).

You seem to be doing checks in serial on something that you want to happen in parallel. You're either going to want to fork a separate process, use threading, or integrate it in with a POE object.
Another option, which may or may not work for your application, is to set your tempdir to something more specific and keep all the files you're working on in there, then just watch the directory as a whole, which would then only require 1 inotify object, if i'm reading this right. (I haven't done anything with this module in particular but I have a pretty good idea of how it works by hooking syscalls to the file system).

Related

Perl File Write Issue

I'm having a really weird problem with this perl script. The basic point is that sometimes a file write/append doesn't happen. On a run of the program, either all of the writes will happen or none of them will. Here is the subroutine, with some comments:
sub process_svs {
my $F;
open($F, '<', $_[0]);
if($log_dups==1) {
open($dfh, '>>',"./duplicates.txt");
}
while (my $line = <$F>) {
chomp $line;
if($line =~ /somepattern/) {
if (! -e "somefile") {
copy("source","dest") or warn ("couldn't copy");
} elsif($log_dups==1) {
system("touch ./duplicates.txt"); # ghetto workaround
print $dfh "./".$_[0]."_files/".$1.",v already exists\n" or die("Couldn't write duplicate"); # problem line
}
}
}
close $F;
}
The print statements to stdout always work, but if I remove the touch ./duplicates.txt crap, nothing is written to duplicates.txt.
The other "weird" thing, is that earlier in the program, I create a directory with perl mkdir, and if the directory exists when the program is run, I don't need the workaround, the duplicates.txt writing works just fine. If I delete the directory, and let the program mkdir it, it doesn't work. Seems relevant, but I can't figure out how since the directory and the text file are not in the same location, or related in any way, that I can think of.
Additionally, I have run it through the debugger, and can see the write call being executed, but inspecting duplicates.txt immediately after the write shows nothing written.
Any possible reasons for this would be greatly appreciated.
If you want to see a modified, but more complete, version of the script, it is here:
use strict;
use warnings;
use File::Copy;
my $svs = $ARGV[0];
my $rhis_str = system("rhis $svs > ./tmp_history");
my $fh;
my $dfh;
my #versions;
my $all_revs = 0;
my $current_rev = "";
my $log_dups = 0;
sub process_svs {
my $F;
open($F, '<', $_[0]);
if($log_dups==1) {
open($dfh, '>>',"./duplicates.txt");
}
while (my $line = <$F>) {
chomp $line;
if($line =~ /something/) {
if (! -e "something") {
copy("source","dest") or warn ("couldn't copy ");
} elsif($log_dups==1) {
system("touch ./duplicates.txt"); # ghetto workaround
print $dfh "something already exists\n" or die("Couldn't write duplicate");
}
}
}
close $F;
}
for(my $i = 0; $i <= scalar #ARGV; $i++) {
my $arg = $ARGV[$i];
if($arg eq "-a") {
$all_revs = 1;
} elsif($arg eq "-r") {
$all_revs = 0;
$current_rev = $ARGV[$i+1];
} elsif($arg eq "--log-dups") {
$log_dups = 1;
}
}
open($fh, '<','./tmp_history') or die(">>> Failed to open ./tmp_history");;
mkdir "./".$svs."_files";
if($all_revs == 1) {
print ">>> Processing all revisions of ".$svs;
if($log_dups==1) {
print" (and logging duplicates)\n";
}
while(my $line = <$fh>) {
chomp $line;
if ($line =~ /something/) {
push #versions, $1;
}
}
}
system("some_cmd &>/dev/null");
process_svs($svs);
}
You're not checking to see if your files opened. This is a very, very basic mistake and you should fix this immediately. Either add or die $! after each open or, better yet, use autodie and it will take care of catching all IO exceptions for you and give you good, consistent error messages.
Most importantly, this will tell you why it failed to open. $! tells you why it failed. You don't have that in your check on print.
print $dfh "./".$_[0]."_files/".$1.",v already exists\n" or die("Couldn't write duplicate"); # problem line
You're checking if print failed, but you're not including $!. Either add $! like die "Couldn't write to duplicate: $!" or use autodie, remove the or die clause, and let autodie take care of it. I recommend the second.
I suspect you'll find that something else is deleting duplicates.txt between the open and the print.
The second thing that grabs my attention is here.
if($log_dups==1) {
open($dfh, '>>',"./duplicates.txt");
}
You're using a global variable $log_dups to decide whether or not to open the file for writing (and not checking if it succeeded). This should be a variable that gets passed into the function, it's just good programming practice. Later you decide whether to print to $dfh based on that global variable.
if (! -e "something") {
copy("source","dest") or warn ("couldn't copy ");
} elsif($log_dups==1) {
system("touch ./duplicates.txt"); # ghetto workaround
print $dfh "something already exists\n" or die("Couldn't write duplicate");
}
Because $log_dups is global it's possible something else is changing $log_dups between deciding to open duplicates.txt and writing to it. To avoid all these problems, and to make the code simpler, $log_dups should be an argument passed into the function.
Furthermore, the filehandle $dfh is inexplicably a global. Same problem, something else could be closing it. It will also not be automatically closed at the end of the function which might leave writes to duplicates.txt buffered until the program exits. $dfh should be a lexical.
Other problems...
my $rhis_str = system("rhis $svs > ./tmp_history");
$rhis_str will contain the exit status of the rhis program. I don't think that's what you want. You don't use this variable anyway.
There's no need to pass ./file to open, it's safe and easier to read to use just pass file. That it's in the current working directory is implied.
If you fix these basic problems and still have trouble, then edit your question with the revised code and we can look again.

Find out in Perl and Windows if a file is writeable/ removable

I would like to build in Perl under Windows a Watch-Dog for a Hot-Folder (I might call it Folder-Watch or, hmm, probably much better: a Hot-Dog).
So far I succeeded in exactly doing that, with Win32::ChangeNotify (see sample below).
But as you might guess reading the source code I ran into a problem when the moving process wants to finish when the copying/creating process of the file in $watchdir has not finished (No such file or directory).
use Win32::ChangeNotifier;
use File::Copy qw(move);
my $notify = Win32::ChangeNotify->new($watchdir, 0, "FILE_NAME");
while (1) {
if ($notify->wait(1_000)) { # 1-second wait cycle
notify->reset;
#foundfiles = File::get_by_ext($watchdir, "csv"); # search and return files in $watchdir with extension "csv"
print "Something has happened! (del/ren/create)\n";
foreach (#foundfiles) {
move($watchdir.$_, $someotherdir.$_) or die "Fehler: $!";
}
#foundfiles = ();
}
}
Is there a way to automatically find out if the file is ready to go, i.e. has been finally created/copied?
I was thinking about something like
while (1) {
move $file if (-w $file) # writeable
wait(1)
}
but that does not seem to work under Windows. I need to solve this under Windows as well as Perl. Other than that I am open to suggestions.
Yes! I solved it (thanks to Сухой27)!
Inserting the following code right before moving the file:
while (1) {
last if writeable($path_in.$_);
print "-";
$| = 1;
sleep(1);
}
...whereas writeable refers to this little sub-marine:
sub writeable {
return open(my $file, ">>", shift);
}
Thanks, and have a nive day! :-)

Perl - Breaking out of a system/backticks command on keypress if it takes a long time

I have a problem I am hoping someone can help with...
I have a foreach loop that executes a backticks command on each iteration, such as greping a folder in a directory for a string (as shown below, greatly simplified for the purposes of explaining my question).
my #folderList = ("/home/bigfolder", "/home/hugefolder", "/home/massivefolder");
my #wordList = ("hello", "goodbye", "dog", "cat");
foreach my $folder (#folderList) {
foreach my $word (#wordList) {
print "Searching for this $word in this $folder\n";
my #output = `grep -R $word $folder`; #this could take hours so the user needs the option to skip/cancel this iteration and go the next one
print "#output\n";
}
}
The problem I am having:
If the folder the backticks grep command is being run against is particularly large or the array of words to check against is particularly large then the backticks command could take hours to complete (which is fine).
But what i want to be able to do is to break out of the inner loop (i.e when a word is being greped for in a folder) and go to the next iteration if it is taking a long time when the user presses a key on the keyboard or enters the word "next" or "exit" for example.
I know that if i wasnt using backticks I could easily break out of a normal loop using something like the following (but the logic of this obviously does not work when a backticks/system call is involved):
use strict;
use warnings;
use Term::ReadKey;
my $n = 0;
while () {
print '.';
last if ReadKey(-1);
$n++;
}
print $n;
There may be a simple solution that I am overlooking but I have never had the need to do this before, so your help is much appreciated, thanks
The solution is to run the long-running program in a background process (and remember the process id of the new process), and keep your user interaction in the foreground process. When the foreground is signaled to break, kill the background process.
All the parts I mentioned are well-explained in previous posts on Stack Overflow.
You are trying to simultaneously run an external command and process keyboard events, so you need to use some asynchronous framework. Asynchronous frameworks are based on either forks, threads, or event loops, and event loops are not appropriate in this case.
Here's an outline of how you could use a fork:
use POSIX ':sys_wait_h'; # defines WNOHANG
foreach my $folder (#folderList) {
foreach my $word (#wordList) {
print "Searching for this $word in this $folder\n";
my $pid = fork();
if ($pid == 0) { # child process
# we are just printing output from the child process; if you want
# to move data from the child process back to the parent, well,
# that's a whole other can of worms
print `grep -R $word $folder`;
exit;
} else { # parent process
while (waitpid($pid, &WNOHANG) != $pid) {
if (Term::ReadKey(-1)) {
kill 'TERM', $pid; # or maybe kill 'KILL', ...
last;
}
}
}
}
}
I understand what people have said regarding background processes, threads and forking and so on, but the option that suited my arrangement the best (and is probably the easier to implement), although I confess may not be the most efficient, best practice or preferred way of doing it, involved using eval and catching user control-c keypresses.
Very Simple Example:
NEXT:foreach $folder (#folders) { #label on the foreach
eval {
$SIG{INT} = sub { break() }; #catches control-c keypress and calls the break subroutine
$var1 = `grep -r "hello" $folder`;
};
sub break {
print "Breaking out of the backticks command and going to next folder \n";
next NEXT;
}
} #ending bracket of foreach loop

Pass argument through command line and popup if the user has not given the input

The code below creates a file and accepts an input argument through the command line.
I want to do two things:
If the user forgot to enter the input on the command line, the system should give some sort of alert or message. Assume if I forgot to give an input argument, then the system should not proceed with the script execution.
Assume if the system tries to create the already existing file, at present we are managing with showing a message like "File already exists", but instead I want to ask something like "File already exists, are you sure you want to override? yes/no". If he answers yes, then simply override the existing one, else the system should ask for another input from the user.
#!/usr/local/bin/perl
#print "content-type: text/html \n\n"; #HTTP HEADER
$numArgs = $#ARGV + 1;
foreach $argnum (0 .. $#ARGV) {
$GET_ALL_USER_INPUTS = "$ARGV[$argnum]\n";
}
#INPUT_ARR = split(/,/, $GET_ALL_USER_INPUTS);
$filename = "DD_WRITE_${INPUT_ARR[0]}.txt";
$GET_ARR_SIZE = scalar #INPUT_ARR;
$CLIENT_NAME = "T-sys";
$DD_CONTENT = "Design Document ${INPUT_ARR[0]} - ${CLIENT_NAME} :-\n";
$DD_CONTENT .= "--------------------------------------";
#get the no length and generate dotted lines
for($i=0;$i<=length(${INPUT_ARR[0]});$i++){
$DD_CONTENT .= "-";
}
$DD_CONTENT .= "--------------\n";
$DD_CONTENT .= "Database Details\n";
if (-e "${filename}") {
print "File exists!";
exit;
}
else {
open(FILE, ">", "$filename") or die "Cannot open $filename - $!";
print FILE "${DD_CONTENT}\n";
close (FILE);
}
I understand the question to be "How do I prompt a user?" because you do not know how to do that. I skip part 1 of the problem description because you already do know about exit.
First, you should replace your command-line argument handling with Getopt::Long. As it is written now, it is needlessly convoluted.
Getting input from a user at run-time is easy with ExtUtils::MakeMaker which already comes with the Perl distribution.
use ExtUtils::MakeMaker qw(prompt);
my $user_answer = prompt 'Okay to overwrite? ';
if ('y' eq $user_answer) { …
I see that you have commented out a piece of code about HTTP. If you intend to run this program under a CGI environment, prompting will not work as you would expect. On the Web, you need a different technology and control flow altogether.
The existence of a command line argument can be determined pretty easily:
if (exists $ARGV[0]) { do_stuff_with_args } else { die "No arguments!"; }

What is a good way to wait until a file updated and then read from it in Perl?

I was wondering if there's a way to wait for a file to be updated, and then read from it once it's updated. So if I have file.txt, I want to wait until something new is written to it, and then read it/process it/etc. Currently I am polling using Time::HiRes::sleep(.01), but I'm wondering if there's a better way. Thanks.
Yes there is a better way. On windows you can use the FileSystemWatcher interface, on Linux, use inotify.
Windows
use Win32::FileSystem::Watcher;
my $watcher = Win32::FileSystem::Watcher->new( "c:\\" );
# or
my $watcher = Win32::FileSystem::Watcher->new(
"c:\\",
notify_filter => FILE_NOTIFY_ALL,
watch_sub_tree => 1,
);
$watcher->start();
print "Monitoring started.";
sleep(5);
# Get a list of changes since start().
my #entries = $watcher->get_results();
# Get a list of changes since the last get_results()
#entries = $watcher->get_results();
# ... repeat as needed ...
$watcher->stop(); # or undef $watcher
foreach my $entry (#entries) {
print $entry->action_name . " " . $entry->file_name . "\n";
}
# Restart monitoring
# $watcher->start();
# ...
# $watcher->stop();
LINUX
use Linux::Inotify2;
my $inotify = new Linux::Inotify2();
foreach (#ARGV)
{
$inotify->watch($_, IN_ALL_EVENTS);
}
while (1)
{
# By default this will block until something is read
my #events = $inotify->read();
if (scalar(#events)==0)
{
print "read error: $!";
last;
}
foreach (#events)
{
printf "File: %s; Mask: %d\n", $_->fullname, $_->mask;
}
}
File::Tail will poll the file, but has a few advantages over your approach:
The poll time is recomputed dynamically based on the number of lines written since the last poll
If the file remains unchanged, polling will slow to avoid using up CPU
File::Tail will detect if the file has been truncated, moved and/or recreated, and silently re-open the file for you
It can tie a regular file handle which you can use like normal without any special API or syntax.
Example from the perldoc:
use File::Tail;
my $ref=tie *FH,"File::Tail",(name=>$name);
while (<FH>) {
print "$_";
}