I am using below code to write log message to text file, the program is getting hanged when different source calls this method in parallel. Is there a way to grant /control parallel writing without breaking the program.
sub sLog {
my $self = 'currentServerDirectory';
my $logMsg = "";
my $fileName = join '_', $self->{LogFilePrefix}, &sTimeStamp("Log");
my $absFileName = "$self->{LogFolder}/$fileName.txt";
open APPHDLER, ">>$absFileName" or &exitErr("Cannot append message to file, $absFileName");
print APPHDLER scalar(localtime(time))." - $logMsg\n";
close APPHDLER;
}
Try using flock -- here is a simple example you can try to understand its behavior:
use strict;
use warnings;
use Fcntl qw/:flock/;
chomp(my $id = `date`);
my $cnt = 0;
while (1) {
open LOG, ">>shared" or die "Couldn't open shared";
flock (LOG, LOCK_EX) or die "Couldn't flock";
print LOG "$id: $cnt\n";
$cnt++;
sleep 1;
close LOG;
}
Say this is saved in flock.pl, then you can run flock.pl& to run one or more instances in the background. Then do tail -f shared to see what happens. Since you are sleeping 1 second between obtaining the lock and releasing it via close LOG , you'll see an update once a second if you have one process. However, if you have N processes, you'll see each one taking N seconds.
In your existing example, you can try adding the use Fcntl and flock lines.
When you open a file for writing, a lock on that file is granted to the process that opened it. This is done to prevent corruption of data, by processes overwriting each other. Possible solutions would be to feed the output data to a single process that handles writing to the log file, making sure that processes close the file and release their locks when they are finished writing, or using a library or file format that is designed for parallel access of files. The first two of those methods would be the easiest and preferred for writing log files like this. There are also probably perl modules (check CPAN) that handle log files.
I have a Perl file. The user opens a file, reads data and displays the data in grid. user edits it and saves it back to the file.
I am trying to use flock so that when the user reads the file, the file gets locked. I tried below code but it didnt work.
Referring to the accepted answer of this post. How do I lock a file in Perl?
use Fcntl ':flock'; #added this at the start
$filename= dsfs.com/folder1/test.txt; #location of my file
open(my $fh, '<', $filename) or die $!; #file open
flock($fh, LOCK_EX) or die "Could not lock '$file' - $!"; #inserted flock before reading starts so that no other user can use this file
#reading of file starts here
#once read, user saves file.
close($fh) or die "Could not write '$file' - $!"; #release lock after user writes.
I guess this is a normal operation without any race around conditions but this doesnot work for me.I am not sure if the perl script is able to detect flock or not.
For testing purposes, i try to open the file before my writing and saving function gets completed. when i try to open the same file before saving gets completed, it means that the lock is not released yet. in this situation if i open the file at backend and edit the file, i am still able to save changes. In practical case, it should not be able to edit anything once the file is locked.
can anyone please suggest me any troubleshooting for this or is my procedure of using flock incorrect ??
There's another problem if your flock implementation is based on lockf(3) or fcntl(2), which it probably is. Namely, LOCK_EX should be used with "write intent", on a file opened for output.
For lockf(3), perldoc -f flock says
Note that the emulation built with lockf(3) doesn't provide shared locks, and it requires that FILEHANDLE be open with write intent.
and for fcntl(2):
Note that the fcntl(2) emulation of flock(3) requires that FILEHANDLE be open with read intent to use LOCK_SH and requires that it be open with write intent to use LOCK_EX.
A workaround for input files or for more complicated synchronized operations is for all processes to sync on a trivial lock file, like:
open my $lock, '>>', "$filename.lock";
flock $lock, LOCK_EX;
# can't get here until our process has the lock ...
open(my $fh, '<', $filename) or die $!; #file open
... read file, manipulate ...
close $fh;
open my $fh2, '>', $filename;
... rewrite file ...
close $fh2;
# done with this operation, can release the lock and let another
# process access the file
close $lock;
There's two problems:
flock will block until it can lock. You therefore need flock ( $file, LOCK_EX | LOCK_NB ) or die $!;
flock (on Unix) is advisory. It won't stop them accessing it unless they also check for a lock.
I opened a file to read from line by line:
open(FH,"<","$myfile") or die "could not open $myfile: $!";
while (<FH>)
{
# ...do something
}
Later on in the program, I try to re-read the file (walk thru the file again):
while (<FH>)
{
# ...do something
}
and realized that it is as if the control within file is at the EOF and will not iterate from first line in the file.... is this default behavior? How to work around this? The file is big and I do not want to keep in memory as array. So is my only option is to close and open the file again?
Use seek to rewind to the beginning of the file:
seek FH, 0, 0;
Or, being more verbose:
use Fcntl;
seek FH, 0, SEEK_SET;
Note that it greatly limits the usefulness of your tool if you must seek on the input, as it can never be used as a filter. It is extremely useful to be able to read from a pipe, and you should strive to arrange your program so that the seek is not necessary.
You have a few options.
Reopen the file handle
Set the position to the beginning of the file using seek, as William Pursell suggested.
Use a module such as Tie::File, which lets you read the file as an array, without loading it into memory.
I end up having my script appending the new changes that I wanted to make to the end of the file instead of in the actual file.
open (INCONFIG, "+<$text") or die $!;
#config = <INCONFIG>;
foreach(#config)
{
if ( $_ =~ m/$checker/ )
{
$_ = $somethingnew;
}
print INCONFIG $_;
}
close INCONFIG or die;
Ultimately I wanted to rewrite the whole text again, but with certain strings modified if it matched the search criterion. But so far it only appends ANOTHER COPY of the entire file(with changes) to the bottom of the old file.
I know that I can just close the file, and use another write file -handle and parse it in. But was hoping to be able to learn what I did wrong, and how to fix it.
As I understand open, using read/write access for a text file isn't a good idea. After all a file just is a byte stream: Updating a part of the file with something of a different length is the stuff headaches are made of ;-)
Here is my approach: Try to emulate the -i "inplace" switch of perl. So essentially we write to a backup file, which we will later rename. (On *nix system, there is some magic with open filehandles keeping deleted files available, so we don't have to create a new file. Lets do it anyway.)
my $filename = ...;
my $tempfile = "$filename.tmp";
open my $inFile, '<', $filename or die $!;
open my $outFile, '>', $tempfile or die $!;
while (my $line = <$inFile>) {
$line = doWeirdSubstitutions($line);
print $outFile $line;
}
close $inFile or die $!;
close $outFile or die $!;
rename $tempfile, $filename
or die "rename failed: $!"; # will break under weird circumstances.
# delete temp file
unlink $tempfile or die $!;
Untested, but obvious code. Does this help with your problem?
Your problem is a misunderstanding of what <+ "open for update" does. It is discussed in the Perl Tutorial at
Mixing Reads and Writes.
What you really want to do is copy the old file to a new file and then rename it after the fact. This is discussed in the perlfaq5 mentioned by daxim. Plus there are entire modules dedicated to doing this safely, such as File::AtomicWrite. These help with the issue of your program aborting and leaving you with a clobbered file.
As others pointed out, there are better ways :)
But if you really want to read and write using +<, you should remember that, after reading the file, you're at the end of the file... That explains that your output is appended after the original content.
What you need to do is reset the file-pointer to the beginning of the file, using seek:
seek(INCONFIG ,0,0);
Then start writing...
perlopentut says this about mixing reads and writes
In fact, when it comes to updating a file, unless you're working on a
binary file as in the WTMP case above, you probably don't want to use
this approach for updating. Instead, Perl's -i flag comes to the
rescue.
Another way is to use the Tie::File module. The code reduces to just this:
tie my #config, 'Tie::File', $text or die $!;
s/$checker/$somethingnew/g for #config;
But remember to back the file up before you modify it until you have debugged your program.
What is the best way to create a lock on a file in Perl?
Is it best to flock on the file or to create a lock file to place a lock on and check for a lock on the lock file?
If you end up using flock, here's some code to do it:
use Fcntl ':flock'; # Import LOCK_* constants
# We will use this file path in error messages and function calls.
# Don't type it out more than once in your code. Use a variable.
my $file = '/path/to/some/file';
# Open the file for appending. Note the file path is quoted
# in the error message. This helps debug situations where you
# have a stray space at the start or end of the path.
open(my $fh, '>>', $file) or die "Could not open '$file' - $!";
# Get exclusive lock (will block until it does)
flock($fh, LOCK_EX) or die "Could not lock '$file' - $!";
# Do something with the file here...
# Do NOT use flock() to unlock the file if you wrote to the
# file in the "do something" section above. This could create
# a race condition. The close() call below will unlock the
# file for you, but only after writing any buffered data.
# In a world of buffered i/o, some or all of your data may not
# be written until close() completes. Always, always, ALWAYS
# check the return value of close() if you wrote to the file!
close($fh) or die "Could not write '$file' - $!";
Some useful links:
PerlMonks file locking tutorial (somewhat old)
flock() documentation
In response to your added question, I'd say either place the lock on the file or create a file that you call 'lock' whenever the file is locked and delete it when it is no longer locked (and then make sure your programs obey those semantics).
The other answers cover Perl flock locking pretty well, but on many Unix/Linux systems there are actually two independent locking systems: BSD flock() and POSIX fcntl()-based locks.
Unless you provide special options to configure when building Perl, its flock will use flock() if available. This is generally fine and probably what you want if you just need locking within your application (running on a single system). However, sometimes you need to interact with another application that uses fcntl() locks (like Sendmail, on many systems) or perhaps you need to do file locking across NFS-mounted filesystems.
In those cases, you might want to look at File::FcntlLock or File::lockf. It is also possible to do fcntl()-based locking in pure Perl (with some hairy and non-portable bits of pack()).
Quick overview of flock/fcntl/lockf differences:
lockf is almost always implemented on top of fcntl, has file-level locking only. If implemented using fcntl, limitations below also apply to lockf.
fcntl provides range-level locking (within a file) and network locking over NFS, but locks are not inherited by child processes after a fork(). On many systems, you must have the filehandle open read-only to request a shared lock, and read-write to request an exclusive lock.
flock has file-level locking only, locking is only within a single machine (you can lock an NFS-mounted file, but only local processes will see the lock). Locks are inherited by children (assuming that the file descriptor is not closed).
Sometimes (SYSV systems) flock is emulated using lockf, or fcntl; on some BSD systems lockf is emulated using flock. Generally these sorts of emulation work poorly and you are well advised to avoid them.
CPAN to the rescue: IO::LockedFile.
Ryan P wrote:
In this case the file is actually unlocked for a short period of time while the file is reopened.
So don’t do that. Instead, open the file for read/write:
open my $fh, '+<', 'test.dat'
or die "Couldn’t open test.dat: $!\n";
When you are ready to write the counter, just seek back to the start of the file. Note that if you do that, you should truncate just before close, so that the file isn’t left with trailing garbage if its new contents are shorter than its previous ones. (Usually, the current position in the file is at its end, so you can just write truncate $fh, tell $fh.)
Also, note that I used three-argument open and a lexical file handle, and I also checked the success of the operation. Please avoid global file handles (global variables are bad, mmkay?) and magic two-argument open (which has been a source of many a(n exploitable) bug in Perl code), and always test whether your opens succeed.
I think it would be much better to show this with lexical variables as file handlers
and error handling.
It is also better to use the constants from the Fcntl module than hard code the magic number 2 which might not be the right number on all operating systems.
use Fcntl ':flock'; # import LOCK_* constants
# open the file for appending
open (my $fh, '>>', 'test.dat') or die $!;
# try to lock the file exclusively, will wait till you get the lock
flock($fh, LOCK_EX);
# do something with the file here (print to it in our case)
# actually you should not unlock the file
# close the file will unlock it
close($fh) or warn "Could not close file $!";
Check out the full documentation of flock and the File locking tutorial on PerlMonks even though that also uses the old style of file handle usage.
Actually I usually skip the error handling on close() as there is not
much I can do if it fails anyway.
Regarding what to lock, if you are working in a single file then lock that file. If you need to lock several files at once then - in order to avoid dead locks - it is better to pick one file that you are locking. Does not really matter if that is one of the several files you really need to lock or a separate file you create just for the locking purpose.
Have you considered using the LockFile::Simple module? It does most of the work for you already.
In my past experience, I have found it very easy to use and sturdy.
use strict;
use Fcntl ':flock'; # Import LOCK_* constants
# We will use this file path in error messages and function calls.
# Don't type it out more than once in your code. Use a variable.
my $file = '/path/to/some/file';
# Open the file for appending. Note the file path is in quoted
# in the error message. This helps debug situations where you
# have a stray space at the start or end of the path.
open(my $fh, '>>', $file) or die "Could not open '$file' - $!";
# Get exclusive lock (will block until it does)
flock($fh, LOCK_EX);
# Do something with the file here...
# Do NOT use flock() to unlock the file if you wrote to the
# file in the "do something" section above. This could create
# a race condition. The close() call below will unlock it
# for you, but only after writing any buffered data.
# In a world of buffered i/o, some or all of your data will not
# be written until close() completes. Always, always, ALWAYS
# check the return value on close()!
close($fh) or die "Could not write '$file' - $!";
My goal in this question was to lock a file being used as a data store for several scripts. In the end I used similar code to the following (from Chris):
open (FILE, '>>', test.dat') ; # open the file
flock FILE, 2; # try to lock the file
# do something with the file here
close(FILE); # close the file
In his example I removed the flock FILE, 8 as the close(FILE) performs this action as well. The real problem was when the script starts it has to hold the current counter, and when it ends it has to update the counter. This is where Perl has a problem, to read the file you:
open (FILE, '<', test.dat');
flock FILE, 2;
Now I want to write out the results and since i want to overwrite the file I need to reopen and truncate which results in the following:
open (FILE, '>', test.dat'); #single arrow truncates double appends
flock FILE, 2;
In this case the file is actually unlocked for a short period of time while the file is reopened. This demonstrates the case for the external lock file. If you are going to be changing contexts of the file, use a lock file. The modified code:
open (LOCK_FILE, '<', test.dat.lock') or die "Could not obtain lock";
flock LOCK_FILE, 2;
open (FILE, '<', test.dat') or die "Could not open file";
# read file
# ...
open (FILE, '>', test.dat') or die "Could not reopen file";
#write file
close (FILE);
close (LOCK_FILE);
Developed off of http://metacpan.org/pod/File::FcntlLock
use Fcntl qw(:DEFAULT :flock :seek :Fcompat);
use File::FcntlLock;
sub acquire_lock {
my $fn = shift;
my $justPrint = shift || 0;
confess "Too many args" if defined shift;
confess "Not enough args" if !defined $justPrint;
my $rv = TRUE;
my $fh;
sysopen($fh, $fn, O_RDWR | O_CREAT) or LOGDIE "failed to open: $fn: $!";
$fh->autoflush(1);
ALWAYS "acquiring lock: $fn";
my $fs = new File::FcntlLock;
$fs->l_type( F_WRLCK );
$fs->l_whence( SEEK_SET );
$fs->l_start( 0 );
$fs->lock( $fh, F_SETLKW ) or LOGDIE "failed to get write lock: $fn:" . $fs->error;
my $num = <$fh> || 0;
return ($fh, $num);
}
sub release_lock {
my $fn = shift;
my $fh = shift;
my $num = shift;
my $justPrint = shift || 0;
seek($fh, 0, SEEK_SET) or LOGDIE "seek failed: $fn: $!";
print $fh "$num\n" or LOGDIE "write failed: $fn: $!";
truncate($fh, tell($fh)) or LOGDIE "truncate failed: $fn: $!";
my $fs = new File::FcntlLock;
$fs->l_type(F_UNLCK);
ALWAYS "releasing lock: $fn";
$fs->lock( $fh, F_SETLK ) or LOGDIE "unlock failed: $fn: " . $fs->error;
close($fh) or LOGDIE "close failed: $fn: $!";
}
One alternative to the lock file approach is to use a lock socket. See Lock::Socket on CPAN for such an implementation. Usage is as simple as the following:
use Lock::Socket qw/lock_socket/;
my $lock = lock_socket(5197); # raises exception if lock already taken
There are a couple of advantages to using a socket:
guaranteed (through the operating system) that no two applications will hold the same lock: there is no race condition.
guaranteed (again through the operating system) to clean up neatly when your process exits, so there are no stale locks to deal with.
relies on functionality that is well supported by anything that Perl runs on: no issues with flock(2) support on Win32 for example.
The obvious disadvantage is of course that the lock namespace is global. It is possible for a kind of denial-of-service if another process decides to lock the port you need.
[disclosure: I am the author of the afor-mentioned module]
Use the flock Luke.
Edit: This is a good explanation.
flock creates Unix-style file locks, and is available on most OS's Perl runs on. However flock's locks are advisory only.
edit: emphasized that flock is portable
Here's my solution to reading and writing in one lock...
open (TST,"+< readwrite_test.txt") or die "Cannot open file\n$!";
flock(TST, LOCK_EX);
# Read the file:
#LINES=<TST>;
# Wipe the file:
seek(TST, 0, 0); truncate(TST, 0);
# Do something with the contents here:
push #LINES,"grappig, he!\n";
$LINES[3]="Gekke henkie!\n";
# Write the file:
foreach $l (#LINES)
{
print TST $l;
}
close(TST) or die "Cannot close file\n$!";
Flock is probably the best but requires you to write all the supporting code around it - timeouts, stale locks, non-existant files etc.
I trued LockFile::Simple but found it started setting the default umask to readonly and not cleaning this up. Resulting in random permissions problems on a multi process/multi-threaded application on modperl
I've settled on wrapping up NFSLock with some empty file handling.