this perl code for loading
When running code
use Term::ProgressBar::IO;
open my $fh, '<', 'passwords.txt' or die "could not open file n.txt: $!";
my $pb = Term::ProgressBar::IO->new($fh);
my $line_count;
while (<$fh>) {
$line_count += 1;
$pb->update();
}
close $fh;
print "total lines $line_count"
C:\Users\USER\Desktop>r.pl
0% [* ]total lines 360
what is the problem
Term::ProgressBar::IO seems incompatible with regular filehandles, and expects an instance of IO::File as seen in the test script, so you'll want to define $fh with
my $fh = IO::File->new('passwords.txt', 'r') or die ...;
The docs say that this module will work with any seekable filehandle, but it still doesn't work (for me, anyway).
The relevant line during construction is:
if (ref($count) and $count->can("seek")) {
When $count is an IO::File type, this condition passes but it fails when $count is a regular GLOB, even one opened for reading and writing. A GLOB will support the seek method, but can("seek") won't return true until after a method has been called on it.
use feature 'say';
open my $fh, '<', 'some-file';
say $fh->can('seek'); # ""
say tell $fh; # 0
say $fh->can('seek'); # ""
say eval { $fh->tell }; # 0
say $fh->can('seek'); # 1
and this suggests another workaround (one that could be implemented inside Term::ProgressBar::IO to address this issue), and that is to make a filehandle method call on the filehandle before you pass it to Term::ProgressBar::IO:
open my $fh, '<', 'passwords.txt' or die "could not open file n.txt: $!";
eval { $fh->tell }; # endow $fh with methods detectable by UNIVERSAL::can
...
Here is a simple process bar code, output as below.
$n = 10;
for($i=1;$i<=$n;$i++){
proc_bar($i,$n);
select(undef, undef, undef, 0.2);
}
sub proc_bar{
local $| = 1;
my $i = $_[0] || return 0;
my $n = $_[1] || return 0;
print "\r [ ".("\032" x int(($i/$n)*50)).(" " x (50 - int(($i/$n)*50)))." ] ";
printf("%2.1f %%",$i/$n*100);
local $| = 0;
}
Related
Limiting the scope of a variable $x to a particular code chunk or subroutine, by means of my $x, saves a coder from a world of "global variable"-caused confusion.
But when it comes to the input record separator, $/, apparently its scope cannot be limited.
Am I correct in this?
As a consequence, if I forget to reset the input record separator at the end of a loop, or inside a subroutine, the code below my call to the subroutine can give unexpected results.
The following example demonstrates this.
#!/usr/bin/perl
use strict; use warnings;
my $count_records; my $infile = $ARGV[0]; my $HANDLEinfile;
open $HANDLEinfile, '<', $infile or die "cannot open $infile for reading";
$count_records = 0;
while(<$HANDLEinfile>)
{
$count_records++;
print "$count_records:\n";
print;
}
close $HANDLEinfile;
look_through_other_file();
print "\nNOW, after invoking look_through_other_file:\n";
open $HANDLEinfile, '<', $infile or die "cannot open $infile for reading";
$count_records = 0;
while(<$HANDLEinfile>)
{
$count_records++;
print "$count_records:\n";
print;
}
close $HANDLEinfile;
sub look_through_other_file
{
$/ = undef;
# here, look through some other file with a while loop
return;
}
Here is how it behaves on an input file:
> z.pl junk
1:
All work
2:
and
3:
no play
4:
makes Jack a dull boy.
NOW, after invoking look_through_other_file:
1:
All work
and
no play
makes Jack a dull boy.
>
Note that if one tries to change to
my $/ = undef;
inside the subroutine, this generates an error.
Incidentally, among the stackoverflow tags, why is there no tag for "input record separator"?
The answer for the my $/ = undef; question is to change it to local $/ = undef;. Then the revised code is as follows.
#!/usr/bin/perl
use strict; use warnings;
my $count_records; my $infile = $ARGV[0]; my $HANDLEinfile;
open $HANDLEinfile, '<', $infile or die "cannot open $infile for reading";
$count_records = 0;
while(<$HANDLEinfile>)
{
$count_records++;
print "$count_records:\n";
print;
}
close $HANDLEinfile;
look_through_other_file();
print "\nNOW, after invoking look_through_other_file:\n";
open $HANDLEinfile, '<', $infile or die "cannot open $infile for reading";
$count_records = 0;
while(<$HANDLEinfile>)
{
$count_records++;
print "$count_records:\n";
print;
}
close $HANDLEinfile;
sub look_through_other_file
{
local $/ = undef;
# here, look through some other file with a while loop
return;
}
Then there is no need to return the input record separator to another value, or to the default, $/ = "\n";, by hand.
You can use local to temporarily update the value of a global variable, including $/.
sub look_through_other_file {
local $/ = undef;
# here, look through some other file with a while loop
return;
}
will use an undefined $/ as long as the look_through_other_file subroutine is in the call stack.
You may encounter this construction in this common idiom, to slurp the entire contents of a file into a variable without altering the value of $/ for the rest of the program:
open my $fh, "<", "/some/file";
my $o = do { local $/; <$fh> };
When I run the below script I get a corrupt yaml file like so
---
1:
name1: abc
name2: abc
---
me3: abc
---
Question
Can anyone see that I am doing wrong?
#!/usr/bin/perl
use strict;
use YAML::Syck;
use Fcntl ':flock', 'SEEK_SET';
use warnings;
use Data::Dumper;
my $acc;
my $acc_fh;
$acc->{1}{name1} = "abc";
unlink 'test.yaml';
# write initial
open F, '>', 'test.yaml';
print F YAML::Syck::Dump($acc);
close F;
($acc, $acc_fh) = read_yaml_with_lock('test.yaml');
$acc->{1}{name2} = "abc";
$acc->{1}{name3} = "abc";
write_yaml_with_lock($acc, $acc_fh);
($acc, $acc_fh) = read_yaml_with_lock('test.yaml');
delete $acc->{1}{name3};
write_yaml_with_lock($acc, $acc_fh);
sub read_yaml_with_lock {
my ($file) = #_;
open my $fh, '+<', $file or die $!;
flock($fh, LOCK_EX) or die $!;
my $obj = YAML::Syck::LoadFile($fh); # this dies on failure
return ($obj, $fh);
}
sub write_yaml_with_lock {
my ($obj, $fh) = #_;
my $yaml = YAML::Syck::Dump($obj);
$YAML::Syck::ImplicitUnicode = 1;
seek $fh, 0, SEEK_SET; # seek back to the beginning of file
print $fh $yaml . "---\n";
close $fh;
}
You write to the same file twice. During the second time the YAML code you're writing is shorter than the first time because you delete that hash key inbetween the calls. However, you neither unlink the file after the first time nor do you truncate it after writing to it the second time. So what you see as corruption is the part of the file that has been written the first time but that hasn't been overwritten the second time.
The "me3" part is what is left of " name3", which gets partially overwritten by "---\n" (4 characters). When you write the first time, you have more data. Then you rewind the file handle position and write a shorter data, which does not overwrite all of the old.
I think your solution "should" be to skip this passing a file handle around and rewinding it and instead use the appropriate open for each subroutine. E.g.:
sub read_yaml {
my $file = shift;
open my $fh, '<', $file or die $!;
...
close $fh;
}
sub write_yaml {
my ($file, $obj) = #_;
open my $fh, '>', $file or die $!;
...
close $fh;
}
Keeping the file handle open in between operations not really that useful or efficient, and it introduces some difficulties.
I have written the two program. One program is write the content to the text file simultaneously. Another program is read that content simultaneously.
But both the program should run at the same time. For me the program is write the file is correctly. But another program is not read the file.
I know that once the write process is completed than only the data will be stored in the hard disk. Then another process can read the data.
But I want both read and write same time with different process in the single file. How can I do that?
Please help me.
The following code write the content in the file
sub generate_random_string
{
my $length_of_randomstring=shift;# the length of
# the random string to generate
my #chars=('a'..'z','A'..'Z','0'..'9','_');
my $random_string;
foreach (1..$length_of_randomstring)
{
# rand #chars will generate a random
# number between 0 and scalar #chars
$random_string.=$chars[rand #chars];
}
return $random_string;
}
#Generate the random string
open (FH,">>file.txt")or die "Can't Open";
while(1)
{
my $random_string=&generate_random_string(20);
sleep(1);
#print $random_string."\n";
print FH $random_string."\n";
}
The following code is read the content. This is another process
open (FH,"<file.txt") or die "Can't Open";
print "Open the file Successfully\n\n";
while(<FH>)
{
print "$_\n";
}
You might use an elaborate cooperation protocol such as in the following. Both ends, reader and writer, use common code in the TakeTurns module that handles fussy details such as locking and where the lock file lives. The clients need only specify what they want to do when they have exclusive access to the file.
reader
#! /usr/bin/perl
use warnings;
use strict;
use TakeTurns;
my $runs = 0;
reader "file.txt" =>
sub {
my($fh) = #_;
my #lines = <$fh>;
print map "got: $_", #lines;
++$runs <= 10;
};
writer
#! /usr/bin/perl
use warnings;
use strict;
use TakeTurns;
my $n = 10;
my #chars = ('a'..'z','A'..'Z','0'..'9','_');
writer "file.txt" =>
sub { my($fh) = #_;
print $fh join("" => map $chars[rand #chars], 1..$n), "\n"
or warn "$0: print: $!";
};
The TakeTurns module is execute-around at work:
package TakeTurns;
use warnings;
use strict;
use Exporter 'import';
use Fcntl qw/ :DEFAULT :flock /;
our #EXPORT = qw/ reader writer /;
my $LOCKFILE = "/tmp/taketurns.lock";
sub _loop ($&) {
my($path,$action) = #_;
while (1) {
sysopen my $lock, $LOCKFILE, O_RDWR|O_CREAT
or die "sysopen: $!";
flock $lock, LOCK_EX or die "flock: $!";
my $continue = $action->();
close $lock or die "close: $!";
return unless $continue;
sleep 0;
}
}
sub writer {
my($path,$w) = #_;
_loop $path =>
sub {
open my $fh, ">", $path or die "open $path: $!";
my $continue = $w->($fh);
close $fh or die "close $path: $!";
$continue;
};
}
sub reader {
my($path,$r) = #_;
_loop $path =>
sub {
open my $fh, "<", $path or die "open $path: $!";
my $continue = $r->($fh);
close $fh or die "close $path: $!";
$continue;
};
}
1;
Sample output:
got: 1Upem0iSfY
got: qAALqegWS5
got: 88RayL3XZw
got: NRB7POLdu6
got: IfqC8XeWN6
got: mgeA6sNEpY
got: 2TeiF5sDqy
got: S2ksYEkXsJ
got: zToPYkGPJ5
got: 6VXu6ut1Tq
got: ex0wYvp9Y8
Even though you went to so much trouble, there are still issues. The protocol is unreliable, so reader has no guarantee of seeing all messages that writer sends. With no writer active, reader is content to read the same message over and over.
You could add all this, but a more sensible approach would be using abstractions the operating system provides already.
For example, Unix named pipes seem to be a pretty close match to what you want, and note how simple the code is:
pread
#! /usr/bin/perl
use warnings;
use strict;
my $pipe = "/tmp/mypipe";
system "mknod $pipe p 2>/dev/null";
open my $fh, "<", $pipe or die "$0: open $pipe: $!";
while (<$fh>) {
print "got: $_";
sleep 0;
}
pwrite
#! /usr/bin/perl
use warnings;
use strict;
my $pipe = "/tmp/mypipe";
system "mknod $pipe p 2>/dev/null";
open my $fh, ">", $pipe or die "$0: open $pipe: $!";
my $n = 10;
my #chars = ('a'..'z','A'..'Z','0'..'9','_');
while (1) {
print $fh join("" => map $chars[rand #chars], 1..$n), "\n"
or warn "$0: print: $!";
}
Both ends attempt to create the pipe using mknod because they have no other method of synchronization. At least one will fail, but we don't care as long as the pipe exists.
As you can see, all the waiting machinery is handled by the system, so you do what you care about: reading and writing messages.
This works.
The writer:
use IO::File ();
sub generate_random_string {...}; # same as above
my $file_name = 'file.txt';
my $handle = IO::File->new($file_name, 'a');
die "Could not append to $file_name: $!" unless $handle;
$handle->autoflush(1);
while (1) {
$handle->say(generate_random_string(20));
}
The reader:
use IO::File qw();
my $file_name = 'file.txt';
my $handle = IO::File->new($file_name, 'r');
die "Could not read $file_name: $!" unless $handle;
STDOUT->autoflush(1);
while (defined (my $line = $handle->getline)) {
STDOUT->print($line);
}
are you on windows or *nix? you might be able to string something like this together on *nix by using tail to get the output as it is written to the file. On windows you can call CreateFile() with FILE_SHARE_READ and/or FILE_SHARE_WRITE in order to allow others to access the file while you have it opened for read/write. you may have to periodically check to see if the file size has changed in order to know when to read (i'm not 100% certain here.)
another option is a memory mapped file.
I have a text file that's composed of fixed length records but all in one line with no line breaks in between. What's the best way to process it in Perl? Thanks!
First, let's open the file, and make sure it's in bin mode:
open my $fh, '<', 'file.name' or die "Cannot open file.name: $!";
binmode $fh;
Now, set input record separator to reference to length of your records (let's assume 120 bytes per record):
local $/ = \120;
Now, let's read the records:
while (my $record = <$fh>) {
And now if you want to get data out of it, you have to write some unpack thing:
my #elements = unpack("......", $record);
Now you can process #elements, and finish while() {} loop:
...
}
Whole "program":
open my $fh, '<', 'file.name' or die "Cannot open file.name: $!";
binmode $fh;
local $/ = \120;
while (my $record = <$fh>) {
my #elements = unpack("......", $record);
...
}
close $fh;
use the read FILEHANDLE,SCALAR,LENGTH function to read a block at a time into a buffer...
use constant LEN => 60;
while (!eof $fh) {
my $len = read $fh, $buf, LEN;
die "short read" if $len < LEN;
# processing...
}
... and process the buffer using regular expressions, unpack, or however you like.
unpack() may be of use here. You can specify the list of characters (using 'c', 'C' or 'W') and it'll unpack automatically into a list. See the pack documentation for the options to use.
I quickly jotted off a Perl script that would average a few files with just columns of numbers. It involves reading from an array of filehandles. Here is the script:
#!/usr/local/bin/perl
use strict;
use warnings;
use Symbol;
die "Usage: $0 file1 [file2 ...]\n" unless scalar(#ARGV);
my #fhs;
foreach(#ARGV){
my $fh = gensym;
open $fh, $_ or die "Unable to open \"$_\"";
push(#fhs, $fh);
}
while (scalar(#fhs)){
my ($result, $n, $a, $i) = (0,0,0,0);
while ($i <= $#fhs){
if ($a = <$fhs[$i]>){
$result += $a;
$n++;
$i++;
}
else{
$fhs[$i]->close;
splice(#fhs,$i,1);
}
}
if ($n){ print $result/$n . "\n"; }
}
This doesn't work. If I debug the script, after I initialize #fhs it looks like this:
DB<1> x #fhs
0 GLOB(0x10443d80)
-> *Symbol::GEN0
FileHandle({*Symbol::GEN0}) => fileno(6)
1 GLOB(0x10443e60)
-> *Symbol::GEN1
FileHandle({*Symbol::GEN1}) => fileno(7)
So far, so good. But it fails at the part where I try to read from the file:
DB<3> x $fhs[$i]
0 GLOB(0x10443d80)
-> *Symbol::GEN0
FileHandle({*Symbol::GEN0}) => fileno(6)
DB<4> x $a
0 'GLOB(0x10443d80)'
$a is filled with this string rather than something read from the glob. What have I done wrong?
You can only use a simple scalar variable inside <> to read from a filehandle. <$foo> works. <$foo[0]> does not read from a filehandle; it's actually equivalent to glob($foo[0]). You'll have to use the readline builtin, a temporary variable, or use IO::File and OO notation.
$text = readline($foo[0]);
# or
my $fh = $foo[0]; $text = <$fh>;
# or
$text = $foo[0]->getline; # If using IO::File
If you weren't deleting elements from the array inside the loop, you could easily use a temporary variable by changing your while loop to a foreach loop.
Personally, I think using gensym to create filehandles is an ugly hack. You should either use IO::File, or pass an undefined variable to open (which requires at least Perl 5.6.0, but that's almost 10 years old now). (Just say my $fh; instead of my $fh = gensym;, and Perl will automatically create a new filehandle and store it in $fh when you call open.)
If you are willing to use a bit of magic, you can do this very simply:
use strict;
use warnings;
die "Usage: $0 file1 [file2 ...]\n" unless #ARGV;
my $sum = 0;
# The current filehandle is aliased to ARGV
while (<>) {
$sum += $_;
}
continue {
# We have finished a file:
if( eof ARGV ) {
# $. is the current line number.
print $sum/$. , "\n" if $.;
$sum = 0;
# Closing ARGV resets $. because ARGV is
# implicitly reopened for the next file.
close ARGV;
}
}
Unless you are using a very old perl, the messing about with gensym is not necessary. IIRC, perl 5.6 and newer are happy with normal lexical handles: open my $fh, '<', 'foo';
I have trouble understanding your logic. Do you want to read several files, which just contains numbers (one number per line) and print its average?
use strict;
use warnings;
my #fh;
foreach my $f (#ARGV) {
open(my $fh, '<', $f) or die "Cannot open $f: $!";
push #fh, $fh;
}
foreach my $fh (#fh) {
my ($sum, $n) = (0, 0);
while (<$fh>) {
$sum += $_;
$n++;
}
print "$sum / $n: ", $sum / $n, "\n" if $n;
}
Seems like a for loop would work better for you, where you could actually use the standard read (iteration) operator.
for my $fh ( #fhs ) {
while ( defined( my $line = <$fh> )) {
# since we're reading integers we test for *defined*
# so we don't close the file on '0'
#...
}
close $fh;
}
It doesn't look like you want to shortcut the loop at all. Therefore, while seems to be the wrong loop idiom.