In a Perl script, we are retrieving configuration details from a pm file. After the user changes the configuration details using the script's interface, the same values should be written back in the pm file.
For example, I have the following config.pm file:
$SourcePrimUserHost = '10.226.33.233';
$SourcePrimUserPort = '33002';
$SourceGroupsHost = '10.226.33.233';
$SourceGroupsPort = '33002';
I'm reading these values from a Perl script. I want to store updated values back to config.pm file.
How can we do this? Looking forward to your help.
This is not a good design choice.
The Perl module may (should!) be installed in such a way that the user can read, but not write it.
If the module is used by multiple users or multiple Perl programs, then the conf would be system-global and not application-specific.
Issues arise if multiple instances of the program are run at the same time.
I would recommend using a data serialization format like YAML, although JSON, Freeze/Thaw and Dumper may be other contestants. This configuration would best be stored in a seperate file.
If you have to store the data in the same file, you could use the __DATA__ token. Everything behind that is accessible inside the code as the DATA filehandle, and will not be executed by perl. It is also trivial to find this token when updating the configuration. If the module is called Foo::Bar:
my $serialized_stuff = ...;
my $self_loc = $INC{"Foo/Bar.pm"}; # %INC holds the location for every loaded module.
my $tempfile = ...;
open $SELF, "<", $self_loc or die ...;
open $TEMP, ">", $tempfile or die ...;
# don't touch anything before __DATA__
while(<$SELF>) {
print $TEMP $_;
last if /^__DATA__$/;
}
print $TEMP $serialized_stuff;
close $TEMP; close $SELF;
rename $tempfile => $self_loc or die ...;
Use one of the many configuration tools like Config::General from CPAN. They are easy to use, support different notations and you can write back your values into a text file.
Related
Was debugging a perl script for the first time in my life and came over this:
$my_temp_file = File::Temp->tmpnam();
system("cmd $blah | cmd2 > $my_temp_file");
open(FIL, "$my_temp_file");
...
unlink $my_temp_file;
This works pretty much like I want, except the obvious race conditions in lines 1-3. Even if using proper tempfile() there is no way (I can think of) to ensure that the file streamed to at line 2 is the same opened at line 3. One solution might be pipes, but the errors during cmd might occur late because of limited pipe buffering, and that would complicate my error handling (I think).
How do I:
Write all output from cmd $blah | cmd2 into a tempfile opened file handle?
Read the output without re-opening the file (risking race condition)?
You can open a pipe to a command and read its contents directly with no intermediate file:
open my $fh, '-|', 'cmd', $blah;
while( <$fh> ) {
...
}
With short output, backticks might do the job, although in this case you have to be more careful to scrub the inputs so they aren't misinterpreted by the shell:
my $output = `cmd $blah`;
There are various modules on CPAN that handle this sort of thing, too.
Some comments on temporary files
The comments mentioned race conditions, so I thought I'd write a few things for those wondering what people are talking about.
In the original code, Andreas uses File::Temp, a module from the Perl Standard Library. However, they use the tmpnam POSIX-like call, which has this caveat in the docs:
Implementations of mktemp(), tmpnam(), and tempnam() are provided, but should be used with caution since they return only a filename that was valid when function was called, so cannot guarantee that the file will not exist by the time the caller opens the filename.
This is discouraged and was removed for Perl v5.22's POSIX.
That is, you get back the name of a file that does not exist yet. After you get the name, you don't know if that filename was made by another program. And, that unlink later can cause problems for one of the programs.
The "race condition" comes in when two programs that probably don't know about each other try to do the same thing as roughly the same time. Your program tries to make a temporary file named "foo", and so does some other program. They both might see at the same time that a file named "foo" does not exist, then try to create it. They both might succeed, and as they both write to it, they might interleave or overwrite the other's output. Then, one of those programs think it is done and calls unlink. Now the other program wonders what happened.
In the malicious exploit case, some bad actor knows a temporary file will show up, so it recognizes a new file and gets in there to read or write data.
But this can also happen within the same program. Two or more versions of the same program run at the same time and try to do the same thing. With randomized filenames, it is probably exceedingly rare that two running programs will choose the same name at the same time. However, we don't care how rare something is; we care how devastating the consequences are should it happen. And, rare is much more frequent than never.
File::Temp
Knowing all that, File::Temp handles the details of ensuring that you get a filehandle:
my( $fh, $name ) = File::Temp->tempfile;
This uses a default template to create the name. When the filehandle goes out of scope, File::Temp also cleans up the mess.
{
my( $fh, $name ) = File::Temp->tempfile;
print $fh ...;
...;
} # file cleaned up
Some systems might automatically clean up temp files, although I haven't care about that in years. Typically is was a batch thing (say once a week).
I often go one step further by giving my temporary filenames a template, where the Xs are literal characters the module recognizes and fills in with randomized characters:
my( $name, $fh ) = File::Temp->tempfile(
sprintf "$0-%d-XXXXXX", time );
I'm often doing this while I'm developing things so I can watch the program make the files (and in which order) and see what's in them. In production I probably want to obscure the source program name ($0) and the time; I don't want to make it easier to guess who's making which file.
A scratchpad
I can also open a temporary file with open by not giving it a filename. This is useful when you want to collect outside the program. Opening it read-write means you can output some stuff then move around that file (we show a fixed-length record example in Learning Perl):
open(my $tmp, "+>", undef) or die ...
print $tmp "Some stuff\n";
seek $tmp, 0, 0;
my $line = <$tmp>;
File::Temp opens the temp file in O_RDWR mode so all you have to do is use that one file handle for both reading and writing, even from external programs. The returned file handle is overloaded so that it stringifies to the temp file name so you can pass that to the external program. If that is dangerous for your purpose you can get the fileno() and redirect to /dev/fd/<fileno> instead.
All you have to do is mind your seeks and tells. :-) Just remember to always set autoflush!
use File::Temp;
use Data::Dump;
$fh = File::Temp->new;
$fh->autoflush;
system "ls /tmp/*.txt >> $fh" and die $!;
#lines = <$fh>;
printf "%s\n\n", Data::Dump::pp(\#lines);
print $fh "How now brown cow\n";
seek $fh, 0, 0 or die $!;
#lines2 = <$fh>;
printf "%s\n", Data::Dump::pp(\#lines2);
Which prints
[
"/tmp/cpan_htmlconvert_DPzx.txt\n",
"/tmp/cpan_htmlconvert_DunL.txt\n",
"/tmp/cpan_install_HfUe.txt\n",
"/tmp/cpan_install_XbD6.txt\n",
"/tmp/cpan_install_yzs9.txt\n",
]
[
"/tmp/cpan_htmlconvert_DPzx.txt\n",
"/tmp/cpan_htmlconvert_DunL.txt\n",
"/tmp/cpan_install_HfUe.txt\n",
"/tmp/cpan_install_XbD6.txt\n",
"/tmp/cpan_install_yzs9.txt\n",
"How now brown cow\n",
]
HTH
I have an application in which scripts will be run that need to be able access stored data. I want to run a script (main.pl) which will create an array. Later, if I run A.pl or B.pl, I want those scripts to be able access the previously created array and change values within it. What do I need to code in main.pl A.pl B.pl so I can achieve that?
Normally one perl instance can not access the variables of another instance. The question then becomes "what can one do that is almost like sharing variables"?
One approach is to store data somewhere it can persist, such as in a database or a CSV file on disk. This means reading the data at the beginning of the program, and writing it or updating it, and naturally leads to questions about race conditions, locking, etc... and greatly expands the scope that any possible answer would need to cover.
Another approach is to write your programs to use CSV or YAML or some other format easily read and written by libraries from CPAN, and use STDIN and STDOUT for input and output. This allows decoupling of storage, and also chaining several tools together with a pipe from the shell prompt.
For an in-memory solution for tieing hashes to shared memory, you can check out IPC::Shareable
http://metacpan.org/pod/IPC::Shareable
Perl memory structures can't be stored and then accessed later by other Perl scripts. However, you can write out those memory structures as a file. This can be done through hard raw coding, or by using a wide variety of Perl modules. The Storable is a standard Perl module and has been around for quite a while.
Since all you're installing is an array, you could have one program write the array to a file, and then have the other file read the array.
use strict;
use warnings;
use autodie;
use constant {
ARRAY_FILE => "$Env{HOME}/perl_arry.txt",
};
my #array;
[...] #Build the array
open my $output_fh, ">", ARRAY_FILE;
while my $item ( #array ) {
say {$output_fh} $item;
}
close $output_fh;
Now, have your second program read in this array:
use strict;
use warnings;
use autodie;
use constant {
ARRAY_FILE => "$Env{HOME}/perl_arry.txt",
};
my #new_array;
open my $input_fh, "<", ARRAY_FILE;
while ( my $item = <$input_fh> ) {
push #new_array, $item;
}
close $input_fh;
More complex data can be stored with Storable, but's it's pretty much the same thing: You need to write Storable to a physical file and then reopen that file to pull in your data once again.
We maintain a huge number of perl modules, actualy its so huge that we don't even know of all modules that we are responsible for. We would like to track what scripts and modules accesses another module in some sort of log, preferably stored by module name, so that we can evaluate whether its a risk to update a module and so that we can know what we might affect.
Is there any simple way to do this?
You could do a simple regex search:
use strict;
use warnings;
my %modules;
foreach my $perl_file (#file_list) {
open FILE, $perl_file or die "Can't open $perl_file ($!)";
while (<FILE>) {
if (/\s*(?:use|require)\s*([^;]+);/) {
$modules{$1}{$perl_file}++;
}
}
}
This is quick and dirty, but it should work pretty well. You end up with a hash of modules, each of which points to a hash of the files that use it.
Of course, it will catch things like use strict; but those will be easy enough to ignore.
If you have things like use Module qw/function/; You will grab the whole thing before the semicolon, but it shouldn't be a big deal. You can just search the keys for your known module names.
A downside is that it doesn't track dependencies. If you need that information you could add it by getting it from cpan or something.
Update: If you want to log this at runtime, you could create a wrapper script and have your perl command point to the wrapper on your system. Then make the wrapper something like this:
use strict;
use warnings;
use Module::Loaded;
my $script = shift #ARGV;
#run program
do $script;
#is_loaded() gets the path of these modules if they are loaded.
print is_loaded('Some::Module');
print is_loaded('Another::Module');
You might run the risk of funny side effects, though, since the method of calling your script has changed. It depends on what you are doing.
Maybe edit sitecustomize.pl so that each time when Perl runs, it would write some info in a log, and then analyse it? Add something like this to sitecustomize.pl:
open (LOG, '>>',"absolutepathto/logfile.txt");
print LOG $0,"\t",$$,"\t",scalar(localtime),"\n";
open SELF, $0;
while (<SELF>) {
print LOG $_ if (/use|require/);
}
close SELF;
print LOG "_" x 80,"\n";
close LOG;
EDIT:
Also, we forgot about %INC hash, so the code above may be rewritten as follows, to include more data about which modules were actually loaded + include files required by do function:
open (LOG, '>>',"absolutepathto/logfile.txt");
print LOG $0,' ',$$,' ',scalar(localtime),"\n";
open SELF, $0;
while (<SELF>) {
print LOG $_ if (/use|require/);
}
close SELF;
END {
local $" = "\n";
print LOG "Files loaded by use, eval, or do functions at the end of this program run:\n";
print LOG "#{[values %INC]}","\n";
print LOG "_" x 80,"\n";
close LOG;
}
My web app runs on Apache mod_perl using CGI::Application. I want to provide a download of a generated file. In the past (before we were using mod_perl and CGI::App) I just spooled out a csv file to STDOUT as it was generated. Now I'm shooting for a little more refinement - creating an Excel spreadsheet using Spreadsheet::WriteExcel - and I can't seem it to get to print it out from the file handle.
sub export_list {
my $self = shift;
binmode(STDOUT);
my $str;
open my $fh, '>', \$str;
my $workbook = Spreadsheet::WriteExcel->new($fh);
my $worksheet = $workbook->add_worksheet();
$worksheet->write_col(0,0, ['some','data','here']);
warn $str;
return $str;
}
The output is just a blank response, and the warn is blank as well.
The method I'm using to write the spreadsheet to a filehandle is pretty much straight out of the documentation, so I assume the problem is due to some CGI::App noobery on my part. The documentation's suggested methods for filehandles and mod_perl proved pretty fruitless as well.
I guess I should mention I'm running on Windows, and that my current workaround is to create a file and provide the user with a link to it. That poses more problems, however, in regards to clearing out the directory and when to do so, and also authentication for access to the generated files.
Suggestions? Scathing criticism?
You shouldn't need to mess with STDOUT; CGI-App should handle that properly for you under the hood. You'll also may need to close the filehandle before you try to send the data.
It looks like you're not setting a proper content type for the Excel data, though. For anything other than text/html, you'll need to set it manually. Try something like this:
sub export_list {
my $self = shift;
my $str;
open my $fh, '>', \$str or die "Can't open to var: $!";
my $workbook = Spreadsheet::WriteExcel->new($fh);
my $worksheet = $workbook->add_worksheet();
$worksheet->write_col(0,0, ['some','data','here']);
$workbook->close;
close $fh;
warn $str;
$self->header_add( -type => 'application/vnd.ms-excel' );
return $str;
}
You may also be interested in CGI::Application::Plugin::Stream
Instead of creating the whole spreadsheet in memory, you should either write it out to a file and them stream it when finished (using CGI::Application::Plugin::Stream helps here, but you'd still need to clean it up afterwards, but really every web app should have a temp directory that periodically gets cleaned up) or print it as you create it (which means making the FH STDIN instead which might be trickier under mod_perl or maybe not).
And then remember to close your workbook when it's done.
You want to close the workbook. Also close the filehandle:
warn "length 1=".length($str);
$workbook->close();
close($fh) or die "error on close: $!";
warn "length 2=".length($str);
length 1=0 at wx.pl line 16.
length 2=5632 at wx.pl line 19.
With Spreadsheet::WriteExcel, I can create a new workbook, but what if I want to open an existing book and modify certain columns? How would I accomplish that?
I could parse all of the data out of the sheet using Spreadsheet::ParseExcel then write it back with new values in certain rows/columns using Spreadsheet::WriteExcel, however. Is there a module that already combines the two?
Mainly I just want to open a .xls, overwrite certain rows/columns, and save it.
Spreadsheet::ParseExcel will read in existing excel files:
my $parser = Spreadsheet::ParseExcel->new();
# $workbook is a Spreadsheet::ParseExcel::Workbook object
my $workbook = $parser->Parse('Book1.xls');
But what you really want is Spreadsheet::ParseExcel::SaveParser, which is a combination of Spreadsheet::ParseExcel and Spreadsheet::WriteExcel. There is an example near the bottom of the documentation.
If you have Excel installed, then it's almost trivial to do this with Win32::OLE. Here is the example from Win32::OLE's own documentation:
use Win32::OLE;
# use existing instance if Excel is already running
eval {$ex = Win32::OLE->GetActiveObject('Excel.Application')};
die "Excel not installed" if $#;
unless (defined $ex) {
$ex = Win32::OLE->new('Excel.Application', sub {$_[0]->Quit;})
or die "Oops, cannot start Excel";
}
# get a new workbook
$book = $ex->Workbooks->Add;
# write to a particular cell
$sheet = $book->Worksheets(1);
$sheet->Cells(1,1)->{Value} = "foo";
# write a 2 rows by 3 columns range
$sheet->Range("A8:C9")->{Value} = [[ undef, 'Xyzzy', 'Plugh' ],
[ 42, 'Perl', 3.1415 ]];
# print "XyzzyPerl"
$array = $sheet->Range("A8:C9")->{Value};
for (#$array) {
for (#$_) {
print defined($_) ? "$_|" : "<undef>|";
}
print "\n";
}
# save and exit
$book->SaveAs( 'test.xls' );
undef $book;
undef $ex;
Basically, Win32::OLE gives you everything that is available to a VBA or Visual Basic application, which includes a huge variety of things -- everything from Excel and Word automation to enumerating and mounting network drives via Windows Script Host. It has come standard with the last few editions of ActivePerl.
There's a section of the Spreadsheet::WriteExcel docs that covers Modifying and Rewriting Spreadsheets.
An Excel file is a binary file within a binary file. It contains several interlinked checksums and changing even one byte can cause it to become corrupted.
As such you cannot simply append or update an Excel file. The only way to achieve this is to read the entire file into memory, make the required changes or additions and then write the file out again.
You can read and rewrite an Excel file using the Spreadsheet::ParseExcel::SaveParser module which is a wrapper around Spreadsheet::ParseExcel and Spreadsheet::WriteExcel. It is part of the Spreadsheet::ParseExcel package.
There's an example as well.
The Spreadsheet::ParseExcel::SaveParser module is a wrapper around Spreadsheet::ParseExcel and Spreadsheet::WriteExcel.
I recently updated the documentation with, what I hope, is a clearer example of how to do this.