How do I execute a unix command containing a perl variable in perl - perl

In the following perl code, I am tring to copy a perl variable $file from one directory to another directory with:
"system("cp $file $Output_Dir);
This command writes down the file name alright but then says:
cp: cannot stat 'tasmax_AFR-44_CNRM-CERFACS-CNRM-CM5_historical_r1i1p1_CLMcom-CCLM4-8-17_v1_day_19910101-19951231.nc': No such file or directory
The command
system("#sixfiles = ls $Vars[$kk]}*");
gives me the error:
sh: 1: =: not found
I wonder what is wrong with this code. Assistance will be appreciated.
#!/usr/bin/perl -w
use strict;
use warnings;
use File::Path;
use File::Copy;
my $debug = 1;
my #Vars = ("pr","tasmin","tasmax");
my $Vars;
my #sixfiles;
my $sixfiles;
my $Input_Dir = "/home/zmumba/DATA/Input_Dir";
my $Output_Dir = "/home/zmumba/DATA/Output_Dir";
for (my $kk=0; $kk < #Vars; ++$kk) {
opendir my $in_dir, $Input_Dir or die "opendir failed on $Input_Dir: $! ($^E)";
while (my $file=readdir $in_dir) {
next unless $file =~ /^$Vars[$kk]/;
next if -d $file;
print "$file\n";
print "Copying $file\n" if $debug;
my $cmd01 = "cp $file $Output_Dir";
print "Doing system ($cmd01)\n" if $debug;
system ($cmd01);
system("#sixfiles = ls $Vars[$kk]}*");
}
}

Try this:
use feature qw(say);
use strict;
use warnings;
use File::Spec;
my #Vars = ("pr","tasmin","tasmax");
my $Input_Dir = "/home/zmumba/DATA/Input_Dir";
my $Output_Dir = "/home/zmumba/DATA/Output_Dir";
opendir my $in_dir, $Input_Dir or die "opendir failed on $Input_Dir: $! ($^E)";
while (my $file=readdir $in_dir) {
next if ($file eq '.') || ($file eq '..');
next if -d $file;
next if !grep { $file =~ /^$_/ } #Vars;
say "Copying $file";
$file = File::Spec->catfile( $Input_Dir, $file );
system "cp", $file, $Output_Dir;
}

system ($cmd01);
Gives:
cp: cannot stat '<long-but-correct-file-name>': No such file or directory
This is almost certainly because you are not running the code from $Input_Dir, so that file doesn't exist in your current directory. You need to either chdir to the correct directory or add the directory path to the front of the file name variable.
system("#sixfiles = ls $Vars[$kk]}*");
This code makes no sense. The code passed to system() needs to be Unix shell code. That's the ls $Vars[$kk]}* bit (but I'm not sure where that } comes from). You can't populate a Perl array inside a shell command. You would need to capture the value returned from the ls command and then parse it somehow to separate it into a list.

You can give a try with the following code:
#!/usr/bin/env perl
use strict;
use warnings;
my $debug = 1;
my #Vars = ("pr", "tasmin", "tasmax");
my $Vars;
my $Input_Dir = "/home/zmumba/DATA/Input_Dir";
my $Output_Dir = "/home/zmumba/DATA/Output_Dir";
my $cpsrc, $cpdest = '';
print "No Write Permission: $!" unless(-w $Output_Dir);
for my $findex (0 .. $#Vars) {
$cpsrc = qq($Input_Dir/$Vars[$findex]);
print "$Vars[$findex]\n";
print "Copying $Vars[$findex]\n" if $debug;
my $cmd01 = "cp $cpsrc $Output_Dir";
print "Doing system ($cmd01)\n" if $debug;
system($cmd01);
}
You don't have to go through each file in source dir. You already know the files to copy from source.

Related

Recovering a specific line in multiple .txt in a directory using Perl

I have the results of a program which gives me the results from some search giving me 2000+ file txt archives. I just need a specific line in each file, this is what I have been trying with Perl:
opendir(DIR, $dirname) or die "Could not open $dirname\n";
while ($filename = readdir(DIR)) {
print "$filename\n";
open ($filename, '<', $filename)or die("Could not open file.");
my $line;
while( <$filename> ) {
if( $. == $27 ) {
print "$line\n";
last;
}
}
}
closedir(DIR);
But there is a problem with the $filename in line 5 and I don't know an alternative to it so I don't have to manually name each file.
Several issues with that code:
Using an old-school bareword identifier for the directory handle instead of a autovivified variable like you are for the file handle.
Using the same variable for the filename and file handle is pretty strange.
You don't check to see if the file is a directory or something else other than a plain file before trying to open it.
$27?
You never assign anything to that $line variable before printing it.
Unless $directory is your program's current working directory, you're running into an issue mentioned in the readdir documentation
If you're planning to filetest the return values out of a readdir, you'd better prepend the directory in question. Otherwise, because we didn't chdir there, it would have been testing the wrong file.
(Substitute open for filetest)
Always use strict; and use warnings;.
Personally, if you just want to print the 27th line of a large number of files, I'd turn to awk and find (Using its -exec test to avoid potential errors about the command line maximum length being hit):
find directory/ -maxdepth 1 -type -f -exec awk 'FNR == 27 { print FILENAME; print }' \{\} \+
If you're on a Windows system without standard unix tools like those installed, or it's part of a bigger program, a fixed up perl way:
#!/usr/bin/env perl
use strict;
use warnings;
use autodie;
use feature qw/say/;
use File::Spec;
my $directory = shift;
opendir(my $dh, $directory);
while (my $filename = readdir $dh) {
my $fullname = File::Spec->catfile($directory, $filename); # Construct a full path to the file
next unless -f $fullname; # Only look at regular files
open my $fh, "<", $fullname;
while (my $line = <$fh>) {
if ($. == 27) {
say $fullname;
print $line;
last;
}
}
close $fh;
}
closedir $dh;
You might also consider using glob to get the filenames instead of opendir/readdir/closedir.
And if you have Path::Tiny available, a simpler version is:
#!/usr/bin/env perl
use strict;
use warnings;
use autodie;
use feature qw/say/;
use Path::Tiny;
my $directory = shift;
my $dir = path $directory;
for my $file ($dir->children) {
next unless -f $file;
my #lines = $file->lines({count => 27});
if (#lines == 27) {
say $file;
print $lines[-1];
}
}

Can't find error "Global symbol #xx requires explicit package name"

I have checked the questions that may already have an answer and none of them have helped.
This is for my semester project for Unix Programming. I have created a script that compares HTML files to one other from a website.
The script worked perfectly as expected until I tried to implement the second website, so in turn I deleted the added code for the second website and now I get the errors
Global symbol "#master" requires explicit package name
Global symbol "#child" requires explicit package name
within the csite_md5 subroutine. I have gone through the code many times over and cannot see the problem.
I am looking for another set of eyes to see if I'm just missing something simple, which usually is the case.
Also I am new to Perl as this is my first time using the language.
#!/usr/bin/perl
use strict;
use warnings;
use Digest::MD5 qw(md5_hex);
use File::Basename;
# Path to the c-site download root directory
my $csite_dir = '/root/websites/c-site/wget/';
opendir my $dh, $csite_dir or die $!;
# Finds the sub directories c-site_'date +%F' where the c-site download is located
my #wget_subdir_csite = sort grep /^[^.]/, readdir $dh;
# Creates the absolute path to the c-site download
my $csite_master_dir = "$csite_dir$wget_subdir_csite[0]/dayzunderground.webs.com";
my $csite_child_dir = "$csite_dir$wget_subdir_csite[1]/dayzunderground.webs.com";
# Call to subroutine to append the .html file name to the absolute path
my #master_csite = &gethtml_master_csite($csite_master_dir);
my #child_csite = &gethtml_child_csite($csite_child_dir);
&csite_md5(\#master_csite, \#child_csite);
sub gethtml_master_csite{
my ($master_path) = #_;
opendir (DIR, $master_path) or die $!;
# Ends with .html and is a file
my #html_master = sort grep {m/\.html$/i && -f "$master_path/$_"} readdir(DIR);
my #files_master = ("$master_path/$html_master[0]","$master_path/$html_master[1]","$master_path/$html_master[2]","$master_path/$html_master[3]");
return #files_master
}
sub gethtml_child_csite{
my ($child_path) = #_;
opendir (DIR, $child_path) or die $!;
# Ends with .html and is a file
my #html_child = sort grep {m/\.html$/i && -f "$child_path/$_"} readdir(DIR);
my #files_child = ("$child_path/$html_child[0]","$child_path/$html_child[1]","$child_path/$html_child[2]","$child_path/$html_child[3]");
return #files_child
}
sub csite_md5{
my ($master, $child) = #_;
if(&md5sum($master[0]) ne &md5sum($child[0])){
my $filename = basename($master[0]);
system("diff -u -d -t --width=100 $master[0] $child[0] > ~/websites/c-site/diff/c-site-$filename-`date +%F`");
#print "1"
}
if(&md5sum($master[1]) ne &md5sum($child[1])){
my $filename2 = basename($master[1]);
system("diff -u -d -t --width=100 $master[1] $child[1] > ~/websites/c-site/diff/c-site-$filename2-`date +%F`");
#print "2"
}
if(&md5sum($master[2]) ne &md5sum($child[2])){
my $filename3 = basename($master[2]);
system("diff -u -d -t --width=100 $master[2] $child[2] > ~/websites/c-site/diff/c-site-$filename3-`date +%F`");
#print "3"
}
if(&md5sum($master[3]) ne &md5sum($child[3])){
my $filename4 = basename($master[3]);
system("diff -u -d -t --width=100 $master[3] $child[3] > ~/websites/c-site/diff/c-site-$filename4-`date +%F`");
#print "4"
}
}
sub md5sum{
my $file = shift;
my $digest = "";
eval{
open(FILE, $file) or die "Can't find file $file\n";
my $ctx = Digest::MD5->new;
$ctx->addfile(*FILE);
$digest = $ctx->hexdigest;
close(FILE);
};
if($#){
print $#;
return "";
}
return $digest
}
$master and $child are array references; use them like $master->[0]. $master[0] uses the array #master, which is a completely separate variable.
I thought it may help to go through your program and point out some practices that are less than optimal
You shouldn't use an ampersand & when calling a Perl subroutine. That was required in Perl 4 which was superseded about 22 years ago
It is preferable to use the File::Spec module to manipulate file paths, both to handle cases like multiple path separators and for portability. File::Spec will also do the job of File::BaseName
It is unnecessary to use the shell to create a date string. Use the Time::Piece module and localtime->ymd generates the same string as date +%F
It is neater and more concise to use map where appropriate instead of writing multiple identical assignments
The gethtml_master_csite and gethtml_child_csite subroutines are identical except that they use different variable names internally. They can be replaced by a single gethtml_csite subroutine
You should use lexical file and directory handles throughout, as you have done with the first opendir. You should also use the three-parameter form of open (with the open mode as the second parameter)
If an open fails then you should include the variable $! in the die string so that you know why it failed. Also, if you end the string with a newline then Perl won't append the source file and line number to the string when it is printed
As you have read, the csite_md5 attempts to use arrays #master and #child which don't exist. You have array references $master and $child instead. Also, the subroutine lends itself to a loop structure instead of writing the four comparisons explicitly
In md5sum you have used an eval to catch the die when the open call fails. It is nicer to check for this explicitly
The standard way of returning a false value from a subroutine is a bare return. If you return '' then it will evaluate as true in list context
With those chnages in place your code looks like this. Please ask if you have any problem understanding it. Note that I haven't been able to test it but it does compile
#!/usr/bin/perl
use strict;
use warnings;
use Digest::MD5 qw(md5_hex);
use File::Spec::Functions qw/ catdir catfile splitpath /;
use Time::Piece 'localtime';
my $csite_dir = '/root/websites/c-site/wget/';
opendir my $dh, $csite_dir or die qq{Unable to open "$csite_dir": $!};
my #wget_subdir_csite = sort grep /^[^.]/, readdir $dh;
my ($csite_master_dir, $csite_child_dir) = map
catdir($csite_dir, $_, 'dayzunderground.webs.com'),
#wget_subdir_csite[0,1];
my #master_csite = gethtml_csite($csite_master_dir);
my #child_csite = gethtml_csite($csite_child_dir);
csite_md5(\#master_csite, \#child_csite);
sub gethtml_csite {
my ($path) = #_;
opendir my $dh, $path or die qq{Unable to open "$path": $!};
my #files = sort grep { /\.html$/i and -f } map catfile($path, $_), readdir $dh;
return #files;
}
sub csite_md5 {
my ($master_list, $child_list) = #_;
for my $i ( 0 .. $#$master_list ) {
my ($master, $child) = ($master_list->[$i], $child_list->[$i]);
if ( md5sum($master) ne md5sum($child) ) {
my $filename = (splitpath($master))[-1]; # Returns (volume, path, file)
my $date = localtime->ymd;
system("diff -u -d -t --width=100 $master $child > ~/websites/c-site/diff/c-site-$filename-$date");
}
}
}
sub md5sum {
my ($file) = #_;
my $digest = "";
open my $fh, '<', $file or do {
warn qq{Can't open file "$file": $!}; # '
return;
};
my $ctx = Digest::MD5->new;
$ctx->addfile($fh);
return $ctx->hexdigest;
}

perl: File::Copy: no such file or directory error on reading existing file

I'm trying to download ruby gems listed in several example rails projects I downloaded via bundle install with the latest version of ruby installed through RVM, and I'm following the directions here to properly downloads the gems listed in the bundler files. I thought I'd speed up the process with this perl script: (Not quite comfortable with bash yet):
#!/usr/bin/perl
use File::Copy qq(copy);
use strict;
use warnings;
my #files = `ls`;
foreach my $dir (#files)
{
chomp $dir;
if( -d $dir)
{
print "\n\n\tabout to enter $dir\n\n\n";
print `cd "$dir"`;
system("echo \"2.2.0\" > .ruby-version");
system("echo \"$dir\" > .ruby-gemset");
copy("Gemfile", "tmp.save") or die "copy: $!\n";
open(RD,"<tmp.save") or die "$dir: tmp.save: $!\n";
open(WR,">Gemfile") or die "$dir: Gemfile: $!\n";
print WR <RD>;
print WR "\n";
print WR "ruby \'2.2.0\'";
print WR "\n\n";
while(<RD>) { print WR $_; print WR "\n"; }
close(RD); close(WR);
print `cd ..`;
print `cd "$dir"`;
print `bundle install`;
print `cd ..`;
}
}
error is:
copy: No such file or directory
The file clearly exists these projects are presumably created with rails <name> command. Why would this happen and how do I fix it?
When you invoke an external command with system or backticks, you spawn a new process. That process gets its own environment, which is not shared with the parent process or with other children:
system('pwd'); # /foo
system('cd "/bar"');
system('pwd'); # still /foo
(Note that if you're not actually storing the output of a command, it's preferable to use system instead of backticks.)
This means that when you `cd "$dir"`, your Perl script's working directory isn't changing. To fix, use native Perl commands instead of invoking shell commands.
Here's a rough example (untested):
use strict;
use warnings;
use File::Copy;
my $parent_dir = 'foo';
opendir my $dh, $parent_dir or die "opendir failed on '$parent_dir': $!";
my #subdirs = map { "$parent_dir/$_" }
grep { $_ !~ /^\.\.?$/ && -d "$parent_dir/$_" } readdir $dh;
foreach my $subdir (#subdirs) {
my $source = "$subdir/Gemfile";
my $target = "$subdir/tmp.save";
if (-f $source) {
copy $source, $target or die "copy failed: $!";
}
}

How to read directories and sub-directories without knowing the directory name in perl?

Hi i want to read directories and sub-directories without knowing the directory name. Current directory is "D:/Temp". 'Temp' has sub-directories like 'A1','A2'. Again 'A1' has sub-directories like 'B1','B2'. Again 'B1' has sub-directories like 'C1','C2'. Perl script doesn't know these directories. So it has to first find directory and then read one file at a time in dir 'C1' once all files are read in 'C1' it should changes to dir 'C2'. I tried with below code here i don't want to read all files in array(#files) but need one file at time. In array #dir elements should be as fallows.
$dir[0] = "D:/Temp/A1/B1/C1"
$dir[1] = "D:/Temp/A1/B1/C2"
$dir[2] = "D:/Temp/A1/B2/C1"
Below is the code i tried.
use strict;
use File::Find::Rule;
use Data::Dumper;
my $dir = "D:/Temp";
my #dir = File::Find::Rule->directory->in($dir);
print Dumper (\#dir);
my $readDir = $dir[3];
opendir ( DIR, $readDir ) || die "Error in opening dir $readDir\n";
my #files = grep { !/^\.\.?$/ } readdir DIR;
print STDERR "files: #files \n\n";
for my $fil (#files) {
open (F, "<$fil");
read (F, my $data);
close (F);
print "$data";
}
use File::Find;
use strict;
use warnings;
my #dirs;
my %has_children;
find(sub {
if (-d) {
push #dirs, $File::Find::name;
$has_children{$File::Find::dir} = 1;
}
}, 'D:/Temp');
my #ends = grep {! $has_children{$_}} #dirs;
print "$_\n" for (#ends);
Your Goal: Find the absolute paths to those directories that do not themselves have child directories.
I'll call those directories of interest terminal directories. Here's the prototype for a function that I believe provides the convenience you are looking for. The function returns its result as a list.
my #list = find_terminal_directories($full_or_partial_path);
And here's an implementation of find_terminal_directories(). Note that this implementation does not require the use of any global variables. Also note the use of a private helper function that is called recursively.
On my Windows 7 system, for the input directory C:/Perl/lib/Test, I get the output:
== List of Terminal Folders ==
c:/Perl/lib/Test/Builder/IO
c:/Perl/lib/Test/Builder/Tester
c:/Perl/lib/Test/Perl/Critic
== List of Files in each Terminal Folder: ==
c:/Perl/lib/Test/Builder/IO/Scalar.pm
c:/Perl/lib/Test/Builder/Tester/Color.pm
c:/Perl/lib/Test/Perl/Critic/Policy.pm
Implementation
#!/usr/bin/env perl
use strict;
use warnings;
use Cwd qw(abs_path getcwd);
my #dir_list = find_terminal_directories("C:/Perl/lib/Test");
print "== List of Terminal Directories ==\n";
print join("\n", #dir_list), "\n";
print "\n== List of Files in each Terminal Directory: ==\n";
for my $dir (#dir_list) {
for my $file (<"$dir/*">) {
print "$file\n";
open my $fh, '<', $file or die $!;
my $data = <$fh>; # slurp entire file contents into $data
close $fh;
# Now, do something with $data !
}
}
sub find_terminal_directories {
my $rootdir = shift;
my #wanted;
my $cwd = getcwd();
chdir $rootdir;
find_terminal_directories_helper(".", \#wanted);
chdir $cwd;
return #wanted;
}
sub find_terminal_directories_helper {
my ($dir, $wanted) = #_;
return if ! -d $dir;
opendir(my $dh, $dir) or die "open directory error!";
my $count = 0;
foreach my $child (readdir($dh)) {
my $abs_child = abs_path($child);
next if (! -d $child || $child eq "." || $child eq "..");
++$count;
chdir $child;
find_terminal_directories_helper($abs_child, $wanted); # recursion!
chdir "..";
}
push #$wanted, abs_path($dir) if ! $count; # no sub-directories found!
}
Perhaps the following will be helpful:
use strict;
use warnings;
use File::Find::Rule;
my $dir = "D:/Temp";
local $/;
my #dirs =
sort File::Find::Rule->exec( sub { File::Find::Rule->directory->in($_) == 1 }
)->directory->in($dir);
for my $dir (#dirs) {
for my $file (<"$dir/*">) {
open my $fh, '<', $file or die $!;
my $data = <$fh>;
close $fh;
print $data;
}
}
local $/; lets us slurp the file's contents into a variable. Delete it if you only want to read the first line.
The sub in the exec() is used to pass only those dirs which don't contain a dir
sort is used to arrange those dirs in your wanted order
A file glob <"$dir/*"> is used to get the files in each dir
Edit: Have modified the code to find only 'terminal directories.' Thanks to DavidRR for this spec clarification.
I would use File::Find
Sample script:
#!/usr/bin/perl
use strict;
use warnings;
use File::Find;
my $dir = "/home/chris";
find(\&wanted, $dir);
sub wanted {
print "dir: $File::Find::dir\n";
print "file in dir: $_\n";
print "complete path to file: $File::Find::name\n";
}
OUTPUTS:
$ test.pl
dir: /home/chris/test_dir
file in dir: test_dir2
complete path to file: /home/chris/test_dir/test_dir2
dir: /home/chris/test_dir/test_dir2
file in dir: foo.txt
complete path to file: /home/chris/test_dir/test_dir2/foo.txt
...
Using backticks, write subdirs and files to a file called filelist:
`ls -R $dir > filelist`

perl - loop through directory to find file.mdb and execute code if file.ldb not found

I am a beginner PERL programmer and I have come across a snag that I can't get by. I have been reading and re-reading web posts and Simon Cozens book at perl.org all day, but can't seem to solve the problem.
My intention with the code below is to loop through files in a directory and when the file has a certain string a name to verify that the same file name doesn't exist with a different extension and if it doesn't, to print me the file name (later I will implement a delete of the file, but for now I want to ensure it will work.) Specifically, I am finding .mdb files and after checking there are no associated .ldb's files, deleting the .mdb file.
right now my code returns this:
RRED_Database_KHOVIS.ldb
RRED_Database_KHOVIS.mdb
I will kill RRED_Database_KHOVIS.mdb
RRED_Database_mkuttler.mdb
I will kill RRED_Database_mkuttler.mdb
RRED_Database_SBreslow.ldb
RRED_Database_SBreslow.mdb
I will kill RRED_Database_SBreslow.mdb
i want it to only return the "I will kill..." after a .mdb file with no associated .ldb file.
My current code is below. I appreciate any help offered...
use strict;
use warnings;
use File::Find;
use diagnostics;
my $dir = "//vfg1msfs01ab/vfgcfs01\$/Regulatory Reporting/Access Database/";
my $filename = "RRED_Database";
my $fullname, my $ext;
opendir DH, $dir or die "Couldn't open the directory: $!";
while ($_ = readdir(DH)) {
my $ext = ".mdb";
if ((/$filename/) && ($_ ne $filename . $ext)) {
print "$_ \n";
unless (-e $dir . s/.mdb/.ldb/) {
s/.ldb/.mdb/;
print "I will kill $_ \n\n" ;
#unlink $_ or print "oops, couldn't delete $_: $!\n";
}
s/.ldb/.mdb/;
}
}
When looping through files, I like to use 'next' statements repeatedly to assure that I'm only looking at exactly what I want. Try this:
use strict;
use warnings;
use File::Find;
use diagnostics;
my $dir = "//vfg1msfs01ab/vfgcfs01\$/Regulatory Reporting/Access Database/";
my $filename = "RRED_Database";
my $fullname, my $ext;
opendir DH, $dir or die "Couldn't open the directory: $!";
while ($_ = readdir(DH)) {
my $ext = ".mdb";
# Jump to next while() iteration unless the file begins
# with $filename and ends with $ext,
# and capture the basename in $1
next unless $_ =~ m|($filename.*)$ext|;
# Jump to next while() iteration if if the file basename.ldb is found
next if -f $1 . ".ldb";
# At this point, we have an mdb file with no matching ldb file
print "$_ \n";
print "I will kill $_ \n\n" ;
#unlink $_ or print "oops, couldn't delete $_: $!\n";
}
While stuart's anwser made it more lean... I was able to also get it to work with the code below... (i changed .mdb to .accdb because I am now dealing with different file type)
use strict;
use warnings;
use File::Spec;
use diagnostics;
my $dir = "//vfg1msfs01ab/vfgcfs01\$/Regulatory Reporting/Access Database/";
my $filename = "RRED_Database";
my $ext;
opendir DH, $dir or die "Couldn't open the directory: $!";
while ($_ = readdir(DH)) {
my $ext = ".accdb";
if ((/$filename/) && ($_ ne $filename . $ext) && ($_ !~ /.laccdb/)) {
# if file contains database name, is not the main database and is not a locked version of db
s/$ext/.laccdb/;
unless (-e File::Spec->join($dir,$_)) {
s/.laccdb/$ext/;
#print "I will kill $_ \n\n";
unlink $_ or print "oops, couldn't delete $_: $!\n";
}
s/.laccdb/$ext/;
}
}