Read content from file and find full file-name on disk - perl

My problem is that I have a bunch of file names without the version appended (version keeps changing everytime). The file names are in a file in a particular sequence and I need to get the latest version from a folder and then sequentially install the same. The logic would be:
scan a file with contents
read a line from the file
using this as a key, access the folder and match the same
if found, write the full file-name to a file with some characters appended
if not found, skip and loop to line 1, till all the lines in the file are finished
What is the best language to use: shell script or Perl for such a task? And if someone can provide some hints in the form of code :-)

I would read in all your partial filenames then loop through the folder matching the full filenames against the partial ones. The exact implementation would depend on some details.
Do the full filenames need to appear in the same order as the partial ones did? Can you derive the partial filename from the full filename?
Update: so, something like (assuming $infile, $outfile, and $indir are already opened file and dirhandles, and a translation routine partial_filename_from_full that returns undef for things like directories or non-relevant files):
chomp( my #partial_filenames = readline( $infile ) );
while ( my $filename = readdir( $indir ) ) {
my $partial_filename = partial_filename_from_full( $filename );
if ( defined $partial_filename ) {
$full_filename{ $partial_filename } = $filename;
}
}
for my $partial_filename ( #partial_filenames ) {
if ( exists $full_filename{ $partial_filename } ) {
print $outfile $full_filename{ $partial_filename }, "\n";
} else {
# error? just skip it? you decide
}
}
If there are multiple full filenames per partial filename, instead of assigning:
$full_filename{ $partial_filename } = $filename;
you would determine if $filename were a better "match" than the previously encountered
one.

Your question is not very clear, but I'm guessing you have a directory containing file names such as:
fileA01
fileA02
fileB03
fileB05
fileB12
fileC02
fileD09
fileE22
The file you scan 'with contents' contains a list of names such as:
fileA
fileB
fileE
And you want code to find the entry in the directory with the highest version number for the corresponding file name:
fileA02
fileB12
fileE22
You will have to decide exactly how versions are compared - I've used 2-digit version numbers, but you haven't stated your constraints.
I would probably use Perl for this. First, I'd read the whole 'file with contents' into memory, and then create a monster regex to recognize the file names - possibly with the version number detection included. I'd use opendir, readdir (and closedir) to process the directory. For each line, I'd match it with the regex, and capture whether the name was the most recent version of any of the sought files. If so, I'd capture the filename in a hash, indexed by the version-less filename (hence, if fileA01 was read first, then I'd have $filelist{fileA} = "fileA01"; except of course both the hash key and the full filename would be in variables.
Doing it in shell would be harder. Using the most powerful features of Bash, it is probably doable; I'd still use Perl (or Python, or any scripting language).

I would use awk.
awk -f myawk.awk
myawk.awk
BEGIN{
}
{
myfilename = $0;
retval = getline otherfile < myfilename;
if (retval == -1) # check the correct syntax
{
# file does not exist. do the necessary error handling
}
else
{
# File exists. so do what you want.
# perhaps you might want to write to a new file with the modified filename
}
}
END{
}

Related

Perl Script did not find the newest file

I use the following perl script from "https://exchange.nagios.org/directory/Plugins/Operating-Systems/Linux/Check-Newest-files-age-and-size-in-Diredtory/"
But in these script is an Error. The script is not showing the newest file. Can someone find the mistake? In the comments of the site have wrote somebody, that in line 22 the mistake is. I can't find it:
Here is the code:
# Check that file exists (can be directory or link)
unless (-d $opt_f) {
print "FILE_AGE CRITICAL: Folder not found - $opt_f\n";
exit $ERRORS{'CRITICAL'};
}
my $list = opendir DIRHANDLE, $opt_f or die "Cant open directory: $!";
while ($_ = readdir(DIRHANDLE))
{
$file=sprintf("%s/%s",$opt_f,$_);
$attrs = stat("$file");
$diff = time()-$attrs->mtime;
if($temp == 0)
{
#$temp=$diff;
$new=$file;
}
if($_ ne "." && $_ ne "..")
{
if($diff<$temp)
{
$temp=$diff;
$new=$_;
}
else
{
$temp=$diff; $new=$_;
}
}
}
$st = File::stat::stat($opt_f."/".$new);
$age = time - $st->mtime;
$size = $st->size;
Example:
I have some files on a filer (backups in a .img File). I use this script, to check the newest file size. If I create a new folder with a new file, the check looks to the correct file. But if I create a second file, the check looks to the old file anytime. If I create a third file, the check goes to the correct file. The fourth file is wrong and the fifth file is correct again(and so on)
An easy (easier?) way to do this would be to use the built-in glob function to read the directory instead of opening it, and then use simple file tests to sort the files by creation or modification time:
my #files = sort {-M($a) <=> -M($b)} glob "*"; # or -C for creation
# $files[0] is the newest file
A list of file test operators is at
https://users.cs.cf.ac.uk/Dave.Marshall/PERL/node69.html
Note that -C and -M relate to when the script started, so for long-running or daemon scripts you might need to do something a bit different.
You want to find the earliest mtime, so we're talking about a simple comparison of the previously-found earlier mtime with the mtime of the current file. But there's so much code beyond that in what you posted ...and the first thing you do with the value you want to compare is change it? What?
Let's start over.
my $earliest_mtime = -1;
my $earliest_qfn;
while (defined( my $fn = readdir($dh) )) {
next if $fn =~ /^\.\.?\z/;
my $qfn = "$dir_qfn/$fn";
my $stats = stat($qfn)
or warn("Can't stat \"$qfn\": $!\n"), next;
my $mtime = $stats->mtime;
if ($mtime < $earliest_mtime) {
$earliest_mtime = $mtime;
$earliest_qfn = $qfn;
}
}
if (defined($earliest_qfn)) {
say $earliest_qfn;
}
The biggest issue with the script seems to be that line 12 calls the core version of stat but line 13 expects the output to be that of File::stat::stat(). I suspect that testing for '.' or '..' should be done at the top of the while loop and all the variables should be defined before they are used.
As Jeremy has said, you're better off sorting an array of the files and pushing/poping the first/last value, depending on what you're looking for.

reading from multiple files using wild character in perl

Suppose I have 5 files: tmp1.txt, tmp2.txt, tmp3.txt, temp1.txt, temp2.txt.
Now is there any way to open multiple files and read from them using wilcards.
Example,
If I write "t*.txt" then data from each file should be read.
If I write "tm*.txt" then only data from 3 files should be read.
Yes, you can use a glob, assuming these files exist in a local directory, and no other files with similar names are in that directory.
print "Read which files? ";
chomp(my $glob = <STDIN>);
my #files_to_read = glob $glob;
Of course, you can assure that you get no other files by filtering them
my %valid = map { $_ => 1 } qw(tmp1 tmp2 tmp3 temp1 temp2);
#files = grep $valid{$_}, glob $glob;
The first statement creates a hash where the valid file name keys have a true value, the other statement runs this check on the elements of the glob list.
You can use glob to find the list of files, and read through them sequentially by assigning the list to #ARGV, which emulates them being passed on the command line.
our #ARGV = glob '/path/to/tm*.txt';
while (<ARGV>) {
print;
}

Debugging Perl one-liner for renaming files

I was testing/trying out this Perl one-liner, and I'm trying to figure out what happened to the files. I don't see the files anymore.
Did I delete them or what went wrong?
Example of file names listed (original):
IMG_0178.JPG
IMG_0182.JPG
IMG_0183.JPG
IMG_0184.JPG
IMG_0186.JPG
I wanted to simply change the file extension to lowercase (.jpg):
perl -e'while(<*.JPG>) { s/JPG$/jpg/; rename <*.jpg>, $_ }'
Don't use rename with a glob. Use scalars. Try to assign the file name to a new variable before the substitution and rename the old name to the modified one, like this:
perl -e'while(<*.JPG>) { ($new = $_) =~ s/JPG$/jpg/; rename $_, $new }'
Check output with ls -1:
IMG_0178.jpg
IMG_0182.jpg
IMG_0183.jpg
IMG_0184.jpg
IMG_0186.jpg
Bizarrely your code should do what you wanted.
A file glob like <*.JPG> in scalar context will return the next file that matches the pattern, and since both while and rename apply scalar context, the two globs return the same value at each iteration.
while (<*.JPG>) {
s/JPG$/jpg/;
rename <*.jpg>, $_;
}
In the first iteration of the loop $_ is set to IMG_0178.JPG by the while, and the substitution sets the file type to lower case.
Then in the rename <*.jpg> is executed in scalar context and again returns IMG_0178.JPG - the first file in the same list because Windows file names are case-insensitive.
So finally the rename performs rename 'IMG_0178.JPG', 'IMG_0178.jpg' as required.
Rewriting rename like this shows this clearly
sub ren($$) {
print "$_[0] -> $_[1]\n";
}
while (my $file = <*.JPG>) {
$file =~ s/JPG$/jpg/;
ren <*.JPG>, $file;
}
output
IMG_0178.JPG -> IMG_0178.jpg
IMG_0182.JPG -> IMG_0182.jpg
IMG_0183.JPG -> IMG_0183.jpg
IMG_0184.JPG -> IMG_0184.jpg
IMG_0186.JPG -> IMG_0186.jpg
So you are lucky, and your files should have been renamed as you wanted.
But don't do this. In particular you should run the program with a print statement in place of any critical operations so that you can see what is going to happen.
This would be better as id more clearly does what is intended
perl -e '($f = $_) =~ s/JPG$/jpg/i and rename $_, $f while <*.JPG>'

How to find a file which exists in different directories under a given path in Perl

I'm looking for a method to looks for file which resides in a few directories in a given path. In other words, those directories will be having files with same filename across. My script seem to have the hierarchy problem on looking into the correct path to grep the filename for processing. I have a fix path as input and the script will need to looks into the path and finding files from there but my script seem stuck on 2 tiers up and process from there rather than looking into the last directories in the tier (in my case here it process on "ln" and "nn" and start processing the subroutine).
The fix input path is:-
/nfs/disks/version_2.0/
The files that I want to do post processing by subroutine will be exist under several directories as below. Basically I wanted to check if the file1.abc do exists in all the directories temp1, temp2 & temp3 under ln directory. Same for file2.abc if exist in temp1, temp2, temp3 under nn directory.
The files that I wanted to check in full path will be like this:-
/nfs/disks/version_2.0/dir_a/ln/temp1/file1.abc
/nfs/disks/version_2.0/dir_a/ln/temp2/file1.abc
/nfs/disks/version_2.0/dir_a/ln/temp3/file1.abc
/nfs/disks/version_2.0/dir_a/nn/temp1/file2.abc
/nfs/disks/version_2.0/dir_a/nn/temp2/file2.abc
/nfs/disks/version_2.0/dir_a/nn/temp3/file2.abc
My script as below:-
#! /usr/bin/perl -w
my $dir = '/nfs/fm/disks/version_2.0/' ;
opendir(TEMP, $dir) || die $! ;
foreach my $file (readdir(TEMP)) {
next if ($file eq "." || $file eq "..") ;
if (-d "$dir/$file") {
my $d = "$dir/$file";
print "Directory:- $d\n" ;
&getFile($d);
&compare($file) ;
}
}
Note that I put the print "Directory:- $d\n" ; there for debug purposes and it printed this:-
/nfs/disks/version_2.0/dir_a/
/nfs/disks/version_2.0/dir_b/
So I knew it get into the wrong path for processing the following subroutine.
Can somebody help to point me where is the error in my script? Thanks!
To be clear: the script is supposed to recurse through a directory and look for files with a particular filename? In this case, I think the following code is the problem:
if (-d "$dir/$file") {
my $d = "$dir/$file";
print "Directory:- $d\n" ;
&getFile($d);
&compare($file) ;
}
I'm assuming the &getFile($d) is meant to step into a directory (i.e., the recursive step). This is fine. However, it looks like the &compare($file) is the action that you want to take when the object that you're looking at isn't a directory. Therefore, that code block should look something like this:
if (-d "$dir/$file") {
&getFile("$dir/$file"); # the recursive step, for directories inside of this one
} elsif( -f "$dir/$file" ){
&compare("$dir/$file"); # the action on files inside of the current directory
}
The general pseudo-code should like like this:
sub myFind {
my $dir = shift;
foreach my $file( stat $dir ){
next if $file -eq "." || $file -eq ".."
my $obj = "$dir/$file";
if( -d $obj ){
myFind( $obj );
} elsif( -f $obj ){
doSomethingWithFile( $obj );
}
}
}
myFind( "/nfs/fm/disks/version_2.0" );
As a side note: this script is reinventing the wheel. You only need to write a script that does the processing on an individual file. You could do the rest entirely from the shell:
find /nfs/fm/disks/version_2.0 -type f -name "the-filename-you-want" -exec your_script.pl {} \;
Wow, it's like reliving the 1990s! Perl code has evolved somewhat, and you really need to learn the new stuff. It looks like you learned Perl in version 3.0 or 4.0. Here's some pointers:
Use use warnings; instead of -w on the command line.
Use use strict;. This will require you to predeclare variables using my which will scope them to the local block or the file if they're not in a local block. This helps catch a lot of errors.
Don't put & in front of subroutine names.
Use and, or, and not instead of &&, ||, and !.
Learn about Perl Modules which can save you a lot of time and effort.
When someone says detect duplicates, I immediately think of hashes. If you use a hash based upon your file's name, you can easily see if there are duplicate files.
Of course a hash can only have a single value for each key. Fortunately, in Perl 5.x, that value can be a reference to another data structure.
So, I recommend you use a hash that contains a reference to a list (array in old parlance). You can push each instance of the file to that list.
Using your example, you'd have a data structure that looks like this:
%file_hash = {
file1.abc => [
/nfs/disks/version_2.0/dir_a/ln/temp1
/nfs/disks/version_2.0/dir_a/ln/temp2
/nfs/disks/version_2.0/dir_a/ln/temp3
],
file2.abc => [
/nfs/disks/version_2.0/dir_a/nn/temp1
/nfs/disks/version_2.0/dir_a/nn/temp2
/nfs/disks/version_2.0/dir_a/nn/temp3
];
And, here's a program to do it:
#! /usr/bin/env perl
#
use strict;
use warnings;
use feature qw(say); #Can use `say` which is like `print "\n"`;
use File::Basename; #imports `dirname` and `basename` commands
use File::Find; #Implements Unix `find` command.
use constant DIR => "/nfs/disks/version_2.0";
# Find all duplicates
my %file_hash;
find (\&wanted, DIR);
# Print out all the duplicates
foreach my $file_name (sort keys %file_hash) {
if (scalar (#{$file_hash{$file_name}}) > 1) {
say qq(Duplicate File: "$file_name");
foreach my $dir_name (#{$file_hash{$file_name}}) {
say " $dir_name";
}
}
}
sub wanted {
return if not -f $_;
if (not exists $file_hash{$_}) {
$file_hash{$_} = [];
}
push #{$file_hash{$_}}, $File::Find::dir;
}
Here's a few things about File::Find:
The work takes place in the subroutine wanted.
The $_ is the name of the file, and I can use this to see if this is a file or directory
$File::Find::Name is the full name of the file including the path.
$File::Find::dir is the name of the directory.
If the array reference doesn't exist, I create it with the $file_hash{$_} = [];. This isn't necessary, but I find it comforting, and it can prevent errors. To use $file_hash{$_} as an array, I have to dereference it. I do that by putting a # in front of it, so it can be #$file_hash{$_} or, #{$file_hash{$_}}.
Once all the file are found, I can print out the entire structure. The only thing I do is check to make sure there is more than one member in each array. If there's only a single member, then there are no duplicates.
Response to Grace
Hi David W., thank you very much for your explainaion and sample script. Sorry maybe I'm not really clear in definding my problem statement. I think I can't use hash in my path finding for the data structure. Since the file*.abc is a few hundred and undertermined and each of the file*.abc even is having same filename but it is actually differ in content in each directory structures.
Such as the file1.abc resides under "/nfs/disks/version_2.0/dir_a/ln/temp1" is not the same content as file1.abc resides under "/nfs/disks/version_2.0/dir_a/ln/temp2" and "/nfs/disks/version_2.0/dir_a/ln/temp3". My intention is to grep the list of files*.abc in each of the directories structure (temp1, temp2 and temp3 ) and compare the filename list with a masterlist. Could you help to shed some lights on how to solve this? Thanks. – Grace yesterday
I'm just printing the file in my sample code, but instead of printing the file, you could open them and process them. After all, you now have the file name and the directory. Here's the heart of my program again. This time, I'm opening the file and looking at the content:
foreach my $file_name (sort keys %file_hash) {
if (scalar (#{$file_hash{$file_name}}) > 1) {
#say qq(Duplicate File: "$file_name");
foreach my $dir_name (#{$file_hash{$file_name}}) {
#say " $dir_name";
open (my $fh, "<", "$dir_name/$file_name")
or die qq(Can't open file "$dir_name/$file_name" for reading);
# Process your file here...
close $fh;
}
}
}
If you are only looking for certain files, you could modify the wanted function to skip over files you don't want. For example, here I am only looking for files which match the file*.txt pattern. Note I use a regular expression of /^file.*\.txt$/ to match the name of the file. As you can see, it's the same as the previous wanted subroutine. The only difference is my test: I'm looking for something that is a file (-f) and has the correct name (file*.txt):
sub wanted {
return if not -f $_ and /^file.*\.txt$/;
if (not exists $file_hash{$_}) {
$file_hash{$_} = [];
}
push #{$file_hash{$_}}, $File::Find::dir;
}
If you are looking at the file contents, you can use the MD5 hash to determine if the file contents match or don't match. This reduces a file to a mere string of 16 to 28 characters which could even be used as a hash key instead of the file name. This way, files that have matching MD5 hashes (and thus matching contents) would be in the same hash list.
You talk about a "master list" of files and it seems you have the idea that this master list needs to match the content of the file you're looking for. So, I'm making a slight mod in my program. I am first taking that master list you talked about, and generating MD5 sums for each file. Then I'll look at all the files in that directory, but only take the ones with the matching MD5 hash...
By the way, this has not been tested.
#! /usr/bin/env perl
#
use strict;
use warnings;
use feature qw(say); #Can use `say` which is like `print "\n"`;
use File::Find; #Implements Unix `find` command.
use Digest::file qw(digest_file_hex);
use constant DIR => "/nfs/disks/version_2.0";
use constant MASTER_LIST_DIR => "/some/directory";
# First, I'm going thorugh the MASTER_LIST_DIR directory
# and finding all of the master list files. I'm going to take
# the MD5 hash of those files, and store them in a Perl hash
# that's keyed by the name of file file. Thus, when I find a
# file with a matching name, I can compare the MD5 of that file
# and the master file. If they match, the files are the same. If
# not, they're different.
# In this example, I'm inlining the function I use to find the files
# instead of making it a separat function.
my %master_hash;
find (
{
%master_hash($_) = digest_file_hex($_, "MD5") if -f;
},
MASTER_LIST_DIR
);
# Now I have the MD5 of all the master files, I'm going to search my
# DIR directory for the files that have the same MD5 hash as the
# master list files did. If they do have the same MD5 hash, I'll
# print out their names as before.
my %file_hash;
find (\&wanted, DIR);
# Print out all the duplicates
foreach my $file_name (sort keys %file_hash) {
if (scalar (#{$file_hash{$file_name}}) > 1) {
say qq(Duplicate File: "$file_name");
foreach my $dir_name (#{$file_hash{$file_name}}) {
say " $dir_name";
}
}
}
# The wanted function has been modified since the last example.
# Here, I'm only going to put files in the %file_hash if they
sub wanted {
if (-f $_ and $file_hash{$_} = digest_file_hex($_, "MD5")) {
$file_hash{$_} //= []; #Using TLP's syntax hint
push #{$file_hash{$_}}, $File::Find::dir;
}
}

Splitting a concatenated file based on header text

I have a few very large files which are basically a concatenation of several small files and I need to split them into their constituent files. I also need to name the files the same as the original files.
For example the files QMAX123 and QMAX124 have been concatenated to:
;QMAX123 - Student
... file content ...
;QMAX124 - Course
... file content ...
I need to recreate the file QMAX123 as
;QMAX123 - Student
... file content ...
And QMAX124 as
;QMAX124 - Course
... file content ...
The original file's header ;QMAX<some number> is unique and only appears as a header in the file.
I used the script below to split the content of the files, but I haven't been able to adapt it to get the file names right.
awk '/^;QMAX/{close("file"f);f++}{print $0 > "file"f}' <filename>
So I can either adapt that script to name the file correctly or I can rename the split files created using the script above based on the content of the file, whichever is easier.
I'm currently using cygwin bash (which has perl and awk) if that has any bearing on your answer.
The following Perl should do the trick
use warnings ;
use strict ;
my $F ; #will hold a filehandle
while (<>) {
if ( / ^ ; (\S+) /x) {
my $filename = $1 ;
open $F, '>' , $filename or die "can't open $filename " ;
} else {
next unless defined $F ;
print $F $_ or warn "can't write" ;
}
}
Note it discards any input before a line with filename next unless defined $F ; You may care to generate an error or add a default file. Let me know and I can change it
With Awk, it's as simple as
awk '/^;QMAX/ {filename = substr($1,2)} {print >> filename}' input_file