looking for shell / perl script to capture the folder time stamp and the files inside the folder - perl

I am looking for script to capture the folder time stamp and the files inside the folder
Example: I have a folder Cform12 with files inside say note1.txt , note2.rtf , note3.ldt
ls -lrt will generate drwxr-xr-x 5 r12applprd dba 4096 Dec 4 02:31 Cform12
and
ls -lrt SCF6761-PROD will generate
total 12
-rwxr-xr-x 3 r12applprd dba 4096 Dec 4 02:30 note1.txt
-rwxr-xr-x 3 r12applprd dba 4096 Dec 4 02:30 note2.rtf
-rwxr-xr-x 26 r12applprd dba 4096 Dec 4 02:31 note3.ldt
Now i have output as
Dec 4 02:31 , Cform12 , note1.txt
Dec 4 02:31 , Cform12 , note2.txt
Dec 4 02:31 , Cform12 , note3.txt
Need help me with the shell or perl script for the same.

I have created the following script to help with my question .Thanks to the contributors for all the directions
use strict;
use warnings;
my $dir = '/tmp/';
opendir(DIR, $dir) or die $!;
while (my $file = readdir(DIR)) {
next unless (-d "$dir/$file");
my $fdir = "$dir/$file";
print "$fdir\n";
my $mtime = (stat $fdir)[9];
$mtime = localtime($mtime);
$mtime = echo "$mtime" | awk '{print \$2 , \$3, \$4}' 2 > /dev/ null;
chomp($mtime);
my #files1 = find $fdir -type f | awk -F "/" '{ print \$NF }';
foreach my $fil (#files1) { print "$mtime,$file,$fil"; }
}
closedir(DIR);
exit 0;

I think this script will help you if my understanding is correct..
use strict;
use warnings;
my $path="/var/inner";#directory absolute path
my $mtime=(stat $path)[9];
my $name=`basename $path`;
chomp($name);
$mtime= localtime($mtime);
$mtime=`echo "$mtime" | awk '{print \$2 , \$3, \$4}'`;
chomp($mtime);
my #files = glob("$path/*");
foreach my $file(#files){
print "$mtime , $name , ".`basename $file`;
}
The script below does the same but recursively. Is this what you want?
use strict;
use warnings;
my $path="/export/home/tarumugam/shellshop";#directory absolute path
sub rotator{
(my $path)=#_;
my $mtime=(stat $path)[9];
my $name=`basename $path`;
chomp($name);
$mtime= localtime($mtime);
$mtime=`echo "$mtime" | awk '{print \$2 , \$3, \$4}' 2> /dev/null`;
chomp($mtime);
my #files = glob("$path/*");
foreach my $file(#files){
if(-f $file){
print "$mtime , $name , ".`basename $file`;
}else{
rotator($file);
}
}
}
rotator($path);

Are you looking for something like the following?
$ stat -c "%x" src; find src -mindepth 1 -maxdepth 1 -exec basename '{}' \;
2013-12-03 22:39:42.911796567 -0500
UndirectedGraphClient.java
UndirectedGraph.java
Bag.java
A bash scripting guide could help you out a lot. Maybe take a look at http://mywiki.wooledge.org/BashGuide or http://wiki.bash-hackers.org/doku.php ?

Related

How to loop through a directory in perl

I have a directory dir1 containing several hundreds of files, which are to be iteratively processed by a speech program called HRest. The program is supposed to take each file one by one, process it and put it in a new directory (as dir2 for first iteration) to be used in next iteration. My problem is that i don't know if the way I've employed to loop through the files in dir1, and also the way I am running the script (trainhmms.pl dir1 1) is correct.
If the files in dir1 are L1, L2, L3, ..., L500, I want HRest to be executed as
HRest -T 1 -I timedlabels_train.mlf -t -i 20 -l dir1/L1 -M dir2 -S train.scp
for the first file, and as
HRest -T 1 -I timedlabels_train.mlf -t -i 20 -l dir1/L2 -M dir2 -S train.scp
for the next file, and so on for all files. Then in next call of the script, I want it to be changed to
HRest -T 1 -I timedlabels_train.mlf -t -i 20 -l dir2/L1 -M dir3 -S train.scp
for the first file, and so on..
Here is the script for the first iteration:
#!/usr/bin/perl
use File::Slurp;
# Usage: trainhmms.pl dir1 1
# dir1: Folder containing models after being initialised by HInit (L1,L2,..,L512)
$file = $ARGV[0];
$iter = $ARGV[1];
my #files = read_dir '/Users/negarolfati/Documents/Detection_rerun/AF_TIMIT/1_state//trainHMMs/dir1';
for my $file ( #files ) {
$iter2 = $iter+1;
$cmd = "HRest -T 1 -I timedlabels_train.mlf -t -i 20 -l '$dir[$iter]/$file' -M '$dir[$iter2]' -S train.scp ";
system("$cmd");
}
You can't just use readdir on a directory string. You have to opendir the string, then readdir from the directory handle that you get, and finally closedir the handle.
You must also remember that readdir returns directory names as well as file names, and the pseudo-directories . and .. too. To filter out just the files, you can use the -f test operator. And it is usually most convenient to chdir to the directory you are reading so that you don't have to append the path to each file name that readdir returns before you do the test.
I don't know what HRest is, but if your command line must be executed from a specific working directory (perhaps to acccess timedlabels_train.mlf and train.scp) then please say so. I will have to remove the chdir statement.
Something like this should get you going. I have used autodie, which does automatic checks on file system operations. It saves having to check chdir and opendir explicitly each time with or die $!.
#!/usr/bin/perl
use strict;
use warnings;
use autodie;
use File::Spec::Functions 'catdir';
my ($file, $iter) = #ARGV;
my $root = '/Users/negarolfati/Documents/Detection_rerun/AF_TIMIT/1_state/trainHMMs';
my $dir1 = catdir $root, 'dir'.$iter;
my $dir2 = catdir $root, 'dir'.($iter+1);
chdir $dir1;
opendir my ($dh), '.';
my #files = grep -f, readdir $dh;
closedir $dh;
for my $file ( #files ) {
my $cmd = "HRest -T 1 -I timedlabels_train.mlf -t -i 20 -l '$dir1/$file' -M '$dir2' -S train.scp";
system($cmd);
}
Update
Here is an alternative version that avoids chdir so that the current working directory remains unchanged.
I have added the secondary loop that was in your bash script. I have also added a print statement so that you can see each command before it is executed.
To allow the system call to go ahead, just delete or comment out the next statement.
#!/usr/bin/perl
use strict;
use warnings;
use autodie;
use File::Spec::Functions qw/ catdir catfile /;
STDOUT->autoflush;
my $root = '/Users/negarolfati/Documents/Detection_rerun/AF_TIMIT/1_state/trainHMMs';
for my $iter (1 .. 4) {
my $dir1 = catdir $root, 'dir'.$iter;
my $dir2 = catdir $root, 'dir'.($iter+1);
opendir my ($dh), $dir1;
while (my $node = readdir $dh) {
my $file = catfile($dir1, $node);
next unless -f $file;
my $cmd = "HRest -T 1 -I timedlabels_train.mlf -t -i 20 -l '$file' -M '$dir2' -S train.scp";
print $cmd, "\n";
next; # Remove for full functionality
system($cmd);
}
closedir $dh;
}
You can do this:
my #files = <$path/*>;
foreach my $filename ( reverse(#files) ) {
...
}

Perl script to compare file contents of dirA with file contents of dirB and output the difference in a separate file

I am just a beginner at perl and I need help with the following.
DirA and DirB have files f1 ,f2 ,f3 ,f4, f5 (not necessarily 5 in number). I need to compare the contents of f1 in DirA with contents of f1 in DirB and output the result in f1_diff. Similarly need to do this for all files in directories A and B. So in the above case assuming the contents of all files are different, Script will output 5 files f1_diff,f2_diff,f3_diff,f4_diff,f5_diff which has the missing lines of both f1 DirA and f1 DirB.
For example, if
f1 dir A has line1, line2, line3, line4xxxx, line5
f1 dir B has line1, line2xxxx, line3, line4, line5
f1_diff should have
line2 --> line2xxxx
line4xxxx -->line4
Can someone please help me with this.
Update:
I have the below script right now and need to add the following.
Filename: # of different lines
File 1 – 1
File 2 - 30
File 3 – missing in dir1
File 3a – missing in dir2
Secondly the number of lines differing. Can someone help me modify the same
#!/usr/bin/perl
package Main; {
use strict;
use warnings;
use Cwd;
my $DirA;
my $DirB;
my $y;
print ("\nChoose one of the entries below\n");
print ("e1\e2\e3\n\n");
print("Enter here --->");
my $dir = <>;
chomp($dir);
if ($dir eq "e1"){
$DirA = "./dir1";
$DirB = "./dir2";
}
elsif ($dir eq "e2"){
$DirA = "./dir3";
$DirB = "./dir4";
}
else{
$DirA = "./dir5";
$DirB = "./dir6";
}
opendir my($dh), "$DirA" or die "Couldn't open dir DirA!";
my #files = readdir $dh;
closedir $dh;
system("rm -rf diffs");
system ("mkdir diffs");
foreach my $file (#files) {
next if -d $file;
my $diff = `diff -y --suppress-common-lines "$DirA/$file" "$DirB/$file"`;
open DIFF_FILE, ">$file"."_diff";
print DIFF_FILE $diff;
close DIFF_FILE;#}
}
chdir("./diffs/");
my $cwd = cwd();
system("mv ../*_diff .");
foreach(glob('*.*')) {
unlink if (-f $_) && !-s _;
}
print("Total number of differences = "); system("ls *_diff | wc -l");print("\n");
}
I recommend the use of CPAN Modules to achieve this tasks:
To find files: File::Find
To compare files: File::Compare
To show file differences: Text::Diff
Check if the output format of Text::Diff is what you need, it offers configuration features
You could use Text::Diff.
#!/usr/bin/perl
use Text::Diff;
use strict;
use warnings;
opendir my($dh), "DirA" or die "Couldn't open dir DirA!";
my #files = readdir $dh;
closedir $dh;
foreach my $file (#files) {
next if -d $file;
my $diff = diff "DirA/$file", "DirB/$file";
open DIFF_FILE, ">$file"."_diff";
print DIFF_FILE $diff;
close DIFF_FILE;
}
It is easy if parallel is available :
ls dirA/f* | parallel 'echo {} - dirB/{/} >>{/}_dif ; diff {} dirB/{/} >> {/}_dif '

Making nested/recursive directories using Perl

I have written a C++ program that reads in a file and outputs the file with the same name with slight modifications. Now I need to browse recursively into the directory (of any package) and then create files in a separate directory having same directory structure.
I can browse into the directory by using
#file_list = `find <package directory> -name '*.cpp'`;
And I'm trying to execute my program like this:
foreach (#file_list) {
# This gives error as sh: other_dir/./lev1/lev2/filename.cpp: not found
./myexe $_ other_dir/$_;
}
Previously I had written a shell script which worked good with packages having files in a single directory. The script is as follows:
1 #!/bin/bash
2 #echo off
3 rm -rf demac_dir
4 mkdir demac_dir
5 for i in `ls *.h *.cpp`
6 do
7 echo processing file ${i} ...
8 ./main ${i} demac_dir/${i}
9 done
Now I'm trying to so the same thing with a package (i.e. wxWidgets) which has .cpp and .h files in nested subdirectories. So I wrote the following Perl script (after ysth's suggestion).
1 #!/usr/bin/perl -l
2 use File::Path 'make_path';
3 use File::Basename 'dirname';
4
5 `rm -rf demac_dir`;
6 `mkdir demac_dir`;
7 #l1 = `find . -name '*.h'`;
8 #l2 = `find . -name '*.cpp'`;
9 #l3 = `find . -name '*.cc'`;
10 push(#l , #l1, #l2, #l3);
11 foreach (#l) {
12 print "processing file $_ ...";
13 make_path( dirname( "demac_dir/$_" ));
14 ## `touch touch demac_dir/$_`;
15 `sudo ./main $_ demac_dir/$_`;
16 }
17
Err, you are missing a system() or something there that I presume is in your real code.
To make the directories you need, try:
use File::Path 'make_path';
use File::Basename 'dirname';
foreach (#file_list) {
make_path( dirname( "other_dir/$_" ) );
...
}
Update: seeing your code, it looks like you are leaving newlines on the end of the filenames, which are probably causing you trouble (though I don't see why your ./main processor would produce the error you say it does...).
Add a chomp(#l); before your foreach loop.
Check the FileHandler.pm and TemplateGenerator.pm from morphus

How can I detect that a symlink is broken in Perl?

I would like to remove a broken symlink in a directory using Perl.
In my mind I just had to list the file of a directory and test is this a symlink (-l) and if it returns false just unlink it.
But it appears that when using readir to list all files my broken symlinks are not recoganized as a file. Because my link is pointing to nothing I understand why.
All the file in $myDir are symlinks, either valid or broken.
When I display #files I only get a list of valid symlink.
opendir DIR, $myDir;
my #files = grep(/$regexp/,readdir(DIR));
closedir DIR;
print "filenames : #files\n";
There are two main relevant system calls, stat() and lstat(). The lstat() call will tell you that it is a symlink (but on other files, behaves the same as stat()). This allows you to determine that the name is a symlink. The stat() system call follows a symlink to its end, and tells you about the file (or directory) at the end of the link. If the stat() call fails on the symlink, then the symlink is broken or you're trying to access a directory or file where you have no permission.
The Perl file test operators include -l to detect whether a name is a symlink. You can use the Perl functions stat and lstat explicitly. Between these, you should be able to sort out whether a symlink is broken or not - but you should probably plan to write a function to do the job.
You probably don't need to use the readlink Perl function. Beware the underlying system readlink() call; it does not return a null-terminated string!
It is interesting that neither Perl nor its POSIX module supports the realpath() function. However, the PathTools module does support it. If realpath fails, on a symlink, the symlink is non-functional (aka broken).
Here's some code I've used to remove broken links:
chdir $dir or die;
opendir(DIR, '.') or die;
foreach my $link (readdir DIR) {
next unless -l $link and not -e readlink($link);
print "Removing broken link $link\n";
unlink $link;
}
closedir DIR;
Note that it's important that the directory containing the links is the current directory. readdir returns only filenames, and the links might be relative.
Combining lstat with stat:
say "dangling link at $fn" if (lstat $fn and not stat $fn);
update: it works for me...
salva#topo:~/t/dl$ perl -E 'opendir $dh, "."; say $_ for grep { !stat $_ and lstat $_ } readdir $dh'
foo
salva#topo:~/t/dl$ ls -l
total 0
-rw-rw-r-- 1 salva salva 0 2011-07-05 12:34 f
lrwxrwxrwx 1 salva salva 11 2011-07-05 12:00 fii -> /etc/shadow
lrwxrwxrwx 1 salva salva 12 2011-07-05 11:59 foo -> /etc/hjdkshf
Check for broken symlinks (checking only the top level if there are symlinks to symlinks):
use strict;
use warnings;
use autodie;
opendir my $dirh, '.';
while (my $file = readdir $dirh) {
if ( -l $file ) {
my $target = readlink $file;
if ( ! -e $target && ! -l $target ) {
print "$file -> $target broken\n";
}
}
}
Use readlink() and stat() the result.
Using the built-in Perl glob function ?
For examples:
#files = <*>;
foreach $file (#files) {
print $file . "\n";
}
For a specific $dir:
#files = <$dir*>;
foreach $file (#files) {
print $file . "\n";
}
A broken symlink is a link (-l) that does not exists (!-e)
perl -e 'print "broken: $_\n" for grep { -l and ! -e } glob("*");'

Perl file test operator help

This is a really basic issue, but I'm new to perl and cannot work out what the issue is. I'm just trying to isolate the files in a directory, but the -d operator keeps treating all the folder contents as files ...
#contents is my array, and when I run this:
use strict;
if ($ARGV[1]) {
die("Error: You can only monitor one directory at a time\n");
}
my $directory = $ARGV[0] || die "Error: No directory defined\n";
opendir(DIR, $directory) || die "Error: Can't open dir $directory: $!";
my #contents = readdir(DIR);
foreach my $item(#contents) {
if (-d $item) { next; }
print"$item is a file\n";
}
closedir (DIR);
I keep getting both folders and files. Alternatively, if I use -f, I get nothing.
edit: this is the output -
file01.txt is a file
folder 01 is a file
folder 02 is a file
Screen shot 2010-04-18 at 1.26.17 PM.png is a file
I'm running this on OSX
edit:dir ls -l output:
aaron ~/Documents/test: ls -l
total 112
-rw-r--r--# 1 aaron staff 51235 18 Apr 13:26 Screen shot 2010-04-18 at 1.26.17 PM.png
-rw-r--r--# 1 aaron staff 7 18 Apr 13:26 file01.txt
drwxr-xr-x 3 aaron staff 102 18 Apr 13:25 folder 01
drwxr-xr-x 2 aaron staff 68 18 Apr 13:25 folder 02
Solution
I was testing with '.' as the directory...you're testing with some other directory. The names read from the directory are then checked relative to the current directory. If I use some other directory name, I'll get almost everything except '.' and '..' listed as files, regardless.
If you prefix the name with the value of $ARGV[0], you'll get the expected result:
#!/bin/perl -w
use strict;
if ($ARGV[1]) {
die("Error: You can only monitor one directory at a time\n");
}
my $directory = $ARGV[0] || die "Error: No directory defined\n";
opendir(DIR, $directory) || die "Error: Can't open dir $directory: $!";
my #contents = readdir(DIR);
foreach my $item(#contents) {
next if -d "$ARGV[0]/$item";
print "$ARGV[0]/$item is a file\n";
}
closedir (DIR);
Previous attempts to explain
This works on MacOS X:
#!/bin/perl -w
use strict;
my #contents = <*>;
foreach my $item (#contents)
{
print "== $item\n";
next if -d $item;
print "$item is a file\n";
}
Test:
MiniMac JL: perl -c xx.pl
xx.pl syntax OK
MiniMac JL: perl xx.pl
== cproto-4.7g
== fpqsort1
fpqsort1 is a file
== fpqsort1.h
fpqsort1.h is a file
== fpqsort2
fpqsort2 is a file
== fpqsort2.c
fpqsort2.c is a file
== gcc-predef.h
gcc-predef.h is a file
== git-1.6.5.7
== go
== makefile
makefile is a file
== qs-test1.c
qs-test1.c is a file
== qs-test2.c
qs-test2.c is a file
== RCS
== rep-report.txt
rep-report.txt is a file
== select.c
select.c is a file
== soq
== xx.pl
xx.pl is a file
MiniMac JL:
Given a marginally modified version of the code in the question:
#!/bin/perl -w
use strict;
if ($ARGV[1]) {
die("Error: You can only monitor one directory at a time\n");
}
my $directory = $ARGV[0] || die "Error: No directory defined\n";
opendir(DIR, $directory) || die "Error: Can't open dir $directory: $!";
my #contents = readdir(DIR);
foreach my $item(#contents) {
print "<<$item>>\n";
next if -d $item;
print"$item is a file\n";
}
closedir (DIR);
Running it on the same directory as before, I get the output:
Minimac JL: perl yy.pl .
<<.>>
<<..>>
<<cproto-4.7g>>
<<fpqsort1>>
fpqsort1 is a file
<<fpqsort1.h>>
fpqsort1.h is a file
<<fpqsort2>>
fpqsort2 is a file
<<fpqsort2.c>>
fpqsort2.c is a file
<<gcc-predef.h>>
gcc-predef.h is a file
<<git-1.6.5.7>>
<<go>>
<<makefile>>
makefile is a file
<<qs-test1.c>>
qs-test1.c is a file
<<qs-test2.c>>
qs-test2.c is a file
<<RCS>>
<<rep-report.txt>>
rep-report.txt is a file
<<select.c>>
select.c is a file
<<soq>>
<<xx.pl>>
xx.pl is a file
<<yy.pl>>
yy.pl is a file
Minimac JL:
Note the Perlish idiom 'next if -d $item;'. Also note the debugging techniques: print the names as they go through the array - using the '<<' and '>>' to surround the name helps identify odd side effects (such as newlines in names). I did double check that the provided code produces the same result - it does. And I'm running on a MacOS X 10.6.3 with the stock Perl.
From the Perldoc on readdir:
If you're planning to filetest the
return values out of a readdir, you'd
better prepend the directory in
question. Otherwise, because we didn't
chdir there, it would have been
testing the wrong file.
Debugged code:
#!/usr/bin/perl
use strict;
if ($ARGV[1]) {
die("Error: You can only monitor one directory at a time\n");
}
my $directory = $ARGV[0] || die "Error: No directory defined\n";
opendir(DIR, $directory) || die "Error: Can't open dir $directory: $!";
my #contents = readdir(DIR);
foreach my $item (#contents) {
print "\$item=$item\n";
print "$item is a file \n" if (-f $directory."/".$item);
print"$item is a dir\n" if (-d $directory."/".$item);
}
closedir (DIR);
I guess your #contents has newlines at the end of the file. Try adding chomp:
foreach $item(#contents) {
chomp($item);
next if (-d $item);
print"$item is a file\n";
}