Generate dynamic file name list - perl

I started programming in perl few months back and this is my first question on stackoverflow.com. I hope I can get a solution.
So I want to copy some files from an ftp server. The file names are in this format:
abc_201149_cde_07_fgh_include_internal
In this example the numeric part gets changed on weekly basis, e.g. 201149 says year = 2011 and week = 49. Similarly, 07 says which version it is.
I have copied all the file names into one file called "sdk_link.txt" and I am reading each file name from this and then copying to my local PC:
use Net::FTP;
use File::Copy;
$Login = "<redacted>";
$Pwd = "<redacted>";
$ftpHost = "<redacted>";
$ftpFolder = "/daj/dhakj/ahdakl/abc_201206_def_05";
$ftp=Net::FTP->new($ftpHost,Timeout=>100);
if ($ftp)
{
print $ftp->message;
}
$ftp->login($Login,$Pwd);
print $ftp->message;
$ftp->cwd($ftpFolder);
print $ftp->message;
open FILE,"sdk_link.txt" or die $!;
while($test=<FILE>)
{
chomp($test);
#Copy the file
copy("$test","/testing");
}
$ftp->quit;
I want to run this script every week on Windows. How can I make the numeric part change so that the correct files get downloaded?

Well, the obvious answer is to keep a template on file, and insert the correct numbers. For example:
echo abc_%s_cde_%s_fgh_include_internal |
perl -MPOSIX -nE 'say sprintf $_, strftime("%Y%U", localtime()), "07";'
Output:
abc_201207_cde_07_fgh_include_internal
So that if you'd have a file with templates, you can use %s to insert strings, and provide arguments either from your own list of arguments, or dynamically, as you prefer. E.g.:
my $year = "2011";
my $week = "49";
my $ver = "07"; # Strings or numbers does not really matter
open my $fh, '<', "sdk_link.txt" or die $!;
while (<$fh>) {
my $file = sprintf $_, $year, $week, $ver;
copy($file, "/testing") or die "Copy failed for file $file: $!";
}
I am not so sure File::Copy::copy works as intended for remote files, but that's another question. I believe Net::FTP::get() might be what you want.

Related

date creation with Image::Exiftools or Date::Handler

I'm trying to merge all dirs containing pictures having different timeZone, in a single dir, uniform file's names with a same format (Hungarian style yyyymmdd_hhmmss) eliminating huge of duplicates, at once.
I was looking for modules (over the stat statement which return only the epoch (stat(file))[9]) that can read the file's creation date, not the last modification or access date.
I fount Immage::ExifTools seem to be the effective and easiest to list.
I noted that after several cycling into the dir the module stop to extract the correct date/time output stuck in a monotone identical wrong date/time value for the next over files. Here is the list and below one of pictures that get me into this trouble: more precisely file's property on win10 give a creation date of 20140626 12:16 pm; running instead the routine I obtain 20021208_120000.
foreach $img(#img){
next if -l $img;
$img =~ /.+(\..+$)/;
$ext = $1;
# %ENV;
$exif=new Image::ExifTool;
$exif->ExtractInfo($dir.$img);
$for = $exif->GetValue('CreateDate');
$for =~ s/$space/\_/g;
$for =~ s/\://g;
$for = '_'.$for;
$size = (stat($dir.$img))[7];
# $date = Date->new($date[9]);
# #data = $date->array;
#tie my %date, 'Date::Tie', utc_epoch => $date{$date[9]}; #tz => $date{tz};
#my $date = Date::Tie->new( epoch => $date[9] );
%date;
# $for = IMG.$for.$ext;
if (!$all{'IMG'.$for.$ext}){
$all{'IMG'.$for.$ext}= $size ;
rename $dir.$img, $dir.'IMG'.$for.$ext;
print "rename $dir.$img, $dir.'IMG'.$for.$ext\n";
}elsif($all{'IMG'.$for.$ext} == $size){
unlink $dir.$img;
print "Deleting $dir.$img\n";
}
Checking the files properties involved, the "wrong" ones seems to have the same properties "working" ones: both working and wrong state the properties of : creation/acquisition date, modification date and last access date..
I can't understand where the module fault.
Have you any recommendation ? Any different module to use ?
Thanks
Simon
On Windows you can use Win32API::File::Time to read and modify the file creation time:
use feature qw(say);
use strict;
use warnings;
use Win32API::File::Time qw(GetFileTime SetFileTime);
use File::Temp qw(tempdir);
my $dir = tempdir( CLEANUP => 1 );
my $fn = 'test.txt';
open (my $fh, '>', $fn) or die "Could not open file '$fn': $!";
say $fh "Foobar";
close $fh;
print_file_times($fn);
my ($atime, $mtime, $ctime) = GetFileTime ($fn);
SetFileTime ($fn, undef, undef, $ctime-180);
print_file_times($fn);
sub print_file_times {
my ($fn) = #_;
my ($atime, $mtime, $ctime) = GetFileTime ($fn);
say "File: $fn:";
say " File access time: $atime";
say " File modification time: $mtime";
say " File creation time: $ctime";
return $ctime;
}
Output:
File: test.txt:
File access time: 1614640101
File modification time: 1614640101
File creation time: 1614639958
File: test.txt:
File access time: 1614640101
File modification time: 1614640101
File creation time: 1614639778

Perl script to compare two files and report new entries in file 1

I am looking to compare two files. Incoming file from a vendor with IPs and a local file we have been using that will soon be updated with the vendor copy. I would like it it print only the new data that it finds that is not in our copy.
IPV4-vendorlist.txt
10.0.0.0
192.168.1.1
192.168.2.2
192.168.3.3
IPV4-outgoing.txt
10.0.0.0
192.168.1.1
192.168.2.2
In this example, I would like it to print "The following will be added: 192.168.3.3".
Here is the code I have thus far that runs, it just doesn't produce any output:
use strict;
my $fname = 'IPV4-vendorlist.txt';
open my $vendor, "<", $fname
or die "Couldn't open $fname: $!";
my %urls;
while (my $url = <$vendor>) {
chomp $url;
$urls{$url} = undef;
}
close $vendor;
$fname = 'IPV4-outgoing.txt';
open my $ourfile, "<", $fname
or die "Couldn't open $fname: $!";
while (my $url = <$ourfile>) {
chomp $url;
next if exists $urls{$url};
print "The following will be added: $url";
}
close $ourfile;
Your script (probably) works. There are no IPs in the "outgoing" list which are not also already on the "vendor" list. (Perhaps you meant the other way around? There are addresses on the "vendor" list which are not on the "outgoing" list.)
For what it's worth, standard Unix tools like diff and cmp and comm already provide basic functionality for comparing lists.
This kind of thing is trivial to do with List::Compare. Read each file into an array, where each line is an element in the array. Then:
use strict;
use warnings;
use List::Compare;
$lc = List::Compare->new(\#vendor_list, \#outgoing);
my #new_ips = $lc->get_unique();

Find a string in one file and append part of it to matching line in another file

I have two files
first:
8237764738;00:78:9E:EE:CA:6F;FTTH;MULTI
8237764738;2C:39:96:52:47:82;FTTH;MULTI
0415535921;E8:BE:81:86:F1:6F;FTTH;MULTI
0415535921;2C:39:96:5B:12:C6;EZ;SINGLE
...etc
second:
00:78:9E:EE:CA:6F;2013/10/28 13:37:50
E8:BE:81:86:F1:6F;2013/11/05 13:38:30
00:78:9E:EC:4A:B0;2013/10/28 13:59:16
2C:E4:12:AA:F7:95;2013/10/31 13:57:55
...etc
and I have to take mac_address (second position) from the first file and find it in the second one
and append (if match) to first file the date at end from the second file.
output:
8237764738;00:78:9E:EE:CA:6F;FTTH;MULTI;2013/10/28 13:37:50
0415535921;E8:BE:81:86:F1:6F;FTTH;MULTI;2013/11/05 13:38:30
I write a simple script to find the mac_address
but I don't know how to put in the script to add the date.
my %iptv;
my #result;
open IN, "/home/terminals.csv";
while (<IN>) {
chomp;
#wynik = split(/;/,$_);
$iptv{$result[1]} = $result[0];
}
close IN;
open IN, "/home/reboots.csv";
open OUT, ">/home/out.csv";
while (<IN>) {
chomp;
my ($mac, $date) = split(/;/,$_);
if (defined $iptv{$mac})
{
print OUT "$date,$mac \n";
}
}
close IN;
close OUT;
Assuming that the first file lists each MAC number once and that you want an output line for each time the MAC appears in the second file, then:
#!/usr/bin/env perl
use strict;
use warnings;
die "Usage: $0 terminals reboots\n" unless scalar(#ARGV) == 2;
my %iptv;
open my $in1, '<', $ARGV[0] or die "Failed to open file $ARGV[0] for reading";
while (<$in1>)
{
chomp;
my #result = split(/;/, $_); # Fix array used here
$iptv{$result[1]} = $_; # Fix what's stored here
}
close $in1;
open my $in2, '<', $ARGV[1] or die "Failed to open file $ARGV[1] for reading";
while (<$in2>)
{
chomp;
my ($mac, $date) = split(/;/,$_);
print "$iptv{$mac};$date\n" if (defined $iptv{$mac});
}
close $in2;
This uses two file names on the command line and writes to standard output; it is a more general purpose program than your original. It also gets me around the problem that I don't have a /home directory.
For your sample inputs, the output is:
8237764738;00:78:9E:EE:CA:6F;FTTH;MULTI;2013/10/28 13:37:50
0415535921;E8:BE:81:86:F1:6F;FTTH;MULTI;2013/11/05 13:38:30
You were actually fairly close to this, but were making some silly little mistakes.
In your code, you either aren't showing everything or you aren't using:
use strict;
use warnings;
Perl experts use both to make sure they don't make silly mistakes; beginners should do so too. It would have pointed out that #wynik was not declared with my and was assigned to but not used, for example. You could have meant to write #result = split...;. You were not saving the correct data; you were not writing out the information from the $iptv{$mac} that you needed to.

How do I change values with perl and regex/sed inside a file?

I'm pretty sure I am doing something stupid and I apologize for this ahead of time. I have looked at the one-liners that were suggested elsewhere on similar searches and I like the idea of them, I'm just not sure how to apply because it's not a direct swap. And if the answer is that this can't be done, then that is fine and I will script around that.
The problem: I have log files I need to send through a parser that requires the dates to be in YYYY-MM-DD. The files can be saved this way; however, some people prefer them in YYYY/MM/DD for their own viewing and send those to me. I can modify one or two dates with sed and this works beautifully; however, when there are 2-3+ years in the files, it would be nice not to have to do it manually for each date.
My code (I have left the debugging commands in place):
use strict;
use File::Copy;
use Getopt::Std;
my %ARGS = ();
getopts('f:v', \%ARGS);
my $file = $ARGS{f};
&main();
sub main($)
{
open (FIN, "<$file") || die ("Cannot open file");
print "you opened the file\n";
while (<FIN>) {
my $line = $_;
if ($line =~ /(\d*)\/(\d*)\/(\d*) /i) {
#print "you are in the if";
my $year = $1;
my $month = $2;
my $day = $3;
print $line;
print "\nyou have year $1\n";
print "you have month $2\n";
print "you have day $3\n";
s/'($1\/$2\/$3)/$1-$2-$3'/;
}
}
close FIN;
}
I can see that the regex is getting the right values into my variables but the original line is not being replaced in the file.
Questions:
1) Should this be possible to do within the same file or do I need to output it to a different file? Looking at other answers, same file should be fine.
2) Does the file need to be opened in another way or somehow set to be written to rather than merely running the replace command like I do with sed? <--I am afraid that the failure may be in here somewhere simple that I am overlooking.
Thanks!
You never write to the file. With sed, you'd use -i, and you can do exactly the same in Perl.
perl -i -pe's{(\d{4})/(\d{2})/(\d{2})}{$1-$2-$3}g' file
Or with a backup:
perl -i~ -pe's{(\d{4})/(\d{2})/(\d{2})}{$1-$2-$3}g' file
That's equivalent to
local $^I = ''; # Or for the second: local $^I = '~';
while (<>) {
s{(\d{4})/(\d{2})/(\d{2})}{$1-$2-$3}g;
print;
}
If you didn't want to rely on $^I, you'd have to replicate its behaviour.
for my $qfn (#ARGV) {
open($fh_in, '<', $qfn)
or do { warn("Can't open $ARGV: $!\n"); next; };
unlink($qfn)
or do { warn("Can't overwrite $ARGV: $!\n"); next; };
open(my $fh_out, '>', $qfn) {
or do { warn("Can't create $ARGV: $!\n"); next; };
while (<$fh_in>) {
s{(\d{4})/(\d{2})/(\d{2})}{$1-$2-$3}g;
print $fh_out $_;
}
}
perl -pi.bak -e 's|(\d{4})/(\d\d)/(\d\d)|$1-$2-$3|g;' input
Replace input with your log file name. A backup file input.bak will be created in case you ever need the original data.

How do I copy a CSV file, but skip the first line?

I want to write a script that takes a CSV file, deletes its first row and creates a new output csv file.
This is my code:
use Text::CSV_XS;
use strict;
use warnings;
my $csv = Text::CSV_XS->new({sep_char => ','});
my $file = $ARGV[0];
open(my $data, '<', $file) or die "Could not open '$file'\n";
my $csvout = Text::CSV_XS->new({binary => 1, eol => $/});
open my $OUTPUT, '>', "file.csv" or die "Can't able to open file.csv\n";
my $tmp = 0;
while (my $line = <$data>) {
# if ($tmp==0)
# {
# $tmp=1;
# next;
# }
chomp $line;
if ($csv->parse($line)) {
my #fields = $csv->fields();
$csvout->print($OUTPUT, \#fields);
} else {
warn "Line could not be parsed: $line\n";
}
}
On the perl command line I write: c:\test.pl csv.csv and it doesn't create the file.csv output, but when I double click the script it creates a blank CSV file. What am I doing wrong?
Your program isn't ideally written, but I can't tell why it doesn't work if you pass the CSV file on the command line as you have described. Do you get the errors Could not open 'csv.csv' or Can't able to open file.csv? If not then the file must be created in your current directory. Perhaps you are looking in the wrong place?
If all you need to do is to drop the first line then there is no need to use a module to process the CSV data - you can handle it as a simple text file.
If the file is specified on the command line, as in c:\test.pl csv.csv, you can read from it without explicitly opening it using the <> operator.
This program reads the lines from the input file and prints them to the output only if the line counter (the $. variable) isn't equal to one).
use strict;
use warnings;
open my $out, '>', 'file.csv' or die $!;
while (my $line = <>) {
print $out $line unless $. == 1;
}
Yhm.. you don't need any modules for this task, since CSV ( comma separated value ) are simply text files - just open file, and iterate over its lines ( write to output all lines except particular number, e.g. first ). Such task ( skip first line ) is so simple, that it would be probably better to do it with command line one-liner than a dedicated script.
quick search - see e.g. this link for an example, there are numerous tutorials about perl input/output operations
http://learn.perl.org/examples/read_write_file.html
PS. Perl scripts ( programs ) usually are not "compiled" into binary file - they are of course "compiled", but, uhm, on the fly - that's why /usr/bin/perl is called rather "interpreter" than "compiler" like gcc or g++. I guess what you're looking for is some editor with syntax highlighting and other development goods - you probably could try Eclipse with perl plugin for that ( cross platform ).
http://www.eclipse.org/downloads/
http://www.epic-ide.org/download.php/
this
user#localhost:~$ cat blabla.csv | perl -ne 'print $_ if $x++; '
skips first line ( prints out only if variable incremented AFTER each use of it is more than zero )
You are missing your first (and only) argument due to Windows.
I think this question will help you: #ARGV is empty using ActivePerl in Windows 7