Download multiple files from SFTP using perl Net::SFTP::Foreign - perl

I am using the below code to download files from SFTP and I am getting error message after one file download. Then I am rerunning the program and then it downloads only one file again and when trying to download the next file in the array it get failed. I couldn't understand the reasons for this failure and kindly review the below code and share opinions.
#!/usr/bin/perl
use strict;
use Net::SFTP::Foreign::Constants qw(:error :status);
use Net::SFTP::Foreign;
use Net::SFTP::Foreign::Attributes;
my $SFTP_CON=Net::SFTP::Foreign->new("sftpserver.test.com",user=>"testuser",password=>"pass");
my #SFTP_Files=('/test/1.txt','/test/2.txt','/test/3.txt');
my $root_dir='/test/';
$local_dir='/u/test/';
foreach my $file_to_dwn(#SFTP_Files){
my $file_to=$file_to_dwn;
$file_to=~s/$root_dir/$local_dir/igs;
unless($SFTP_CON->get("$file_to_dwn","$file_to")){
if (($SFTP_CON->error == SFTP_ERR_REMOTE_STAT_FAILED or
$SFTP_CON->error == SFTP_ERR_REMOTE_OPEN_FAILED) and
$SFTP_CON->status == SSH2_FX_NO_SUCH_FILE) {
print "Remote file does not exist!";
exit;
}
else {
print "Transfer Failed";
exit;
}
print $SFTP_CON->error."\n";
exit;
}
}
I am getting "Transfer Failed", when the program tries to download the one after another (second file download).
Please do needful.

Related

Read and compare images in folder using Perl

I am trying to read and compare large number of image files in a folder. But I am not able to read the image file.
Code snippet :
use strict;
use warnings;
use Image::Compare;
opendir my $ref_dir, "./" or die "Cannot open directory: $!";
my #ref_files = readdir $ref_dir;
closedir $ref_dir;
print $#ref_files;
my($cmp) = Image::Compare->new();
$cmp->set_image1(
img => './'.$ref_files[0], #one instance - reading first file in the folder
type => 'bmp',
);
The code is throwing the following error - "Unable to read image data from file './.': 'Could not open ./.: Permission denied' at C:/Perl64/site/lib/Image/Compare.pm line 162. "
The code works fine if I provide the file name directly - img => './image.bmp'. So it can't be a permission issue.
The error message seems pretty clear.
Unable to read image data from file './.': 'Could not open ./.: Permission denied
The first file you're getting back from opendir() is the current directory (.) - and it's no surprise that Image::Compare can't get the image data that it wants from that file!
Perhaps you should add a filter to only return the files you're interested in.
my #ref_files = grep { -f } readdir $ref_dir;

perl script to access sec edgar master files returns file not found when file exists on ftp server

It would be great if someone can help. I really am stuck.
I am downloading the master files from SEC edgar and I got the script from—http://brage.bibsys.no/bi/bitstream/URN:NBN:no-bibsys_brage_38213/1/Norli_SRFE_2012.pdf (page 14..published now)
I get the error 404 master.gz not found
While debugging i made it paste the url and when i use the same in browser I can download the file. It is parsing the url correctly till QTR1 but after that it is not able to find the file when it actually exists ..please help.
1) for debugging reasons now I changed the code to 1995 (but later plan to add years 1995 to 2012)
2) It did not work for any file. When I said QTR1 abovr - I meant that the same code without the file name (just for testing ) -- ....full-index/1995/QTR1/ (without the file name) returns a status code OK but ...ftp.sec.gov/edgar/full-index/1995/QTR1/master.gz returns 404 file not found error. It does not work for any quarter.
I wasted so much time on this seemingly simple thing which is supposed to work but it is just not working.... could you copy past this and run..is it working for you?
The code below gets the master files from QTR folders. Pasting my code ::
—————-
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->timeout(250);
$ua->env_proxy;
for($year=1995; $year<1996; $year=$year+1){
for($i=1; $i<5; $i=$i+1){
$quarter = “QTR” . $i;
$filegrag = “ftp://ftp.sec.gov/edgar/full-index/” . $year . “/” . $quarter . “/master.gz”;
print $filegrag;
# This command gets the file from EDGAR
my $response = $ua->get($filegrag);
print $response;
print $response->status_line;
# Now just pipe the output to a file named appropriately
$filename = $year . $quarter . “master”;
open(MYOUTFILE, “>” . $filename);
if ($response->is_success) {
print MYOUTFILE $response->decoded_content;
}
else {
die $response->status_line;
}
close(MYOUTFILE);
}
}
I realized that there were some firewall issues that were causing the problem I had. Now things are fine.

Open2 Api failure while installing Mead software

I am running into some issues trying to install a software called MEAD . I would appreciate if someone could have alook .
I get the following error while installing
/mead/bin # ./mead.pl GA3
Using system rc-file: /home/karosh/mead/bin/../.meadrc
Warning: Can't find user rc-file
Cluster: /home/karosh/mead/bin/../data/GA3/GA3.cluster
open2: exec of /home/karosh/mead/bin/driver.pl failed at ./mead.pl line 230
THe mead software is not written by me so I have not changed any of the perl scrips . I line 230 in the driver.pl file is
sub run_mead {
my %options = #_;
my $reader = FileHandle->new();
my $writer = FileHandle->new();
unless ( open2($reader, $writer, "$FindBin::Bin/driver.pl") ) {
die "Unable to run MEAD.\n";
}
...
...
}
Does this error mean that open2 was not found . The mead folks have put the following line in the file:
use strict;
use File::Spec;
use FileHandle;
use IPC::Open2;
Or does it mean that i need to install the rpm that contains the API . I see that this API is a part of the core perl bundle http://perldoc.perl.org/IPC/Open2.html. So why was it not installed ? Do i need to install perl again .
Someone has earlier faced this problem - http://www.summarization.com/~radev/mead/email/0160.html but the solution is not working for me . I find no Perl files with the incorrect perl directives . The mead team has been dissolved and there is no one to ask questions but I need to use this software.
I think if some one can explain me the meaning of the error than I can do deeper. Anyone?
It probably means that .../driver.pl doesn't have execute permission. Change the file permissions or call it like
open2($reader, $writer, "perl $FindBin::Bin/driver.pl")
open2($reader, $writer, "$^X $FindBin::Bin/driver.pl")

Error reason from Perl Net::FTP

I'm using Net::FTP to transfer files up to a mainframe and I'm testing failure conditions.
My code is basically along the following lines:
my $ftp = Net::FTP->new ("mainframe.com", Timeout => 20);
if (! $ftp) {
logMessage ("Could not connect to host: $!");
return;
}
if (! $ftp->login ("paxdiablo", "demigodemporeroftheuniverse")) {
logMessage ("Could not log in to host: $!");
$ftp->quit ();
return;
}
if (! $ftp->put ("myfile.txt", "'CANT.WRITE.TO.THIS'")) {
logMessage ("Could not put file: $!");
$ftp->quit ();
return;
}
I know I can't create the data set CANT.WRITE.TO.THIS since I don't have the required permissions but, when I try, the only message I see is:
Could not put file:
There is no indication in $! as to what the problem was. I've looked in the Net::FTP doco and all it says is:
put ( LOCAL_FILE [, REMOTE_FILE ] )Put a file on the remote server. LOCAL_FILE may be a name or a filehandle. If LOCAL_FILE is a filehandle then REMOTE_FILE must be specified. If REMOTE_FILE is not specified then the file will be stored in the current directory with the same leafname as LOCAL_FILE.Returns REMOTE_FILE or the generated remote filename if REMOTE_FILE is not given.
I also cannot find anything there about retrieving the specific error (like $ftp->getLastError() or something similar).
How can I indicate to the user why the transfer failed?
On an earlier iteration, I resorted to putting the file, then getting it again and checking contents locally. I'd really rather not have to inflict such kludgy code on people again.
From Net::FTP:
$ftp = Net::FTP->new("some.host.name", Debug => 0)
or die "Cannot connect to some.host.name: $#"
Note $#.
$ftp->cwd("/pub")
or die "Cannot change working directory ", $ftp->message;
Note $ftp->message

Why can't I connect to my CAS server with Perl's AuthCAS?

I'm attempting to use an existing CAS server to authenticate login for a Perl CGI web script and am using the AuthCAS Perl module (v 1.3.1). I can connect to the CAS server to get the service ticket but when I try to connect to validate the ticket my script returns with the following error from the IO::Socket::SSL module:
500 Can't connect to [CAS Server]:443 (Bad hostname '[CAS Server]')
([CAS Server] substituted for real server name)
Symptoms/Tests:
If I type the generated URL for the authentication into the web browser's location bar it returns just fine with the expected XML snippet. So it is not a bad host name.
If I generate a script without using the AuthCAS module but using the IO::Socket::SSL module directly to query the CAS server for validation on the generated service ticket the Perl script will run fine from the command line but not in the browser.
If I add the AuthCAS module into the script in item 2, the script no longer works on the command line and still doesn't work in the browser.
Here is the bare-bones script that produces the error:
#!/usr/bin/perl
use strict;
use warnings;
use CGI;
use AuthCAS;
use CGI::Carp qw( fatalsToBrowser );
my $id = $ENV{QUERY_STRING};
my $q = new CGI;
my $target = "http://localhost/cgi-bin/testCAS.cgi";
my $cas = new AuthCAS(casUrl => 'https://cas_server/cas');
if ($id eq ""){
my $login_url = $cas->getServerLoginURL($target);
printf "Location: $login_url\n\n";
exit 0;
} else {
print $q->header();
print "CAS TEST<br>\n";
## When coming back from the CAS server a ticket is provided in the QUERY_STRING
print "QUERY_STRING = " . $id . "</br>\n";
## $ST should contain the received Service Ticket
my $ST = $q->param('ticket');
my $user = $cas->validateST($target, $ST); #### This is what fails
printf "Error: %s\n", &AuthCAS::get_errors() unless (defined $user);
}
Any ideas on where the conflict might be?
The error is coming from the line directly above the snippet Cebjyre quoted namely
$ssl_socket = new IO::Socket::SSL(%ssl_options);
namely the socket creation. All of the input parameters are correct. I had edited the module to put in debug statements and print out all the parameters just before that call and they are all fine. Looks like I'm going to have to dive deeper into the IO::Socket::SSL module.
As usually happens when I post questions like this, I found the problem. It turns out the Crypt::SSLeay module was not installed or at least not up to date. Of course the error messages didn't give me any clues. Updating it and all the problems go away and things are working fine now.
Well, from the module source it looks like that IO::Socket error is coming from get_https2
[...]
unless ($ssl_socket) {
$errors = sprintf "error %s unable to connect https://%s:%s/\n",&IO::Socket::SSL::errstr,$host,$port;
return undef;
}
[...]
which is called by callCAS, which is called by validateST.
One option is to temporarily edit the module file to put some debug statements in if you can, but if I had to guess, I'd say the casUrl you are supplying isn't matching up to the _parse_url regex properly - maybe you have three slashes after the https?