It has been awhile since I have used perl and I am attempting to print out a list of files on a SFTP server.
This is my Perl script -
#! /usr/bin/env perl
use strict;
use warnings;
use feature qw(say);
use autodie;
use Net::SFTP::Foreign;
# US Server Configuration
use constant {
HOST => "Server_Name",
REMOTE_DIR => "\\",
LOCAL_DIR => "sample/local",
PORT => "3235",
USER_NAME => "name",
PASSWORD => "password",
BACKEND => "Net_SSH2",
DEBUG => "0",
};
my $stfp = Net::SFTP::Foreign->new (
HOST,
backend => BACKEND,
timeout => 240,
user => USER_NAME,
password => PASSWORD,
port => PORT,
autodie => 1,
);
#
# List remote directory contents
#
my $remotefiles;
$remotefiles = $stfp->ls(REMOTE_DIR);
#
# Loop through remote files and print each filename
#
foreach ($remotefiles){
my $file = $_;
my $filename = $file->{filename};
if($filename ne "." && $filename ne ".."){
print"the filename is $filename";
}
}
$stfp->disconnect;
I am getting the following error - Not a HASH reference at this line -> my $filename = $file->{filename};
Not sure what the problem is or how to fix it.
Here's the documentation for Net::SFTP::Foreign. This is what it says about the ls() method:
Returns a reference to a list of entries. Every entry is a
reference to a hash with three keys: filename, the name of the
entry; longname, an entry in a "long" listing like ls -l; and
a, a Net::SFTP::Foreign::Attributes object containing file
atime, mtime,permissions and size
So when you call this method:
$remotefiles = $sftp->ls(REMOTE_DIR);
What you get back is a reference to an array. When you try to iterate across that array:
foreach ($remotefiles)
You're not doing what you think you're doing. You're iterating over a list with a single element - which is an array reference. So $file gets an array reference, not the hash reference that you're looking for.
So how to fix it? Well, it's simple enough. You need to dereference $remotefiles which will turn it into an array (rather than an array reference).
# Just add a '#' to dereference the reference and
# get to the underlying array
foreach (#$remotefiles)
Now, each time round the loop, $_ (and, therefore, $file) will contain one of the hash references described in my quote above, and everything will work as you expect.
The important lesson to learn here is that you need to read and understand the documentation for the code that you are using. Or, at least, you need to be better at copying code from the examples. The ls() example clearly uses the dereference syntax.
my $ls = $sftp->ls('/home/foo')
or die "unable to retrieve directory: ".$sftp->error;
print "$_->{filename}\n" for (#$ls);
$filenane is an array reference. Dereference the variable to get at the referant. See perlreftut for the details. It's easy when you get the hang of it, but doing it the first time may be a little painful.
Related
I am doing some archeology of a program from 2005. It's been happily writing backups like this:
use Storable qw(nstore_fd);
my %data =
(
"Games" => \%Games,
"Members" => \%Members,
"Scenarios" => \%Scenarios,
"Passwords" => \%Passwords,
"TourneyNames" => \%TourneyNames
);
nstore_fd(\%data, *BACKUP) or die "Can't write backup: $!";
I need to recover some of this data.
I did this:
use Storable qw(fd_retrieve);
$data = fd_retrieve(*STDIN); # the backup file
print keys %{$data}, "\n";
foreach $thing (keys(%{$data})) {
print "$thing\n";
print scalar keys %{$data->{$thing}}, "\n";
}
and got:
$ perl ~/foo.pl < BobsterBackup2018-7-9-35131.bak
GamesMembersPasswordsScenariosTourneyNames
Games
15556
Members
Can't locate object method "FIRSTKEY" via package "Hash::Case::Lower" at /Users/mgregory/foo.pl line 9.
$
I see that the original source has:
tie %Members, 'Hash::Case::Lower'; # index by non-case-sensitive member-name.
... but I'm at a loss what to make of it!
When you restore objects that are
tied
to packages, you first need to load their classes.
Storable
as well as, for example,
Sereal
will not load them for you. It recreates the objects as they are stored, and Perl expects the packages to exist.
All you need to do is load the module before deserializing your backup.
use Storable qw(fd_retrieve);
use Hash::Case::Lower;
my $data = fd_retrieve(*STDIN); # the backup file
I have read a few post similar to this, but they dont quite have the answer I want. Basically, I am struggling to do pass by reference with this File::find function. The original code is huge. So I take out pieces of the code.
Note that
$node,$ref_p, $ref_alphaSUM ,$ref_rank, $ref_sorted_alpha, $output_file
are all variables or references of global variables. I need to recursively traverse through the directories and apply the &eachFile subroutine to each file, which will update my variables stated above. The "&whichFiles" subroutine just sorts the filenames.
First problem: Since not all files in the directories contains the string nfcapd, i have added the line if(/^nfcapd./) to check the if the name of the file contains this string. But since I am also passing those variables above to eachFile, the "$_" cannot be used now.
Thus, I believe This is why I kept getting the following error:(please correct me if I am wrong):
Use of uninitialized value $File::Find::name in pattern match (m//) at
./pvalues.pl line 178, <PRE> line 65184 (#1)
Line 178 is the line where if(/^nfcapd./) appears in the code below.
This leads to 2nd problem: How to do pass by reference within find function and at the same time preserving a variable for the name of file such that I can still check if the "&eachFile" is being applied to the correct files?
#!/usr/bin/perl
use strict;
use warnings;
use File::Find;
find ({wanted => \&eachFile($node,$ref_p, $ref_alphaSUM ,$ref_rank, $ref_sorted_alpha, $output_file), preprocess => \&whichFiles},$flow_data_path);
sub eachFile {
my ($node , $ref_p, $ref_alphaSUM ,$ref_rank, $ref_sorted_alpha , $output_file) = #_;
if(/^nfcapd\./){
#do something
}
}
To pass variables to your wanted sub, you need a closure:
sub eachFile {
my ($node , $ref_p, $ref_alphaSUM ,$ref_rank, $ref_sorted_alpha , $output_file) = #_;
if(/^nfcapd\./){
#do something
}
}
my $wanted = sub { eachFile($node,$ref_p, $ref_alphaSUM ,$ref_rank, $ref_sorted_alpha, $output_file) };
find({wanted => $wanted, ...});
I wrote this script but I'm not sure if it is correct.
What I want to do is process a JSON file by reading its content, decoding it, and looping through each item as $item. The contents from a certain URL with the ID defined as $items[$i]['paper_item_id'] are saved with that ID into the defined destination.
But the code doesn't seem to function. I'm not sure on where I went wrong but any help or tips to improve the code and make it work would be good.
I'm not asking you to do the job, just need help seeing on where I went wrong and correct it for me.
The script should basically decode the JSON and then download the swf files from a certain directory URL to a directory on my PC using the IDs.
This is the code
use LWP::Simple;
$items = 'paper_items.json';
my $s = $items or die;
$dcode = decode_json($items);
for ($i = 0 ; $i < $count ($items) ; $i++) {
use File::Copy;
$destination = "paper/";
copy(
"http://media1.clubpenguin.com/play/v2/content/global/clothing/paper/"
. $items[$i]['paper_item_id'] . ".swf",
$destination . $items[$i]['paper_item_id'] . ".swf"
);
The program can be broken down into three steps:
Fetch the JSON source.
Parse the JSON.
Iterate over decoded data structure. We expect an array of hashes. Mirror files denoted by the paper_item_id to the working directory.
We will use LWP::Simple functions here.
Our script has the following header:
#!/usr/bin/perl
use strict; # disallow bad constructs
use warnings; # warn about possible bugs
use LWP::Simple;
use JSON;
Fetching the JSON
my $json_source = get "http://media1.clubpenguin.com/play/en/web_service/game_configs/paper_items.json";
die "Can't access the JSON source" unless defined $json_source;
That was easy: we dispatch a get request on that URL. If the output is undefined, we throw a fatal exception.
Parsing the JSON
my $json = decode_json $json_source;
That was easy; we expect the $json_source to be an UTF-8 encoded binary string.
If we want to inspect what is inside that data structure, we can print it out like
use Data::Dumper; print Dumper $json;
or
use Data::Dump; dd $json;
If everything works as expected, this should give a screenfull of an array of hashes.
Iterating
The $json is an array reference, so we'll loop over all items:
my $local_path = "paper";
my $server_path = "http://media1.clubpenguin.com/play/v2/content/global/clothing/paper";
for my $item (#$json) {
my $filename = "$item->{paper_item_id}.swf";
my $response = mirror "$server_path/$filename" => "$local_path/$filename";
warn "mirror failed for $filename with $response" unless $response == 200;
}
Perl has a concept of references, which is similar to pointers. Because data structures like hashes or arrays can only contain scalars, other arrays or hashes are only referenced. Given an array reference, we can access the array like #$reference or #{ $reference }.
To access an entry, the subscript operator [...] for arrays or {...} for hashes is seperated by the dereference operator ->.
Thus, given %hash and $hashref to the same hash,
my %hash = (key => "a", otherkey => "b");
my $hashref = \%hash;
then $hashref->{key} eq $hash{key} holds.
Therefore, we loop over the items in #$json. All of these items are hash references, therefore we use $item->{$key}, not $hash{key} syntax.
What you are trying to do is to download the Shockwave Flash resources from Disney's Club Penguin game site.
I cannot imagine Disney would be too happy about this, and the site's terms of use say this under "Use of Content" ("DIMG" is Disney Interactive Media Group)
Except as we specifically agree in writing, no Content from any DIMG Site may be used, reproduced, transmitted, distributed or otherwise exploited in any way other than as part of the DIMG Site ...
Code is untested.
use File::Slurp qw(read_file);
use JSON qw(decode_json);
use LWP::Simple qw(mirror);
for my $item (#{ decode_json read_file 'paper_items.json' }) {
my $id = $item->{paper_item_id};
mirror "http://media1.clubpenguin.com/play/v2/content/global/clothing/paper/$id.swf", "paper/$id.swf";
}
I'm using Perl 5.12 on Mac 10.5.7. I have a JAR file, that I wish to unzip, and then process a file matching a file pattern. I'm having trouble figuring out how to iterate over the results of unzipping. I have ...
### build an Archive::Extract object ###
my $ae = Archive::Extract->new( archive => $dbJar );
### what if something went wrong?
my $ok = $ae->extract or die $ae->error;
### files from the archive ###
my #files = $ae->files;
for (#files) {
print "file : $_\n";
But there is only one thing returned from the loop -- "ARRAY(0x7fd5c22f7cd8)". I know there are lots of files in the archive, so I'm curious what I"m doing wrong here. - Dave
$ae->files returns an array reference, not an array. Try this:
my $files = $ae->files;
for(#$files) {
print "file : $_\n";
}
From the Perldoc of Archive::Extract:
$ae->files
This is an array ref holding all the paths from the archive. See extract() for details.
It's quite common for methods to return not arrays and hashes, but references to them. It's also quite common for methods to take references to arrays and hashes as arguments. This is because less data has to be passed back and forth between the method and your call.
You can do this:
for my $file ( #{ $ae->files } ) {
print "$file\n";
}
The #{...} dereferences the reference and makes it into a simple array. And yes, you can put method calls that return an array reference in the #{...} like I did.
As already mentioned, a very helpful package is Data::Dumper. This can take a reference and show you the data structure contained therein. It also will tell you if that data structure represents a blessed object which might be a clue that there is a method you can use to pull out the data.
For example, imagine an object called Company. One of the methods might be $company->Employees which returns an array reference to Employee objects. You might not realize this and discover that you get something like ARRAY{0x7fd5c22f7cd8) returned, pushing this through Data::Dumper might help you see the structure:
use Data::Dumper;
use Feature qw(say);
use Company;
[...]
#employee_list = $company->Employees;
# say join "\n", #employee_list; #Doesn't work.
say Dumper #employee_list;
This might print:
$VAR = [
{
FIRST => 'Marion',
LAST => 'Librarian',
POSITION => 'Yes Man',
} Employee,
{
FIRST => 'Charlie',
LAST => 'australopithecus`,
POSITION => 'Cheese Cube Eater'
} Employee,
]
Not only do you see this is an array of hash references, but these are also objects Employee too. Thus, you should use some methods from the Employee class to parse the information you want.
use Feature qw(say);
use Company;
use Employee;
[...]
for my $employee ( #{ $company->Employees } ) {
say "First Name: " . $employee->First;
say "Last Name: " . $employee->Last;
say "Position: " . $employee->Position;
print "\n";
}
Try doing this :
use Data::Dumper;
print Dumper #files;
That way, you will see the content of #files.
If you don't know how to process your data structure, paste the output of Dumper here
Fooling around more with the Perl Plucene module and, having created my index, I am now trying to search it and return results.
My code to create the index is here...chances are you can skip this and read on:
#usr/bin/perl
use Plucene::Document;
use Plucene::Document::Field;
use Plucene::Index::Writer;
use Plucene::Analysis::SimpleAnalyzer;
use Plucene::Search::HitCollector;
use Plucene::Search::IndexSearcher;
use Plucene::QueryParser;
use Try::Tiny;
my $content = $ARGV[0];
my $doc = Plucene::Document->new;
my $i=0;
$doc->add(Plucene::Document::Field->Text(content => $content));
my $analyzer = Plucene::Analysis::SimpleAnalyzer->new();
if (!(-d "solutions" )) {
$i = 1;
}
if ($i)
{
my $writer = Plucene::Index::Writer->new("solutions", $analyzer, 1); #Third param is 1 if creating new index, 0 if adding to existing
$writer->add_document($doc);
my $doc_count = $writer->doc_count;
undef $writer; # close
}
else
{
my $writer = Plucene::Index::Writer->new("solutions", $analyzer, 0);
$writer->add_document($doc);
my $doc_count = $writer->doc_count;
undef $writer; # close
}
It creates a folder called "solutions" and various files to it...I'm assuming indexed files for the doc I created. Now I'd like to search my index...but I'm not coming up with anything. Here is my attempt, guided by the Plucene::Simple examples of CPAN. This is after I ran the above with the param "lol" from the command line.
#usr/bin/perl
use Plucene::Simple;
my $plucy = Plucene::Simple->open("solutions");
my #ids = $plucy->search("content : lol");
foreach(#ids)
{
print $_;
}
Nothing is printed, sadly )-=. I feel like querying the index should be simple, but perhaps my own stupidity is limiting my ability to do this.
Three things I discovered in time:
Plucene is a grossly inefficient proof-of-concept and the Java implementation of Lucene is BY FAR the way to go if you are going to use this tool. Here is some proof: http://www.kinosearch.com/kinosearch/benchmarks.html
Lucy is a superior choice that does the same thing and has more documentation and community (as per the comment on the question).
How to do what I asked in this problem.
I will share two scripts - one to import a file into a new Plucene index and one to search through that index and retrieve it. A truly working example of Plucene...can't really find it easily on the Internet. Also, I had tremendous trouble CPAN-ing these modules...so I ended up going to the CPAN site (just Google), getting the tar's and putting them in my Perl lib (I'm on Strawberry Perl, Windows 7) myself, however haphazard. Then I would try to run them and CPAN all the dependencies that it cried for. This is a sloppy way to do things...but it's how I did them and now it works.
#usr/bin/perl
use strict;
use warnings;
use Plucene::Simple;
my $content_1 = $ARGV[0];
my $content_2 = $ARGV[1];
my %documents;
%documents = (
"".$content_2 => {
content => $content_1
}
);
print $content_1;
my $index = Plucene::Simple->open( "solutions" );
for my $id (keys %documents)
{
$index->add($id => $documents{$id});
}
$index->optimize;
So what does this do...you call the script with two command line arguments of your choosing - it creates a key-value pair of the form "second argument" => "first argument". Think of this like the XMLs in the tutorial at the apache site (http://lucene.apache.org/solr/api/doc-files/tutorial.html). The second argument is the field name.
Anywho, this will make a folder in the directory the script was run in - in that folder will be files made by lucene - THIS IS YOUR INDEX!! All we need to do now is search that index using the power of Lucene, something made easy by Plucene. The script is the following:
#usr/bin/perl
use strict;
use warnings;
use Plucene::Simple;
my $content_1 = $ARGV[0];
my $index = Plucene::Simple->open( "solutions" );
my (#ids, $error);
my $query = $content_1;
#ids = $index->search($query);
foreach(#ids)
{
print $_."---seperator---";
}
You run this script by calling it from the command line with ONE argument - for example's sake let it be the same first argument as you called the previous script. If you do that you will see that it prints your second argument from the example before! So you have retrieved that value! And given that you have other key-value pairs with the same value, this will print those too! With "---seperator---" between them!