500 Can't connect to example.com:443 (LWP::Protocol::https::Socket: SSL connect attempt failed with unknown error SSL wants a read first) - perl

I want to export issue from the GitLab CE using this script.
I ran the following commands on macOS 10.10.5:
sudo /usr/bin/perl -MCPAN -e'install Text::CSV_XS'
sudo cpan Mozilla::CA
sudo cpan install LWP
sudo cpan LWP::Protocol::https
However, I still get this error:
500 Can't connect to gitlab.lrz.de:443 (LWP::Protocol::https::Socket: SSL connect attempt failed with unknown error SSL wants a read first) /Users/kiaora/Downloads/get-all-project-issues.pl line 41.`
This is literally my first experience with perl. Any help is greatly appreciated.
I only changed these lines:
my $PROJECT_ID="myProjectID"; # numeric project id, can be found in project -> general settings
my $GITLAB_API_PRIVATE_TOKEN='myToken(scope read_repository)'; # obtained from https://gitlab.lrz.de/profile/personal_access_tokens
my $baseurl = "https://gitlab.lrz.de/"; # change if using a private install
EDIT:
#!/usr/bin/perl -w
use strict;
use LWP::UserAgent;
use Text::CSV_XS qw( csv );
use JSON::PP qw(decode_json);
# Uncomment these for debugging
# use LWP::ConsoleLogger::Easy qw( debug_ua );
# use Data::Dumper;
my $PROJECT_ID="--my-project-id--"; # numeric project id, can be found in project -> general settings
my $GITLAB_API_PRIVATE_TOKEN='--my-api-private-token--'; # obtained from https://gitlab.com/profile/personal_access_tokens
my $baseurl = "https://gitlab.lrz.de/"; # change if using a private install
$baseurl .= "api/v4/";
my $issuesurl = $baseurl."projects/".$PROJECT_ID."/issues";
my #issues = ();
my $page = 1;
my $totalpages;
do
{
my %query_hash = (
'per_page' => 100,
'page' => $page
);
print "Fetching page $page".(defined($totalpages)?" (of $totalpages)":"")."\n";
my $ua = LWP::UserAgent->new();
# debug_ua($ua);
$ua->default_header('PRIVATE-TOKEN' => $GITLAB_API_PRIVATE_TOKEN);
my $uri = URI->new($issuesurl);
$uri->query_form(%query_hash);
my $resp = $ua->get($uri);
if (!$resp->is_success) {
die $resp->status_line;
}
$totalpages = int($resp->header("X-Total-Pages"));
my $resptext;
$resptext = $resp->decoded_content;
my $issuedata = decode_json($resptext);
push(#issues, #{$issuedata});
}
while ($page++ < $totalpages);
my $outputfname = "issues.csv";
my $csv = Text::CSV_XS->new ({ binary => 1, eol => $/ });
open my $fh, ">", $outputfname or die "$outputfname: $!";
my #headings = [
"URL",
"Milestone",
"Author",
"Title",
"Description",
"State",
"Assignees",
"Labels",
"Created At",
"Updated At",
"Closed At",
"Due date",
"Confidental",
"Weight",
"Locked",
"Time estimate",
"Time spent",
"Human Time estimate",
"Human Time spent",
];
$csv->print ($fh, #headings) or $csv->error_diag;
foreach my $i (#issues)
{
# print Dumper([$i])."\n";
my #values = [
$i->{'web_url'},
$i->{'milestone'}->{'title'},
$i->{'author'}->{'username'},
$i->{'title'},
$i->{'description'},
$i->{'state'},
join(',', map {$_->{'username'}} #{$i->{'assignees'}}),
join(',', #{$i->{'labels'}}),
$i->{'created_at'},
$i->{'updated_at'},
$i->{'closed_at'},
$i->{'due_date'},
$i->{'confidential'},
$i->{'weight'},
$i->{'discussion_locked'},
$i->{'time_stats'}->{'time_estimate'},
$i->{'time_stats'}->{'total_time_spent'},
$i->{'time_stats'}->{'human_time_estimate'},
$i->{'time_stats'}->{'human_total_time_spent'},
$i->{'closed_by'}->{'username'},
];
$csv->print ($fh, #values) or $csv->error_diag;
}
close $fh or die "$outputfname: $!";
print "Issues saved to $outputfname\n";

Related

Perl script as a router between icecast2 and VLC client

I wrote a Perl script to act as router between an icecast2 server and a VLC client. I start the script via
plackup --listen :8000 testStreamingServer.pl --debug
The problem is, I always get to the print "Starting server\n"; line but neither VLC, Firefox or curl can connect to the audio stream. There is no further info on the server side. No debug message, nothing.
Here is the code:
#!/usr/bin/env perl
use strict;
use warnings;
use Data::Dumper;
use LWP::UserAgent;
use Plack::Runner;
use File::Temp qw(tempfile);
my $source_url = 'http://server1.example.com/stream.mp3';
my $redirect_url = 'http://server2.example.com/stream.mp3';
my $ua = LWP::UserAgent->new();
print "Starting server\n";
my $app = sub {
print "Request received\n";
my $response = $ua->get($source_url);
if ($response->is_success) {
print "Source server is online, stream is playable\n";
my ($fh, $filename) = tempfile();
my $content_response = $ua->get($source_url, ':content_file' => $filename);
my $headers = $content_response->headers();
print Dumper $headers;
my $size = -s $filename;
open(my $file, "<", $filename);
my $file_content = do { local $/; <$file> };
close $file;
return [
200,
[ 'Content-Type' => 'audio/mpeg',
'Content-Length' => $size,
'icy-metaint' => 8192,
'icy-name' => 'My Stream',
'icy-description' => 'My stream description',
'icy-genre' => 'variety',
'icy-pub' => 1,
'icy-br' => 128,
'Connection' => 'close' ],
[ $file_content ],
];
} else {
print "Source server is offline, redirecting to backup stream\n";
return [
302,
[ 'Location' => $redirect_url, 'Connection' => 'close' ],
[],
];
}
};
my $runner = Plack::Runner->new;
$runner->parse_options(qw(--listen :8000 --server HTTP::Server::PSGI));
$runner->run($app);
print "Server started\n";
I tried print debugging, line by line. I am clueless right now.

Transfer file from Remote machine to local machine using Net::OpenSSH

I have built a script which should get a file from remote machine to local machine.
#!/usr/bin/perl
use strict;
use warnings;
use Net::OpenSSH;
use Data::Dumper;
my $local_dir = "/LOCAL/DIR/LOCATION/"
print "[LOCAL DIR]-> $local_dir\n";
my $remote_dir = "/REMOTE/DIR/LOCATION/";
print "[REMOTE DIR]-> $remote_dir\n";
my ($host, $user, $password) = ("remote.machine.ip.address", "userid", "password");
my $ssh = Net::OpenSSH->new($host,
user => $user,
password => $passwd,
master_opts => [-o => "StrictHostKeyChecking=no"]
);
$ssh->error and die "Couldn't establish SSH connection: ". $ssh->error;
my #file = $ssh->capture("cd $remote_dir && ls -1tr | grep Report | tail -1");
print "[FILE]:\n".Dumper(\#file);
$ssh->scp_get({glob => 1}, "$remote_dir$file[0]", $local_dir)
or die "scp failed: " . $ssh->error;
undef $ssh;
In the above code its able to print the Dumper value for #file but unable to get the file in local system.
Here is the error it throws at the end:
[FILE]:
$VAR1 = [
'Report_Managable_20200705.csv
'
];
scp: /REMOTE/DIR/LOCATION/Report_Managable_20200705.csv
protocol error: expected control record
scp failed: scp failed: child exited with code 1 at file_get_test.pl line 22.
Can anybody help me to fix this issue. TIA.
The list returned by $ssh->capture() has new lines at the end of each item. Try use chomp #file to remove the newlines.

Perl SNMP trap generator for scale testing?

I've hacked the script below together to let me generate traps to a test server. What I really need is something that will generate traps at a large scale so that I can check my tools on the receiving end to find out where the bottleneck is, such as UDP, Net::SNMP, Perl, etc.
I had hoped this script would let me generate something like 10k events/second but I am sadly mistaken.
Does anyone know if I can do this in Perl or have a suggestion of another way to do it?
#! /usr/bin/perl
use strict;
use warnings;
use Log::Fast;
use FindBin;
use Getopt::Long;
use File::Basename;
use Cwd qw(abs_path);
my $ROOT_DIR = abs_path("$FindBin::Bin/..");
use POSIX qw/strftime/;
use Net::SNMP qw(:ALL);
use Time::HiRes qw( time sleep );
#FIXME - I had to add below for Perl 5.10 users.
# on Perl 5.10, I would get the following when running:
# perl -e"autoflush STDOUT, 1;"
# Can't locate object method "autoflush" via package "IO::Handle" at -e line 1.
use FileHandle;
# Create default logger, will reconfigure it as soon as we read configuration from database
my $log = Log::Fast->global();
my $myname = $0;
$myname =~ s{.*/}{}; # leave just program name without path
# Command line options
my $options = {
debug => 0,
verbose => 0,
logfile => "./$myname.log",
help => 0,
community => "public",
trapsource => "127.0.0.1",
timelimit => 1,
};
sub usage_and_exit {
my ($exit_code) = #_;
print STDERR qq{
This program is used to generate SNMP traps to a specified host at a specified rate
Usage: $myname [-o --option]
-h : this (help) message
-d : debug level (0-5) (0 = disabled [default])
-v : Also print results to STDERR
-l : log file (defaults to local dir
-r : Rate (events/sec)
-ts : host to generate messages FROM
-td : host to generate messages TO
-tl : Run for this many seconds (default 1)
-c : community
Example: $myname -td 192.168.28.29 -r 1 -tl 5 -v
};
exit($exit_code);
}
GetOptions(
'debug|d=i' => \$options->{debug},
'help|h!' => \$options->{help},
'verbose|v!' => \$options->{verbose},
'logfile|l=s' => \$options->{logfile},
'rate|r=i' => \$options->{rate},
'trapsource|ts=s' => \$options->{trapsource},
'trapdest|td=s' => \$options->{trapdest},
'community|c=s' => \$options->{community},
'timelimit|tl=i' => \$options->{timelimit},
) or usage_and_exit(1); # got some invalid options
if ( $options->{help} ) {
usage_and_exit(0);
}
# Reconfigure log to use logfile (as we finally got it from $settings), also
# set proper level and output based on $options{verbose} and $options{debug}
setup_log();
# Finally we are initialized, announce this to the world :-)
$log->INFO("Program initialized successfully");
my $date = strftime "%Y-%m-%d %H:%M:%S", localtime;
# start func
my $period = 1 / $options->{rate};
my $start = time();
my $limit = time() + $options->{timelimit};
my $total = $options->{rate} * $options->{timelimit};
$log->INFO("Generating $options->{rate} trap(s) every second for $options->{timelimit} seconds (1 every $period seconds, $total total events)");
while($start < $limit) {
my $elapsed = time() - $start;
if ($elapsed < $period) {
sleep($period - $elapsed);
my ($session, $error) = Net::SNMP->session(
-hostname => $options->{trapdest},
-community => $options->{community},
-port => SNMP_TRAP_PORT, # Need to use port 162
-version => 'snmpv2c'
);
if (!defined($session)) {
$log->INFO("ERROR: %s.", $error);
exit 1;
}
my $result = $session->snmpv2_trap(
-varbindlist => [
'1.3.6.1.2.1.1.3.0', TIMETICKS, 600,
'1.3.6.1.6.3.1.1.4.1.0', OBJECT_IDENTIFIER, '1.3.6.1.4.1.326',
'1.3.6.1.6.3.18.1.3.0', IPADDRESS, $options->{trapsource}
]
);
if (!defined($result)) {
$log->INFO("ERROR: %s.", $session->error());
} else {
$log->INFO("SNMPv2-Trap-PDU sent from $options->{trapsource} to $options->{trapdest}.");
}
} else {
$start = time();
}
}
#-------------------------------------------
# There should only be subs from here down
#-------------------------------------------
# =================================================================================================
# Helper functions
# =================================================================================================
# commify not used yet
sub commify {
my $text = reverse $_[0];
$text =~ s/(\d\d\d)(?=\d)(?!\d*\.)/$1,/g;
return scalar reverse $text;
}
sub setup_log {
my $log_dir = dirname($options->{logfile});
# Create log dir, and build log path if not provided by command line option
if ( !-d $log_dir ) {
mkdir( $log_dir, 0755 ) or die("mkdir $log_dir: $!");
}
if ( !$options->{logfile} ) {
$options->{logfile} = $log_dir . "/" . basename( $0, '.pl' ) . '.log';
}
my $log_options = {};
# Set up output to file or both file and stderr
if ( $options->{verbose} ) {
# make multiplexer FH sending data both to file and STDERR
open( my $fh, '>>:tee', $options->{logfile}, \*STDERR )
or die("$options->{logfile}: $!");
$fh->autoflush(1);
$log_options->{fh} = $fh;
}
else {
open( my $fh, '>>', $options->{logfile} ) or die("$options->{logfile}: $!");
$log_options->{fh} = $fh;
}
# Setup extra information to put in every log line, depending on debug level
if ( $options->{debug} > 1 ) {
$log_options->{prefix} = "%D %T %S [%L] ";
}
else {
$log_options->{prefix} = "%D %T [%L] ";
}
$log_options->{level} = $options->{debug} > 0 ? 'DEBUG' : 'INFO';
$log->config($log_options);
$SIG{__WARN__} = sub {
my $msg = shift;
$msg =~ s/\n//;
$log->WARN($msg);
};
$log->INFO("Starting logging to $options->{logfile} with pid $$");
}
sub DEBUG {
my ( $level, #log_args ) = #_;
if ( $options->{debug} >= $level ) {
$log->DEBUG(#log_args);
}
}
Perhaps use something like Parallel::ForkManager ? In addition, with specific regard to testing scalability of your SNMP collector, you'll probably be interested in the use case of receiving the traps from many HOSTS, not just a single host sending traps at a high rate. For that, you might want to look at using pssh.
One problem might be the slowness of Net::SNMP in pure-perl - perhaps exectuting snmptest or snmptrap via the shell might be faster ? Worth a try.

Perl get md5 hash of a fetched file

I'm trying to create a hash of a file that i have downloaded using the module File::Fetch. I'm trying to store the downloaded file name and its hash in the database as a record. However, none of the methods of File::Fetch objects provide the downloaded file. Is there any way to download the file into a variable in order to create a hash of it?
Here is a snippet of my code,
eval{
$uri_handle = File::Fetch->new(uri => $nurl) or die "Could'nt create fetch object\n";
$getfile = $uri_handle ->output_file or die "There is no file to be fetched\n";
my $dir_handle = $uri_handle->fetch( to => $dir2 ) or die "Couldn't fetch file: $uri_handle->error\n";
# print "$getfile was successfully saved to $dir\n";
};
if ($#){
print "\n There is no file at $url\n\n";
}else{
my $file_data_handle = DBI->connect($database_connection_string,$database_user,$database_pass) or die "Couldn't open database: $DBI::errstr\n";
my $file_statement_handle = $file_data_handle->prepare('insert into files (filename,filehash,sourceurl,originalurl) VALUES ($getfile,$filehash,$nurl,$url)') or die "Couldn't prepare statement: $DBI::errstr\n";
$file_statement_handle->execute() or die "Couldn't execute statement: $DBI::errstr\n";
$file_data_handle->disconnect();
}
It writes the result to disk directly. Use LWP instead.
use Digest::MD5 qw( md5_hex );
use LWP::UserAgent qw( );
my $ua = LWP::UserAgent->new();
my $response = $ua->get($url);
die $response->status_line if !$response->is_success;
my $file = $response->decoded_content( charset => 'none' );
my $md5_hex = md5_hex($file);

Perl Hash + File + While

well, the idea is to remove a file a direction with their description and store it in a hash
this is content in file /home/opmeitle/files-pl/bookmarks2
}, {
"date_added": "12989744094664781",
"id": "1721",
"name": "Perl DBI - dbi.perl.org",
"type": "url",
"url": "http://dbi.perl.org/"
}, {
"date_added": "12989744373130384",
"id": "1722",
"name": "DBD::mysql - MySQL driver for the Perl5 Database Interface (DBI) - metacpan.org",
"type": "url",
"url": "https://metacpan.org/module/DBD::mysql"
}, {
now, the code in perl.
use strict;
open(FILE, '/home/opmeitle/files-pl/bookmarks2');
my #lines = <FILE>;
my #list55;
my $count = 1;
my $n = 0;
my %hash=(); #$hash{$lines[$n]}=$lines[$n];
while ($lines[$n]) {
if ($lines[$n] =~ /(http:|https:|name)/) {
if ($lines[$n] =~ s/("|: |,|id|url|name|\n)//g) {
if ($lines[$n] =~ s/^\s+//){
if ($lines[$n] =~ /http:|https/){
$hash{$lines[$n]} = '';
}
else {
$hash{$n} = $lines[$n];
}
}
}
}
$n++;
$count++;
}
close(FILE);
# print hash
my $key;
my $value;
while( ($key,$value) = each %hash){
print "$key = $value\n";
}
result after executing the script.
http://dbi.perl.org/ =
https://metacpan.org/module/DBD::mysql =
3 = Perl DBI - dbi.perl.org
9 = DBD::mysql - MySQL driver for the Perl5 Database Interface (DBI) - metacpan.org
but i need something like this
http://dbi.perl.org/ = Perl DBI - dbi.perl.org
Perl DBI - dbi.perl.org = DBD::mysql - MySQL driver for the Perl5 Database Interface (DBI) - metacpan.org
thanks for you answers.
As #amon hinted, Chrome bookmarks are JSON format, for which there are several good modules on CPAN.
use strict;
use warnings;
use JSON;
my $file = '/home/opmeitle/files-pl/bookmarks2';
open my $fh, '<', $file or die "$file: $!\n";
my $inhash = decode_json(join '', <$fh>);
close $fh;
my %outhash = map traverse($_), values %{ $inhash->{roots} };
sub traverse
{
my $hashref = shift;
if (exists $hashref->{children}) {
return map traverse($_), #{ $hashref->{children} };
} else {
return $hashref->{url} => $hashref->{name};
}
}
Now %outhash has the data you wanted.
EDIT: to help understand what's going on here:
use Data::Dumper;
print Dumper($inhash); # pretty-print the structure returned by decode_json
As others have said, the best thing to do is to load the JSON data into a Perl datastructure. This is easily done using the JSON module. Before we can do this, we need to read in the file. There are two ways to do this. The non-CPAN way:
# always ...
use strict;
use warnings;
my $file = '/home/opmeitle/files-pl/bookmarks2';
my $text = do {
open my $fh, '<', $file or die "Cannot open $file: $!\n";
local $/; #enable slurp
<$fh>;
};
or the CPAN way
# always ...
use strict;
use warnings;
use File::Slurp;
my $text = read_file $file;
Once you have the file read in, then decode
use JSON;
my $data = decode_json $text;
Please post a whole file and a better description of what you want and I would be glad to comment on a more formal way of traversing the datastructure.