We have private MediaWiki installation inside our company. Based on daily builds on our source code, we update the wiki with Perforce labels so that people can use the build that is labeled for streamlined process. We tried to automate this using Perl scripts on a Windows server using MediaWiki::Bot and MediaWiki::API.
use MediaWiki::Bot;
use MediaWiki::API;
my $mw = MediaWiki::API->new();
$mw->{config}->{api_url} = 'http://somewiki/w/index.php/title#feature_List';
# log in to the wiki
$mw->login({
lgname => 'username',
lgpassword => 'password'
|| die $mw->{error}->{code} . ': ' . $mw->{error}->{details};
# get a list of articles in category
my $articles = $mw->list({
action => 'query',
list => 'categorymembers',
cmtitle => 'Category:Perl',
cmlimit => 'max'
}) || die $mw->{error}->{code} . ': ' . $mw->{error}->{details};
# and print the article titles
foreach (#{$articles}) {
print "$_->{title}\n";
}
Output:
2: Failed to decode JSON returned by http://vaporwiki/w/index.php/Executor#Execu
tor_Feature_List
Decoding Error:
malformed JSON string, neither array, object, number, string or atom, at charact
er offset 0 (before "<!DOCTYPE html PUBLI...") at C:/Perl/lib/MediaWiki/API.pm l
ine 398
Returned Data: <whole page data>
The API URL is wrong. Try http://vaporwiki/w/api.php.
Related
I am sending email using Dancer2 via the Dancer2::Plugin::Email package. The main code that I have for this is:
sub sendEmail {
my ($params,$email_address,$template) = #_;
my $text = '';
my $tt = Template->new({
INCLUDE_PATH => config->{views},
INTERPOLATE => 1,
OUTPUT => \$text
}) || die "$Template::ERROR\n";
my $out = $tt->process($template,$params);
my $email = email {
from => XXXXX,
to => $email_address,
subject => XXXXX,
body => $text,
'Content-Type' => 'text/html'
};
}
where I have hidden a couple of the fields. I have gotten the following error:
Route exception: open body: Invalid argument at
/usr/local/share/perl/5.22.1/MIME/Entity.pm line 1878. in
/usr/local/share/perl/5.22.1/Dancer2/Core/App.pm l. 1454
It is not occurring all of the time and I haven't been able to find a consistent piece of code that always fails.
I have set the host parameter of the mail server that I am using in the configuration as explained here: https://metacpan.org/pod/Dancer2::Plugin::Email Simple tests show it works, but I get sporadic errors that I can't track down.
I'm attempting to use a particular web service, and I can successfully perform the upload with the following command:
curl -X POST --header "Transfer-Encoding: chunked" -d #Downloads/file.pdf https://some.webservice/upload
I get back a json response indicate success.
However, I'm unable to figure out how to do the same with WWW::Mechanize.
$mech->post("https://" . $server . "/upload", Content_Type => 'multipart/form-data', Content => [upID => $upid, name => $dlfile, userID => 0, userK => 0, file_0 => [$dlfile]]);
This receives a similar json response with a big fat error message in it.
Do I need to explicitly set the Transfer-Encoding header first? Is there some other trick to it? Google's not shedding much light on this, Perlmonks neither, and the documentation's a little obtuse.
You can do it using HTTP::Request::StreamingUpload
my $starttime = time();
my $req = HTTP::Request::StreamingUpload->new(
POST => $url,
path => $file,
headers => HTTP::Headers->new(
'Transfer-Encoding' => 'chunked'
),
);
my $gen = $req->content;
die unless ref($gen) eq "CODE";
my $total = 0;
$req->content(sub {
my $chunk = &$gen();
$total += length($chunk);
print "\r$total / $size bytes ("
. int($total/$size*100)
. "%) sent, "
. int($total/1000/(time()-$starttime+1))
. " k / sec ";
return $chunk;
});
my $resp = $ua->request($req);
print "\n";
unless ($resp->is_success) {
die "Failed uploading the file: ", $resp->status_line;
}
my $con = $resp->content;
return $con;
Do you really need WWW::Mechanize? It is a subclass of LWP::UserAgent with additional functionality that gives browser-like functionality like filling in and submitting forms, clicking links, a page history with a "back" operation etc. If you don't need all of that then you may as well use LWP::UserAgent directly
Either way, the post method is inherited unchanged from LWP::UserAgent, and it's fine to use it directly as you have done
The way to send a chunked POST is to set the Content to a reference to a subroutine. The subroutine must return the next chunk of data each time it is called, and finally ann empty string or undef when there is no more to send
Is the data supposed to be a JSON string?
It's easiest to write a factory subroutine that returns a closure, like this
sub make_callback {
my ($data) = shift;
sub { substr($data, 0, 512, "") }
}
Then you can call post like this
my $payload = to_json(...);
$mech->post(
"https://$server/upload",
Content_Type => 'multipart/form-data',
Content => make_callback($payload)
);
Please be aware that all of this is untested
I'm pretty new to Slack. I've created a team, and I've added an integration with our SVN repository. I've set up a "post-commit" script and I get a notification in the intended Slack channel. All is good as far as that goes.
I would like to customise the notification in the slack channel, and I'm failing.
The post-commit script is Perl-based; a number of templates are available. I've used one and added a line to include the files changed thus:
my $files = `/usr/bin/svnlook changed -r $ARGV[1] $ARGV[0]`;
my $date = `/usr/bin/svnlook date -r $ARGV[1] $ARGV[0]`;
my $log = `/usr/bin/svnlook log -r $ARGV[1] $ARGV[0]`;
my $who = `/usr/bin/svnlook author -r $ARGV[1] $ARGV[0]`;
chomp $who;
my $payload = {
'revision' => $ARGV[1],
'date' => $date,
'author' => $who,
'log' => $log,
'files' => $files,
};
my $ua = LWP::UserAgent->new;
$ua->timeout(15);
my $req = POST( "https://${opt_domain}/services/hooks/subversion?token=${opt_token}", ['payload' => encode_json($payload)] );
The only message I get in the Slack channel is of the form:
<rev number>:<author>-<log message>
I don't know why I don't get the date and the list of files changed appear in the Slack message. The relevant svnlook commands, when run manually on the server produce the expected results. If I change the relevant bit of the script to:
my $payload = {
'wibble' => $ARGV[1],
'date' => $date,
'author' => $who,
'log' => $log,
'files' => $files,
};
and then manually invoke the hook with
post-commit /var/svn/repo 5966
(for example) I get an output that includes:
payload=%7B%22wibble%22%3A%225966%22
(as I expected/hoped), but then
Response:
HTTP/1.1 500 Server Error
Connection: close
...
X-Cache: Error from cloudfront
X-Frame-Options: SAMEORIGIN
invalid_payload
I was naively expecting the payload to specify the message, so you could craft a message something like:
revision: 5696
date: 2016/03/14 12:00:00
author: ...
log: <log text>
files: myfile.c, myfile.h
In the above example, I was wondering if I'd get
wibble: 5696
So, I have to come to the conclusion that something at the Slack end requires only certain fields in the received JSON and then formats them into the channel message according to some rule. I have found NO clue what fields are acceptable and how one might exercise some control over the formatting. For example, the fact I don't get an error about an invalid payload suggests that the date and files field names are valid, but how do I get them to appear in the Slack message?
If you want to create a custom message you should use Incoming Webhooks in the API. I suspect you are using a Subversion hook, and it probably has a fixed layout of the message.
Post the text you want in the text key in the payload. That way you can format it as you would a normal Slack message.
Came across wanting to do the same thing. What I ended doing was changing the $payload's log value to include $files ... like so:
my $files = `/usr/bin/svnlook changed -r $ARGV[1] $ARGV[0]`;
my $payload = {
'revision' => $ARGV[1],
'url' => $url,
'author' => $who,
'log' => "$log\n$files"
};
(starting with the sample here: https://github.com/tinyspeck/services-examples/blob/master/subversion.pl)
I am new in Perl and also RRDs.
I have tried to implement a simple example, and although it seems that is operating correctly the output is not displayed. The pictures are produced normally but there is no data in the graphs.
I have been following the CPAN documentation for implementation RRD::Simple and theoretically I am doing something wrong. I tried to debug the code and it seems fine, but when it comes to print the graphs there is no data.
#!/usr/bin/perl
use strict;
use warnings;
use RRD::Simple ();
use Data::Dumper;
$| = 1; # Flush the output
my ($rrd, $unixtime, $file);
$file = "perl.txt";
my $path = '/home/os/Desktop/Test_Perl/';
my $period = '3years';
my $rrdfile = 'myfile.rrd';
while (sleep 15) {
open(FH, ">>", $file) || die "Unable to open $file: $!\n";
my $range = 50;
my $minimum = 100;
my $random_number_in = int(rand($range)) + $minimum;
my $random_number_out = int(rand($range)) + $minimum;
my $random_number_sec = int(rand($range)) + $minimum;
# Create an interface object
$rrd = RRD::Simple->new(
file => $rrdfile,
cf => [qw( AVERAGE MIN MAX LAST )],
#default_dstype => "DERIVE",
);
unless (-e $rrdfile) {
# Create a new RRD file with 3 data sources called
# bytesIn, bytesOut and faultsPerSec.
$rrd->create(
$period,
step => 5, # 5 sec interval
bytesIn => "GAUGE",
bytesOut => "GAUGE",
faultsPerSec => "GAUGE"
);
}
# Put some arbitary data values in the RRD file for the same
# 3 data sources called bytesIn, bytesOut and faultsPerSec.
$rrd->update(
bytesIn => $random_number_in,
bytesOut => $random_number_out,
faultsPerSec => $random_number_sec
);
print FH "This is the bytes_in: $random_number_in\n";
print FH "This is the bytes_out: $random_number_out\n";
print FH "This is the bytes_sec: $random_number_sec\n";
# Generate graphs:
# /home/os/Desktop/Test_Perl/myfile-hourly.png, /home/os/Desktop/Test_Perl/myfile-daily.png
# /home/os/Desktop/Test_Perl/myfile-weekly.png, /home/os/Desktop/Test_Perl/myfile-monthly.png
my %rtn = $rrd->graph(
$rrdfile,
destination => $path,
basename => "my_graph",
timestamp => "both", # graph, rrd, both or none
periods => [qw(hour day week month)], # omit to generate all graphs
sources => [qw(bytesIn bytesOut faultsPerSec)],
source_colors => [qw(ff0000 aa3333 000000)],
source_labels => [("Bytes In", "Bytes Out", "Faults Per Second")],
source_drawtypes => [qw(LINE1 AREA LINE)],
line_thickness => 2,
extended_legend => 1,
title => "Network Interface eth0",
vertical_label => "Bytes/Faults",
width => 800,
height => 500,
interlaced => "", # If images are interlaced they become visible to browsers more quickly
);
printf("Created %s\n", join(", ", map { $rtn{$_}->[0] } keys %rtn));
# Return information about an RRD file
my $info = $rrd->info($rrdfile); # This method will return a complex data structure containing details about the RRD file, including RRA and data source information.
print Data::Dumper::Dumper($info);
my #sources = $rrd->sources($rrdfile);
my $seconds = $rrd->retention_period($rrdfile); # This method will return the maximum period of time (in seconds) that the RRD file will store data for.
# Get unixtime of when RRD file was last updated
$unixtime = $rrd->last($rrdfile);
print FH "myfile.rrd was last updated at " . scalar(localtime($unixtime)) . "\n";
# Get list of data source names from an RRD file
my #dsnames = $rrd->sources;
print "Available data sources: " . join(", ", #dsnames) . "\n";
my $heartbeat_In = $rrd->heartbeat($rrdfile, "bytesIn");
my $heartbeat_Out = $rrd->heartbeat($rrdfile, "bytesOut");
my $heartbeat_sec = $rrd->heartbeat($rrdfile, "faultsPerSec"); # This method will return the current heartbeat of a data source.
printf "This is the heartbeat_in: %s\n", $heartbeat_In;
my #rtn_In = $rrd->heartbeat($rrdfile, "bytesIn", 10);
my #rtn_Out = $rrd->heartbeat($rrdfile, "bytesOut", 10);
my #rtn_sec = $rrd->heartbeat($rrdfile, "faultsPerSec", 10); # This method will set a new heartbeat of a data source.
close(FH);
}
Part of the output:
'myfilerrd' => {
'last_ds' => 'U',
'value' => undef,
'min' => '0',
'max' => undef,
'minimal_heartbeat' => 120,
'index' => 3,
'type' => 'DERIVE',
'unknown_sec' => 15
}
I do not understand why the value is undefined?
After 3-4 days of testing and searching over the Internet for more information I just found the answer to my problem. RRD is a very simple to use tool but very very powerful. I would recommend anybody to use it through Perl especially with RRD::Simple module is very easy.
Answer:
I was adjusting the heart beat of my RRD to 10 sec, while my step (data collection time) is 300 by default. If the user do not specify the step "sampling frequency" by default the system will use 300. In result the graph takes 0 values so there is not output. More information and very nice analysis can be found here HeartBeat
Based on my experimentation, I found that since I am using a while loop inside the create function I have to first give the command:
my $rrd = RRD::Simple->new( file => "myfile.rrd" );
and as a second step I had to kill the process and set the step by entering the command:
my $rrd = RRD::Simple->new(
file => "myfile.rrd",
step => 50 );
Based on my experimentation I found that I had to remove this block of code below had to be added to the file as a second step. First had to make the creation and then add it on my loop. This is because initially the "myfile.rrd" has to be created with all the settings, before the user start modifying them.
unless (-f "myfile.rrd") {
$rrd->create(
step => 50,
bytesIn => "GAUGE",
bytesOut => "GAUGE",
faultsPerSec => "COUNTER"
);
}
Another point that worth mentioning here is that by default RRD Data Source (DS) is set to GAUGE. More information can be found here RRDtool
The Perl module can be found easily CPAN RRD::Simple which provides analysis and extra "features" that you can add to your code.
In conclusion RRD::Simple is very simple, it can be executed by copy-paste into your program. Any further modifications (e.g sample rates, Average/Max/Min values etc.) need a bit of reading upon but definitely worth the effort. Unfortunately there is not much of examples online so some testing need it to be done in order to understand what I need to modify in my code to make it work. By writing this short analysis and providing some links to read upon I hope to save someone else from spending a few days to come up with the answer to his problem.
Again I encourage anyone to try implementing RRD's is a very powerful tool to graphically view your results and store the data up to 3 years.
Another update that I think is useful to some people maybe. Instead of following all this process by adding and removing code in order to make the rrd file working.
After modifications and experimentation I found another solution.
use strict;
use RRD::Simple;
use RRDs;
my $rrd = RRD::Simple->new(
file => "myfile.rrd",
rrdtool => "/usr/local/rrdtool-1.2.11/bin/rrdtool", #optional
tmpdir => "/var/tmp", #optional
cf => [ qw(AVERAGE MAX) ], #optional
default_dstype => "COUNTER", #optional
on_missing_ds => "add", #optional
RRDs::tune("myfile.rrd", "-i", "Source_Name:0") #optional -i or --minimum
RRDs::tune("myfile.rrd", "-a", "Source_Name:200") #optional -a or --maximum
);
There are several optional values that someone can use, but I recommend to use all of them so you can take full control of the program.
I am using:
default_dstype => "COUNTER", #optional
Because by default RRD's will set GAUGE as Data Source (DS). By setting the DS to COUNTER the user can set the minimum and maximum values. Short examples can be found here also RRD::Simple::Examples.
I'm trying to use Net::Google::Spreadsheets to manipulate a Google Docs spreadsheet (side note: you may have seen my previous question where I was trying to work from the inside of Google Docs, now I'm trying a different angle)
I'm trying an example pretty much straight out of the perldoc:
#!/usr/bin/perl
use strict;
use warnings;
use Net::Google::Spreadsheets;
my $service = Net::Google::Spreadsheets->new(
username => 'my.email#gmail.com',
password => 'mypassword'
);
my #spreadsheets = $service->spreadsheets();
# find a spreadsheet by key
my $spreadsheet = $service->spreadsheet(
{
title => 'Perl Test' # This is a spreadsheet I manually created already
}
);
# find a worksheet by title
my $worksheet = $spreadsheet->worksheet(
{
title => 'Sheet1'
}
);
my $cell = $worksheet->cell({col => 1, row => 1});
# update input value of a cell
$cell->input_value('new value');
When I run my code, I get this error:
request for 'https://spreadsheets.google.com/feeds/worksheets/tNdoUPkz7MhRAtVoBaaZVHQ/private/full?title=Sheet1' failed:
500 Internal Server Error
Internal Error
at /usr/local/share/perl/5.10.1/Net/Google/DataAPI/Role/Service.pm line 96
Net::Google::DataAPI::Role::Service::request('Net::Google::Spreadsheets=HASH(0x167ce60)', 'HASH(0x1c63ba8)') called at /usr/local/share/perl/5.10.1/Net/Google/DataAPI/Role/Service.pm line 158
Net::Google::DataAPI::Role::Service::get_feed('Net::Google::Spreadsheets=HASH(0x167ce60)', 'https://spreadsheets.google.com/feeds/worksheets/tNdoUPkz7MhR...', 'HASH(0x1a38d58)') called at /usr/local/share/perl/5.10.1/Net/Google/DataAPI.pm line 106
Net::Google::Spreadsheets::Spreadsheet::worksheets('Net::Google::Spreadsheets::Spreadsheet=HASH(0x1a36460)', 'HASH(0x1a38d58)') called at /usr/local/share/perl/5.10.1/Net/Google/DataAPI.pm line 119
Net::Google::Spreadsheets::Spreadsheet::worksheet('Net::Google::Spreadsheets::Spreadsheet=HASH(0x1a36460)', 'HASH(0x1a38d58)') called at spreadsheet_test.pl line 22
And then if I try to open up the Perl Test spreadsheet from within Google Docs, Google itself gives me the equivalent of a 500 error.
So what am I doing wrong?