SOAP::Lite log transport request/response with custom identifier - perl

I would like to log SOAP::Lite transport request/response contents using a custom identifier (e.g. a transaction-id or txn_id in my example below):
#!/usr/bin/perl
use strict;
use warnings;
use Data::Dumper;
use SOAP::Lite +trace => [ transport => \&log_transport, ];
sub log_transport {
my ($in) = #_;
if (ref($in) eq "HTTP::Request") {
# INSERT INTO logs ( txn_id, request ) VALUES ( $tnx_id, $in->content )
say STDERR Dumper(ref($in), $in->content);
}
elsif (ref($in) eq "HTTP::Response") {
# UPDATE logs SET response = '$in->content' WHERE txn_id = $tnx_id
say STDERR Dumper(ref($in), $in->content);
}
}
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME}=0;
my $soap = SOAP::Lite->proxy('https://www.dataaccess.com/webservicesserver/NumberConversion.wso?op=NumberToWords');
$soap->serializer()->register_ns('http://www.dataaccess.com/webservicesserver/' ,"ns");
sub getWords
{
my ($number, $txn_id) = #_ ;
my $method = SOAP::Data->name("ns:NumberToWords");
my #params = ( SOAP::Data->name("ubiNum" => $number) );
my $response = $soap->call($method => #params);
if (!$response->fault) {
say STDOUT "NumberToWords = " . $response->result;
}
else {
say STDERR "error: " . (defined $response->faultstring? $response->faultstring : $soap->transport->status);
}
}
getWords(444, '123abc');
In my example above, how can I pass the transaction-id 123abc to my logger?
P.S. I do not wish to use:
$soap->outputxml(1)->call($method => #params)

It does not seem like the SOAP::Trace transport callback supports extra argument passing. As a workaround you could use a lexical variable declared in the outer scope like this:
use strict;
use warnings;
use Data::Dumper;
my $TXN_ID;
use SOAP::Lite +trace => [ transport => \&log_transport, ];
sub log_transport {
my ($in) = #_;
say STDERR "Logging transaction id: $TXN_ID:";
if (ref($in) eq "HTTP::Request") {
# INSERT INTO logs ( txn_id, request ) VALUES ( $tnx_id, $in->content )
say STDERR Dumper(ref($in), $in->content);
}
elsif (ref($in) eq "HTTP::Response") {
# UPDATE logs SET response = '$in->content' WHERE txn_id = $tnx_id
say STDERR Dumper(ref($in), $in->content);
}
}
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME}=0;
my $soap = SOAP::Lite->proxy('https://www.dataaccess.com/webservicesserver/NumberConversion.wso?op=NumberToWords');
$soap->serializer()->register_ns('http://www.dataaccess.com/webservicesserver/' ,"ns");
sub getWords
{
my ($number, $txn_id) = #_ ;
$TXN_ID = $txn_id;
my $method = SOAP::Data->name("ns:NumberToWords");
my #params = ( SOAP::Data->name("ubiNum" => $number) );
my $response = $soap->call($method => #params);
if (!$response->fault) {
say STDOUT "NumberToWords = " . $response->result;
}
else {
say STDERR "error: " . (defined $response->faultstring? $response->faultstring : $soap->transport->status);
}
}
getWords(444, '123abc');

Related

Mojo::UserAgent non-blocking vs blocking performance

I have the following code:
my $ua = Mojo::UserAgent->new ();
my #ids = qw(id1 id2 id3);
foreach (#ids) {
my $input = $_;
my $res = $ua->get('http://my_site/rest/id/'.$input.'.json' => sub {
my ($ua, $res) = #_;
print "$input =>" . $res->result->json('/net/id/desc'), "\n";
});
}
Mojo::IOLoop->start unless Mojo::IOLoop->is_running;
Why when I run the above code (non-blocking) does it take about 6 seconds while when running the code as blocking, i.e. inside the loop something like:
my $res = $ua->get('http://my_site/rest/id/'.$input.'.json');
print "$input =>" . $res->result->json('/net/id/desc'), "\n";
without the latest line it takes about 1 second?
Why is the blocking code faster than the non-blocking code?
The first thing to check when things happened. I couldn't get the same delay. Remember to try each way several times to spot outliers where there's a network hiccup. Note that the second argument to the non-blocking sub is a transaction object, normally written as $tx, where the response object is normally written res:
use Mojo::Util qw(steady_time);
say "Begin: " . steady_time();
END { say "End: " . steady_time() }
my $ua = Mojo::UserAgent->new ();
my #ids = qw(id1 id2 id3);
foreach (#ids) {
my $input = $_;
my $res = $ua->get(
$url =>
sub {
my ($ua, $tx) = #_;
print "Fetched\n";
}
);
}
One possibility is that keep-alive is holding an open connection. What happens if you turn that off?
my $res = $ua->get(
$url =>
{ Connection => 'close' }
sub {
my ($ua, $tx) = #_;
print "Fetched\n";
}
);
Here's a version that uses promises, which you'll want to get used to as more Mojo stuff moves to it:
use feature qw(signatures);
no warnings qw(experimental::signatures);
use Mojo::Promise;
use Mojo::Util qw(steady_time);
use Mojo::UserAgent;
my $ua = Mojo::UserAgent->new;
say "Begin: " . steady_time();
END { say "End: " . steady_time() }
my #ids = qw(id1 id2 id3);
my #gets = map {
$ua->get_p( 'http://www.perl.com' )->then(
sub ( $tx ) { say "Fetched: " . steady_time() },
sub { print "Error: #_" }
);
} #ids;
Mojo::Promise->all( #gets )->wait;

Perl: Issue with blessed object

I am creating a bot that connects to a Matrix server. For that I use Net::Async::Matrix.
The code:
#!/usr/bin/perl
use strict;
use warnings;
use Net::Async::Matrix;
use Net::Async::Matrix::Utils qw ( parse_formatted_message );
use IO::Async::Loop;
use Data::Dumper;
my $loop = IO::Async::Loop->new;
my $matrix = Net::Async::Matrix->new(
server => 'matrix.server.net',
on_error => sub {
my ( undef, $message ) = #_;
warn "error: $message\n";
},
);
$loop->add( $matrix );
$matrix->login(
user_id => '#bot:matrix.server.net',
password => 'password',
)->get;
my $room = $matrix->join_room( '#Lobby:matrix.server.net' )->get;
$room->configure(
on_message => sub {
my ( undef, $member, $content, $event ) = #_;
my $msg = parse_formatted_message( $content );
my $sendername = $member->displayname;
print Dumper $sendername;
&sendmsg("$sendername said: $msg");
},
);
my $stream = $matrix->start;
sub sendmsg {
my $input = shift;
if ($input) {
$room->send_message(
type => "m.text",
body => $input,
),
}
}
$loop->run;
Basically, I want the bot to echo what was said.
I get following output:
$VAR1 = 'm1ndgames'; Longpoll failed - encountered object 'm1ndgames
said: test', but neither allow_blessed, convert_blessed nor
allow_tags settings are enabled (or TO_JSON/FREEZE method missing) at
/usr/local/share/perl/5.24.1/Net/Async/Matrix.pm line 292.
and I don't understand it. When I enter a string like test into the body, it gets sent to the room.
parse_formatted_message returns a String::Tagged object. This class overloads concatenation so that "$sendername said: $msg" also returns a String::Tagged object. This object is passed to sendmsg which tries to serialize it into JSON, but it refuses to serialize objects.
Fix: Replace
my $msg = parse_formatted_message( $content );
with
my $msg = parse_formatted_message( $content )->str;
I'd guess that this is a quoting error. If you look at Net::Async::Matrix::Room:
sub send_message
{
my $self = shift;
my %args = ( #_ == 1 ) ? ( type => "m.text", body => shift ) : #_;
my $type = $args{msgtype} = delete $args{type} or
croak "Require a 'type' field";
$MSG_REQUIRED_FIELDS{$type} or
croak "Unrecognised message type '$type'";
foreach (#{ $MSG_REQUIRED_FIELDS{$type} } ) {
$args{$_} or croak "'$type' messages require a '$_' field";
}
if( defined( my $txn_id = $args{txn_id} ) ) {
$self->_do_PUT_json( "/send/m.room.message/$txn_id", \%args )
->then_done()
}
else {
$self->_do_POST_json( "/send/m.room.message", \%args )
->then_done()
}
}
The type you sent is handled by this sub, and then the actual message gets handed off to _do_POST_json in Net::Async::Matrix.
But you've sent a string containing a :.
So I think what's happening is it's encoding like this:
use JSON;
use Data::Dumper;
my $json = encode_json ( {body => "m1ndgames: said test"});
print Dumper $json;
But the response that's coming back, at line 292 which is:
if( length $content and $content ne q("") ) {
eval {
$content = decode_json( $content );
1;
} or
return Future->fail( "Unable to parse JSON response $content" );
return Future->done( $content, $response );
}
So I think is what is happening is the remote server is sending you a broken error code, and the module isn't handling it properly - it's expecting JSON but it isn't actually getting it.
My best guess would be - try dropping the : out of your message, because I would guess there's some bad quoting happening. But without seeing the code on the server side, I can't quite tell.

Is it possible to register a function to preprocess log messages with Log::Log4perl?

In this example:
$logger->debug({
filter => \&Data::Dumper::Dumper,
value => $ref
});
I can pretty print my references instead of ARRAY(0xFFDFKDJ). But it's too boring to type that long code every time. I just want:
$logger->preprocessor({
filter => \&Data::Dumper::Dumper,
value => $ref
});
$logger->debug( $ref, $ref2 );
$logger->info( $array );
And $ref, $ref2, and $array will be dumped by Data::Dumper.
It there a way to do this?
UPD
With help of your answers I do the patch
Now you just:
log4perl.appender.A1.layout=FallbackLayout
log4perl.appender.A1.layout.chain=PatternLayout
log4perl.appender.A1.layout.chain.ConversionPattern=%m%n
log4perl.appender.A1.warp_message = sub { $#_ = 2 if #_ > 3; \
return #_; }
# OR
log4perl.appender.A1.warp_message = main::warp_my_message
sub warp_my_message {
my( #chunks ) = #_;
use Data::Dump qw/ pp /;
for my $msg ( #chunks ) {
$msg = pp $msg if ref $msg;
}
return #chunks;
}
UPD2
Or you can use this small module
log4perl.appender.SomeAPP.warp_message = Preprocess::Messages::msg_filter
log4perl.appender.SomeAPP.layout = Preprocess::Messages
package Preprocess::Messages;
sub msg_filter {
my #chunks = #_;
for my $msg ( #chunks ) {
$msg = pp $msg if ref $msg;
}
return #chunks;
};
sub render {
my $self = shift;
my $layout = Log::Log4perl::Layout::PatternLayout->new(
'%d %P %p> %c %F:%L %M%n %m{indent=2}%n%n'
);
$_[-1] += 1; # increase level of the caller
return $layout->render( join $Log::Log4perl::JOIN_MSG_ARRAY_CHAR, #{ shift() }, #_ );
}
sub new {
my $class = shift;
$class = ref ($class) || $class;
return bless {}, $class;
}
1;
Yes, of course you can set 'warp_message = 0' and combine msg_filter and render together.
log4perl.appender.SomeAPP.warp_message = 0
log4perl.appender.SomeAPP.layout = Preprocess::Messages
sub render {
my($self, $message, $category, $priority, $caller_level) = #_;
my $layout = Log::Log4perl::Layout::PatternLayout->new(
'%d %P %p> %c %F:%L %M%n %m{indent=2}%n%n'
);
for my $item ( #{ $message } ) {
$item = pp $item if ref $item;
}
$message = join $Log::Log4perl::JOIN_MSG_ARRAY_CHAR, #$message;
return $layout->render( $message, $category, $priority, $caller_level+1 );
}
The easy way: use warp_message
The easiest way to do this is to create a custom appender and set the warp_message parameter so you can get the original references that were passed to the logger:
package DumpAppender;
use strict;
use warnings;
use Data::Dumper;
$Data::Dumper::Indent = 0;
$Data::Dumper::Terse = 1;
sub new {
bless {}, $_[0];
}
sub log {
my($self, %params) = #_;
print ref($_) ? Dumper($_) : $_ for #{ $params{message} };
print "\n";
}
package main;
use strict;
use warnings;
use Log::Log4perl;
Log::Log4perl->init(\q{
log4perl.rootLogger=DEBUG,Dump
log4perl.appender.Dump=DumpAppender
log4perl.appender.Dump.layout=NoopLayout
log4perl.appender.Dump.warp_message=0
});
my $logger = Log::Log4perl->get_logger;
$logger->debug(
'This is a string, but this is a reference: ',
{ foo => 'bar' },
);
Output:
This is a string, but this is a reference: {'foo' => 'bar'}
Unfortunately, if you take this approach, you're stuck writing your own code to handle layouts, open files, etc. I wouldn't take this approach except for very simple projects that only need to print to screen.
A better way: composite appender
A better approach is to write your own composite appender. A composite appender forwards messages on to another appender after manipulating them somehow, e.g. filtering or caching them. With this approach, you can write only the code for dumping the references and let an existing appender do the heavy lifting.
The following shows how to write a composite appender. Some of this is explained in the docs for Log::Log4perl::Appender, but I copied much of it from Mike Schilli's Log::Log4perl::Appender::Limit:
package DumpAppender;
use strict;
use warnings;
our #ISA = qw(Log::Log4perl::Appender);
use Data::Dumper;
$Data::Dumper::Indent = 0;
$Data::Dumper::Terse = 1;
sub new {
my ($class, %options) = #_;
my $self = {
appender => undef,
%options
};
# Pass back the appender to be limited as a dependency to the configuration
# file parser.
push #{ $options{l4p_depends_on} }, $self->{appender};
# Run our post_init method in the configurator after all appenders have been
# defined to make sure the appenders we're connecting to really exist.
push #{ $options{l4p_post_config_subs} }, sub { $self->post_init() };
bless $self, $class;
}
sub log {
my ($self, %params) = #_;
# Adjust call stack so messages are reported with the correct caller and
# file
local $Log::Log4perl::caller_depth = $Log::Log4perl::caller_depth + 2;
# Dump all references with Data::Dumper
$_ = ref($_) ? Dumper($_) : $_ for #{ $params{message} };
$self->{app}->SUPER::log(
\%params,
$params{log4p_category},
$params{log4p_level}
);
}
sub post_init {
my ($self) = #_;
if(! exists $self->{appender}) {
die "No appender defined for " . __PACKAGE__;
}
my $appenders = Log::Log4perl->appenders();
my $appender = Log::Log4perl->appenders()->{$self->{appender}};
if(! defined $appender) {
die "Appender $self->{appender} not defined (yet) when " .
__PACKAGE__ . " needed it";
}
$self->{app} = $appender;
}
package main;
use strict;
use warnings;
use Log::Log4perl;
Log::Log4perl->init(\q{
log4perl.rootLogger=DEBUG, Dump
log4perl.appender.Dump=DumpAppender
log4perl.appender.Dump.appender=SCREEN
log4perl.appender.SCREEN=Log::Log4perl::Appender::Screen
log4perl.appender.SCREEN.layout=PatternLayout
log4perl.appender.SCREEN.layout.ConversionPattern=%d %p %m%n
});
my $logger = Log::Log4perl->get_logger;
$logger->debug(
'This is a string, but this is a reference: ',
{ foo => 'bar' },
);
Output:
2015/09/14 13:38:47 DEBUG This is a string, but this is a reference: {'foo' => 'bar'}
Note that you have to take some extra steps if you initialize Log::Log4perl via the API instead of via a file. This is documented in the composite appenders section of the Log::Log4perl::Appender documentation.

how to get session id from cookie jar in perl?

My question is very simple.. It is how to get session id from cookie jar ... I have tried below code :-
use warnings;
use HTTP::Cookies;
use HTTP::Request::Common;
use LWP::UserAgent;
$ua = new LWP::UserAgent;
if ( !$ua ) {
print "Can not get the page :UserAgent fialed \n";
return 0;
}
my $cookies = new HTTP::Cookies( file => './cookies.dat', autosave => 1 );
$ua->cookie_jar($cookies);
# push does all magic to exrtact cookies and add to header for further reqs. useragent should be newer
push #{ $ua->requests_redirectable }, 'POST';
$result = $ua->request(
POST "URL",
{ Username => 'admin',
Password => 'admin',
Submit => 'Submit',
}
);
my $session_id = $cookies->extract_cookies($result);
print $session_id->content;
print "\n\n";
$resp = $result->content;
#print "Result is \n\n\n $resp \n";
$anotherURI = URL;
$requestObject = HTTP::Request::Common::GET $anotherURI;
$result = $ua->request($requestObject);
$resp = $result->content;
#print $resp."\n";
I am not getting where the session id is stored and how to fetch it ?
Note:- URL contains the URL of the page.
I wrote HTTP::CookieMonster to make this kind of thing a bit easier. If you don't know which cookie you're looking for, you can do something like this:
use strict;
use warnings;
use HTTP::CookieMonster;
use WWW::Mechanize;
my $mech = WWW::Mechanize->new;
my $monster = HTTP::CookieMonster->new( $mech->cookie_jar );
my $url = 'http://www.nytimes.com';
$mech->get( $url );
my #all_cookies = $monster->all_cookies;
foreach my $cookie ( #all_cookies ) {
printf( "key: %s value: %s\n", $cookie->key, $cookie->val);
}
If you already know the cookie's key, you can something like:
my $cookie = $monster->get_cookie('RMID');
my $session_id = $cookie->val;
Have a look at HTTP::Cookies->scan.
Something like this should do the trick (should add a constraint on the domain at least):
my $session_id;
$cookie_jar->scan(
sub {
my ($key, $val, $path, $domain, $port,
$path_spec, $secure, $expires, $discard, $hash
) = #_;
if ( $key eq "session_id" ) {
$session_id = $val;
}
}
);

Perl - Parse URL to get a GET Parameter Value

How to get the value of a parameter code using URI::URL Perl module?
From this link:
http://www.someaddress.com/index.html?test=value&code=INT_12345
It can be done using URI::URL or URI (I know the first one is kind of obsolete). Thanks in advance.
Create a URI object and use the query_form method to get the key/value pairs for the query. If you know that the code parameter is only specified once, you can do it like this:
my $uri = URI->new("http://www.someaddress.com/index.html?test=value&code=INT_12345");
my %query = $uri->query_form;
print $query{code};
Alternatively you can use URI::QueryParam whichs adds soem aditional methods to the URI object:
my $uri = URI->new("http://www.someaddress.com/index.html?test=value&code=INT_12345");
print $uri->query_param("code");
use URI;
my $uri = URI->new("http://someaddr.com/index.html?test=FIRST&test=SECOND&code=INT_12345");
my %query = $uri->query_form;
use Data::Dumper;
print Dumper \%query;
We can see:
$VAR1 = {
'test' => 'SECOND',
'code' => 'INT_12345'
};
Unfortunately, this result is wrong.
There is possible solution:
use URI::Escape;
sub parse_query {
my ( $query, $params ) = #_;
$params ||= {};
foreach $var ( split( /&/, $query ) ){
my ( $k, $v ) = split( /=/, $var );
$k = uri_unescape $k;
$v = uri_unescape $v;
if( exists $params->{$k} ) {
if( 'ARRAY' eq ref $params->{$k} ) {
push #{ $params->{$k} }, $v;
} else {
$params->{$k} = [ $params->{$k}, $v ];
}
} else {
$params->{$k} = $v;
}
}
return $params;
}