Reading specific values in hash of hashes - Perl - perl

I have an hash of hash map as below. Please note that the hash map is very huge which contains PluginsStatus as Success or Error. When PluginsStatus for a key is Success then I need not process anything (I have handled this scenario) but if its Error I need to to display in the order - PluginsStatus, PluginspatchLogName, PluginsLogFileName_0, PluginsLogFileLink_0, PluginsLogFileErrors_0 and so on.
Please note, I do not know exactly how many keys (in hash of a hash) i.e. PluginsLogFileName, PluginsLogFileLink, PluginsLogFileErrors exists i.e. it is dynamic.
$VAR1 = { 'Applying Template Changes' => {
'PluginsLogFileErrors_2' => 'No Errors',
'PluginsStatus' => 'Error',
'PluginsLogFileName_1' => 'Applying_Template_Changes_2015-05-12_02-57-40AM.log',
'PluginsLogFileName_2' => 'ApplyingTemplates.log',
'PluginsLogFileErrors_1' => 'ERROR: FAPSDKEX-00024 : Error in undeploying template.Cause : Unknown.Action : refer to log file for more details.',
'PluginspatchLogName' => '2015-05-11_08-14-28PM.log',
'PluginsLogFileLink_0' => '/tmp/xpath/2015-05-11_08-14-28PM.log',
'PluginsLogFileName_0' => '2015-05-11_08-14-28PM.log',
'PluginsLogFileErrors_0' => 'No Errors',
'PluginsLogFileLink_2' => 'configlogs/ApplyingTemplates.log',
'PluginsLogFileLink_1' => 'configlogs/Applying_Template_Changes_2015-05-12_02-57-40AM.log'
},
'Configuring Keystore Service' => {
'PluginsStatus' => 'Error',
'PluginsLogFileName_1' => 'Configuring_Keystore_Service_2015-05-11_11-11-37PM.log',
'PluginsLogFileErrors_1' => 'ERROR: Failed to query taxonomy attribute AllProductFamilyAndDomains.',
'PluginspatchLogName' => '2015-05-11_08-14-28PM.log',
'PluginsLogFileLink_0' => '/tmp/xpath/2015-05-11_08-14-28PM.log',
'PluginsLogFileName_0' => '2015-05-11_08-14-28PM.log',
'PluginsLogFileErrors_0' => 'No Errors',
'PluginsLogFileLink_1' => 'configlogs/Configuring_Keystore_Service_2015-05-11_11-11-37PM.log'
},
'Applying Main Configuration' => {
'PluginsStatus' => 'Error',
'PluginspatchLogName' => '2015-05-11_08-14-28PM.log',
'PluginsLogFileName_0' => 'Applying_Main_Configuration_2015-05-12_01-11-21AM.log',
'PluginsLogFileLink_0' => '/tmp/xpath/2015-05-11_08-14-28PM.log',
'PluginsLogFileErrors_0' => 'ERROR: Failed to query taxonomy attribute AllProductFamilyAndDomains.apps.ad.common.exception.ADException: Failed to query taxonomy attribute AllProductFamilyAndDomains.... 104 lines more'
}
};
Below is an output snippet I am looking for:
Plugin name is = Applying Template Changes
PluginsStatus = Error
PluginspatchLogName = 2015-05-11_08-14-28PM.log
PluginsLogFileName_0 = 2015-05-11_08-14-28PM.log
PluginsLogFileLink_0 = /tmp/xpath/2015-05-11_08-14-28PM.log
PluginsLogFileErrors_0 = No Errors
PluginsLogFileName_1 = Applying_Template_Changes_2015-05-12_02-57-40AM.log
PluginsLogFileLink_1 = configlogs/Applying_Template_Changes_2015-05-12_02- 57-40AM.log
PluginsLogFileErrors_1 = ERROR: FAPSDKEX-00024 : Error in undeploying template.Cause : Unknown.Action : refer to log file for more details.,
PluginsLogFileName_2 = ApplyingTemplates.log
PluginsLogFileLink_2 = configlogs/ApplyingTemplates.log
PluginsLogFileErrors_2 = No Errors`
Please let me know if someone could help me here ?

You have built a hash that is less than ideal for your purposes. You should create a LogFile hash element that has an array as its value. After that the process is trivial
{
"Applying Main Configuration" => {
LogFile => [
{
Errors => "ERROR: Failed to query taxonomy attribute AllProductFamilyAndDomains.apps.ad.common.exception.ADException: Failed to query taxonomy attribute AllProductFamilyAndDomains.... 104 lines more",
Link => "/tmp/xpath/2015-05-11_08-14-28PM.log",
Name => "Applying_Main_Configuration_2015-05-12_01-11-21AM.log",
},
],
patchLogName => "2015-05-11_08-14-28PM.log",
Status => "Error",
},
"Applying Template Changes" => {
LogFile => [
{
Errors => "No Errors",
Link => "/tmp/xpath/2015-05-11_08-14-28PM.log",
Name => "2015-05-11_08-14-28PM.log",
},
{
Errors => "ERROR: FAPSDKEX-00024 : Error in undeploying template.Cause : Unknown.Action : refer to log file for more details.",
Link => "configlogs/Applying_Template_Changes_2015-05-12_02-57-40AM.log",
Name => "Applying_Template_Changes_2015-05-12_02-57-40AM.log",
},
{
Errors => "No Errors",
Link => "configlogs/ApplyingTemplates.log",
Name => "ApplyingTemplates.log",
},
],
patchLogName => "2015-05-11_08-14-28PM.log",
Status => "Error",
},
"Configuring Keystore Service" => {
LogFile => [
{
Errors => "No Errors",
Link => "/tmp/xpath/2015-05-11_08-14-28PM.log",
Name => "2015-05-11_08-14-28PM.log",
},
{
Errors => "ERROR: Failed to query taxonomy attribute AllProductFamilyAndDomains.",
Link => "configlogs/Configuring_Keystore_Service_2015-05-11_11-11-37PM.log",
Name => "Configuring_Keystore_Service_2015-05-11_11-11-37PM.log",
},
],
patchLogName => "2015-05-11_08-14-28PM.log",
Status => "Error",
},
}

Just iterate over the keys of the hash. Use the $hash{key}{inner_key} syntax to get into the nested hash.
#!/usr/bin/perl
use warnings;
use strict;
use feature qw{ say };
my %error = ( 'Applying Template Changes' => {
'PluginsLogFileErrors_2' => 'No Errors',
'PluginsStatus' => 'Error',
'PluginsLogFileName_1' => 'Applying_Template_Changes_2015-05-12_02-57-40AM.log',
# ...
'PluginsLogFileLink_0' => '/tmp/xpath/2015-05-11_08-14-28PM.log',
'PluginsLogFileErrors_0' => 'ERROR: Failed to query taxonomy attribute AllProductFamilyAndDomains.apps.ad.common.exception.ADException: Failed to query taxonomy attribute AllProductFamilyAndDomains.... 104 lines more',
},
);
for my $step (keys %error) {
print "Plugin name is = $step\n";
for my $detail (sort keys %{ $error{$step} }) {
print "$detail = $error{$step}{$detail}\n";
}
}

Related

Perl PAWS SelectObjectContent - "Precondition Failed"

I receive a "Precondition Failed" error when I try to run:
my $obj = $s3->SelectObjectContent(
Bucket => 'MyBucket',
Expression => 'SELECT * FROM s3object s',
ExpressionType => 'SQL',
InputSerialization => {
JSON => {
Type => 'LINES'
}
},
Key => 'MyKey',
OutputSerialization => {
JSON => {}
} );
I followed https://metacpan.org/pod/Paws::S3::SelectObjectContent#RequestProgress-=%3E-Paws::S3::RequestProgress
I have successfully called "GetObject" using my bucket and key, so those are not responsible for the error.

Elasticsearch searching with perl client

I'm attempting to do something that should be simple but I cannot get it to work. I've looked and search all over to find detailed doc for perl search::elsticsearch. I can only find CPAN doc and as far as search is concerned it is barely mentioned. I've search here and cannot find a duplicate question.
I have elasticsearch and filebeat. Filebeat is sending syslog to elasticsearch. I just want to search for messages with matching text and date range. I can find the messages but when I try to add date range the query fails. Here is the query from kibana dev tools.
GET _search
{
"query": {
"bool": {
"filter": [
{ "term": { "message": "metrics" }},
{ "range": { "timestamp": { "gte": "now-15m" }}}
]
}
}
}
I don't get exactly what I'm looking for but there isn't an error.
Here is my attempt with perl
my $results=$e->search(
body => {
query => {
bool => {
filter => {
term => { message => 'metrics' },
range => { timestamp => { 'gte' => 'now-15m' }}
}
}
}
}
);
This is the error.
[Request] ** [http://x.x.x.x:9200]-[400]
[parsing_exception]
[range] malformed query, expected [END_OBJECT] but found [FIELD_NAME],
with: {"col":69,"line":1}, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__
at ./elasticsearchTest.pl line 15.
With vars: {'body' => {'status' => 400,'error' => {
'root_cause' => [{'col' => 69,'reason' => '[range]
malformed query, expected [END_OBJECT] but found [FIELD_NAME]',
'type' => 'parsing_exception','line' => 1}],'col' => 69,
'reason' => '[range] malformed query, expected [END_OBJECT] but found [FIELD_NAME]',
'type' => 'parsing_exception','line' => 1}},'request' => {'serialize' => 'std',
'path' => '/_search','ignore' => [],'mime_type' => 'application/json',
'body' => {
'query' => {
'bool' =>
{'filter' => {'range' => {'timestamp' => {'gte' => 'now-15m'}},
'term' => {'message' => 'metrics'}}}}},
'qs' => {},'method' => 'GET'},'status_code' => 400}
Can someone help me figure out how to search with the search::elasticsearch perl module?
Multiple filter clauses must be passed as separate JSON objects within an array (like in your initial JSON query), not multiple filters in the same JSON object. This maps to how you must create the Perl data structure.
filter => [
{term => { message => 'metrics' }},
{range => { timestamp => { 'gte' => 'now-15m' }}}
]

Can't update description of listing with eBay API

I'm trying to write a Perl script that will update our eBay listings descriptions without having to keep logging in (running across multiple marketplaces if proving tricky to keep stock levels, descriptions etc updated). Here is what I have so far:
my $ebay = new Net::eBay( {
SiteLevel => 'prod',
DeveloperKey => 'x',
ApplicationKey => 'x',
CertificateKey => 'x',
Token => 'x',
} );
$ebay->setDefaults( { API => 2, compatibility => 900 } );
my $new_desc = q|<meta name="viewport" content="width=device-width, initial-scale=1.0">
<p>We are proud to announce our first ever badge! With an easy-to-iron
on backing, fitting couldn't be any easier! We have designed the path to
be a perfect addition to any piece of cosplay costume. Please do send
in the photos of it being used on your costumes, as we would love to
share.</p>
<p>The badge is 7 x 7 cm / 2 x 2 inches in size, and 2mm thi<br></p>|;
my $result = $ebay->submitRequest( "ReviseItem",
{
DetailLevel => "ReturnAll",
ErrorLevel => "1",
SiteId => "1",
Item => {
Description => \$new_desc,
ItemID => 253430606975
},
ItemID => 253430606975
}) || die;
print "Result: " . Dumper( $result ) . "\n";
I get an error when running it though:
'Errors' => [
{
'ShortMessage' => 'Return Policy Attribute Not Valid',
'ErrorClassification' => 'RequestError',
'ErrorCode' => '21920200',
'LongMessage' => 'Return Policy Attribute returnDescription Not Valid On This Site',
'SeverityCode' => 'Warning',
'ErrorParameters' => {
'Value' => 'returnDescription',
'ParamID' => '0'
}
},
{
'ShortMessage' => 'Description is missing.',
'ErrorClassification' => 'RequestError',
'ErrorCode' => '106',
'SeverityCode' => 'Error',
'LongMessage' => 'A description is required.'
}
],
Am I misunderstanding what gets passed in? from what I can understand, you just pass in the params you want to change?
UPDATE: As suggested by Dave, I'm giving Marketplace::Ebay a go. Just testing by trying to select one of my items:
my $ebay = Marketplace::Ebay->new(
production => 1,
site_id => 3,
developer_key => 'xx',
application_key => 'xx',
certificate_key => 'xxx',
token => 'xx',
xsd_file => 'ebaySvc.xsd',
);
my $res = $ebay->api_call('GetItem', { ItemID => 253430606975 });
print Dumper($res);
But I get some weird error:
error: element `{urn:ebay:apis:eBLBaseComponents}GiftIcon' not
processed for {urn:ebay:apis:eBLBaseComponents}GetItemResponse/Item at
//[5]/*[6] $VAR1 = undef;
Any ideas?
Ah ha - got it! The issue seemed to be around the way the HTML was being passed along. If I put it inside a CDATA tag, it works fine:
my $new_desc = q|<![CDATA[
some html etc here
]]>|;
my $result = $ebay->submitRequest( "ReviseItem",
{
DetailLevel => "ReturnAll",
ErrorLevel => "1",
SiteId => "1",
Item => {
Description => $new_desc,
ItemID => 253430606975
},
ItemID => 253430606975
}) || die;
...and updates perfectly

ElasticSearch (search_context_missing_exception) with Search::ElasticSearch::Scroll

I'm using Search::Elasticsearch and Search::Elasticsearch::Scroll for search and scroll into my elasticsearch server.
In scrolling process, for some querys, I'm seeing the next errors while I'm scrolling the search results:
2016/03/22 11:03:38 - 265885 FATAL: [Daemon.pm][8221]: Something gone wrong, error $VAR1 = bless( {
'msg' => '[Missing] ** [http://localhost:9200]-[404] Not Found, called from sub Search::Elasticsearch::Scroll::next at searcher.pl line 92. With vars: {\'body\' => {\'hits\' => {\'hits\' => [],\'max_score\' => \'0\',\'total\' => 5215},\'timed_out\' => bless( do{\\(my $o = 0)}, \'JSON::XS::Boolean\' ),\'_shards\' => {\'failures\' => [{\'index\' => undef,\'reason\' => {\'reason\' => \'No search context found for id [4920053]\',\'type\' => \'search_context_missing_exception\'},\'shard\' => -1},{\'index\' => undef,\'reason\' => {\'reason\' => \'No search context found for id [5051485]\',\'type\' => \'search_context_missing_exception\'},\'shard\' => -1},{\'index\' => undef,\'reason\' => {\'reason\' => \'No search context found for id [4920059]\',\'type\' => \'search_context_missing_exception\'},\'shard\' => -1},{\'index\' => undef,\'reason\' => {\'reason\' => \'No search context found for id [5051496]\',\'type\' => \'search_context_missing_exception\'},\'shard\' => -1},{\'index\' => undef,\'reason\' => {\'reason\' => \'No search context found for id [5051500]\',\'type\' => \'search_context_missing_exception\'},\'shard\' => -1}],\'failed\' => 5,\'successful\' => 0,\'total\' => 5},\'_scroll_id\' => \'c2NhbjswOzE7dG90YWxfaGl0czo1MjE1Ow==\',\'took\' => 2},\'request\' => {\'serialize\' => \'std\',\'path\' => \'/_search/scroll\',\'ignore\' => [],\'mime_type\' => \'application/json\',\'body\' => \'c2Nhbjs1OzQ5MjAwNTM6bHExbENzRDVReEc0OV9UMUgzd3Vkdzs1MDUxNDg1OnJrQ3lsUkRKVHRxRWRWeURoOTB4WVE7NDkyMDA1OTpscTFsQ3NENVF4RzQ5X1QxSDN3dWR3OzUwNTE0OTY6cmtDeWxSREpUdHFFZFZ5RGg5MHhZUTs1MDUxNTAwOnJrQ3lsUkRKVHRxRWRWeURoOTB4WVE7MTt0b3RhbF9oaXRzOjUyMTU7\',\'qs\' => {\'scroll\' => \'1m\'},\'method\' => \'GET\'},\'status_code\' => 404}
',
'stack' => [
[
'searcher.pl',
92,
'Search::Elasticsearch::Scroll::next'
]
],
'text' => '[http://localhost:9200]-[404] Not Found',
'vars' => {
'body' => {
'hits' => {
'hits' => [],
'max_score' => '0',
'total' => 5215
},
'timed_out' => bless( do{\(my $o = 0)}, 'JSON::XS::Boolean' ),
'_shards' => {
'failures' => [
{
'index' => undef,
'reason' => {
'reason' => 'No search context found for id [4920053]',
'type' => 'search_context_missing_exception'
},
'shard' => -1
},
{
'index' => undef,
'reason' => {
'reason' => 'No search context found for id [5051485]',
'type' => 'search_context_missing_exception'
},
'shard' => -1
},
{
'index' => undef,
'reason' => {
'reason' => 'No search context found for id [4920059]',
'type' => 'search_context_missing_exception'
},
'shard' => -1
},
{
'index' => undef,
'reason' => {
'reason' => 'No search context found for id [5051496]',
'type' => 'search_context_missing_exception'
},
'shard' => -1
},
{
'index' => undef,
'reason' => {
'reason' => 'No search context found for id [5051500]',
'type' => 'search_context_missing_exception'
},
'shard' => -1
}
],
'failed' => 5,
'successful' => 0,
'total' => 5
},
'_scroll_id' => 'c2NhbjswOzE7dG90YWxfaGl0czo1MjE1Ow==',
'took' => 2
},
'request' => {
'serialize' => 'std',
'path' => '/_search/scroll',
'ignore' => [],
'mime_type' => 'application/json',
'body' => 'c2Nhbjs1OzQ5MjAwNTM6bHExbENzRDVReEc0OV9UMUgzd3Vkdzs1MDUxNDg1OnJrQ3lsUkRKVHRxRWRWeURoOTB4WVE7NDkyMDA1OTpscTFsQ3NENVF4RzQ5X1QxSDN3dWR3OzUwNTE0OTY6cmtDeWxSREpUdHFFZFZ5RGg5MHhZUTs1MDUxNTAwOnJrQ3lsUkRKVHRxRWRWeURoOTB4WVE7MTt0b3RhbF9oaXRzOjUyMTU7',
'qs' => {
'scroll' => '1m'
},
'method' => 'GET'
},
'status_code' => 404
},
'type' => 'Missing'
}, 'Search::Elasticsearch::Error::Missing' );
The code I'm using is the next one (simplified) :
# Retrieve scroll
my $scroll = $self->getScrollBySignature($item);
# Retrieve all affected documents ids
while (my #docs = $scroll->next(500)) {
# Do stuff with #docs
}
The function getScrollBySignature have the next code in order to call to elasticSearch
my $scroll = $self->{ELASTIC}->scroll_helper(
index => $self->{INDEXES},
search_type => 'scan',
ignore_unavailable => 1,
body => {
size => $self->{PAGINATION},
query => {
filtered => {
filter => {
bool => {
must => [{term => {signature_id => $item->{profileId}}}, {terms => {channel_type_id => $type}}]
}
}
}
}
}
);
As you can see, I'm doing the scroll without passing scroll parameter then as documentation says, the time that scroll is alive is 1 min.
The elasticSearch is a cluster of 3 servers, and the query that ends with that error retrieves a bit more than 5000 docs.
My first solution was to update the life time for scroll to 5 minutes and the error didn't appear.
The question is, as I understand every time I'm calling $scroll->next() the life time off scroll affected is upgraded 1m more, then how is possible to receive those context related errors?
I'm doing something in a bad manner?
Thank you all.
The first thing that comes to mind is that the timer is not updated. Have you checked this? You can do a query every 10 seconds for example and see if at the 6th query it gives you the error ...
Well, a good rule of thumb is inside a ->next() block, don't stay by iteration more than time that you've configured in scroll.
Between each call of ->next() you cannot stay more than that time configured. If you stay more, the scroll may be not be there and the error earch_context_missing_exception will appear.
My solution for this problem was inside next block only store data into array/hash structure and once the scroll process ended work with all data.
The solution of the question example:
# Retrieve scroll
my $scroll = $self->getScrollBySignature($item);
# Retrieve all affected documents ids
my #allDocs;
while (my #docs = $scroll->next(500)) {
push #allDocs, map {$_->{_id}} #docs
}
foreach (#allDocs) {
# Do stuff with doc
}

Trouble with Perl 5.10.1 error Type of arg 1 to each must be hash

I am at my wits end on this one. I have to use a older version of Perl since our Linux standard continues to be RHEL 6. So I have Perl 5.10.1. I developed my script in 5.14 and it works great. Moving to RHEL and older Perl it barfs.
Here is the DataDump of the information brought back as $response, followed by the code used in 5.14 and the things I have tried in 5.10.1. Any help would be appreciated.
response => {
md5 => {
organizations => "b157f81f9469e88fd1ac2435559f558e",
scanners => "40e782276cc521ef799cc111a9472cf4",
zones => "a41b4d543756320418fce473d97a3b8d",
},
organizations => [
{ description => "", id => 1, name => "Our Corporation" },
],
scanners => [
{ description => "", id => 5, name => "BR549-A", status => 1 },
{
description => "Test VM Scanner in BFE",
id => 16,
name => "BFENessus01",
status => 1,
},
{
description => "Our other Nessus Scanner VM",
id => 17,
name => "OHTHERNESSUS01",
status => 1,
},
{ description => "", id => 49, name => "NYCNESSUS02", status => 1 },
{ description => "", id => 50, name => "LAX1NESSUS03", status => 1 },
{ description => "", id => 51, name => "LAX1NESSUS04", status => 1 },
{ description => "", id => 52, name => "LAX1NESSUS05", status => 1 },
{ description => "", id => 54, name => "MK-NESSUS", status => 1 },
{
description => "Networking team's scanner",
id => 55,
name => "NETEAMNESSUS06",
status => 1,
},
},
timestamp => 1400177639,
type => "regular",
warnings => [],
}
PERL Version 5.14 (WORKING)
while (($key, $value) = each($response->{'response'}{'scanners'} )){
switch ($value->{'status'}){
case 1 {print "Status OK \t\t\t"}
case 2 {print " !! CLOSED !! \t\t\t"}
case 4 {print " !! TIMEOUT !! \t\t\t"}
case 16384 {print " ** DISABLED **\t\t\t"}
case 1024 {print " Updating Plug-Ins \t\t\t"}
case 1025 {print " ** Updating Plug-Ins **\t\t"}
case 1281 {print " Attempting to Update\t\t"}
case 256 {print " !! Out of Date !! \t\t\t"}
case 257 {print "Plug-Ins Out of Sync !! \t\t"}
else { print "Status BAD ($value->{'status'}) \t\t"}
}
print "$value->{'name'} $value->{'description'} \n";
}
PERL VERSION 5.10.1 (DOES NOT WORK)
while (($key, $value) = each($response->{'response'}{'scanners'} )){
errors out with error: Type of arg 1 to each must be hash (not hash element) at
Changing to:
while (($key, $value) = each(%{$response->{'response'}{'scanners'}} )){
Allows it to run, but stops with Not a HASH reference at
$response->{'response'}{'scanners'} contains a reference to an array, so your code is equivalent to the following:
while (my ($key, $value) = each(#{ $response->{response}{scanners} })) {
...
}
However, each didn't work on arrays in 5.10. The following is equivalent:
for my $key (0..$#{ $response->{response}{scanners} }) {
my $value = $response->{response}{scanners}[$key];
...
}
Since you don't actually use $key, you could also use the following:
for my $value (#{ $response->{response}{scanners} }) {
...
}
Note: I recommend against using each($ref). Aside from the backwards-compatibility issues, it's experimental, and it doesn't work with some special hashes and arrays.