I have some perl code
my $res = $ua->get( $access->to_url );
if ($res->is_success) {
my $ref = XMLin( $res->content );
my $xml = new XML::Simple;
$data = $xml->XMLin($res->content,ForceArray => 1);
#print $res->content;
for my $purchase ( #{ $data->{PurchaseOrders}->{PurchaseOrder}} )
This bit is fine....
How ever when i try loop through child elements, if there is only one child element
i get the "not an array reference" error
for my $item ( #{$purchase->{LineItems}->{LineItem}} )
{
$itemCode = $item->{ItemCode};
}
The XML structure is something like this
PurchaseOrders
PurchaseOrder
LineItems
LineItem
i am aware of an issue with xml simple where i have to forceArray, but i am not sure how to forceArray on the child Nodes
I found this article on stackoverflow that seems very close to my exact problem, but i am struggling on how to execute it with in my code
perl, parsing XML using XML::Simple
$VAR1 = {
'PurchaseOrderID' => '82fa50d6-fd45-4fd2-b42d-035aaaa39a2c',
'LineAmountTypes' => 'Exclusive',
'SentToContact' => 'true',
'AttentionTo' => 'sxxxx',
'Status' => 'AUTHORISED',
'LineItems' => {
'LineItem' => {
'LineAmount' => '57.61',
'Quantity' => '1.0000',
'UnitAmount' => '57.6100',
'LineItemID' => 'e295d55d-68bd',
'Description' => 'xxx',
'ItemCode' => 'xxx',
'TaxAmount' => '11.52',
'AccountCode' => '310',
'TaxType' => 'INPUT2'
}
},
'UpdatedDateUTC' => '2018-10-26T14:19:19.053',
'CurrencyCode' => 'GBP',
'Contact' => {
Included a snipet from my print dumper - please note,its just a snipet of the important part, everything is fine until it hits line items
Also here is the XML file
<PurchaseOrder>
<PurchaseOrderID>82fa50</PurchaseOrderID>
<PurchaseOrderNumber>PO-0029</PurchaseOrderNumber>
<Date>2018-10-26T00:00:00</Date>
<DeliveryDate>2018-10-28T00:00:00</DeliveryDate>
<DeliveryAddress>Address/DeliveryAddress>
<AttentionTo>XXX</AttentionTo>
<SentToContact>true</SentToContact>
<Reference>000000078</Reference>
<CurrencyRate>1.000000</CurrencyRate>
<CurrencyCode>GBP</CurrencyCode>
<Contact>
<ContactID>f203ed00-8cd1-4e4d-9b76-f5e7d90a3c19</ContactID>
<ContactStatus>ACTIVE</ContactStatus>
<Name>XXX</Name>
<FirstName>XXXy</FirstName>
<LastName>XXX</LastName>
<Addresses>
<Address>
<AddressType>XXX</AddressType>
<AddressLine1>XXX</AddressLine1>
<AddressLine2>XXX</AddressLine2>
<City>XXX</City>
<Region>XXX</Region>
<PostalCode>XXX</PostalCode>
<Country>GBR</Country>
</Address>
<Address>
<AddressType>XXX</AddressType>
<AddressLine1>Unit 1-3</AddressLine1>
<AddressLine2>XXX</AddressLine2>
<City>XXX</City>
<Region>West Yorkshire</Region>
<PostalCode>POSTCODE</PostalCode>
<Country>GBR</Country>
</Address>
</Addresses>
<UpdatedDateUTC>2018-10-08T17:19:55.083</UpdatedDateUTC>
<DefaultCurrency>GBP</DefaultCurrency>
</Contact>
<BrandingThemeID>2ffe566f-7a88-486a-938c-639d27966197</BrandingThemeID>
<Status>AUTHORISED</Status>
<LineAmountTypes>Exclusive</LineAmountTypes>
<LineItems>
<LineItem>
<ItemCode>xxx</ItemCode>
<Description>des</Description>
<UnitAmount>57.6100</UnitAmount>
<TaxType>INPUT2</TaxType>
<TaxAmount>11.52</TaxAmount>
<LineAmount>57.61</LineAmount>
<AccountCode>310</AccountCode>
<Quantity>1.0000</Quantity>
<LineItemID>e295d55d-68bd-41b0-a0b1-cf1f2d5b7a4f</LineItemID>
</LineItem>
</LineItems>
<SubTotal>57.61</SubTotal>
<TotalTax>11.52</TotalTax>
<Total>69.13</Total>
<UpdatedDateUTC>2018-10-26T14:19:19.053</UpdatedDateUTC>
<HasAttachments>false</HasAttachments>
</PurchaseOrder>
You can avoid issues with ForceArray and confusing data structures by using an XML parser that returns an object that understands the XML tree. Mojo::DOM is a nice one if you know CSS.
use Mojo::DOM;
my $dom = Mojo::DOM->new->xml(1)->parse($res->decoded_content);
for my $purchase ($dom->find('PurchaseOrders > PurchaseOrder')->each) {
# $purchase is a Mojo::DOM object representing a PurchaseOrder element
for my $item ($purchase->find('LineItems > LineItem')->each) {
# It's unclear if ItemCode is an an attribute or a sub-element; assuming sub-element
my $itemCode = $item->at('ItemCode')->text;
...
}
}
XML::LibXML is another option that can be used similarly but using XPath or DOM instead of CSS to locate elements.
use XML::LibXML qw( );
my $doc = XML::LibXML->load_xml(string => $res->decoded_content);
for my $purchase ($doc->findodes('/PurchaseOrders/PurchaseOrder')) {
# $purchase is a XML::LibXML::Element object representing a PurchaseOrder element
for my $item ($purchase->findnodes('LineItems/LineItem')) {
# It's unclear if ItemCode is an an attribute or a sub-element; assuming sub-element
my $itemCode = $item->findvalue('ItemCode');
...
}
}
Related
I have a Perl script that is successfully getting a response from my ShoreTel Phone server. The server provides information on what calls are currently connected for the extension entered. However I am having issues looping through the sub arrays to get more than one response when there are multiple items. In this case I want to get each of the caller IDs that is currently connected.
My SOAP:LITE request is successfully pulling data from the server using the following code:
use strict;
use warnings;
use SOAP::Lite;
use CGI;
use Data::Dumper;
my $myWebService = SOAP::Lite
-> uri('http://www.ShoreTel.com/ProServices/SDK/Web')
-> proxy('http://10.1.##.##:8070/ShoreTelWebSDK/WebService')
-> on_action(sub {sprintf '%s/ShoreTelWebService/%s', $_[0], $_[1]});
my $query = new CGI;
my $ip = $query->remote_host; # IP address of remote party...use later as unique identifier
my $myClientID = $query->param('MyClientID'); # Possible client ID from previous script passed into us.
my $extnNr = $query->param('MyExtn'); # Has to be at least an extension number so we know who to status.
my $url = CGI::url(-path_info=>1); # What is my URL?
# There should be an extension number given, else what would we status.
if (defined($refreshNr) && defined($extnNr) && ($extnNr ne '') && ($refreshNr ne ''))
{
# If there is a client ID defined, use it...otherwise registering and getting a client ID
# is the first thing we need to do when using our web service.
unless (defined($myClientID))
{
# To use our service, we need to register ourselves as a client...use remote IP address
# as a unique name for association to this session.
my $regClientResult = $myWebService->RegisterClient(SOAP::Data->name('clientName' => $ip));
if ($regClientResult->fault)
{
print '<p>FAULT', $myClientID->faultcode, ', ', $myClientID->faultstring;
}
else
{
# Retrieve client ID which we will be using for subsequent communication.
$myClientID = $regClientResult->valueof('//RegisterClientResponse/RegisterClientResult/');
}
}
if (defined($myClientID))
{
# Use our web service to open the line. This is necessary to get a line ID.
# print '<br>Client ID ', $myClientID, ' has been registered.<br>';
my $openResult = $myWebService->OpenLine(SOAP::Data->name('clientHandle' => $myClientID), SOAP::Data->name('lineAddress' => $extnNr));
my $lineID = $openResult->valueof('//OpenLineResponse/OpenLineResult/lineID/');
my $lineType = $openResult->valueof('//OpenLineResponse/OpenLineResult/lineType/');
my $lineName = $openResult->valueof('//OpenLineResponse/OpenLineResult/lineName/');
my $lineState = $openResult->valueof('//OpenLineResponse/OpenLineResult/lineState/');
# Call GetActiveCalls to see if anything is going on with this line.
my $result = $myWebService->GetActiveCalls(SOAP::Data->name('clientHandle' => $myClientID), SOAP::Data->name('lineID' => $lineID));
my $callID = $result->valueof('//GetActiveCallsResponse/GetActiveCallsResult/ShoreTelCallStateInfo/callInfo/callID/');
if ($callID ne '')
{
# print '<br>Call ID is ', $callID;
my $isExternal = $result->valueof('//GetActiveCallsResponse/GetActiveCallsResult/ShoreTelCallStateInfo/callInfo/isExternal/');
my $isInbound = $result->valueof('//GetActiveCallsResponse/GetActiveCallsResult/ShoreTelCallStateInfo/callInfo/isInbound/');
my $callReason = $result->valueof('//GetActiveCallsResponse/GetActiveCallsResult/ShoreTelCallStateInfo/callInfo/callReason/');
my $connectedID = $result->valueof('//GetActiveCallsResponse/GetActiveCallsResult/ShoreTelCallStateInfo/callInfo/connectedID/');
my $connectedIDName = $result->valueof('//GetActiveCallsResponse/GetActiveCallsResult/ShoreTelCallStateInfo/callInfo/connectedIDName/');
my $callerID = $result->valueof('//GetActiveCallsResponse/GetActiveCallsResult/ShoreTelCallStateInfo/callInfo/callerID/');
my $callerIDName = $result->valueof('//GetActiveCallsResponse/GetActiveCallsResult/ShoreTelCallStateInfo/callInfo/callerIDName/');
my $calledID = $result->valueof('//GetActiveCallsResponse/GetActiveCallsResult/ShoreTelCallStateInfo/callInfo/calledID/');
my $calledIDName = $result->valueof('//GetActiveCallsResponse/GetActiveCallsResult/ShoreTelCallStateInfo/callInfo/calledIDName/');
my $callState = $result->valueof('//GetActiveCallsResponse/GetActiveCallsResult/ShoreTelCallStateInfo/callState/');
my $callStateDetail = $result->valueof('//GetActiveCallsResponse/GetActiveCallsResult/ShoreTelCallStateInfo/callStateDetail/');
# Print call information.
print <<EndOfCallInfo;
HTML CODE
EndOfCallInfo
}
else
{
print <<EndOfCallInfo2;
HTML CODE
EndOfCallInfo2
}
}
}
But I am only able to access the first result in the multidimensional array.
I have tried looping through the results using
for my $t ($result->result({ShoreTelCallStateInfo}{callInfo}')) {
print $t->{callerID} . "\n";}
But I am getting absolutely no results. It appears that the the loop is not even entered.
The following code I have works fine, but only pulls the first caller ID, in this case 1955.
my $callerID = $result->valueof('//GetActiveCallsResponse/GetActiveCallsResult/ShoreTelCallStateInfo/callInfo/callerID/');
What can I do to make my loop work?
So that you can see what I am receiving from the server I have included the response from the SOAP Server using DUMP :
$VAR1 = { 'ShoreTelCallStateInfo' => [
{ 'callStateDetail' => 'Active',
'callState' => 'OnHold',
'callInfo' =>
{ 'callerIDName' => 'Joel LASTNAME',
'callID' => '69105', 'lineID' => '3947',
'connectedIDName' => 'VM-Forward',
'calledID' => '2105',
'callerID' => '1955',
'isInbound' => 'false',
'calledIDName' => 'VM-Forward',
'callReason' => 'None',
'callUniqueID' => '1369702515',
'connectedID' => '2105',
'isExternal' => 'false',
'callGUID' => '{00030000-66C2-537E-3FD8-0010492377D9}'
}
},
{ 'callStateDetail' => 'Active',
'callState' => 'Connected',
'callInfo' =>
{ 'callerIDName' => 'LASTNAME Joel ',
'callID' => '71649',
'lineID' => '3947',
'connectedIDName' => 'LASTNAME Joel ',
'calledID' => '1955',
'callerID' => '+1385#######',
'isInbound' => 'true',
'calledIDName' => 'Joel LASTNAME',
'callReason' => 'None',
'callUniqueID' => '1117287558',
'connectedID' => '+1385#######',
'isExternal' => 'true',
'callGUID' => '{00030000-66C5-537E-3FD8-0010492377D9}'
}
}
]
};
Just a guess...
The following code I have works fine, but only pulls the first caller
ID, in this case 1955.
my $callerID = $result->valueof('//GetActiveCallsResponse/GetActiveCallsResult/ShoreTelCallStateInfo/callInfo/callerID/');
What can I do to make my loop work?
SOAP::Lite docs say:
valueof()
Returns the value of a (previously) matched node. It accepts a node
path. In this case, it returns the value of matched node, but does not
change the current node. Suitable when you want to match a node and
then navigate through node children:
$som->match('/Envelope/Body/[1]'); # match method
$som->valueof('[1]'); # result
$som->valueof('[2]'); # first out parameter (if present)
The returned value depends on the context. In a scalar context it will
return the first element from matched nodeset. In an array context it
will return all matched elements.
Does this give the behavior you expect? It imposes list context on the valueof method.
for my $callerID ($result->valueof('//GetActiveCallsResponse/GetActiveCallsResult/ShoreTelCallStateInfo/callInfo/callerID/')) {
...
# do something with each callerID
}
or
my #callerIDs = $result->valueof('//GetActiveCallsResponse/GetActiveCallsResult/ShoreTelCallStateInfo/callInfo/callerID/');
I am having problems creating an array that I can pass as a form using LWP. Basic code is
my $ua = LWP::UserAgent->new();
my %form = { };
$form->{'Submit'} = '1';
$form->{'Action'} = 'check';
for (my $i=0; $i<1; $i++) {
$form->{'file_'.($i+1)} = [ './test.txt' ];
$form->{'desc_'.($i+1)} = '';
}
$resp = $ua->post('http://someurl/test.php', 'Content_Type' => 'multipart/form-data'
, 'Content => [ \%form ]');
if ($resp->is_success()) {
print "OK: ", $resp->content;
}
} else {
print $claimid->as_string;
}
I guess I am not creating the form array correctly or using the wrong type as when I check the _POST variables in test.php nothing has been set :(
The problem is that for some reason you've enclosed your form values in single quotes. You want to send the data structure. E.g.:
$resp = $ua->post('http://someurl/test.php',
'Content_Type' => 'multipart/form-data',
'Content' => \%form);
You want to either send the hash reference of %form, not the has reference contained within an array reference as you had ([ \%form ]). If you had wanted to send the data as an array reference, then you'd just use[ %form ]` which populates the array with the key/value pairs from the hash.
I'd suggest that you read the documentation for HTTP::Request::Common, the POST section in particular for a cleaner way of doing this.
I decided to use eXist as a database for an application that I am writing in Perl and
I am experimenting with it. The problem is that I have stored a .xml document with the following structure
<foo-bar00>
<perfdata datum="GigabitEthernet3_0_18">
<cli cmd="whatsup" detail="GigabitEthernet3/0/18" find="" given="">
<input_rate>3</input_rate>
<output_rate>3</output_rate>
</cli>
</perfdata>
<timeline>2011-5-23T11:15:33</timeline>
</foo-bar00>
and it is located in the "/db/LAB/foo-bar00/2011/5/23/11_15_33.xml" collection.
I can successfully query it, like
my $xquery = 'doc("/db/LAB/foo-bar00/2011/5/23/11_15_33.xml")' ;
or $xquery can be equal to
= doc("/db/LAB/foo-bar00/2011/5/23/11_15_33.xml")/foo-bar00/perfdata/cli/data(output_rate)
or
= doc("/db/LAB/foo-bar00/2011/5/23/11_15_33.xml")/foo-bar00/data(timeline)
my ($rc1, $set) = $eXist->executeQuery($xquery) ;
my ($rc2, $count) = $eXist->numberOfResults($set) ;
my ($rc3, #data) = $eXist->retrieveResults($set) ;
$eXist->releaseResultSet($set) ;
print Dumper(#data) ;
And the result is :
$VAR1 = {
'hitCount' => 1,
'foo-bar00' => {
'perfdata' => {
'cli' => {
'given' => '',
'detail' => 'GigabitEthernet3/0/18',
'input_rate' => '3',
'cmd' => 'whatsup',
'output_rate' => '3',
'find' => ''
},
'datum' => 'GigabitEthernet3_0_18'
},
'timeline' => '2011-5-23T11:15:33'
}
};
---> Given that I know the xml document that I want to retrieve info from.
---> Given that I want to retrieve the timeline information.
When I am writing :
my $db_xml_doc = "/db/LAB/foo-bar00/2011/5/23/11_15_33.xml" ;
my ($db_rc, $db_datum) = $eXist->queryXPath("/foo-bar00/timeline", $db_xml_doc, "") ;
print Dumper($db_datum) ;
The result is :
$VAR1 = {
'hash' => 1717362942,
'id' => 3,
'results' => [
{
'node_id' => '1.2',
'document' => '/db/LAB/foo-bar00/2011/5/23/11_15_33.xml'
}
]
};
The question is : How can I retrieve the "timeline" info ? Seems that the "node_id" variable (=1.2) can points to the "timeline" info, but how can I use it ?
Thank you.
use XML::LibXML qw( );
my $parser = XML::LibXML->new();
my $doc = $parser->parse_file('a.xml');
my $root = $doc->documentElement();
my ($timeline) = $root->findnodes('timeline');
if ($timeline) {
print("Exists: ", $timeline->textContent(), "\n");
}
or
my ($timeline) = $root->findnodes('timeline/text()');
if ($timeline) {
print("Exists: ", $timeline->getValue(), "\n");
}
I could have used /foo-bar00/timeline instead of timeline, but I didn't see the need.
Don't know if you're still interested, but you could either retrieve the doc as DOM and apply an xquery to the DOM, or, probably better, only pull out the info you want in the query that you submit to the server.
Something like this:
for $p in doc("/db/LAB/foo-bar00/2011/5/23/11_15_33.xml")//output_rate
return
<vlaue>$p</value>
I have the following script that scrapes my schools CS department to get a list of all the courses. I want to be able to extract the CRN (course number) and other important information to put into a database which I can let users browse through a web app.
Here is an example URL:
http://courses.illinois.edu/cis/2011/spring/schedule/CS/411.html
I would like to extract info from pages like this. The first level of the scraper just constructs the individual sites from a list of all of the courses. Once I'm at a course specific catalog page, I use the second scraper to attempt to get all of this info i want. For some reason, although the CRN's and Course Instructors are all 'td' elements. My scraper seems to be returning nothing when scraping. I tried to scrape specifically for 'div' instead and I get a bunch of info for each relevant page. So somehow I'm failing to get the 'td' element, but I'm scraping from the right page.
my $tweets = scraper {
# Parse all LIs with the class "status", store them into a resulting
# array 'tweets'. We embed another scraper for each tweet.
# process "h4.ws-ds-name.detail-title", "array[]" => 'TEXT';
process "div.ws-row", "array[]" => 'TEXT';
};
my $res = $tweets->scrape( URI- >new("http://courses.illinois.edu/cis/2011/spring/schedule/CS/index.html?skinId=2169") );
foreach my $elem (#{$res->{array}}){
my $coursenum = substr($elem,2,4);
my $secondLevel = scraper{
process "td.ws-row", "array2[]" => 'TEXT';
};
my $res2 = $secondLevel->scrape(URI- >new("http://courses.illinois.edu/cis/2011/spring/schedule/CS/$coursenum.html"));
my $num = #{$res2->{array2}};
print $num;
print "---------------------", "\n";
my #curr = #{$res2->{array2}};
foreach my $elem2 (#curr){
$num++;
print $elem2, " ", "\n";
}
print "---------------------", "\n";
}
Any ideas?
Thanks
Looks to me like
my $coursenum = substr($elem,2,4)
should be
my $coursenum = substr($elem,3,3)
The easiest way to go in this case is use
HTML::TableExtract
In case you are looking for data from the table only.
I played a bit with your problem. You can get course id, title and link to individual course page within initial scraper:
my $courses = scraper {
process 'div.ws-row',
'course[]' => scraper {
process 'div.ws-course-number', 'id' => 'TEXT';
process 'div.ws-course-title', 'title' => 'TEXT';
process 'div.ws-course-title a', 'link' => '#href';
};
result 'course';
};
The result of scraping is arrayref with hashrefs like this:
{ id => "CS 103",
title => "Introduction to Programming",
link => bless(do{\(my $o = "http://courses.illinois.edu/cis/2011/spring/schedule/CS/103.html?skinId=2169")}, "URI::http"),
},
....
Then you can do additional scraping for each course from their individual pages and add such information into original structure:
for my $course (#$res) {
my $crs_scraper = scraper {
process 'div.ws-description', 'desc' => 'TEXT';
# ... add more items here
};
my $additional_data = $crs_scraper->scrape(URI->new($course->{link}));
# slice assignment to add them into course definition
#{$course}{ keys %$additional_data } = values %$additional_data;
}
Source combined together is as follows:
use strict; use warnings;
use URI;
use Web::Scraper;
use Data::Dump qw(dump);
my $url = 'http://courses.illinois.edu/cis/2011/spring/schedule/CS/index.html?skinId=2169';
my $courses = scraper {
process 'div.ws-row',
'course[]' => scraper {
process 'div.ws-course-number', 'id' => 'TEXT';
process 'div.ws-course-title', 'title' => 'TEXT';
process 'div.ws-course-title a', 'link' => '#href';
};
result 'course';
};
my $res = $courses->scrape(URI->new($url));
for my $course (#$res) {
my $crs_scraper = scraper {
process 'div.ws-description', 'desc' => 'TEXT';
# ... add more items here
};
my $additional_data = $crs_scraper->scrape(URI->new($course->{link}));
# slice assignment to add them into course definition
#{$course}{ keys %$additional_data } = values %$additional_data;
}
dump $res;
Can anybody please show me an example for generating XML from XSD using XML::Compile::Schema.
I am trying to post my script which I am trying along with the XSD but I am not able to do that. so I am looking for a any sample example.
I wrote a tutorial on this a while ago: http://blogs.perl.org/users/brian_e_lozier/2011/10/using-xmlcompile-to-output-xsd-compliant-xml.html
In short words, you'll need to do:
Convert the XSD format to Perl hash structure
Construct this Hash, fill in the data
Convert the Hash to XML
Packages required:
XML::Compile::Schema
XML::LibXML::Document
Following code create a Perl structure from XSD definition.
use XML::Compile::Schema;
use Data::Dumper;
my $filename = $ARGV[0] || "";
if(!$filename){
warn "Please provide the WSDL definition file.";
exit 10;
}
my $schema = XML::Compile::Schema->new($filename);
my $hash;
print Dumper $schema->template('PERL' => 'Application');
Then the Perl data structure created by this program looks like:
{
MakeName =>
{
UniqueID => "anything",
_ => "example", },
MakeDetails =>
{
Name =>
{
UniqueID => "anything",
_ => "example", },
},
};
So the rest of your job will create the same structure in your program, fill in the content like:
my $hash = {
MakeName => {
UniqueID => 'xxxx',
_ => 'Name of the Make',
},
OtherFields => foo_bar_get_other_hash(),
};
....
## breathtaking moment, create the XML from this $hash
my $schema = XML::Compile::Schema->new("/opt/data/your.xsd");
my $doc = XML::LibXML::Document->new();
my $writer = $schema->compile(WRITER => 'Application');
my $xml;
## Create $xml in the memory based on the Schema and your $hash
eval{ $xml = $writer->($doc, $hash);};
if($#){
# Useful if the format is invalid against the Schema definition
# Or if there are other errors may occurs
$err_msg = $#->{message}->toString();
return ("", $err_msg);
}
## If you want save this $xml to file, convert it to string format first
$doc->setDocumentElement($xml);
my $ori_content = $doc->toString(1);
## Now $ori_content holds the full XML content.