parsing rss feed, description field - perl

Question

I use SimpleXml in perl to extract data in tag
<description><strong>CUSIP:</strong> 912828UC2<br /><strong>Term and Type:</strong> 3-Year Note<br /><strong>Offering Amount:</strong> $32,000,000,000<br /><strong>Auction Date:</strong> 12/11/2012<br /><strong>Issue Date:</strong> 12/17/2012<br /><strong>Maturity Date:</strong> 12/15/2015<br /><a href="http://www.treasurydirect.gov/instit/annceresult/press/preanre/2012/A_20121206_6.pdf">PDF version of the announcement</a><br /><a href="http://www.treasurydirect.gov/xml/A_20121206_6.xml">XML version of the announcement</a><br /></description>
I now have trouble extracting individual symbols. For example for Auction Date, use
if ($desc=~m/Auction\sDate:<\/strong>\s+(\d\d\/\d\d\/\d\d\d\d)<br/)
{
}
but I feel it's not robust enough. What is the standard way to extract fields?

Answer 1

As Dan1111 points out in his answer, if you're already using an XML parser (Simple::XML?) you should stick with it for parsing the data within your description tags. It isn't a good idea to attempt to parse data out of an XML/HTML feed; use a parser built for that purpose.
Because of the formatting of the data in your post, I am assuming that you don't have valid HTML that a parser can help you with. In this case, there's no 'standard' way to extract fields, but here's the way I'd approach this problem:
print "$desc\n";
my #parts = split(/;br /, $desc);
my %dates;
foreach my $part (#parts) {
if ($part =~ m/(\w+\s+Date).+(\d{2}\/\d{2}\/\d{4})/) {
$dates{$1} = $2;
}
}
foreach my $label (keys %dates) {
printf "%-16s%12s\n", "${label}:", $dates{$label};
}
Looking at the original string, I can see that there are 3 dates, and several other records, so the first thing to do is to split them up. I found that each record in the string is delimited by the characters ';br ', so I used that for the split:
my #parts = split(/;br /, $desc);
After doing that, I have an array that contains each of the different data parts from your string. now, I just needed to parse each part. because your question is interested in the Auction Date value, I wrote a regular expression that will capture the date. anticipating that the other dates might be valuable as well, I modified my regex so that I could capture the label (Auction, Issue, Maturity), and I stored each label-date pair in a hash (%dates) :
foreach my $part (#parts) {
if ($part =~ m/(\w+\s+Date).+(\d{2}\/\d{2}\/\d{4})/) {
$dates{$1} = $2;
}
}
Finally, I just printed out my hash:
foreach my $part (#parts) {
if ($part =~ m/(\w+\s+Date).+(\d{2}\/\d{2}\/\d{4})/) {
$dates{$1} = $2;
}
}
Make sense?

Answer 2

What is more robust depends on your expected input and what you are looking for. However, here is something that you might find helpful.
I used XML::Twig for this. XML::Simple (which I assume is what you are using now) is not recommended for new development due to various quirks.
use Modern::Perl;
use XML::Twig;
my $twig = XML::Twig->new();
$twig->parse(<DATA>);
my %params;
my $key;
for my $child (map {$_->text} $twig->root->children)
{
if ($child =~ /(.*):/)
{
$key = $1;
}
else
{
$params{$key} = $child if (defined $key);
undef $key;
}
}
say "$_ is $params{$_}" foreach (keys %params);
__DATA__
<description><strong>CUSIP:</strong> 912828UC2<br /><strong>Term and Type:</strong> 3-Year Note<br /><strong>Offering Amount:</strong> $32,000,000,000<br /><strong>Auction Date:</strong> 12/11/2012<br /><strong>Issue Date:</strong> 12/17/2012<br /><strong>Maturity Date:</strong> 12/15/2015<br />PDF version of the announcement<br />XML version of the announcement<br /></description>
This takes any element that ends with a colon as a key, then assumes the next element in the tree is the value. Obviously that makes some assumptions of what kind of input you will get, but it is pretty robust as long as all of the "key" elements will be enclosed in tags.
Another approach would be stripping out all of the tags first, then searching for key-value pairs in just the text. You can do this with XML::Twig as well; simply calling $twig->root->text will get the text from the entire element. However, in this approach it would be tricky to determine where one key ends and another value begins.

Answer 3

The <description> elements in the RSS feed you show contain valid XHTML fragments as PCDATA. This solution extracts those elements and decodes them, and parses them in turn to access the text of the <strong> elements and their corresponding values.
Note that the XHTML contains multiple elements, and as XHTML is allowed only a single root element I have wrapped it in a dummy <root> elements in $twig->parse("<root>$desc</root>").
Hopefully you will be able to extrapolate from this to access the data you require.
use strict;
use warnings;
use LWP::Simple;
use XML::Twig;
my $xml = get 'http://www.treasurydirect.gov/RI/TreasuryOfferingAnnouncements.rss';
my $twig = XML::Twig->new;
$twig->parse($xml);
for my $desc ($twig->get_xpath('/rss/channel/item/description')) {
$desc = $desc->text;
my $twig = XML::Twig->new;
$twig->parse("<root>$desc</root>");
for my $strong ($twig->get_xpath('/root/strong')) {
my ($key, $val) = ($strong->trimmed_text, $strong->next_sibling->trimmed_text);
$key =~ s/:$//;
print "$key => $val\n";
}
print "\n";
}
output
CUSIP -> 912810QY7
Term and Type -> 29-Year 11-Month Bond
Offering Amount -> $13,000,000,000
Auction Date -> 12/13/2012
Issue Date -> 12/17/2012
Maturity Date -> 11/15/2042
CUSIP -> 912796DT3
Term and Type -> 3-Day Bill
Offering Amount -> $10,000,000,000
Auction Date -> 12/13/2012
Issue Date -> 12/14/2012
Maturity Date -> 12/17/2012
CUSIP -> 912828UE8
Term and Type -> 5-Year Note
Offering Amount -> $35,000,000,000
Auction Date -> 12/18/2012
Issue Date -> 12/31/2012
Maturity Date -> 12/31/2017
CUSIP -> 912828UD0
Term and Type -> 2-Year Note
Offering Amount -> $35,000,000,000
Auction Date -> 12/17/2012
Issue Date -> 12/31/2012
Maturity Date -> 12/31/2014
CUSIP -> 912796AM1
Term and Type -> 26-Week Bill
Offering Amount -> $28,000,000,000
Auction Date -> 12/17/2012
Issue Date -> 12/20/2012
Maturity Date -> 06/20/2013
CUSIP -> 912828UF5
Term and Type -> 7-Year Note
Offering Amount -> $29,000,000,000
Auction Date -> 12/19/2012
Issue Date -> 12/31/2012
Maturity Date -> 12/31/2019
CUSIP -> 912828SQ4
Term and Type -> 4-Year 4-Month TIPS
Offering Amount -> $14,000,000,000
Auction Date -> 12/20/2012
Issue Date -> 12/31/2012
Maturity Date -> 04/15/2017
CUSIP -> 9127957M7
Term and Type -> 13-Week Bill
Offering Amount -> $32,000,000,000
Auction Date -> 12/17/2012
Issue Date -> 12/20/2012
Maturity Date -> 03/21/2013
CUSIP -> 912828TY6
Term and Type -> 9-Year 11-Month Note
Offering Amount -> $21,000,000,000
Auction Date -> 12/12/2012
Issue Date -> 12/17/2012
Maturity Date -> 11/15/2022
CUSIP -> 912828UC2
Term and Type -> 3-Year Note
Offering Amount -> $32,000,000,000
Auction Date -> 12/11/2012
Issue Date -> 12/17/2012
Maturity Date -> 12/15/2015
CUSIP -> 912796AK5
Term and Type -> 52-Week Bill
Offering Amount -> $25,000,000,000
Auction Date -> 12/11/2012
Issue Date -> 12/13/2012
Maturity Date -> 12/12/2013
CUSIP -> 9127955V9
Term and Type -> 4-Week Bill
Offering Amount -> $40,000,000,000
Auction Date -> 12/11/2012
Issue Date -> 12/13/2012
Maturity Date -> 01/10/2013
CUSIP -> 912796AL3
Term and Type -> 26-Week Bill
Offering Amount -> $28,000,000,000
Auction Date -> 12/10/2012
Issue Date -> 12/13/2012
Maturity Date -> 06/13/2013
CUSIP -> 9127957L9
Term and Type -> 13-Week Bill
Offering Amount -> $32,000,000,000
Auction Date -> 12/10/2012
Issue Date -> 12/13/2012
Maturity Date -> 03/14/2013
CUSIP -> 912796DT3
Term and Type -> 11-Day Bill
Offering Amount -> $25,000,000,000
Auction Date -> 12/04/2012
Issue Date -> 12/06/2012
Maturity Date -> 12/17/2012
CUSIP -> 9127956Z9
Term and Type -> 4-Week Bill
Offering Amount -> $40,000,000,000
Auction Date -> 12/04/2012
Issue Date -> 12/06/2012
Maturity Date -> 01/03/2013