Perl WWW:Mechanize Accessing data in a hash reference - perl

I have a question I'm hoping you could help with?
This is the last part I need help with in understanding hash references
Code:
my $content_lengths; # this is at the top
foreach my $url ( # ... more stuff
# compare
if ( $mech->response->header('Content-Length') != $content_length ) {
print "$child_url: different content length: $content_length vs "
. $mech->response->header('Content-Length') . "!\n";
# store the urls that are found to have different content
# lengths to the base url only if the same url has not already been stored
$content_lengths->{$url}->{'different'}->{$child_url} = $mech->response->header('Content-Length');
} elsif ( $mech->response->header('Content-Length') == $content_length ) {
print "Content lengths are the same\n";
# store the urls that are found to have the same content length as the base
# url only if the same url has not already been stored
$content_lengths->{$url}->{'equal'}->{$child_url} = $mech->response->header('Content-Length');
}
What it looked like using Data::Dumper
$VAR1 = {
'http://www.superuser.com/' => {
'difference' => {
'http://www.superuser.com/questions' => '10735',
'http://www.superuser.com/faq' => '13095'
},
'equal' => {
'http://www.superuser.com/ ' => '20892'
}
},
'http://www.stackoverflow.com/' => {
'difference' => {
'http://www.stackoverflow.com/faq' => '13015',
'http://www.stackoverflow.com/questions' => '10506'
},
'equal' => {
'http://www.stackoverflow.com/ ' => '33362'
}
}
};
What I need help with:
I need help understanding the various ways of accessing the different parts in the hash reference and using them to do things, such as print them.
So for example how do I print all the $url from the hash reference (i.e from Data::Dumper that will be http://www.superuser.com/ and http://www.stackoverflow.com/)
and how do I print all the $child_url or a particular one/subset from $child_url and so on?
Your help with this is much appreciated,
thanks a lot

You can navigate your hashref thusly:
$hashref->{key1}{key2}{keyN};
For example, if you want the superuser equal branch:
my $urlArrayref = $hashref->{'http://www.superuser.com/'}{'equal'};
More to the point, to print the urls (first level key) of the hashref, you would do:
foreach my $key ( keys( %{$hashref} ) ) {
print( "key is '$key'\n" );
}
Then if you wanted the second level keys:
foreach my $firstLevelKey ( keys( %{$hashref} ) ) {
print( "first level key is '$firstLevelKey'\n" );
foreach my $secondLevelKey ( keys( %{$hashref->{$firstLevelKey}} ) ) {
print( "\tfirst level key is '$secondLevelKey'\n" );
}
}
And so forth...
----- EDIT -----
This is working sample code from your example above:
#!/usr/bin/perl
use strict;
use warnings;
my $content_lengths = {
'http://www.superuser.com/' => {
'difference' => {
'http://www.superuser.com/questions' => '10735',
'http://www.superuser.com/faq' => '13095'
},
'equal' => {
'http://www.superuser.com/ ' => '20892'
}
},
'http://www.stackoverflow.com/' => {
'difference' => {
'http://www.stackoverflow.com/faq' => '13015',
'http://www.stackoverflow.com/questions' => '10506'
},
'equal' => {
'http://www.stackoverflow.com/ ' => '33362'
}
}
};
foreach my $key1 ( keys( %{$content_lengths} ) ) {
print( "$key1\n" );
foreach my $key2 ( keys( %{$content_lengths->{$key1}} ) ) {
print( "\t$key2\n" );
foreach my $key3 ( keys( %{$content_lengths->{$key1}{$key2}} ) ) {
print( "\t\t$key3\n" );
}
}
}
Which results in this output:
http://www.superuser.com/
difference
http://www.superuser.com/questions
http://www.superuser.com/faq
equal
http://www.superuser.com/
http://www.stackoverflow.com/
difference
http://www.stackoverflow.com/faq
http://www.stackoverflow.com/questions
equal
http://www.stackoverflow.com/

Related

Grouping with Perl: finding a faster solution to recursion

The Perl code below works, but it doesn't scale well even with considerable computer resources. I hoping that someone can help me find more efficient code such as by replacing recursion with iteration, if that's the problem.
my data structure looks like this:
my %REV_ALIGN;
$REV_ALIGN{$dna}{$rna} = ();
Any dna key may have multiple rna sub keys. The same rna sub key may appear with multiple different dna keys. The purpose is to group rna ( transcripts ) based on shared dna sequence elements. For example, if dnaA has RNA1, RNA8, RNA9, and RNA4, and dnaB has RNA11, RNA4, and RNA99, then we group all these transcripts together ( RNA1, RNA9, RNA4, RNA11, RNA99 ) and continue to proceed to try and add to the group by selecting other dna. My recusive solution to this problem works but doesn't scale so well when using data from whole genome to transcriptome alignment.
SO MY QUESTION IS: WHAT IS A MORE EFFICIENT SOLUTION TO THIS PROBLEM? THANK YOU VERY MUCH
my #groups;
while ( my $x =()= keys %REV_ALIGN )
{
my #DNA = keys %REV_ALIGN;
my $dna = shift #DNA;
# the corresponding list of rna
my #RNA = keys %{$REV_ALIGN{$dna}};
delete $REV_ALIGN{$dna};
if ( $x == 1 )
{
push #groups, \#RNA;
last;
}
my $ref = group_transcripts ( \#RNA, \%REV_ALIGN );
push #groups, $ref;
}
sub group_transcripts
{
my $tran_ref = shift;
my $align_ref = shift;
my #RNA_A = #$tran_ref;
my %RNA;
# create a null hash with seed list of transcripts
#RNA{#RNA_A} = ();
# get a list of all remaining dna sequences in the alignment
my #DNA = keys %{$align_ref};
my %count;
# select a different list of transcripts
for my $dna ( #DNA )
{
next unless exists $align_ref->{$dna};
my #RNA_B = keys %{$align_ref->{$dna}};
# check to see two list share and transcripts
for my $element ( #RNA_A, #RNA_B )
{
$count{$element}++;
}
for my $rna_a ( keys %count )
{
# if they do, add any new transcripts to the current group
if ( $count{$rna_a} == 2 )
{
for my $rna_b ( #RNA_B )
{
push #RNA_A, $rna_b if $count{$rna_b} == 1;
}
delete $align_ref->{$dna};
delete $count{$_} foreach keys %count;
# recurse to try and continue adding to list
#_ = ( \#RNA_A, $align_ref );
goto &group_transcripts;
}
}
delete $count{$_} foreach keys %count;
}
# if no more transcripts can be added, return a reference to the group
return \#RNA_A;
}
You have a loops nested four deep. It's an pretty safe bet that's why your code scales poorly.
If I understand correctly what you are trying to accomplish, the input
my %REV_ALIGN = (
"DNA1" => { map { $_ => undef } "RNA1", "RNA2" }, # \ Linked by RNA1 \
"DNA2" => { map { $_ => undef } "RNA1", "RNA3" }, # / \ Linked by RNA3 > Group
"DNA3" => { map { $_ => undef } "RNA3", "RNA4" }, # / /
"DNA4" => { map { $_ => undef } "RNA5", "RNA6" }, # \ Linked by RNA5 \ Group
"DNA5" => { map { $_ => undef } "RNA5", "RNA7" }, # / /
"DNA6" => { map { $_ => undef } "RNA8" }, # > Group
);
should result in
my #groups = (
[
dna => [ "DNA1", "DNA2", "DNA3" ],
rna => [ "RNA1", "RNA2", "RNA3", "RNA4" ],
],
[
dna => [ "DNA4", "DNA5" ],
rna => [ "RNA5", "RNA6", "RNA7" ],
],
[
dna => [ "DNA6" ],
rna => [ "RNA8" ],
],
);
If so, you can use the following:
use strict;
use warnings;
use Graph::Undirected qw( );
my %REV_ALIGN = (
"DNA1" => { map { $_ => undef } "RNA1", "RNA2" },
"DNA2" => { map { $_ => undef } "RNA1", "RNA3" },
"DNA3" => { map { $_ => undef } "RNA3", "RNA4" },
"DNA4" => { map { $_ => undef } "RNA5", "RNA6" },
"DNA5" => { map { $_ => undef } "RNA5", "RNA7" },
"DNA6" => { map { $_ => undef } "RNA8" },
);
my $g = Graph::Undirected->new();
for my $dna (keys(%REV_ALIGN)) {
for my $rna (keys(%{ $REV_ALIGN{$dna} })) {
$g->add_edge("dna:$dna", "rna:$rna");
}
}
my #groups;
for my $raw_group ($g->connected_components()) {
my %group = ( dna => [], rna => [] );
for (#$raw_group) {
my ($type, $val) = split(/:/, $_, 2);
push #{ $group{$type} }, $val;
}
push #groups, \%group;
}
use Data::Dumper qw( Dumper );
print(Dumper(\#groups));
If you just want the RNA, the final section simplifies to the following:
my #groups;
for my $raw_group ($g->connected_components()) {
my #group;
for (#$raw_group) {
my ($type, $val) = split(/:/, $_, 2);
push #group, $val if $type eq 'rna';
}
push #groups, \#group;
}

Perl accessing the elements in a hash/hash reference data structure

I have a question I'm hoping you could help with as I am new to hashes and hash reference stuff?
I have the following data structure:
$VAR1 = {
'http://www.superuser.com/' => {
'difference' => {
'http://www.superuser.com/questions' => '10735',
'http://www.superuser.com/faq' => '13095'
},
'equal' => {
'http://www.superuser.com/ ' => '20892'
}
},
'http://www.stackoverflow.com/' => {
'difference' => {
'http://www.stackoverflow.com/faq' => '13015',
'http://www.stackoverflow.com/questions' => '10506'
},
'equal' => {
'http://www.stackoverflow.com/ ' => '33362'
}
}
If I want to access all the URLs in the key 'difference' so I can then perform some other actions on the URLs, what is the correct or preferred method of accessing those elements?
e.g I will end up with the following URLs that I can then do stuff to in a foreach loop with:
http://www.superuser.com/questions
http://www.superuser.com/faq
http://www.stackoverflow.com/faq
http://www.stackoverflow.com/questions
------EDIT------
Code to access the elements further down the data structure shown above:
my #urls;
foreach my $key1 ( keys( %{$VAR1} ) ) {
print( "$key1\n" );
foreach my $key2 ( keys( %{$VAR1->{$key1}} ) ) {
print( "\t$key2\n" );
foreach my $key3 ( keys( %{$VAR1->{$key1}{$key2}} ) ) {
print( "\t\t$key3\n" );
push #urls, keys %{$VAR1->{$key1}{$key2}{$key3}};
}
}
}
print "#urls\n";
Using the code above why do I get the following error?
Can't use string ("13238") as a HASH ref while "strict refs" in use at ....
It is not difficult, just take the second level of keys off every key in the variable:
my #urls;
for my $key (keys %$VAR1) {
push #urls, keys %{$VAR1->{$key}{'difference'}};
}
If you're struggling with dereferencing, just keep in mind that all values in a hash or array can only be a scalar value. In a multilevel hash or array the levels are just single hashes/arrays stacked on top of each other.
For example, you could do:
for my $value (values %$VAR1) {
push #urls, keys %{$value->{'difference'}};
}
Or
for my $name (keys %$VAR1) {
my $site = $VAR1->{$name};
push #urls, keys %{$site->{'difference'}};
}
..taking the route either directly over the value (a reference to a hash) or over a temporary variable, representing the value via the key. There is more to read in perldoc perldata.

Perl Hash of Hashes

I am new to Perl, and I am trying something with a hash. I have a hash of hashes like below:
%HoH =
(
"Test1" => { checked => 1, mycmd => run1 },
"Test2" => { checked => 1, mycmd => run2 },
)
Using the below code I will get the output given below:
for $family ( keys %HoH )
{
print "$family: ";
for $role ( keys %{ $HoH{$family} } )
{
print "$role=$HoH{$family}{$role} ";
}
print "\n";
}
Output:
Test1: checked=1 mycmd=run1
Test2: checked=1 mycmd=run2
My question is, how can I access individual checked & cmd separately? By accessing separately, I can compare what is checked and do my task.
It's pretty straight-forward to just use the keyword(s) literally:
%HoH =
(
"Test1" => { checked => 1, cmd => run1 },
"Test2" => { checked => 1, cmd => run2 },
);
if ($HoH{"Test1"}{checked}) {
print "Test1 is Checked with cmd: " . $HoH{"Test1"}{cmd} . "\n";
}
Test1 is Checked with cmd: run1
Did I understand your question correctly?
for my $family ( keys %HoH )
{
if ($HoH{$family}->{checked}) {
# Do what you want...
}
}

Perl Working On Two Hash References

I would like to compare the values of two hash references.
The data dumper of my first hash is this:
$VAR1 = {
'42-MG-BA' => [
{
'chromosome' => '19',
'position' => '35770059',
'genotype' => 'TC'
},
{
'chromosome' => '2',
'position' => '68019584',
'genotype' => 'G'
},
{
'chromosome' => '16',
'position' => '9561557',
'genotype' => 'G'
},
And the second hash is similar to this but with more hashes in the array. I would like to compare the genotype of my first and second hash if the position and the choromosome matches.
map {print "$_= $cave_snp_list->{$_}->[0]->{chromosome}\n"}sort keys %$cave_snp_list;
map {print "$_= $geno_seq_list->{$_}->[0]->{chromosome}\n"}sort keys %$geno_seq_list;
I could do that for the first array of the hashes.
Could you help me in how to work for all the arrays?
This is my actual code in full
#!/software/bin/perl
use strict;
use warnings;
use Getopt::Long;
use Benchmark;
use Config::Config qw(Sequenom.ini);
useDatabase::Conn;
use Data::Dumper;
GetOptions("sam=s" => \my $sample);
my $geno_seq_list = getseqgenotypes($sample);
my $cave_snp_list = getcavemansnpfile($sample);
#print Dumper($geno_seq_list);
print scalar %$geno_seq_list, "\n";
foreach my $sam (keys %{$geno_seq_list}) {
my $seq_used = $geno_seq_list->{$sam};
my $cave_used = $cave_snp_list->{$sam};
print scalar(#$geno_seq_list->{$_}) if sort keys %$geno_seq_list, "\n";
print scalar(#$cave_used), "\n";
#foreach my $seq2com (# {$seq_used } ){
# foreach my $cave2com( # {$cave_used} ){
# print $seq2com->{chromosome},":" ,$cave2com->{chromosome},"\n";
# }
#}
map { print "$_= $cave_snp_list->{$_}->[0]->{chromosome}\n" } sort keys %$cave_snp_list;
map { print "$_= $geno_seq_list->{$_}->[0]->{chromosome}\n" } sort keys %$geno_seq_list;
}
sub getseqgenotypes {
my $snpconn;
my $gen_list = {};
$snpconn = Database::Conn->new('live');
$snpconn->addConnection(DBI->connect('dbi:Oracle:pssd.world', 'sn', 'ss', { RaiseError => 1, AutoCommit => 0 }),
'pssd');
#my $conn2 =Database::Conn->new('live');
#$conn2->addConnection(DBI->connect('dbi:Oracle:COSI.world','nst_owner','nst_owner', {RaiseError =>1 , AutoCommit=>0}),'nst');
my $id_ind = $snpconn->execute('snp::Sequenom::getIdIndforExomeSample', $sample);
my $genotype = $snpconn->executeArrRef('snp::Sequenom::getGenotypeCallsPosition', $id_ind);
foreach my $geno (#{$genotype}) {
push #{ $gen_list->{ $geno->[1] } }, {
chromosome => $geno->[2],
position => $geno->[3],
genotype => $geno->[4],
};
}
return ($gen_list);
} #end of sub getseqgenotypes
sub getcavemansnpfile {
my $nstconn;
my $caveman_list = {};
$nstconn = Database::Conn->new('live');
$nstconn->addConnection(
DBI->connect('dbi:Oracle:CANP.world', 'nst_owner', 'NST_OWNER', { RaiseError => 1, AutoCommit => 0 }), 'nst');
my $id_sample = $nstconn->execute('nst::Caveman::getSampleid', $sample);
#print "IDSample: $id_sample\n";
my $file_location = $nstconn->execute('nst::Caveman::getCaveManSNPSFile', $id_sample);
open(SNPFILE, "<$file_location") || die "Error: Cannot open the file $file_location:$!\n";
while (<SNPFILE>) {
chomp;
next if /^>/;
my #data = split;
my ($nor_geno, $tumor_geno) = split /\//, $data[5];
# array of hash
push #{ $caveman_list->{$sample} }, {
chromosome => $data[0],
position => $data[1],
genotype => $nor_geno,
};
} #end of while loop
close(SNPFILE);
return ($caveman_list);
}
The problem that I see is that you're constructing a tree for generic storage of data, when what you want is a graph, specific to the task. While you are constructing the record, you could also be constructing the part that groups data together. Below is just one example.
my %genotype_for;
my $record
= { chromosome => $data[0]
, position => $data[1]
, genotype => $nor_geno
};
push #{ $gen_list->{ $geno->[1] } }, $record;
# $genotype_for{ position }{ chromosome }{ name of array } = genotype code
$genotype_for{ $data[1] }{ $data[0] }{ $sample } = $nor_geno;
...
return ( $caveman_list, \%genotype_for );
In the main line, you receive them like so:
my ( $cave_snp_list, $geno_lookup ) = getcavemansnpfile( $sample );
This approach at least allows you to locate similar position and chromosome values. If you're going to do much with this, I might suggest an OO approach.
Update
Assuming that you wouldn't have to store the label, we could change the lookup to
$genotype_for{ $data[1] }{ $data[0] } = $nor_geno;
And then the comparison could be written:
foreach my $pos ( keys %$small_lookup ) {
next unless _HASH( my $sh = $small_lookup->{ $pos } )
and _HASH( my $lh = $large_lookup->{ $pos } )
;
foreach my $chrom ( keys %$sh ) {
next unless my $sc = $sh->{ $chrom }
and my $lc = $lh->{ $chrom }
;
print "$sc:$sc";
}
}
However, if you had limited use for the larger list, you could construct the specific case
and pass that in as a filter when creating the longer list.
Thus, in whichever loop creates the longer list, you could just go
...
next unless $sample{ $position }{ $chromosome };
my $record
= { chromosome => $chromosome
, position => $position
, genotype => $genotype
};
...

How can I force list context in Template Toolkit with RDBO?

I have a TT plugin that does the trivial unique ids:
sub get_unique_uid_tt {
my ( $classname, $o ) = #_;
my %h;
foreach my $item ( #{$o} ) {
unless ( exists $h{ $item->id } ) {
$h{ $item->id } = 1;
}
}
return keys %h;
}
where the template call is simply:
[% Namespace.get_unique_uid_tt( data.users ) %]
and "data" is an RDB Object, users being one of its relationships. I have verified that the ".users" returns a list in Perl directly, whether the relationship has one or many elements.
However, it appears that TT returns the element for single-element lists, while properly returning lists for multiple element.
I looked this up and found that you can force list context with ".list":
[% Namespace.get_unique_uid_tt( data.users.list ) %]
This does not work as intended for single-element lists, as a Data::Dumper revealed:
$VAR1 = [
{
'value' => 1,
'key' => '__xrdbopriv_in_db'
},
{
'value' => bless(
... snip ...
),
'key' => 'db'
},
{
'value' => '1',
'key' => 'id'
}
];
instead of the expected
$VAR1 = [
bless( {
'__xrdbopriv_in_db' => 1,
'id' => '1',
'db' => ... snip ...
}, 'DataClass' )
];
Is there any other simple way in TT to get a list of objects, even on single-element lists? (One approach is to rewrite the function, but one that does not would be preferable)
Found this on the TT mailing list:
http://lists.template-toolkit.org/pipermail/templates/2009-December/011061.html
seems like TT's ".list" has trouble converting objects to lists in general, not just RDBOs.
The suggestion is make a vmethod:
$Template::Stash::LIST_OPS->{ as_list } = sub {
return ref( $_[0] ) eq 'ARRAY' ? shift : [shift];
};
I added this to my context object (same idea):
$context->define_vmethod(
'list',
'as_list',
sub {
return ref( $_[0] ) eq 'ARRAY' ? shift : [shift];
},
);
It's not quite what you're after, but could you alter the TT plugin to handle both lists and single items?
sub get_unique_uid_tt {
my ( $classname, $o ) = #_;
my %h;
if (ref $o eq 'ARRAY') {
foreach my $item ( #{$o} ) {
unless ( exists $h{ $item->id } ) {
$h{ $item->id } = 1;
}
}
}
else {
return ($o->id);
}
return keys %h;
}