Using perl hash for comparing columns of 2 files - perl

Question

I have asked this question(sorry for asking again, this time it is different and difficult) but I have tried a lot but did not achieve the results.
I have 2 big files (tab delimited).
first file ->
Col1 Col2 Col3 Col4 Col5 Col6 Col7 Col8
101_#2 1 H F0 263 278 2 1.5
102_#1 1 6 F1 766 781 1 1.0
103_#1 2 15 V1 526 581 1 0.0
103_#1 2 9 V2 124 134 1 1.3
104_#1 1 12 V3 137 172 1 1.0
105_#1 1 17 F2 766 771 1 1.0
second file ->
Col1 Col2 Col3 Col4
97486 H 262 279
67486 9 118 119
87486 9 183 185
248233 9 124 134
If col3 value/character (of file1) and col2 value/character (of file 2) are same and then compare col5 and col6 of file 1(like a range value) with col3 and col4 of file2, if range of file 1 is present in file 2 then return that row (from file1) and also add the extra column1 from file2 in output.
Expected output ->
Col1 Col2 Col3 Col4 Col5 Col6 Col7 Col8 Col9
101_#2 1 H F0 263 278 2 1.5 97486
103_#1 2 9 V2 124 134 1 1.3 248233
So far I have tried something with hashes->
#ARGV or die "No input file specified";
open my $first, '<',$ARGV[0] or die "Unable to open input file: $!";
open my $second,'<', $ARGV[1] or die "Unable to open input file: $!";
print scalar (<$first>);
while(<$second>){
chomp;
#line=split /\s+/;
$hash{$line[2]}=$line[3];
}
while (<$first>) {
#cols = split /\s+/;
$p1 = $cols[4];
$p2 = $cols[5];
foreach $key (sort keys %hash){
if ($p1>= "$key"){
if ($p2<=$hash{$key})
{
print join("\t",#cols),"\n";
}
}
else{ next; }
}
}
But there is no comparison of col3 value/character (of file1) and col2 value/character (of file 2)in above code.
But this is also taking lot of time and memory.Can anybody suggest how I can make it fast using hashes or hashes of hashes.Thanks a lot.
Hello everyone,
Thanks a lot for your help. I figured out an efficient way for my own question.
#ARGV or die "No input file specified";
open $first, '<',$ARGV[0] or die "Unable to open input file: $!";
open $second,'<', $ARGV[1] or die "Unable to open input file: $!";
print scalar (<$first>);
while(<$second>){
chomp;
#line=split /\s+/;
$hash{$line[1]}{$line[2]}{$line[3]}= $line[0];
}
while (<$first>) {
#cols = split /\s+/;
foreach $key1 (sort keys %hash) {
foreach $key2 (sort keys %{$hash{$key1}}) {
foreach $key3 (sort keys %{$hash{$key1}{$key2}}) {
if (($cols[2] eq $key1) && ($cols[4]>=$key2) && ($cols[5]<=$key3)){
print join("\t",#cols),"\t",$hash{$key1}{$key2}{$key3},"\n";
}
last;
}
}
}
}
Is it right?

Answer 1

You don't need two hash tables. You just need one hash table built from entries in the first file, and when you loop through the second file, check if there's a key in the first-file hash table using defined.
If there is a key, do your comparisons on the values of other columns (we store values from the first file in the hash table for the third column's key).
If there's no key, then either warn, die, or have the script just keep going without saying anything, if that's what you want:
#!/usr/bin/perl -w
use strict;
use warnings;
my $firstHashRef;
open FIRST, "< $firstFile" or die "could not open first file...\n";
while (<FIRST>) {
chomp $_;
my #elements = split "\t", $_;
my $col3Val = $elements[2]; # Perl arrays are zero-indexed
my $col5Val = $elements[4];
my $col6Val = $elements[5];
# keep the fifth and sixth column values on hand, for
# when we loop through the second file...
if (! defined $firstHashRef->{$col3Val}) {
$firstHashRef->{$col3Val}->{Col5} = $col5Val;
$firstHashRef->{$col3Val}->{Col6} = $col6Val;
}
}
close FIRST;
open SECOND, "< $secondFile" or die "could not open second file...\n";
while (<SECOND>) {
chomp $_;
my #elements = split "\t", $_;
my $col2ValFromSecondFile = $elements[1];
my $col3ValFromSecondFile = $elements[2];
my $col4ValFromSecondFile = $elements[3];
if (defined $firstHashRef->{$col2ValFromSecondFile}) {
# we found a matching key
# 1. Compare $firstHashRef->{$col2ValFromSecondFile}->{Col5} with $col3ValFromSecondFile
# 2. Compare $firstHashRef->{$col2ValFromSecondFile}->{Col6} with $col4ValFromSecondFile
# 3. Do something interesting, based on comparison results... (this is left to you to fill in)
}
else {
warn "We did not locate entry in hash table for second file's Col2 value...\n";
}
}
close SECOND;

Answer 2

How about using just awk for this -
awk '
NR==FNR && NR>1{a[$3]=$0;b[$3]=$5;c[$3]=$6;next}
($2 in a) && ($3<=b[$2] && $4>=c[$2]) {print a[$2],$1}' file1 file2
Input Data:
[jaypal:~/Temp] cat file1
Col1 Col2 Col3 Col4 Col5 Col6 Col7 Col8
101_#2 1 H F0 263 278 2 1.5
109_#2 1 H F0 263 278 2 1.5
102_#1 1 6 F1 766 781 1 1.0
103_#1 2 15 V1 526 581 1 0.0
103_#1 2 9 V2 124 134 1 1.3
104_#1 1 12 V3 137 172 1 1.0
105_#1 1 17 F2 766 771 1 1.0
[jaypal:~/Temp] cat file2
Col1 Col2 Col3 Col4
97486 H 262 279
67486 9 118 119
87486 9 183 185
248233 9 124 134
Test:
[jaypal:~/Temp] awk '
NR==FNR && NR>1{a[$3]=$0;b[$3]=$5;c[$3]=$6;next}
($2 in a) && ($3<=b[$2] && $4>=c[$2]) {print a[$2],$1}' file1 file2
101_#2 1 H F0 263 278 2 1.5 97486
103_#1 2 9 V2 124 134 1 1.3 248233