perl combine specific columns from multiple files - perl

Question

I'd like to create a perl script that combines columns from multiple files. I have to respect a series of criteria (folder/file structure). I'll try to represent what I have and what I have.I have two folders with a bunch of files. The files inside each folders have the same names.
Folder1: File1, File2, File3, ...
Folder2: File1, File2, File3, ...
Folder1:File1 content looks like this (tab delimited):
aaaaa 233
bbbbb 34
ccccc 853
...
All the other files look like this one, except the numerical values are different. I want to create a single file (a report) that will look like this:
aaaaa value_Folder1:File1 value_Folder2:File1 value_Folder1:File2 value_Folder2:File2 ...
...
It would be nice to have the file name on top of the columns from which the values are coming from (just the file name, the folder is not important).
I have some code evolving, but it's not doing what I want right now! I tried to make it work via loops, but I feel that it might not be the solution... One other problem is that I don't know how to add columns to my report file. In the following code, I just append the value a the end of the file. Even if it's not super nice, here's my code:
#!/usr/bin/perl -w
use strict;
use warnings;
my $outputfile = "/home/duceppemo/Desktop/count.all.txt";
my $queryDir = "/home/duceppemo/Desktop/query_count/";
my $hitDir = "/home/duceppemo/Desktop/hit_count/";
opendir (DIR, "$queryDir") or die "Error opening $queryDir: $!"; #Open the directory containing the files with sequences to look for
my #queryFileNames = readdir (DIR);
opendir (DIR, "$hitDir") or die "Error opening $hitDir: $!"; #Open the directory containing the files with sequences to look for
my #hitFileNames = readdir (DIR);
my $index = 0;
$index ++ until $queryFileNames[$index] eq ".";
splice(#queryFileNames, $index, 1);
$index = 0;
$index ++ until $queryFileNames[$index] eq "..";
splice(#queryFileNames, $index, 1);
$index = 0;
$index ++ until $hitFileNames[$index] eq ".";
splice(#hitFileNames, $index, 1);
$index = 0;
$index ++ until $hitFileNames[$index] eq "..";
splice(#hitFileNames, $index, 1);
#counter for query file number opened
my $i = 0;
foreach my $queryFile (#queryFileNames) #adjust the file name according to the subdirectory
{
$i += 1; #keep track of the file number opened
$queryFile = $queryDir . $queryFile;
open (QUERY, "$queryFile") or die "Error opening $queryFile: $!";
my #query = <QUERY>; #Put the query sequences from the count file into an array
close (QUERY);
my $line = 0;
open (RESULT, ">>$outputfile") or die "Error opening $outputfile: $!";
foreach my $lineQuery (#query) #look into the query file
{
my #columns = split(/\s+/, $lineQuery); #Split each line into a new array, when it meets a whitespace character (including tab)
if ($i == 1)
{
#open (RESULT, ">>$outputfile") or die "Error opening $outputfile: $!";
print RESULT "$columns[0]\t";
print RESULT "$columns[1]\n";
#close (RESULT);
$line += 1;
}
else
{
open (RESULT, ">>$outputfile") or die "Error opening $outputfile: $!";
print RESULT "$columns[1]\n";
close (RESULT);
$line += 1;
}
}
$line = 0;
}
close (RESULT);
closedir (DIR);
P.S. Any other advises on code optimisation be gratefully accepted!

Answer 1

The main problem is that you don't seem to understand what is a FILEHANDLE. You should research on this.
A Filehandle is a sort of reference to an open file, and since everything is a file, it can be a command or a directory.
When you make opendir(DIR, ...) "DIR" is not a keyword but a filehandle that can have any name. That means your 2 opendir() have the same filehandle, which does not make sense.
It should be more like:
opendir(QDIR, $queryDir) or die "Error opening $queryDir: $!";
my #queryFileNames = readdir(QDIR);
opendir(HDIR, $hitDir) or die "Error opening $hitDir: $!";
my #hitFileNames = readdir(HDIR);
Also, since you should always close every open filehandle, you must call close() at the same level and make sure close() will be called.
e.g. the opening of the filehandle RESULT and its close after the loop in which it was opened does not make sense... How many times will you open it without closing it?
You probably need to open it before the loop, and you don't have to open it twice with the same filehandle...
In general you want to avoid open/close in loops. You simply open before and close after.

Answer 2

That code is doing pretty much what I want:
#!/usr/bin/perl
use strict;
use warnings;
#my $queryDir = "ARGV[0]";
my $queryDir = "C:/Users/Marco/Desktop/query_count/";
opendir (DIR1, "$queryDir") or die "Error opening $queryDir: $!"; #Open the directory containing the files with sequences to look for
my #queryFileName = readdir (DIR1);
#my $hitDir = "ARGV[1]";
my $hitDir = "C:/Users/Marco/Desktop/hit_count/";
opendir (DIR2, "$hitDir") or die "Error opening $hitDir: $!"; #Open the directory containing the files with sequences to look for
my #hitFileName = readdir (DIR2);
my $index = 0;
$index ++ until $queryFileName[$index] eq ".";
splice(#queryFileName, $index, 1);
$index = 0;
$index ++ until $queryFileName[$index] eq "..";
splice(#queryFileName, $index, 1);
$index = 0;
$index ++ until $hitFileName[$index] eq ".";
splice(#hitFileName, $index, 1);
$index = 0;
$index ++ until $hitFileName[$index] eq "..";
splice(#hitFileName, $index, 1);
foreach my $queryFile (#queryFileName) #adjust the queryFileName according to the subdirectory
{
$queryFile = "$queryDir" . $queryFile;
}
foreach my $hitFile (#hitFileName) #adjust the queryFileName according to the subdirectory
{
$hitFile = "$hitDir" . $hitFile;
}
my $outputfile = "C:/Users/Marco/Desktop/out.txt";
my %hash;
foreach my $queryFile (#queryFileName)
{
my $i = 0;
open (QUERY, "$queryFile") or die "Error opening $queryFile: $!";
while (<QUERY>)
{
chomp;
my $val = (split /\t/)[1];
$i++;
$hash{$i}{$queryFile} = $val;
}
close (QUERY);
}
foreach my $hitFile (#hitFileName)
{
my $i = 0;
open (HIT, "$hitFile") or die "Error opening $hitFile: $!";
while (<HIT>)
{
chomp;
my $val = (split /\t/)[1];
$i++;
$hash{$i}{$hitFile} = $val;
}
close (HIT);
}
open (RESULT, ">>$outputfile") or die "Error opening $outputfile: $!";
foreach my $qfile (#queryFileName)
{
print RESULT "\t$qfile";
}
foreach my $hfile (#hitFileName)
{
print RESULT "\t$hfile";
}
print RESULT "\n";
foreach my $id (sort keys %hash)
{
print RESULT "$id\t";
print RESULT "$hash{$id}{$_}\t" foreach (#queryFileName, #hitFileName);
print RESULT "\n";
}
close (RESULT);