How can I optimise the search of unique lines? - perl

Question

I have large tab separated files like the following example:
scaffold1443 182629 182998 chr1.1.1.241051.241420 367 99.80
scaffold1443 131948 132412 chr1.1.2.291778.292242 462 99.80
scaffold1443 96142 96474 chr1.1.3.327471.327803 330 99.70
scaffold1443 53153 53479 chr1.1.4.370342.370668 322 99.40
scaffold526 2870014 2870523 chr1.1.5.488372.488881 507 99.90
scaffold526 2865956 2866314 chr1.1.6.490869.491234 357 98.10
scaffold526 2867666 2868024 chr1.1.6.490869.491234 357 98.10
scaffold526 2485557 2485867 chr1.1.7.610677.610987 310 100.00
I want to print in a new file only the lines that the 4th column is unique.
In the previous example, all the lines should be printed except the 2 lines that have the "chr1.1.6.490869.491234" in the 4th column.
The following script that I wrote (it is a part of a larger pipeline) does the job but it is extremely slow, especially when the input file is very big.
#!/usr/bin/perl
use strict;
use warnings;
#This script takes the best hits output and finds the unique elements that up to only one scaffold.
my $target = $ARGV[0];
my $chromosome = $ARGV[1];
my #mykeys = `cat OUTPUT_$target/psl_score_byname_$target/$chromosome.table| awk '{print \$4}'| sort -u`;
foreach (#mykeys)
{
my $key = $_;
chomp($key);
my $command = "cat OUTPUT_$target/psl_score_byname_$target/$chromosome.table|grep -w $key";
my #belongs= `$command`;
chomp(#belongs);
my $count = scalar(#belongs);
if ($count == 1)
{
open FILE, ">>OUTPUT_$target/unique_hces_$target/$chromosome.txt" or die $!;
print FILE "#belongs\n";
#belongs = ();
}
else {
#belongs = ();
}
}
Is there any smarter and faster way to do it?
Thank you very much in advance.

Answer 1

Given that you do not want to print lines that have duplicates at all, you need to see the whole file before any printing, to first find those lines with duplicates. Then go back and print others.
This can be done by keeping the whole file in memory along with ancillary data structures, or by making two passes. Since the file is "very big" here is a less memory-straining way
use warnings;
use strict;
my $file = 'skip.txt';
open my $fh, '<', $file or die "Can't open $file: $!";
my (%seen, %dupe);
while (<$fh>)
{
my $patt = (split)[3];
# Record line numbers if the 4th field has been seen
if (exists $seen{$patt}) {
$dupe{ $seen{$patt} }++; # num of line with it seen first, with count
$dupe{$.} = 1; # this line's number as well
}
else { $seen{$patt} = $. } # first time this 4th field is seen
}
# Now we know all lines which carry duplicate fourth field
my $outfile = 'filtered_' . $file;
open my $fh_out, '>', $outfile or die "Can't open $outfile: $!";
seek $fh, 0, 0; # rewind to the beginning
$. = 0; # seek doesn't reset $.
while (<$fh>) {
print $fh_out $_ if not exists $dupe{$.}
}
close $fh_out;
The first time a duplicate is found its original line also need be recorded, $dupe{$seen{$patt}}++, in that branch. This need be done only once, and while we can check (whether it's already been recorded) we may well pick up a potentially useful duplicates' count instead.
I've added a few more duplicates (some more than twice) to your posted sample and this produces the correct output.
Comment on the posted code
The posted code checks the fourth field on each line against the whole file, thus processing the file as many times as there are lines. That is a lot of work and it has to take time, specially for big files.
Also, there is no reason to use external programs for that job.

Answer 2

As oneliner:
perl -F"\t" -lanE 'push #l,[#F];$s{$F[3]}++}{say join"\t",#$_ for grep{$s{$_->[3]}==1}#l' <<EOF
scaffold1443 182629 182998 chr1.1.1.241051.241420 367 99.80
scaffold1443 131948 132412 chr1.1.2.291778.292242 462 99.80
scaffold1443 96142 96474 chr1.1.3.327471.327803 330 99.70
scaffold1443 53153 53479 chr1.1.4.370342.370668 322 99.40
scaffold526 2870014 2870523 chr1.1.5.488372.488881 507 99.90
scaffold526 2865956 2866314 chr1.1.6.490869.491234 357 98.10
scaffold526 2867666 2868024 chr1.1.6.490869.491234 357 98.10
scaffold526 2485557 2485867 chr1.1.7.610677.610987 310 100.00
EOF
output
scaffold1443 182629 182998 chr1.1.1.241051.241420 367 99.80
scaffold1443 131948 132412 chr1.1.2.291778.292242 462 99.80
scaffold1443 96142 96474 chr1.1.3.327471.327803 330 99.70
scaffold1443 53153 53479 chr1.1.4.370342.370668 322 99.40
scaffold526 2870014 2870523 chr1.1.5.488372.488881 507 99.90
scaffold526 2485557 2485867 chr1.1.7.610677.610987 310 100.00
more readable:
perl -F"\t" -lanE '
push #lines, [ #F ]; $seen{ $F[3] }++;
END {
say join("\t",#$_) for grep { $seen{ $_->[3] } == 1 } #lines
}
'
You can translate it to full script if want, I created this as oneliner because you said: it is a part of a larger pipeline.
Also note, the above reads the whole file into the memory first - so very the big files could cause problems.

Answer 3

The simple approach involves using an associative array to identify duplicates.
perl -F'\t' -lane'
push #{ $h{ $F[3] } }, $_;
END {
for (values(%h)) {
print(#$_) if #$_ == 1;
}
}
' file.tsv
The above approach requires as much memory as the file is large. That's a no-go if you files are truly large.
If you have truly large files, the simple approach is to sort the file using the sort command line utility (which is rather fast, and can handle arbitrarily large files). By first rearranging the file such that duplicates are next to each other, we can easily filtered out the duplicates without worrying about memory issues.
sort -t$'\t' -k 4,4 file.tsv | perl -F'\t' -lane'
if ($key ne $F[3]) {
print(#buf) if #buf == 1;
#buf = ();
}
$key = $F[3];
push #buf, $_;
END { print(#buf) if #buf == 1; }
'
If you have truly large files, another relatively simple approach is to load the data in a database (e.g. an sqlite3 database). You could easily maintain the original order with this approach.