Perl: how to compare array to hash and print out results - perl

I'm quite new to Perl, so I'm sorry if this is somewhat rudimentary.
I'm working with a Perl script that is working as a wrapper for some Python, text formatting, etc. and I'm struggling to get my desired output.
The script takes a folder, for this example, the folder contains 6 text files (test1.txt through test6.txt). The script then extracts some information from the files, runs a series of command line programs and then outputs a tab-delimited result. However, that result contains only those results that made it through the rest of the processing by the script, i.e. the result.
Here are some snippets of what I have so far:
use strict;
use warnings;
## create array to capture all of the file names from the folder
opendir(DIR, $folder) or die "couldn't open $folder: $!\n";
my #filenames = grep { /\.txt$/ } readdir DIR;
closedir DIR;
#here I run some subroutines, the last one looks like this
my $results = `blastn -query $shortname.fasta -db DB/$db -outfmt "6 qseqid sseqid score evalue" -max_target_seqs 1`;
#now I would like to compare what is in the #filenames array with $results
Example of tab delimited result - stored in $results:
test1.txt 200 1:1-20 79 80
test3.txt 800 1:1-200 900 80
test5.txt 900 1:1-700 100 2000
test6.txt 600 1:1-1000 200 70
I would like the final output to include all of the files that were run through the script, so I think I need a way to compare two arrays or perhaps compare an array to a hash?
Example of the desired output:
test1.txt 200 1:1-20 79 80
test2.txt 0 No result
test3.txt 800 1:1-200 900 80
test4.txt 0 No result
test5.txt 900 1:1-700 100 2000
test6.txt 600 1:1-1000 200 70
Update
Ok, so I got this to work with suggestions by #terdon by reading the file into a hash and then comparing. So I was trying to figure out how to do this with out writing to file and the reading the file back in - I still can't seem to get the syntax correct. Here's what I have, however it seems like I'm not able to match the array to the hash - meaning the hash must not be correct:
#!/usr/bin/env perl
use strict;
use warnings;
#create variable to mimic blast results
my $blast_results = "file1.ab1 9 350 0.0 449 418 418 403479 403042 567
file3.ab1 2 833 0.0 895 877 877 3717226 3718105 984";
#create array to mimic filename array
my #filenames = ("file1.ab1", "file2.ab1", "file3.ab1");
#header for file
my $header = "Query\tSeq_length\tTarget found\tScore (Bits)\tExpect(E-value)\tAlign-length\tIdentities\tPositives\tChr\tStart\tEnd\n";
#initialize hash
my %hash;
#split blast results into array
my #row = split(/\s+/, $blast_results);
$hash{$row[0]}=$_;
print $header;
foreach my $file (#filenames){
## If this filename has an associated entry in the hash, print it
if(defined($hash{$file})){
print "$row[0]\t$row[9]\t$row[1]:$row[7]-$row[8]\t$row[2]\t$row[3]\t$row[4]\t$row[5]\t$row[6]\t$row[1]\t$row[7]\t$row[8]\n";
}
## If not, print this.
else{
print "$file\t0\tNo Blast Results: Sequencing Rxn Failed\n";
}
}
print "-----------------------------------\n";
print "$blast_results\n"; #test what results look like
print "-----------------------------------\n";
print "$row[0]\t$row[1]\n"; #test if array is getting split correctly
print "-----------------------------------\n";
print "$filenames[2]\n"; #test if other array present
The result from this script is (the #filenames array is not matching the hash):
Query Seq_length Target found Score (Bits) Expect(E-value) Align-length Identities Positives Chr Start End
file1.ab1 0 No Blast Results: Sequencing Rxn Failed
file2.ab1 0 No Blast Results: Sequencing Rxn Failed
file3.ab1 0 No Blast Results: Sequencing Rxn Failed
-----------------------------------
file1.ab1 9 350 0.0 449 418 418 403479 403042 567
file3.ab1 2 833 0.0 895 877 877 3717226 3718105 984
-----------------------------------
file1.ab1 9
-----------------------------------
file3.ab1

I'm not entirely sure what you need here but the equivalent of awk's A[$1]=$0 is done using hashes in Perl. Something like:
my %hash;
## Open the output file
open(my $fh, "<","text_file");
while(<$fh>){
## remove newlines
chomp;
## split the line
my #A=split(/\s+/);
## Save this in a hash whose keys are the 1st fields and whose
## values are the associated lines.
$hash{$A[0]}=$_;
}
close($fh);
## Now, compare the file to #filenames
foreach my $file (#filenames){
## Print the file name
print "$file\t";
## If this filename has an associated entry in the hash, print it
if(defined($hash{$file})){
print "$hash{$file}\n";
}
## If not, print this.
else{
print "0\tNo result\n";
}
}

Related

How can I optimise the search of unique lines?

I have large tab separated files like the following example:
scaffold1443 182629 182998 chr1.1.1.241051.241420 367 99.80
scaffold1443 131948 132412 chr1.1.2.291778.292242 462 99.80
scaffold1443 96142 96474 chr1.1.3.327471.327803 330 99.70
scaffold1443 53153 53479 chr1.1.4.370342.370668 322 99.40
scaffold526 2870014 2870523 chr1.1.5.488372.488881 507 99.90
scaffold526 2865956 2866314 chr1.1.6.490869.491234 357 98.10
scaffold526 2867666 2868024 chr1.1.6.490869.491234 357 98.10
scaffold526 2485557 2485867 chr1.1.7.610677.610987 310 100.00
I want to print in a new file only the lines that the 4th column is unique.
In the previous example, all the lines should be printed except the 2 lines that have the "chr1.1.6.490869.491234" in the 4th column.
The following script that I wrote (it is a part of a larger pipeline) does the job but it is extremely slow, especially when the input file is very big.
#!/usr/bin/perl
use strict;
use warnings;
#This script takes the best hits output and finds the unique elements that up to only one scaffold.
my $target = $ARGV[0];
my $chromosome = $ARGV[1];
my #mykeys = `cat OUTPUT_$target/psl_score_byname_$target/$chromosome.table| awk '{print \$4}'| sort -u`;
foreach (#mykeys)
{
my $key = $_;
chomp($key);
my $command = "cat OUTPUT_$target/psl_score_byname_$target/$chromosome.table|grep -w $key";
my #belongs= `$command`;
chomp(#belongs);
my $count = scalar(#belongs);
if ($count == 1)
{
open FILE, ">>OUTPUT_$target/unique_hces_$target/$chromosome.txt" or die $!;
print FILE "#belongs\n";
#belongs = ();
}
else {
#belongs = ();
}
}
Is there any smarter and faster way to do it?
Thank you very much in advance.
Given that you do not want to print lines that have duplicates at all, you need to see the whole file before any printing, to first find those lines with duplicates. Then go back and print others.
This can be done by keeping the whole file in memory along with ancillary data structures, or by making two passes. Since the file is "very big" here is a less memory-straining way
use warnings;
use strict;
my $file = 'skip.txt';
open my $fh, '<', $file or die "Can't open $file: $!";
my (%seen, %dupe);
while (<$fh>)
{
my $patt = (split)[3];
# Record line numbers if the 4th field has been seen
if (exists $seen{$patt}) {
$dupe{ $seen{$patt} }++; # num of line with it seen first, with count
$dupe{$.} = 1; # this line's number as well
}
else { $seen{$patt} = $. } # first time this 4th field is seen
}
# Now we know all lines which carry duplicate fourth field
my $outfile = 'filtered_' . $file;
open my $fh_out, '>', $outfile or die "Can't open $outfile: $!";
seek $fh, 0, 0; # rewind to the beginning
$. = 0; # seek doesn't reset $.
while (<$fh>) {
print $fh_out $_ if not exists $dupe{$.}
}
close $fh_out;
The first time a duplicate is found its original line also need be recorded, $dupe{$seen{$patt}}++, in that branch. This need be done only once, and while we can check (whether it's already been recorded) we may well pick up a potentially useful duplicates' count instead.
I've added a few more duplicates (some more than twice) to your posted sample and this produces the correct output.
Comment on the posted code
The posted code checks the fourth field on each line against the whole file, thus processing the file as many times as there are lines. That is a lot of work and it has to take time, specially for big files.
Also, there is no reason to use external programs for that job.
As oneliner:
perl -F"\t" -lanE 'push #l,[#F];$s{$F[3]}++}{say join"\t",#$_ for grep{$s{$_->[3]}==1}#l' <<EOF
scaffold1443 182629 182998 chr1.1.1.241051.241420 367 99.80
scaffold1443 131948 132412 chr1.1.2.291778.292242 462 99.80
scaffold1443 96142 96474 chr1.1.3.327471.327803 330 99.70
scaffold1443 53153 53479 chr1.1.4.370342.370668 322 99.40
scaffold526 2870014 2870523 chr1.1.5.488372.488881 507 99.90
scaffold526 2865956 2866314 chr1.1.6.490869.491234 357 98.10
scaffold526 2867666 2868024 chr1.1.6.490869.491234 357 98.10
scaffold526 2485557 2485867 chr1.1.7.610677.610987 310 100.00
EOF
output
scaffold1443 182629 182998 chr1.1.1.241051.241420 367 99.80
scaffold1443 131948 132412 chr1.1.2.291778.292242 462 99.80
scaffold1443 96142 96474 chr1.1.3.327471.327803 330 99.70
scaffold1443 53153 53479 chr1.1.4.370342.370668 322 99.40
scaffold526 2870014 2870523 chr1.1.5.488372.488881 507 99.90
scaffold526 2485557 2485867 chr1.1.7.610677.610987 310 100.00
more readable:
perl -F"\t" -lanE '
push #lines, [ #F ]; $seen{ $F[3] }++;
END {
say join("\t",#$_) for grep { $seen{ $_->[3] } == 1 } #lines
}
'
You can translate it to full script if want, I created this as oneliner because you said: it is a part of a larger pipeline.
Also note, the above reads the whole file into the memory first - so very the big files could cause problems.
The simple approach involves using an associative array to identify duplicates.
perl -F'\t' -lane'
push #{ $h{ $F[3] } }, $_;
END {
for (values(%h)) {
print(#$_) if #$_ == 1;
}
}
' file.tsv
The above approach requires as much memory as the file is large. That's a no-go if you files are truly large.
If you have truly large files, the simple approach is to sort the file using the sort command line utility (which is rather fast, and can handle arbitrarily large files). By first rearranging the file such that duplicates are next to each other, we can easily filtered out the duplicates without worrying about memory issues.
sort -t$'\t' -k 4,4 file.tsv | perl -F'\t' -lane'
if ($key ne $F[3]) {
print(#buf) if #buf == 1;
#buf = ();
}
$key = $F[3];
push #buf, $_;
END { print(#buf) if #buf == 1; }
'
If you have truly large files, another relatively simple approach is to load the data in a database (e.g. an sqlite3 database). You could easily maintain the original order with this approach.

Getting member size from zip using Archive::Zip::MemberRead

I am trying to read each member file size from a zip without actually extracting. I iterate through all member names, then use Archive::Zip::MemberRead to get a file handle for each member, against which I was hoping to be able to use the stat method to get the size. However, stat on a file handle from a zip file element returns an empty array so I can't get my file size. Here is my code:
my $zip = Archive::Zip->new($zipFilePath);
my #mbrs = $zip->memberNames();
foreach my $mbrName(#mbrs)
{
my $fh = Archive::Zip::MemberRead->new($zip, $mbrName);
my #fileStats = stat($fh);
my $size = $fileStats[7];
print "\n".$mbrName." -- ".$size;
}
However, the output I get does not display any file size:
dir/fileName1.txt --
dir/fileName2.txt --
The question is how to retrieve member file sizes without actually extracting them.
Why not just use the Archive::Zip module itself? This seems to work for me:
#!/usr/bin/perl
use strict;
use warnings;
use Archive::Zip qw(:ERROR_CODES :CONSTANTS);
my $filename = "somezipfile.zip";
# Read in the ZIP file
my $zip = Archive::Zip->new();
unless ($zip->read($filename) == AZ_OK) {
die "Read error\n";
}
# Loop through the members, printing their name,
# compressed size, and uncompressed size.
my #members = $zip->members();
foreach (#members)
{
print " - " . $_->fileName() . ": " . $_->compressedSize() .
" (" . $_->uncompressedSize() . ")\n";
}
Here is one way only if you have 7-zip installed:
#!/usr/bin/env perl
use warnings;
use strict;
## List files from zip file provided as first argument to the script, the format
## is like:
# Date Time Attr Size Compressed Name
#------------------- ----- ------------ ------------ ------------------------
#2012-10-19 16:56:38 ..... 139 112 1.txt
#2012-10-19 16:56:56 ..... 126 105 2.txt
#2012-10-19 16:57:24 ..... 71 53 3.txt
#2012-10-03 14:39:54 ..... 155 74 A.txt
#2012-09-29 17:53:44 ..... 139 70 AA.txt
#2011-12-08 10:41:16 ..... 30 30 AAAB.txt
#2011-12-08 10:41:16 ..... 18 18 AAAC.txt
# ...
for ( map { chomp; $_ } qx/7z l $ARGV[0]/ ) {
# Omit headers and footers with this flip-flop.
if ( my $l = ( m/^(?:-+\s+){2,}/ ... m/^(?:-+\s+){2,}/ ) ) {
## Don't match flip-flop boundaries.
next if $l == 1 || $l =~ m/E0$/;
## Extract file name and its size.
my #f = split ' ';
printf qq|%s -- %d bytes\n|, $f[5], $f[3];
}
}
I run it like:
perl script.pl files.zip
That yiedls in my test (with some output suppressed):
1.txt -- 139 bytes
2.txt -- 126 bytes
3.txt -- 71 bytes
A.txt -- 155 bytes
AA.txt -- 139 bytes
AAAB.txt -- 30 bytes
AAAC.txt -- 18 bytes
B.txt -- 40 bytes
BB.txt -- 131 bytes
C.txt -- 4 bytes
CC.txt -- 184 bytes
File1.txt -- 177 bytes
File2.txt -- 250 bytes
aaa.txt -- 30 bytes
...

Loading Big files into Hashes in Perl (BLAST tables)

I'm a perl beginner, please help me out with my query... I'm trying to extract information from a blast table (a snippet of what it looks like is below):
It's a standard blast table input... I basically want to extract any information on a list of reads (Look at my second script below , to get an idea of what I want to do).... Anyhow this is precisely what I've done in the second script:
INPUTS:
1) the blast table:
38.1 0.53 59544 GH8NFLV01A02ED GH8NFLV01A02ED rank=0113471 x=305.0 y=211.5 length=345 1 YP_003242370 Dynamin family protein [Paenibacillus sp. Y412MC10] -1 0 48.936170212766 40.4255319148936 47 345 1213 13.6231884057971 3.87469084913438 31 171 544 590
34.3 7.5 123828 GH8NFLV01A03QJ GH8NFLV01A03QJ rank=0239249 x=305.0 y=1945.5 length=452 1 XP_002639994 Hypothetical protein CBG10824 [Caenorhabditis briggsae] 3 0 52.1739130434783 32.6086956521739 46 452 367 10.1769911504425 12.5340599455041 111 248 79 124
37.7 0.70 62716 GH8NFLV01A09B8 GH8NFLV01A09B8 rank=0119267 x=307.0 y=1014.0 length=512 1 XP_002756773 PREDICTED: probable G-protein coupled receptor 123-like, partial [Callithrix jacchus] 1 0 73.5294117647059 52.9411764705882 34 512 703 6.640625 4.83641536273115 43 144 273 306
37.7 0.98 33114 GH8NFLV01A0H5C GH8NFLV01A0H5C rank=0066011 x=298.0 y=2638.5 length=573 1 XP_002756773 PREDICTED: probable G-protein coupled receptor 123-like, partial [Callithrix jacchus] -3 0 73.5294117647059 52.9411764705882 34 573 703 5.93368237347295 4.83641536273115 131 232 273 306
103 1e-020 65742 GH8NFLV01A0MXI GH8NFLV01A0MXI rank=0124865 x=300.5 y=644.0 length=475 1 ABZ08973 hypothetical protein ALOHA_HF4000APKG6B14ctg1g18 [uncultured marine crenarchaeote HF4000_APKG6B14] 2 0 77.9411764705882 77.9411764705882 68 475 151 14.3157894736842 45.0331125827815 2 205 1 68
41.6 0.053 36083 GH8NFLV01A0QKX GH8NFLV01A0QKX rank=0071366 x=301.0 y=1279.0 length=526 1 XP_766153 hypothetical protein [Theileria parva strain Muguga] -1 0 66.6666666666667 56.6666666666667 30 526 304 5.70342205323194 9.86842105263158 392 481 31 60
45.4 0.003 78246 GH8NFLV01A0Z29 GH8NFLV01A0Z29 rank=0148293 x=304.0 y=1315.0 length=432 1 ZP_04111769 hypothetical protein bthur0007_56280 [Bacillus thuringiensis serovar monterrey BGSC 4AJ1] 3 0 51.8518518518518 38.8888888888889 54 432 193 12.5 27.979274611399 48 209 97 150
71.6 4e-011 97250 GH8NFLV01A14MR GH8NFLV01A14MR rank=0184885 x=317.5 y=609.5 length=314 1 ZP_03823721 DNA replication protein [Acinetobacter sp. ATCC 27244] 1 0 92.5 92.5 40 314 311 12.7388535031847 12.8617363344051 193 312 13 52
58.2 5e-007 154555 GH8NFLV01A1KCH GH8NFLV01A1KCH rank=0309994 x=310.0 y=2991.0 length=267 1 ZP_03823721 DNA replication protein [Acinetobacter sp. ATCC 27244] 1 0 82.051282051282 82.051282051282 39 267 311 14.6067415730337 12.540192926045 142 258 1 39
2) The reads list:
GH8NFLV01A09B8
GH8NFLV01A02ED
etc
etc
3) the output I want:
37.7 0.70 62716 GH8NFLV01A09B8 GH8NFLV01A09B8 rank=0119267 x=307.0 y=1014.0 length=512 1 XP_002756773 PREDICTED: probable G-protein coupled receptor 123-like, partial [Callithrix jacchus] 1 0 73.5294117647059 52.9411764705882 34 512 703 6.640625 4.83641536273115 43 144 273 306
38.1 0.53 59544 GH8NFLV01A02ED GH8NFLV01A02ED rank=0113471 x=305.0 y=211.5 length=345 1 YP_003242370 Dynamin family protein [Paenibacillus sp. Y412MC10] -1 0 48.936170212766 40.4255319148936 47 345 1213 13.6231884057971 3.87469084913438 31 171 544 590
I want a subset of the information in the first list, given a list of read names I want to extract (that is found in the 4th column)
Instead of hashing the reads list (only?) I want to hash the blast table itself, and use the information in Column 4 (of the blast table)as the keys to extract the values of each key, even when that key may have more than one value(i.e: each read name might actually have more than one hit , or associated blast result in the table), keeping in mind, that the value includes the WHOLE row with that key(readname) in it.
My greplist.pl script does this, but is very very slow, I think , ( and correct me if i'm wrong) that by loading the whole table in a hash, that this should speed things up tremendously ...
Thank you for your help.
My scripts:
The Broken one (mambo5.pl)
#!/usr/bin/perl -w
# purpose: extract blastX data from a list of readnames
use strict;
open (DATA,$ARGV[0]) or die ("Usage: ./mambo5.pl BlastXTable readslist");
open (LIST,$ARGV[1]) or die ("Usage: ./mambo5.pl BlastXTable readslist");
my %hash = <DATA>;
close (DATA);
my $filename=$ARGV[0];
open(OUT, "> $filename.bololom");
my $readName;
while ( <LIST> )
{
#########;
if(/^(.*?)$/)#
{
$readName=$1;#
chomp $readName;
if (exists $hash{$readName})
{
print "bingo!";
my $output =$hash{$readName};
print OUT "$output\n";
}
else
{
print "it aint workin\n";
#print %hash;
}
}
}
close (LIST);
The Slow and quick cheat (that works) and is very slow (my blast tables can be about 400MB to 2GB large, I'm sure you can see why it's so slow)
#!/usr/bin/perl -w
##
# This script finds a list of names in a blast table and outputs the result in a new file
# name must exist and list must be correctly formatted
# will not output anything using a "normal" blast file, must be a table blast
# if you have the standard blast output use blast2table script
use strict;
my $filein=$ARGV[0] or die ("usage: ./listgrep.pl readslist blast_table\n");
my $db=$ARGV[1] or die ("usage: ./listgrep.pl readslist blast_table\n");
#open the reads you want to grep
my $read;
my $line;
open(READSLIST,$filein);
while($line=<READSLIST>)
{
if ($line=~/^(.*)$/)
{
$read = $1;
print "$read\n";
system("grep \"$read\" $db >$read\_.out\n");
}
#system("grep $read $db >$read\_.out\n");
}
system("cat *\_.out >$filein\_greps.txt\n");
system("rm *.out\n");
I don't know how to define that 4th column as the key : maybe I could use the split function, but I've tried to find a way that does this for a table of more than 2 columns to no avail... Please help!
If there is an easy way out of this please let me know
Thanks !
I'd do the opposite i.e read the readslist file into a hash then walk thru the big blast file and print the desired lines.
#!/usr/bin/perl
use strict;
use warnings;
use 5.010;
# Read the readslist file into a hash
open my $fh, '<', 'readslist' or die "Can't open 'readslist' for reading:$!";
my %readslist = map { chomp; $_ => 1 }<$fh>;
close $fh;
open my $fh_blast, '<', 'blastfile' or die "Can't open 'blastfile' for reading:$!";
# loop on all the blastfile lines
while (<$fh_blast>) {
chomp;
# retrieve the key (4th column)
my ($key) = (split/\s+/)[3];
# print the line if the key exists in the hash
say $_ if exists $readslist{$key};
}
close $fh_blast;
I suggest you build an index to turn your blasts file temporarily into an indexed-sequential file. Read through it and build a hash of addresses within the file where every record for each key starts.
After that it is just a matter of seeking to the correct places in the file to pick up the records required. This will certainly be faster than most simple solutions, as it entails read the big file only once. This example code demonstrates.
use strict;
use warnings;
use Fcntl qw/SEEK_SET/;
my %index;
open my $blast, '<', 'blast.txt' or die $!;
until (eof $blast) {
my $place = tell $blast;
my $line = <$blast>;
my $key = (split ' ', $line, 5)[3];
push #{$index{$key}}, $place;
}
open my $reads, '<', 'reads.txt' or die $!;
while (<$reads>) {
next unless my ($key) = /(\S+)/;
next unless my $places = $index{$key};
foreach my $place (#$places) {
seek $blast, $place, SEEK_SET;
my $line = <$blast>;
print $line;
}
}
Voila, 2 ways of doing this, one with nothing to do with perl :
awk 'BEGIN {while ( i = getline < "reads_list") ar[$i] = $1;} {if ($4 in ar) print $0;}' blast_table > new_blast_table
Mambo6.pl
#!/usr/bin/perl -w
# purpose: extract blastX data from a list of readnames. HINT: Make sure your list file only has unique names , that way you save time.
use strict;
open (DATA,$ARGV[0]) or die ("Usage: ./mambo5.pl BlastXTable readslist");
open (LIST,$ARGV[1]) or die ("Usage: ./mambo5.pl BlastXTable readslist");
my %hash;
my $val;
my $key;
while (<DATA>)
{
#chomp;
if(/((.*?)\t(.*?)\t(.*?)\t(.*?)\t(.*?)\t(.*?)\t(.*?)\t(.*?)\t(.*?)\t(.*?)\t(.*?)\t(.*?)\t(.*?)\t(.*?)\t(.*?)\t(.*?)\t(.*?)\t(.*?)\t(.*?)\t(.*?)\t(.*?))$/)
{
#print "$1\n";
$key= $5;#read
$val= $1;#whole row; notice the brackets around the whole match.
$hash{$key} .= exists $hash{$key} ? "$val\n" : $val;
}
else {
print "something wrong with format";
}
}
close (DATA);
open(OUT, "> $ARGV[1]\_out\.txt");
my $readName;
while ( <LIST> )
{
#########;
if(/^(.*?)$/)#
{
$readName=$1;#
chomp $readName;
if (exists $hash{$readName})
{
print "$readName\n";
my $output =$hash{$readName};
print OUT "$output";
}
else
{
#print "it aint workin\n";
}
}
}
close (LIST);
close (OUT);
The oneliner is faster, and probably better than my script, I'm sure some people can find easier ways to do it... I just thought I'd put this up since it does what I want.

How can I correctly process this file containing tab separated values in Perl?

I am fairly new to Perl and know next to nothing about Perl's 'proper' syntax.
I have a text file that I use everyday with a listing of names, and other info for our users. This file changes daily and sometimes has two rows in it(tab delimited), and other times has 100+ rows in it.
The file also varies between 6-9 columns of data in a row. I have put together a Perl script that uses the split function on tabs, but the issue I am running into is that if I take row a, which has 5 columns in it and then add a second row b that has 6 columns in it that are all populated with data.
I cannot figure out how to get Perl to see that row a only has 5 columns of data and to continue parsing the text file from that point forward. It continues, but the output wraps lines strangely. How can I get around this issue? I hope that made sense.
You will have to post some code and possibly some sample data, but here's a code that is parsing rows of different lengths without issue.
Script:
#!/usr/bin/perl
use strict;
while (<STDIN>)
{
chomp;
my #info = split("\t");
print join(";", #info), "\n";
}
exit;
Test File:
jsmith 101 777-222-5555 Office 1 Building 1 Manager
aposse 104 777-222-5556 Office 2 Building 2 Stock Clerk
jbraza 105 777-222-5557 Office 3
mcuzui 102 777-222-5557 Office 3 Building 3 Cashier
ghines 107 777-222-5557 Office 3
Output:
%> test.pl < file.txt
jsmith;101;777-222-5555;Office 1;Building 1;Manager
aposse;104;777-222-5556;Office 2;Building 2;Stock Clerk
jbraza;105;777-222-5557;Office 3
mcuzui;102;777-222-5557;Office 3;Building 3;Cashier
ghines;107;777-222-5557;Office 3
You should post some sample data and code and explain desired behavior in terms of what the code currently does and what you want it to do. split will give you as many fields as there are in the input.
#!/usr/bin/perl
use strict; use warnings;
while ( my $row = <DATA> ) {
last unless $row =~ /\S/;
chomp $row;
my #cells = split /\t/, $row;
print "< #cells >\n";
}
__DATA__
1 2 3 4 5
a b c d e f
Text::CSV module can be used for parsing tab-separated-values as well. In reality, Text::CSV could parse values delimited by any character.
Relevant excerpt from its POD:
The module accepts either strings or
files as input and can utilize any
user-specified characters as
delimiters, separators, and escapes so
it is perhaps better called ASV
(anything separated values) rather
than just CSV.
#!/usr/bin/env perl
use strict;
use warnings;
use Text::CSV;
my $csv = Text::CSV->new( { 'sep_char' => "\t" } );
open my $fh, '<', 'data.tsv' or die "Unable to open: $!";
my #rows;
while ( my $row_ref = $csv->getline($fh) ) {
push #rows, $row_ref;
}
$csv->sep_char('|');
for my $row_ref (#rows) {
$csv->combine(#$row_ref);
print $csv->string(), "\n";
}

How can I randomly sample the contents of a file?

I have a file with contents
abc
def
high
lmn
...
...
There are more than 2 million lines in the files.
I want to randomly sample lines from the files and output 50K lines. Any thoughts on how to approach this problem? I was thinking along the lines of Perl and its rand function (Or a handy shell command would be neat).
Related (Possibly Duplicate) Questions:
Randomly Pick Lines From a File Without Slurping It With Unix
How can I get exactly n random lines from a file with Perl?
Assuming you basically want to output about 2.5% of all lines, this would do:
print if 0.025 > rand while <$input>;
Shell way:
sort -R file | head -n 50000
From perlfaq5: "How do I select a random line from a file?"
Short of loading the file into a database or pre-indexing the lines in the file, there are a couple of things that you can do.
Here's a reservoir-sampling algorithm from the Camel Book:
srand;
rand($.) < 1 && ($line = $_) while <>;
This has a significant advantage in space over reading the whole file in. You can find a proof of this method in The Art of Computer Programming, Volume 2, Section 3.4.2, by Donald E. Knuth.
You can use the File::Random module which provides a function for that algorithm:
use File::Random qw/random_line/;
my $line = random_line($filename);
Another way is to use the Tie::File module, which treats the entire file as an array. Simply access a random array element.
Perl way:
use CPAN. There is module File::RandomLine that does exactly what you need.
If you need to extract an exact number of lines:
use strict;
use warnings;
# Number of lines to pick and file to pick from
# Error checking omitted!
my ($pick, $file) = #ARGV;
open(my $fh, '<', $file)
or die "Can't read file '$file' [$!]\n";
# count lines in file
my ($lines, $buffer);
while (sysread $fh, $buffer, 4096) {
$lines += ($buffer =~ tr/\n//);
}
# limit number of lines to pick to number of lines in file
$pick = $lines if $pick > $lines;
# build list of N lines to pick, use a hash to prevent picking the
# same line multiple times
my %picked;
for (1 .. $pick) {
my $n = int(rand($lines)) + 1;
redo if $picked{$n}++
}
# loop over file extracting selected lines
seek($fh, 0, 0);
while (<$fh>) {
print if $picked{$.};
}
close $fh;