Splitting large file into small multiple files based on the column concatenation - perl

Question

I have an input file as following. I need to break them into multiple files based on the columns 2,3&5. The file has more columns but i have used cut command to get only the required columns.
12,Accounts,India,free,Internal
13,Finance,China,used,Internal
16,Finance,China,free,Internal
12,HR,India,free,External
19,HR,China,used,Internal
33,Finance,Japan,free,Internal
39,Accounts,US,used,External
14,Accounts,Japan,used,External
11,Finance,India,used,External
11,HR,US,used,External
10,HR,India,used,External
Output files:
Accounts_India_Internal --
12,Accounts,India,free,Internal
Finance_China_Internal --
13,Finance,China,used,Internal
16,Finance,China,free,Internal
HR_India_External --
12,HR,India,free,External
10,HR,India,used,External
HR_China_Internal --
19,HR,China,used,Internal
and so on..
Please let me know how to achieve this.
As of now, I am thinking to sort the file based on these columns (2,3,5) and then run a loop on each record and start creating files. If a file does not exist, then create and add the record. Otherwise open the old file and add the record.
Is it possible to do this using shell scripting (bash)?

Answer 1

Is it possible to do this using shell scripting (bash)?
If you simply want to split the files based on fields 2, 3 and 5 you can do that quickly with awk:
awk -F, '{print >> $2"_"$3"_"$5}' infile.txt
That appends each line to a file whose name is made up of fields 2, 3 and 5.
Example:
[me#home]$ awk -F, '{print >> $2"_"$3"_"$5}' infile.txt
[me#home]$ cat Accounts_India_Internal
12,Accounts,India,free,Internal
[me#home]$ cat Finance_China_Internal
13,Finance,China,used,Internal
16,Finance,China,free,Internal
If you do want output sorted, you can first run the file through sort.
sort -k2,3 -k5,5 -t, infile.txt | awk -F, '{print >> $2"_"$3"_"$5}'
That sorts the lines on fields 2, 3, and 5 before passing them on to the awk command.
Do note that the we're appending to the files so if you repeat the command without deleting the output files, you'll end up with duplicate data in the output files. To address this, as well as include your additional requirements (using first line as header for all new files) as mentioned in the chat, see this solution.

Answer 2

I suggest you keep a hash of file handles keyed by their corresponding file names
This program demonstrates. The input file is expected as a parameter on the command line
use strict;
use warnings;
my %fh;
while (<>) {
chomp;
my $filename = join '_', (split /,/)[1,2,4];
if (not $fh{$filename}) {
open $fh{$filename}, '>', $filename or die "Unable to open '$filename' for output: $!";
print "$filename created\n";
}
print { $fh{$filename} } $_, "\n";
}
output
Accounts_India_Internal created
Finance_China_Internal created
HR_India_External created
HR_China_Internal created
Finance_Japan_Internal created
Accounts_US_External created
Accounts_Japan_External created
Finance_India_External created
HR_US_External created

Answer 3

Note: To use the code, simply change <DATA> to <> and use the file name as argument. The Data::Dumper print is there only for demonstration purposes and can also be removed.
use strict;
use warnings;
use Data::Dumper;
my %h;
while (<DATA>) {
chomp;
my #data = split /,/;
my $file = join "_", #data[1,2,4];
push #{$h{$file}}, $_;
}
print Dumper \%h;
__DATA__
12,Accounts,India,free,Internal
13,Finance,China,used,Internal
16,Finance,China,free,Internal
12,HR,India,free,External
19,HR,China,used,Internal
33,Finance,Japan,free,Internal
39,Accounts,US,used,External
14,Accounts,Japan,used,External
11,Finance,India,used,External
11,HR,US,used,External
10,HR,India,used,External
To print the files, you could use a subroutine like so:
for my $key (keys %h) {
print_file($key, $h{$key};
}
sub print_file {
my ($file, $data) = #_;
open my $fh, ">", $file or die $!;
print $fh "$_\n" for #$data;
}

Answer 4

save input text as foo, then:
cat foo | perl -nle '$k = join "_", (split ",", $_)[1,2,4]; $t{$k} = [#{$t{$k}}, $_]; END{for (keys %t){print join "\n", "$_ --", #{$t{$_}}, undef }}' | csplit -sz - '/^$/' {*}