Modifying Large files line by line using perl script - perl

Question

I am having two huge .csv files one is around 8 GB and other is 3.4 GB file size. I want only few values from each line inside that .csv files.
Its taking huge time to modify the data and copy it into new file.
Could anyone help in modifying the code.So that the modification will be completed in a reasonable time.
Below is the lines of code:
#!/usr/bin/perl
use strict;
use warnings;
use Text::CSV;
require "$ENV{'SAI_HOME'}/bin/utils/Logging.pl";
require "$ENV{'SAI_HOME'}/bin/utils/Utilities.pl";
my $date1 = `date '+%d-%m-%Y_%H-%M-%Ss'`;
chomp($date1);
our $LOGPATH = "$ENV{'SAI_HOME'}/logs/SP6migrationcsv_$date1.log";
my $status = 0;
log_info("Refer $LOGPATH log file for more information");
my $csv = Text::CSV->new( { binary => 1, eol => $/, sep_char => ',' } );
my $file1 = $ARGV[0] or die "Please provide Subscriber and Subscription CSV files on the command line\n";
my $file2 = $ARGV[1] or die "Please provide Subscriber and Subscription CSV files on the command line\n";
my $subscriberFile = "";
my $subscriptionFile = "";
if ( ( grep /SUBSCRIBER/i, $file1 ) && ( grep /SUBSCRIPTION/i, $file2 ) ) {
$subscriberFile = $file1;
$subscriptionFile = $file2;
} elsif ( ( grep /SUBSCRIBER/i, $file2 ) && ( grep /SUBSCRIPTION/i, $file1 ) ) {
$subscriptionFile = $file1;
$subscriberFile = $file2;
} else {
log_error("Invalid CSV files input");
exit -1;
}
my $SP6DIR = `dirname $0`;
chomp $SP6DIR;
$SP6DIR = "${SP6DIR}/SP6";
`mkdir -p $SP6DIR` or checkExit( $?, "Unable to carete $SP6DIR directory" );
my $newSubscriberFile = "Subscriber.csv";
my $newSubscriptionFile = "Subscription.csv";
my $subscriptionimsifile = "$SP6DIR/.IMSI_$newSubscriptionFile";
my $subscriberimsifile = "$SP6DIR/.IMSI_$newSubscriberFile";
$newSubscriberFile = "${SP6DIR}/$newSubscriberFile";
$newSubscriptionFile = "${SP6DIR}/$newSubscriptionFile";
`dos2unix $subscriptionFile $subscriberFile 2>/dev/null`
or checkExit( $?, "Unable to perform dos2unix on input files" );
`cut -d "," -f3 $subscriptionFile > $subscriptionimsifile`
or checkExit( $?, "Failed to get IMSI details from $subscriptionFile" );
`cut -d "," -f1 $subscriberFile > $subscriberimsifile`
or checkExit( $?, "Failed to get IMSI details from $subscriberFile" );
my $isSubscriptionHeaderPresesnt = "false";
my $isSubscriberHeaderPresesnt = "false";
$status = system("head -1 $subscriptionimsifile | grep 'IMSI' >>/dev/null");
if ( $status == 0 ) {
$isSubscriptionHeaderPresesnt = "true";
}
$status = system("head -1 $subscriberimsifile | grep 'IMSI' >>/dev/null");
if ( $status == 0 ) {
$isSubscriberHeaderPresesnt = "true";
}
open( my $subscriptionData, '<:encoding(utf8)', $subscriptionFile )
or die "Could not open '$subscriptionFile' $!\n";
open( NEWSUBSCRIBERDATA, "> $newSubscriberFile" ) or die "Could not open '$newSubscriberFile' $!\n";
open( NEWSUBSCRIPTIONDATA, "> $newSubscriptionFile" ) or die "Could not open '$newSubscriptionFile' $!\n";
if ( "$isSubscriptionHeaderPresesnt" eq "true" ) {
my $subscriptionHeader = <$subscriptionData>;
if ( $csv->parse($subscriptionHeader) ) {
my #subscriptionHeaderFields = $csv->fields();
print NEWSUBSCRIPTIONDATA "\"$subscriptionHeaderFields[0]\",\"$subscriptionHeaderFields[2]\",\"$subscriptionHeaderFields[4]\",\"$subscriptionHeaderFields[5]\",\"$subscriptionHeaderFields[6]\",\"$subscriptionHeaderFields[8]\",\"$subscriptionHeaderFields[13]\",\"$subscriptionHeaderFields[14]\",\"$subscriptionHeaderFields[15]\",\"$subscriptionHeaderFields[16]\",\"$subscriptionHeaderFields[17]\",\"$subscriptionHeaderFields[18]\",\"$subscriptionHeaderFields[25]\",\"$subscriptionHeaderFields[26]\",\"$subscriptionHeaderFields[27]\"\n";
print NEWSUBSCRIBERDATA "\"IMSI\",\"IMEI\",\"MSISDN\",\"$subscriptionHeaderFields[21]\",\"$subscriptionHeaderFields[22]\",\"$subscriptionHeaderFields[12]\",\"$subscriptionHeaderFields[9]\",\"$subscriptionHeaderFields[1]\",\"$subscriptionHeaderFields[0]\"\n";
} else {
log_error("Line could not be parsed: $subscriptionHeader\n");
exit 1;
}
} else {
log_only("No header info in subscription file");
}
if ( "$isSubscriptionHeaderPresesnt" eq "false" && "$isSubscriberHeaderPresesnt" eq "true" ) {
print NEWSUBSCRIBERDATA "\"IMSI\",\"IMEI\",\"MSISDN\",\"CUSTOMER_SEGMENTATION\",\"CUST_SUBCATEGORY\",\"SUBS_TYPE\",\"SUBSCRIPTION_PLAN\",\"CONTRACT_IDREF\",\"SUBSCRIPTION_IDREF\"\n";
} else {
log_only("No header info in subscriber file");
}
my $subscriberHeader = "";
my #subscriptionFields = {};
my #subscriberFields = {};
while ( my $eachSubscriptionLine = <$subscriptionData> ) {
chomp $eachSubscriptionLine;
if ( $csv->parse($eachSubscriptionLine) ) {
#subscriptionFields = $csv->fields();
$status = system("grep \"^[\\\"]*${subscriptionFields[2]}[\\\"]*\\\$\" $subscriberimsifile >> /dev/null");
if ( $status == 0 ) {
my $lastMatchedSubscriberdata = `grep "^[\\\"]*${subscriptionFields[2]}[\\\"]*," $subscriberFile | tail -1`;
chomp $lastMatchedSubscriberdata;
if ( $csv->parse($lastMatchedSubscriberdata) ) {
#subscriberFields = $csv->fields();
if ( "${subscriberFields[0]}" eq "${subscriptionFields[2]}" ) {
#log_only("Updating \"#subscriberFields\" subscriber details from subscription data");
print NEWSUBSCRIBERDATA "\"$subscriberFields[0]\",\"$subscriberFields[1]\",\"$subscriptionFields[2]\",\"$subscriptionFields[21]\",\"$subscriptionFields[22]\",\"$subscriptionFields[12]\",\"$subscriptionFields[9]\",\"$subscriptionFields[1]\",\"$subscriptionFields[0]\"\n";
} else {
log_error("Unable to process #subscriberFields record");
exit -1;
}
} else {
log_error("Line could not be parsed: $lastMatchedSubscriberdata\n");
exit 1;
}
} else {
log_only("Adding new subscriber details from subscription : \"#subscriptionFields\"");
print NEWSUBSCRIBERDATA "\"$subscriptionFields[2]\",,\"$subscriptionFields[3]\",\"$subscriptionFields[21]\",\"$subscriptionFields[22]\",\"$subscriptionFields[12]\",\"$subscriptionFields[9]\",\"$subscriptionFields[1]\",\"$subscriptionFields[0]\"\n";
}
print NEWSUBSCRIPTIONDATA "\"$subscriptionFields[0]\",\"$subscriptionFields[2]\",\"$subscriptionFields[4]\",\"$subscriptionFields[5]\",\"$subscriptionFields[6]\",\"$subscriptionFields[8]\",\"$subscriptionFields[13]\",\"$subscriptionFields[14]\",\"$subscriptionFields[15]\",\"$subscriptionFields[16]\",\"$subscriptionFields[17]\",\"$subscriptionFields[18]\",\"$subscriptionFields[25]\",\"$subscriptionFields[26]\",\"$subscriptionFields[27]\"\n";
} else {
log_error("Line could not be parsed: $eachSubscriptionLine\n");
exit 1;
}
}
close(NEWSUBSCRIPTIONDATA);
open( my $subscriberData, '<:encoding(utf8)', $subscriberFile ) || die "Could not open '$subscriberFile' $!\n";
if ( "$isSubscriberHeaderPresesnt" eq "true" ) {
$subscriberHeader = <$subscriberData>;
}
while ( my $eachSubscriberLine = <$subscriberData> ) {
chomp $eachSubscriberLine;
if ( $csv->parse($eachSubscriberLine) ) {
#subscriberFields = $csv->fields();
$status = system("grep \"^[\\\"]*${subscriberFields[0]}[\\\"]*\\\$\" $subscriptionimsifile >>/dev/null");
if ( $status != 0 ) {
log_only(
"Adding back subscriber details, because unable to get IMSI details from subscription file : \"#subscriberFields\""
);
print NEWSUBSCRIBERDATA "\"$subscriberFields[0]\",\"$subscriberFields[1]\",\"$subscriberFields[2]\",\"$subscriberFields[6]\",,\"$subscriberFields[7]\",,,\n";
}
} else {
log_error("Line could not be parsed: $eachSubscriberLine\n");
exit 1;
}
}
close(NEWSUBSCRIBERDATA);
`sed -i -e '1 s|SUBSCRIPTION_ID|SUBSCRIPTION_IDREF|g' -e '1 s|SUBS_CATEGORY|SUBSCRIPTION_PLAN|g' -e '1 s|SUBS_STATE|SUBS_TYPE|g' -e '1 s|CUST_CATEGORY|CUSTOMER_SEGMENTATION|g' $newSubscriberFile`
or checkExit( $?, "Unable to update header info in subscriber fi le" );

Answer 1

General advice:
Don't use backticks to parse your input. Perl is perfectly capable of doing this with a while loop and split.
Misspelling variable names is going to screw you. Don't. isSubscriptionHeaderPresesnt
mixing your open calls - 3 argument with lexicals is generally preferred, but mismatching isn't nice.
Using the text string "false" in lieu of a boolean is horrible. Don't do it. Someone someday is going to do the equivalent of print "true" if "false" and it's going to break.
The most 'expensive' operation your script will be doing is reading the files. That's almost always true. So find something other than syscalls to grep or sed require full-rereading of the files you're targeting. Assuming subscriptionFile and subscriberFile are your biggies, you're reading them multiple times - you're running a cut that reads the whole thing. A dos2unix that reads the whole thing. A grep that reads the whole thing. And then you're opening it, and reading the whole thing.
your last line is a sed which will... re-read your output file, entirely, and apply a line by line transform to it.