perl log db query errors into a log file - perl

So I started to get familiar with Perl and I wrote my first Db script.
Now I am trying to select data from atable which is huge and trying to insert into a summary table based on some criteria.
Now there are chances , that select query may fail or the insert query may fail due to timeout or other database issues that is beyond my control.
Eventually my script is going to be cron script.
Can I log just the errors that i encounter for the connection,inserts and selects into a file generated in the script?
$logfile = $path.'logs/$currdate.log';
here is my code:
my $SQL_handled="SELECT division_id,region_id, NVL(COUNT(*),0) FROM super_tab GROUP BY division_id,region_id;";
my $result_handled = $dbh->prepare($SQL_handled);
$result_handled->execute();
while (my ($division_id,$region_id,$count ) = $result_handled->fetchrow_array()){
my $InsertHandled="INSERT INTO summary_tab (date_hour, division_id, region_id,volume) VALUES ('$current',$division_id,$region_id,$market_id,'$service_type','$handled',$count);";
my $result_insert_handled = $dbh->prepare($InsertHandled);
$result_insert_handled->execute();
}
something like
if(DBI-query failed ) {
// log the error onto the above logpath
}

Its usually done like this
my $SQL_handled="SELECT division_id,region_id, NVL(COUNT(*),0) FROM super_tab GROUP BY division_id,region_id;";
my $result_handled = $dbh->prepare($SQL_handled);
my $retval = $result_handled->execute();
if(!$retval){
#open a log file and write errors
writelog();
die "Error executing SQL SELECT - $dbh->errstr";
}
while(my ($division_id,$region_id,$count ) = $result_handled->fetchrow_array()){....
}
---------------------------------
sub writelog{
my $path = "/path/to/logfile";
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
$year += 1900;
$mon++;
my $currdate = "$mon$mday$year";
$logfile = $path . "/$currdate.log";
open (OUT, ">>$logfile");
print OUT "There was an error encountered while executing SQL- $dbh->errstr \n";
close(OUT);
}
You can also use $dbh->err; which returns the native Oracle error code to trap the error and exit accordingly.
The above, basic exception handling can be performed for every execute() method call in your script. Remember, DBI will have AutoCommit set to 1 (enabled) by default, unless explicitly disabled. So your transactions would be auto committed per insert, in order to handle the ATOMICITY of the entire transaction, you can disable autocommit and use $dbh->commit and $dbh->rollback to handle when you want to commit, or may be use some custom commit point (for larger sets of data).
Or the below can be used while connecting to the DB
$dbh = DBI->connect( "dbi:Oracle:abcdef", "username", "password" , {
PrintError => 0, ### Don't report errors via warn( )
RaiseError => 1 ### Do report errors via die( )
} );
this would automatically report all errors via die. The RaiseError is usually turned off by default.
Also if I understand you correctly then, by cron you mean you would be calling it from a shell cron job. In that case, call to your perl script from the cron itself can be redirected to log files something like below
perl your_perl.pl >> out.log 2>> err.log
out.log will contain regular logs and err.log will contain errors (specifically thrown by DBI prepare() or execute() methods too). In this case, you also need to make sure you use proper verbiage in print or die so that the logs look meaningful.

First, bear in mind that if you put an email address at the top of your crontab file any output from the cron job will be emailed to you:
MAILTO=me#mydomain.com
Second, if you set DBI's RaiseError to 1 when you connect you do not need to check every call, DBI will raise an error whenever one happens.
Third, DBI has an error handler callback. You register a handler and it is called whenever an error occurs with the handle in error and error text etc. If you return false from the error handler, DBI works as it would without the handler and goes on to die or warn. As a result, it is easier to set RaiseError and create an error handler than as Annjawn suggested.
Lastly, if you don't want to do this yourself, you can use something like DBIx::Log4perl and simply ask for it to log errors and nothing else. Any errors will be written to your Log4perl file and they include the SQL being executed, parameters etc.

Related

"Insecure dependency error while running with -T switch" using cicindela2

I am applying the cicindela2 recommendation engine
It uses Apache mod_perl and the Perl DBI module.
Here is the rough flow of how it works
Data input by Record Handler
Data is passed through the filter chain for batch processing
Temporary tables are output from batch processing
Recommendation result is requested by accessing the Recommend Handler which trigger the action of Recommender
I configured an aggregation and ran the project batch script. I know that the batch processing succeeded because I saw the output of processing from DB. But when I tried to access the recommendation result with URL that triggers the Recommend Handler, I saw a blank white page and the log said
FATAL: Insecure dependency in parameter 1 of DBIx::ContextualFetch::db=HASH(0x7f2a76169e78)->prepare_cached method call while running with -T switch at /usr/local/share/perl5/Ima/DBI.pm line 398.
This is where the error was thrown from the
Ima::DBI
base module
/usr/local/share/perl5/Ima/DBI.pm.
sub _mk_sql_closure {
my ($class, $sql_name, $statement, $db_meth, $cache) = #_;
return sub {
my $class = shift;
my $dbh = $class->$db_meth();
# Everything must pass through sprintf, even if #_ is empty.
# This is to do proper '%%' translation.
my $sql = $class->transform_sql($statement => #_);
return $cache # Line 398
? $dbh->prepare_cached($sql)
: $dbh->prepare($sql);
};
}
It seems that the SQL query prepared by the program is insecure, right?
What is reason for this error?
Is it related to the function of cache management of DBI?
Would it be solved if I clear the cache regularly?
Also, I tried to log the SQL statement generated, but the output failed even when I placed something like $LOGGER->warn("123") in the handle subroutine of the Recommend Handler.
How come the log failed and how to log it correctly?
Insecure dependency... while running with -T switch is Perl's way of telling you that you're running with taint mode active and attempting to do something with tainted data which could be potentially unsafe. In this particular case, $sql is tainted, because some or all of its content came from sources external to the program - probably user input, although it could also have been read from a file.
To fix this, you need to think about where $sql came from, so that you can work out the appropriate way to clean it up.
In the most likely scenario, you've asked a user to supply search terms and then inserted those terms directly into your SQL string. This is a bad idea in general, as it opens you up to the possibility of SQL injection attacks. (Obligatory Bobby Tables link.) Revise your SQL handling to make use of SQL placeholders instead of inserting user input into the WHERE clause and this vulnerability should go away.
If tainted data is making its way into $sql in some other way, you need to clean up the tainted data by using a regular expression to validate it and capture the validated data, then assign the captured data to your variable. e.g.,
my $tainted = <STDIN>;
$tainted =~ /([A-Z]*)/; # Only allow uppercase characters
my $clean = $1; # No longer tainted because it came from $1
If you need to take this route, DO NOT use .* as your regex to untaint the data without serious, serious consideration, because, if you just blindly accept any and all data, you will be discarding any and all benefit provided by taint mode.

Perl dbi : fetchrow_array limit result :

When I use the same request in mySQL Workbench, the count of results is 41100.
In my script, the count is 1015. I have no error in my code (I use die for see it).
What is the cause of this difference?
# Create the statement
# select all id in table contacts
$statement = "select distinct contacts.id from $database.contacts";
# Prepare and execute the SQL query
$sqlQuery = $connectGrc->prepare($statement)
# Execute the statement
$sqlQuery->execute()
# Make id in array
while($oneId = $sqlQuery->fetchrow_array())
{
push(#ArrayId,$oneId);
}
There's nothing obviously wrong (except for the lack of error checking and that your comments are not program comments). Try connecting with RaiseError on, 'use strict;', and see if there are problems which you otherwise might not see. Also, if all you want is a one-column array of results, and can settle for an array reference, consider selectcol_arrayref() for convenience.

Unable to execute an oracle update statement within perl

I have a problem in a perl script that I'm writing. When I run the script it hangs after prepare(). I've tried to run the update statement from SQL Developer and it works fine.
I've also tried to print out all parameters and they are correct.
What am I missing here?
my $upd = 'update ngs.pp_subscr_data set address=?, city=?, postalcode=?, kennitala=?, email=?, firstname=?, lastname=?, last_upd=systimestamp where snb=?';
my $s = $dbh->prepare ($upd) || exitError(-9802, 'Couldn\'t prepare update statement.');
$s->execute($addr, $city, $pcode, $ktala, $email, $fname, $lname, $snb) || exitError(-9803, 'Couldn\'t execute statement: ' . $s->errstr);
Thanks.
First, what version of Oracle?
Ok, I see couple of problems with your description. When you say "hang", is it really a hang? Could it be spinning?
Also, second, you say "... it hangs after prepare()". Does that mean it hangs after you call prepare(), or after it returns from prepare()?
Is it hanging in the database, and your client program is waiting for a database call to complete?
You need to run the program, then look at V$SESSION, identify the SID that corresponds to the database session of your program, and see what it's doing. Look at the EVENT column in V$SESSION. Also, look at the STATUS column to tell if the session is currently in a database call (ACTIVE), or waiting for the client program to call the database (INACTIVE).
Report some information back, and I may be able to provide further insight.
Hope that helps.

Parsing help in Perl

My (existing) perl files creates a Log file using Log4Perl in the following format
[2011-11-21 08:50:22,406] States_Sync INFO Logger.pm:33 script starts
[2011-11-21 08:50:22,610] States_Sync ERROR Logger.pm:36 Error occurred ....
[2011-11-21 08:50:22,406] States_Sync INFO Logger.pm:33 ...
[2011-11-21 08:50:22,610] States_Sync ERROR Logger.pm:36 Error occurred ....
[2011-11-21 08:50:22,406] States_Sync INFO Logger.pm:33 ...
[2011-11-21 08:50:22,610] States_Sync ERROR Logger.pm:36 Error occurred ....
The above is only an example of my log file. I use the following formatter
$layout = Log::Log4perl::Layout::PatternLayout->new("[%d{ISO8601}] %c %p %F{1}:%L %m%n");
Currently I have to send an email in case of error.
Instead of modifying the existing script, I thought of parsing the generated log files only for Error and send all the messages related to "Error" from the log file as email
Is there any easy way of parsing the log file ?
Regards,
Karthik
use grep(1):
grep ERROR log.file
or use perl:
perl -ne 'print if /ERROR/' log.file
I might try this:
if ( /^[^:]+?\s+ERROR\s/ ) {
# pull fields
# send email
}
Try that one out and see if it gives you too many lines. More elaborate version might be:
if ( /^\[\d{4}-\d\d-\d\d \d\d:\d\d:\d\d,\d{3}\]\s+([\w\s])?\s+ERROR\s+/ ) {
but only if the data was more complicated.
Since you're using log4perl, you could use the built-in methods there to check for an ERROR (or above) and send an e-mail based on that.
if ($layout->is_error()){
# Put e-mail logic here
}
And if you wanted just error information and not warn, info, debug or trace you can do the following:
if($layout->is_error() && ! $layout->is_warn()){
# Put e-mail logic here
}
You did not specify how often do you have to check the log file and send the e-mails. If it once a day then the solution provided by tadmc would be your best bet.
However, if you would like to automate it and send an e-mail after a specific interval of time (see maxinterval) for each new ERROR entry encountered, you may want to check the following.
NOTE #1 Adjust the interval and maxinterval as per your requirements to not flood people with e-mails
NOTE #2 Run it in accordance to your log rotate intervals
#!/usr/bin/perl
use strict;
use warnings;
use File::Tail;
my #logs_to_email;
my $log_file = "file.log";
my $error_pattern = qr(^\[.*?\]\s*States_Sync\s*ERROR);
my $tail = File::Tail->new(
name => $log_file,
maxinterval => 60,
interval => 10,
adjustafter => 10,
);
while (defined (my $line = $tail->read)) {
chomp $line;
next if $line =~ /^\s*$/;
next unless $line =~ $error_pattern;
push #logs_to_email, $line;
##
## put e-mail logic to send
## #logs_to_email here
##
}

After querying DB I can't print data as well as text anymore to browser

I'm in a web scripting class, and honestly and unfortunately, it has come second to my networking and design and analysis classes. Because of this I find I encounter problems that may be mundane but can't find the solution to it easily.
I am writing a CGI form that is supposed to work with a MySQL DB. I can insert and delete into the DB just fine. My problem comes when querying the DB.
My code compiles fine and I don't get errors when trying to "display" the info in the DB through the browser but the data and text doesn't in fact display. The code in question is here:
print br, 'test';
my $dbh = DBI->connect("DBI:mysql:austinc4", "*******", "*******", {RaiseError => 1} );
my $usersstatement = "select * from users";
my $projstatment = "select * from projects";
# Get the handle
my $userinfo = $dbh->query($usersstatement);
my $projinfo = $dbh->query($projstatement);
# Fetch rows
while (#userrow = $userinfo->fetchrow()) {
print $userrow[0], br;
}
print 'end';
This code is in an if statement that is surrounded by the print header, start_html, form, /form, end_html. I was just trying to debug and find out what was happening and printed the statements test and end. It prints out test but doesn't print out end. It also doesn't print out the data in my DB, which happens to come before I print out end.
What I believe I am doing is:
Connecting to my DB
Forming a string the contains the command/request to the DB
Getting a handle for my query I perform on the DB
Fetching a row from my handle
Printing the first field in the row I fetched from my table
But I don't see why my data wouldn't print out as well as the end text. I looked in DB and it does in fact contain data in the DB and the table that I am trying to get data from.
This one has got me stumped, so I appreciate any help. Thanks again. =)
Solution:
I was using a that wasn't supported by the modules I was including. This leads me to another question. How can I detect errors like this? My program does in fact compile correctly and the webpage doesn't "break". Aside from me double checking that all the methods I do use are valid, do I just see something like text not being displayed and assume that an error like this occurred?
Upon reading the comments, the reason your program is broken is because query() does not execute an SQL query. Therefore you are probably calling an undefined subroutine unless this is a wrapper you have defined elsewhere.
Here is my original posting of helpful hints, which still apply:
I hope you have use CGI, use DBI, etc... and use CGI::Carp and use strict;
Look in /var/log/apache2/access.log or error.log for the bugs
Realize that the first thing a CGI script prints MUST be a valid header or the web server and browser become unhappy and often nothing else displays.
Because of #3 print the header first BEFORE you do anything, especially before you connect to the database where the script may die or print something else because otherwise the errors or other messages will be emitted before the header.
If you still don't see an error go back to #2.
CGIs that use CGI.pm can be run from a command line in a terminal session without going through the webserver. This is also a good way to debug.