I have written a CGI script in perl. I am receiving POST request to that script from two different locations simultaneously.
From POST request i am populating five different tables. I get 500 rows in each POST request and then i populate each table with 100 rows.
My issue is that after some time one or two of the table start showing less number of rows. Both of these tables store ip in one of their column. Every time only these two tables are the one which is missing data which is weird for me. I am not able to understand why this is happening.
Although i receive something in postgresql logs and number of such logs are same as the number of rows missing from table.
UTC STATEMENT: insert into post (ip, count, location, source_server) values ($1, $2, $3, $4)
UTC STATEMENT: insert into post (ip, count, location, source_server) values ($1, $2, $3, $4)
UTC STATEMENT: insert into post (ip, count, location, source_server) values ($1, $2, $3, $4)
This is my CGI code.
use strict;
use warnings;
use CGI;
use DBI;
use JSON;
#Taking parameters passed as post request
my $conn = CGI->new;
my $data = $conn->param('POSTDATA');
#Details regarding database
my $host = 'host';
my $user = 'user';
my $password = 'password';
my $database = 'db';
my ( $location, $table, $entry );
#Creating connection to DB
my $dsn = "dbi:Pg:dbname=$database;host=$host";
my $dbh
= DBI->connect( $dsn, $user, $password,
{ RaiseError => 0, PrintError => 0 } )
or die("Connection to DB failed $!\n");
#Decoding json received in post request to perl hash
$data = decode_json $data;
my ($source_server) = keys %$data;
my $location = 'XX';
my $main_data = $$data{$source_server};
my #db = qw /a b c d e/;
my %mapping = (
'a' => 'A',
'b' => 'B',
'c' => 'C',
'd' => 'D',
'e' => 'E'
);
foreach (#$main_data) {
my $hash = $_;
#Deciding which table data will be pushed into
$table = shift #db;
$entry = $mapping{$table};
my $sth = $dbh->prepare(
"insert into $table ($entry, count, location, source_server) values (?, ?, ?, ?)");
while ( ( my $key, my $value ) = each %$hash ) {
$sth->execute( $key, $value, $location, $source_server );
}
$sth->finish;
}
#Sending 200 status code back to client
print "Status: 200\n\n";
Data is passed to my CGI script in JSON as {KEY : [{HASH},{HASH},{HASH},{HASH},{HASH}]}. Second and fifth hash contains data for tables which are missing it after some time.
Every hash will have 100 key:value pairs thus making total 500 entries.
Well after simbabque comment above i enabled RaiseError and PrintError. After this i am able to get error in server error logs. Error logs indicate that because of simultaneous insertion of rows coming from multiple places i am getting primary key violation which was causing this strange behaviour.
I tweaked my primary key and all is well :)
Related
# I am saving output to an array and the array looks like this:-
60=20130624-09:45:02.046|21=1|38=565|52=20130624-09:45:02.046|35=D|10=085|40=1|9=205|100=MBTX|49=11342|553=2453|34=388|1=30532|43=Y|55=4323|54=1|56=MBT|11=584|59=0|114=Y|8=FIX.4.4|
# Then i converted this array to scalar variable like this:-
$scal=join('' , #arr);
# And now I am trying to save this into db:-
my $st = qq(INSERT INTO demo (fix)
VALUES ($scal));
my $r = $dbh->do($st) or die $DBI::errstr;
#And my table schema is:-
CREATE TABLE demo (fix varchar);
And I keep getting errors :- DBD::SQLite::db do failed: near ":45": syntax error at pdb.pl line 92, <STDIN> line 1.
DBD::SQLite::db do failed: near ":45": syntax error at pdb.pl line 92, <STDIN> line 1.
Any help will be appreicated
The way you denote your array is a bit weird. Usually you would write it as
my #arr = ( '60=20130624-09:45:02.046',
'21=1',
'38=565',
... );
or whatever your actual content is. But this is not the problem here because you flatten it to the string $scal anyway.
One way to insert this string into your DB is to put ticks (') around it:
my $st = qq(INSERT INTO demo (fix) VALUES ('$scal'));
my $r = $dbh->do($st) or die $DBI::errstr;
But this is bad because it's vulnerable to SQL injection (http://imgs.xkcd.com/comics/exploits_of_a_mom.png).
Consider the case your string is foo'); delete from demo; --. The final result would then be
INSERT INTO demo (fix) VALUES ('foo'); delete from demo; --')
The second reason why this is bad: Your string could contain ticks ($scal="foo's bar") and that also would mess up the resulting INSERT statement:
INSERT INTO demo (fix) VALUES ('foo's bar');
Conclusion: it's always better to use parameterized queries:
my $st = 'INSERT INTO demo (fix) VALUES (?)';
my $r = $dbh->do($st, undef, $scal) or die $DBI::errstr;
The undef is for additional SQL options (I've rarely seen anything different from undef here). The following parameters are replaced for the ?s in the statement. The DB driver does all the quoting for you. The more ? you use, the more parameters you must supply to do():
my $st = 'INSERT INTO sample_tbl (col1, col2, col3) VALUES (?, ?, ?)';
my $r = $dbh->do($st, undef, 'foo', 42, $scal) or die $DBI::errstr;
I am trying to run a simple database connection with Perl using postgreSQL
use DBI;
$database = "postgres";
$user = "postgres";
$password = "admin";
my $dbh = DBI->connect( "dbi:Pg:dbname=$database"
, $user
, $password
)
or die "Can't Connect to database: $DBI::errstr\n";
# get data from the URL string
my $firstname = "haroon";
my $lastname ="ash";
my $age = 24;
# insert the data into the database
my $query = "INSERT INTO people (firstname, lastname, age)
VALUES ('$firstname','$lastname', '$age')";
$dbh->do($query);
# get the ID of the inserted person
$query = "SELECT MAX(id) FROM people";
my $sth = $dbh->prepare($query);
my $rv =$sth->execute;
if($rv < 0){
print $DBI::errstr;
}
else {
my $row = $sth->fetchrow_hashref;
my $person_id = $row->{'max'};
print $firstname, $lastname
. "was successfully inserted at position "
. $person_id;
}
I am trying to print the person id which i had entered latest. But my $person_id = $row->{'max'} seems to give me the correct answer instead of my $person_id = $row->{'id'};. I am not understanding why is that.
You might want to set column alias for query,
$query = "SELECT MAX(id) AS id FROM people";
as postgres is putting his own alias for you, and that is max.
If all you want is last inserted id, you can
my $query = "INSERT INTO people (firstname, lastname, age)
VALUES (?,?,?)
returning id
";
and fetch query as you would do with select. (check pg docs)
You can use the RETURNING keyword to return the id associated with the row you just inserted:
my $query = '
INSERT INTO people (firstname, lastname, age)
VALUES ($1, $2, $3)
RETURNING id';
my $sth = $dbh->prepare($query);
$sth->execute($firstname, $lastname, $age);
my $rv = $sth->fetchrow_hashref();
printf "%s, %s was successfully inserted at position %d\n",
$firstname, $lastname, $rv->{id};
You did not enter a PERSON_ID in your insert statement.
Instead it seems the ID field was populated by the DB, an auto increment value during insert, it seems.
Without knowing the table definition (and potential insert trigger on the table) it is hard/impossible to give you a better answer.
When I fetch the data this way is it possible then to access the column names and the column types or do I need an explicit prepare to reach this?
use DBI;
my $dbh = DBI->connect( ... );
my $select = "...";
my #arguments = ( ... );
my $ref = $dbh->selectall_arrayref( $select, {}, #arguments, );
Update:
With prepare I would do it this way:
my $sth = $dbh->prepare( $select );
$sth->execute( #arguments );
my $col_names = $sth->{NAME};
my $col_types = $sth->{TYPE};
my $ref = $sth->fetchall_arrayref;
unshift #$ref, $col_names;
The best solution is to use prepare to get a statement handle, as you describe in the second part of your question. If you use selectall_hashref or selectall_arrayref, you don't get a statement handle, and have to query the column type information yourself via $dbh->column_info (docs):
my $sth = $dbh->column_info('','',$table,$column); # or $column='' for all
my $info = $sth->fetchall_arrayref({});
use Data::Dumper; print Dumper($info);
(specifically, the COLUMN_NAME and TYPE_NAME attributes).
However, this introduces a race condition if the table changes schema between the two queries.
Also, you may use selectall_arrayref with the Slice parameter to fetch all the columns into a hash ref, it needs no prepared statement and will return an array ref of the result set rows, with each rows columns the key's to a hash and the values are the column values. ie:
my $result = $dbh->selectall_arrayref( qq{
SELECT * FROM table WHERE condition = value
}, { Slice => {} }) or die "Error: ".$dbh->errstr;
$result = [
[0] = { column1 => 'column1Value', column2 => 'column2Value', etc...},
[1] = { column1 => 'column1Value', column2 => 'column2Value', etc...},
];
Making it easy to iterate over results.. ie:
for my $row ( #$results ){
print "$row->{column1Value}, $row->{column2Value}\n";
}
You can also specify which columns to extract but it's pretty useless due to the fact it's more efficient to do that in your SQL query syntax.
{ Slice => { column1Name => 1, column2Name => 1 } }
That would only return the values for column1Name and column2Name just like saying in your SQL:
SELECT column1Name, column2Name FROM table...
I have been trying to use sql NOW() function while I update a table. But nothing happens to that date field, I think DBI just ignores that value. Code is :
dbUpdate($id, (notes => 'This was an update', filesize => '100.505', dateEnd => 'NOW()'));
and the function is :
sub dbUpdate {
my $id = shift;
my %val_hash = #_;
my $table = 'backupjobs';
my #fields = keys %val_hash;
my #values = values %val_hash;
my $update_stmt = "UPDATE $table SET ";
my $count = 1;
foreach ( #fields ) {
$update_stmt .= "$_ = ? ";
$update_stmt .= ', ' if ($count != scalar #fields);
$count++;
}
$update_stmt .= " WHERE ID = ?";
print "update_stmt is : $update_stmt\n";
my $dbo = dbConnect();
my $sth = $dbo->prepare( $update_stmt );
$sth->execute( #values, $id ) or die "DB Update failed : $!\n";
$dbo->disconnect || die "Failed to disconnect\n";
return 1;
}#dbUpdate
As you can see this is a "dynamic" generation of the sql query and hence I dont know where an sql date function(like now()) come.
In the given example the sql query will be
UPDATE backupjobs SET filesize = ? , notes = ? , dateEnd = ? WHERE ID = ?
with param values
100.55, This was an update, NOW(), 7
But the date column still shows 0000-00-........
Any ideas how to fix this ?
I have been trying to use sql NOW() function while I update a table. But
nothing happens to that date field, I think DBI just ignores that
value.
No, it doesn't. But the DB thinks you are supplying a datetime or timestamp data type when you are in fact trying to add the string NOW(). That of course doesn't work. Unfortunately DBI cannot find that out for you. All it does is change ? into an escaped (via $dbh->quote()) version of the string it got.
There are several things you could do:
Supply a current timestamp string in the appropriate format instead of the string NOW().
If you have it available, you can use DateTime::Format::MySQL like this:
use DateTime::Format::MySQL;
dbUpdate(
$id,
(
notes => 'This was an update',
filesize => '100.505',
dateEnd => DateTime::Format::MySQL->format_datetime(DateTime->now),
)
);
If not, just use DateTime or Time::Piece.
use DateTime;
my $now = DateTime->now->datetime;
$now =~ y/T/ /;
dbUpdate(
$id,
(notes => 'This was an update', filesize => '100.505', dateEnd => $now));
Tell your dbUpdate function to look for things like NOW() and react accordingly.
You can do something like this. But there are better ways to code this. You should also consider that there is e.g. CURRENT_TIMESTAMP() as well.
sub dbUpdate {
my $id = shift;
my %val_hash = #_;
my $table = 'backupjobs';
my $update_stmt = "UPDATE $table SET ";
# Check for the NOW() value
# This could be done with others too
foreach ( keys %val_hash ) {
if ($val_hash{$_} =~ m/^NOW\(\)$/i) {
$update_stmt .= "$_ = NOW()";
$update_stmt .= ', ' if scalar keys %val_hash > 1;
delete $val_hash{$_};
}
}
# Put everything together, join keeps track of the trailing comma
$update_stmt .= join(', ', map { "$_=?" } keys %val_hash );
$update_stmt .= " WHERE ID = ?";
say "update_stmt is : $update_stmt";
say "values are: ", join(', ', values %val_hash);
my $dbo = dbConnect();
my $sth = $dbo->prepare( $update_stmt );
$sth->execute( values %val_hash, $id ) or die "DB Update failed : $!\n";
$dbo->disconnect || die "Failed to disconnect\n";
return 1;
}
Write your queries yourself.
You're probably not going to do it and I'll not add an example since you know how to do it anyway.
Here's something else: Is this the only thing you do with your database while your program runs? It is not wise to connect and disconnect the database every time you make a query. It would be better for performance to connect the database once you need it (or at the beginning of the program, if you always use it) and just use this dbh/dbo everywhere. It saves a lot of time (and code).
Sorry but you can't use NOW() as an interpolated value like that.
when a ? is used in an SQL statement, the corresponding value is (via some mechanism or other) passed to the database escaped, so whatever value is interpreted as a string, not as SQL. So you are in effect attempting to use the string 'NOW()' as a date value, not the function NOW().
To use NOW() as a function, you will have to insert it into the SQL itself rather than pass it as a bound value. This means you will either have to use a hack of your dbupdate function or write a new one, or obtain the time as a string in perl and pass the resulting string the dbupdate.
I need to insert values in database using Perl's DBI module. I have parsed a file to obtain these values and hence these values are present in an arrays, say #array1, #array2, #array3. I know how to insert one value at a time but not from an arrays.
I know insert one value at a time:
$dbh = DBI->connect("dbi:Sybase:server=$Srv;database=$Db", "$user", "$passwd") or die "could not connect to database";
$query= "INSERT INTO table1 (id, name, address) VALUES (DEFAULT, tom, Park_Road)";
$sth = $dbh->prepare($query) or die "could not prepare statement\n";
$sth-> execute or die "could not execute statement\n $command\n";
I am not sure if I have array1 containing ids, array2 containing names, and array3 containing address, how would I insert values.
Since you have parallel arrays, you could take advantange of execute_array:
my $sth = $dbh->prepare('INSERT INTO table1 (id, name, address) VALUES (?, ?, ?)');
my $num_tuples_executed = $sth->execute_array(
{ ArrayTupleStatus => \my #tuple_status },
\#ids,
\#names,
\#addresses,
);
Please note that this is a truncated (and slightly modified) example from the documentation. You'll definitely want to check out the rest of it if you decide to use this function.
Use placeholders.
Update: I just realized you have parallel arrays. That is really not a good way of working with data items that go together. With that caveat, you can use List::MoreUtils::each_array:
#!/usr/bin/perl
use strict; use warnings;
use DBI;
use List::MoreUtils qw( each_array );
my $dbh = DBI->connect(
"dbi:Sybase:server=$Srv;database=$Db",
$user, $passwd,
) or die sprintf 'Could not connect to database: %s', DBI->errstr;
my $sth = $dbh->prepare(
'INSERT INTO table1 (id, name, address) VALUES (?, ?, ?)'
) or die sprintf 'Could not prepare statement: %s', $dbh->errstr;
my #ids = qw( a b c);
my #names = qw( d e f );
my #addresses = qw( g h i);
my $it = each_array(#ids, #names, #address);
while ( my #data = $it->() ) {
$sth->execute( #data )
or die sprintf 'Could not execute statement: %s', $sth->errstr;
}
$dbh->commit
or die sprintf 'Could not commit updates: %s', $dbh->errstr;
$dbh->disconnect;
Note that the code is not tested.
You might also want to read the FAQ: What's wrong with always quoting "$vars"?.
Further, given that the only way you are handling error is by dying, you might want to consider specifying { RaiseError => 1 } in the connect call.
How could you not be sure what your arrays contain? Anyway the approach would be the iterate through one array and assuming the other arrays have corresponding values put those into the insert statement
Another way would be to use a hash as an intermediate storage area. IE:
my $hash = {};
foreach(#array1) {
$hash->{id} = $array1[$_];
$hash->{name} = $array2[$_];
$hash->{address} = $array3[$_];
}
foreach( keys %$hash ) {
$sql = "insert into table values(?,?,?)";
$sth = $dbh->prepare($sql) or die;
$sth->execute($hash->{id}, $hash->{name}, $hash->{address}) or die;
}
Though again this depends on the three arrays being synced up. However you could modify this to do value modifications or checks or greps in the other arrays within the first loop through array1 (ie: if your values in array2 and array3 are maybe stored as "NN-name" and "NN-address" where NN is the id from the first array and you need to find the corresponding values and remove the NN- with a s// regex). Depends on how your data is structured though.
Another note is to check out Class::DBI and see if it might provide a nicer and more object oriented way of getting your data in.