perl DBI and placeholders - perl

I have this query select * from table where ID in (1,2,3,5...)
How is it possible to build this query with the DBI using placeholders ?
for example :
my #list = (1, 2, 3, 4, 5);
my $sql = "select * from table where ID in (?)";
$sth->prepare($sql);
$sth->execute();
What argument should I send to execute? Is it a list or a string separated by , or something else?

This should build your query dynamically according to the number of items in your array
my #list =(1,2,3,4,5);
my $sql ="select * from table where ID in (#{[join',', ('?') x #list]})";

It's not possible in that way. You need to specify a placeholder for each item in your array:
my #list = (1,2,3,4,5);
my $sql = "select * from table where ID in (?,?,?,?,?)";
$sth->prepare($sql);
$sth->execute(#list);
If your #list is not a fixed size, you need to build the $sql with the proper number of placeholders.

Quoting DBI documentation:
Also, placeholders can only represent single scalar values. For example, the following statement won't work as expected
for more than one value:
SELECT name, age FROM people WHERE name IN (?) # wrong
SELECT name, age FROM people WHERE name IN (?,?) # two names
Rewrite to:
my $sql = 'select * from table where ID in ( ?, ?, ?, ?, ? )';
$sth->prepare($sql);
$sth->execute(#list);

If you are using DBI to access a PostgreSQL database with the DBD::Pg driver, you can use:
my #list = (1, 2, 3, 4, 5);
my $sql = "select * from table where ID = ANY(?::INT[]);";
$sth->prepare ($sql);
$sth->execute (\#list);

Unless you know the exact number of elements you cannot use placeholders. Try this:
my #list = (1, 2, 3, 4, 5); # any number of elements
my $in = join(',', map { $dbh->quote($_) } #list);
my $sql = "select * from table where someid IN ($in)";

If you switch to DBIx::Simple you can just say:
$db->query('INSERT INTO foo VALUES (??)', $foo, $bar, $baz);
?? Means "as many as needed"
Edit:
Actually, I was a little too optimistic: "If the string (??) is present in the query, it is replaced with a list of as many question marks as #values."
So this does not seem to work:
$db->query( "SELECT * FROM foo WHERE id IN (??) AND stuff=?", #ids, $stuff )
Still useful though..
For the curious, the code in the module is:
# Replace (??) with (?, ?, ?, ...)
sub _replace_omniholder {
my ($self, $query, $binds) = #_;
return if $$query !~ /\(\?\?\)/;
my $omniholders = 0;
my $q = $self->{dbd} =~ /mysql/ ? $quoted_mysql : $quoted;
$$query =~ s[($q|\(\?\?\))] {
$1 eq '(??)'
? do {
Carp::croak('There can be only one omniholder')
if $omniholders++;
'(' . join(', ', ('?') x #$binds) . ')'
}
: $1
}eg;
}

I found a sure way for this to work summarizing all of the above advice. My Production query (I posted a much simpler version here) uses IN <>, where neither the codes nor their quantity is unknown. It could be a single Code (e.g. FIN), or a series of them (FITLC FITLD FITLU FITSC FITSD FITSU MDYLC MDYLD MDYLU). Some function returns that as a list.
The code that makes this happen is
#codes = get_muni_evcode( $category );
my $in = join( ', ', ('?') x #codes );
print "\n\nProcessing Category: $category --> Codes: #codes .. in: $in\n";
my $sql = "select distinct cusip9
from material_event
where event_date between (trunc(sysdate) - 1) + 2/3 and trunc(sysdate) + 2/3
and event_code in ($in)";
my $sth2 = $dbh->prepare($sql);
$sth2->execute( #codes );
while (my $s2 = $sth2->fetchrow_hashref('NAME_lc'))
{
my $cusip9 = $s2->{cusip9};
print "$cusip9\t";
.................. further processing ..............
}
The result sample:
Processing Category: RatingChange --> Codes: FITLC FITLD FITLU FITSC FITSD FITSU MDYLC MDYLD MDYLU MDYSC MDYSD MDYSU SPLD SPLPR SPLU SPSD SPSPR SPSU .. in: ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?
359496HQ2 359496GB6 359496GH3 359496GL4 359496HU3 359496HS8 359496HA7 359496HF6 359496GM2 359496HM1 359496HR0 359496HT6 359496GY6 359496GJ9 359496HL3 359496GU4 359496HK5 359496HN9 359496HP4 359496GW0 359496GZ3 359496HC3 359496GC4 359496GK6 359496GP5 359496GV2 359496GX8 359496GN0
I'm extremely grateful to everybody who posted their ideas here that finally made me find the right way to do this. It must be a pretty common problem I think.

Related

why autoincrement primary key column not autoincrementing in sqlite3

I have an sqlite3 db for an address book, and I created the address table with
my $dbcreate = qq(CREATE TABLE IF NOT EXISTS address
(ID INTEGER PRIMARY KEY AUTOINCREMENT,
LASTNAME TEXT NOT NULL,
FIRSTNAME TEXT NOT NULL,
COMPANY TEXT,
STREET TEXT,
CITY TEXT,
STATE TEXT,
COUNTRY TEXT,
ZIPCODE TEXT,
MOBILE TEXT,
OTHERPHONE TEXT,
FAX TEXT,
EMAIL TEXT,
EMAIL2 TEXT,
WEBSITE TEXT,
WEBSITE2 TEXT,
DOB DATE,
NOTES TEXT,
TAGS TEXT););
So the first column SHOULD, as I understand it, be an autoincrementing primary ID, only, it's not autoincrementing. My understanding from reading sqlite tutorials is that I should only have to enter the other 18 values, and that column should autoincrement, but if I insert 18 values, I'm told I'm missing one.
I've only been able to insert rows if I manually set the ROWID.
What gives?
Here's how I'm adding rows (unsuccessfully)
#!/usr/bin/perl -w
# script to enter a new addressbook entry
use DBI;
use strict;
use warnings;
print "Please enter a DB username? \n";
my $userid=<STDIN>;
chomp($userid);
print "Please enter a DB password? \n";
my $password=<STDIN>;
chomp($password);
my $driver = "SQLite";
my $database = "myaddress.db";
my $dsn = "DBI:$driver:dbname=$database";
my $dbh = DBI->connect($dsn, $userid, $password, { RaiseError => 1 })
or die $DBI::errstr;
print "Successfull DB connection\n";
print "Creating new address book entry\n";
my $arval = 0;
my #dbits = ('LASTNAME','FIRSTNAME','COMPANY','STREET','CITY','STATE','COUNTRY','ZIPCODE','MOBILE','OTHERPHONE','FAX','EMAIL','EMAIL2','WEBSITE','WEBSITE2','DOB','NOTES','TAGS');
# print "#dbits";
# for $b (0 .. 17) {
# print "\'$dbits[$b]\', ";
# }
print "-----\n";
my #nubits = ();
foreach my $databit(#dbits) {
print "Enter $databit: \n";
my $nubit = <STDIN>;
chomp($nubit);
print "$databit = $nubit\n\n";
push(#nubits,$nubit);
print "$databit = $nubits[$arval]\n";
my $arval = (++$arval);
print "$arval\n";
}
# print "new entry will be:\n";
# for $b (0 .. 17) {
# print "$nubits[$b] | ";
# }
my $entry = qq(INSERT INTO address VALUES('$nubits[0]','$nubits[1]','$nubits[2]','$nubits[3]','$nubits[4]','$nubits[5]','$nubits[6]','$nubits[7]','$nubits[8]','$nubits[9]','$nubits[10]','$nubits[11]','$nubits[12]','$nubits[13]','$nubits[14]','$nubits[15]','$nubits[16]','$nubits[17]'););
my $retval = $dbh->do($entry);
if($retval < 0){
print $DBI::errstr;
} else {
print "entry created successfully\n";
}
my $query = qq(select * from address);
print $query;
$dbh->disconnect();
I can manually use sqlite3 from the cli to enter rows, but, as mentioned, I have to manually set the ROWID/Primary ID.
In the past, I've only used sqlite with tcl/tk, never perl, but even though I'm a complete perl n00b, I don't think perl is my problem. sqlite3 is not behaving as expected (unless I've completely misread about a dozen tutorials that state that a primary key id set to autoincrement should, well, autoincrement).
It's not auto incrementing because you're already giving it an id, $nubits[0].
With no column list, INSERT INTO address VALUES (...) inserts all columns in the table. You could use NULL as #ReenactorRob suggests, but that just hides another problem with that query.
INSERT INTO address VALUES (...) requires knowledge of the order in which address was created to know that element 5 is street. If anything changes the ordering in the table, your INSERT breaks. If a column is added, your query breaks. If you put a value in the wrong slot (as you did) its difficult to tell that. You're much better off using an explicit column list.
INSERT INTO address
(lastname, firstname, company, street, ...)
VALUES
(...)
Now your ID will increment and you're protected from future table changes. If that seems like a lot of work, make it a function that takes a hash of values to build a query. It'll be much more readable than remembering what $nubits[12] is. But before you do that, look at DBIx::Class which has already done this for you.
I would be remiss if I didn't mention bind parameters. These are faster, allowing you to use prepared statements, and they protect you against SQL injection attacks.
Try using null in the insert like this:
my $entry = qq(INSERT INTO address VALUES(null,'$nubits[0]','$nubits[1]',...
As has been explained, the problem is that the items in the VALUES section of the INSERT are assigned one-for-one to the table's columns in the order they were first created. If you need to provide values for only a subset of the columns then you also need to provide a list of the column names that should receive the values.
Without the column names the values will be assigned to columns in the order that they were created, so it is the columns at the end that will not receive a value. In this case the database will allocate the value for the LASTNAME column to ID, and it is the TAGS element at the end that doesn't have a value.
It is also much better to use placeholders for the values in your SQL statement, and use prepare and execute instead of just do. That way the DBI module will quote and escape the actual values appropriately according to their data type.
This program shows how to do that
#!/usr/bin/perl
use strict;
use warnings;
use DBI;
use DBD::SQLite;
STDOUT->autoflush;
my ($driver, $dbname) = qw/ SQLite myaddress.db /;
my $dsn = "DBI:$driver:dbname=$dbname";
print 'Please enter a DB username: ';
my $username = <STDIN>;
chomp($username);
print 'Please enter a DB password: ';
my $password = <STDIN>;
chomp($password);
my $dbh = DBI->connect($dsn, $username, $password, { RaiseError => 1 });
print "Successful DB connection\n";
print "Creating new address book entry...\n";
print "-----\n";
my #dbits = qw/
LASTNAME FIRSTNAME COMPANY STREET
CITY STATE COUNTRY ZIPCODE
MOBILE OTHERPHONE FAX EMAIL
EMAIL2 WEBSITE WEBSITE2 DOB
NOTES TAGS
/;
my #nubits;
for my $databit (#dbits) {
print "Enter $databit: ";
my $nubit = <STDIN>;
chomp($nubit);
push #nubits, $nubit;
print "$databit = $nubits[-1]\n";
print scalar #nubits, "\n";
}
my $columns = join ', ', #dbits;
my $placeholders = join ', ', ('?') x #dbits;
my $insert = <<"__END__SQL__";
INSERT INTO address (
$columns
)
VALUES (
$placeholders
)
__END__SQL__
$insert = $dbh->prepare($insert);
$insert->execute(#nubits);
print "Entry created successfully\n";
$dbh->disconnect;
This creates an SQL statement in $insert that looks like this
INSERT INTO address (
LASTNAME, FIRSTNAME, COMPANY, STREET, CITY, STATE, COUNTRY, ZIPCODE, MOBILE, OTHERPHONE, FAX, EMAIL, EMAIL2, WEBSITE, WEBSITE2, DOB, NOTES, TAGS
)
VALUES (
?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?
)

How to use placeholders with "in (...)" clause?

I know I can do this with interpolation. Can I do it using placeholders?
I am getting this error:
DBD::Pg::st execute failed: ERROR: invalid input syntax for integer: "{"22,23"}" at ./testPlaceHolders-SO.pl line 20.
For this script:
#!/usr/bin/perl -w
use strict;
use DBI;
# Connect to database.
my $dbh = DBI->connect("dbi:Pg:dbname=somedb;host=localhost;port=5432", "somedb", "somedb");
my $typeStr = "22,23";
my #sqlParms = [ $typeStr ];
my $sqlStr = << "__SQL_END";
SELECT id
FROM states
WHERE typeId in (?)
ORDER BY id;
__SQL_END
my $query = $dbh->prepare($sqlStr);
$query->execute(#sqlParms);
my $id;
$query->bind_columns(\$id);
# Process rows
while ($query->fetch())
{
print "Id: $id\n";
}
Is there a way around it besides interpolation?
DBD::PG has support for PostgreSQL arrays, so you can simply write a query like this:
WHERE typeid = ANY( ARRAY[1,2,3] )
or, with a parameter...
WHERE typeid = ANY(?)
Then just use the array support
my #targets = (1,2,3);
# ...
$query->execute(\#targets);
Posting comment as answer, as requested.
Generate your own placeholder string. Like so:
my #nums = (22,23);
my $placeholder = join ",", ("?") x #nums;
$query->execute(#nums);
Yes. You must use placeholders for each value, such as IN (?, ?, ?). You can however generate the correct number of question marks using something like this (untested):
my #values = (22, 23, ...);
# will produce "?, ?, ..."
my $in = join ", ", ("?") x #values;
my $sqlStr = "SELECT id FROM states WHERE typeId in ($in) ORDER BY id;";
my $query = $dbh->prepare($sqlStr);
$query->execute(#values);
Note that if you use an ORM such as DBIx::Class instead, this sort of ugly hack gets abstracted away.
You have to build the SQL statement with the correct number of question marks and then set the parameter values. There is no way to bind a list to a single question mark.

Add char to the array elements

Is there any easier way to add char to the beginning and end of array elements in perl.
Ex:my $str ='table1,table2,table3'
my #arry = split (/,/,$str)
I like the output should look like
'table1','table2','table3'
So it can be used in sql query
Thanks.
join(',', map "'$_'", #arry)
is what you are asking for.
But it's much better to use placeholders:
my $str = 'table1,table2,table3';
my #arry = split(/,/, $str);
if (#arry) {
my $query = 'select * from tablename where colname in (' . join(',',('?') x #arry) . ')';
my $sth = $dbh->prepare($query);
$sth->execute(#arry);
...
}
If you are going to use DBI (https://metacpan.org/module/DBI) to work with your database, it can do it for you if you use placeholders in your queries, so that you don't have to quote values.
For example, if you want to insert something into a table:
$dbh->do("INSERT INTO table VALUES(?, ?, ?)", undef, #arry)
or die $dbh->errstr;
where #arry is an array containing exactly 3 values to be inserted into a table.
More on this can be found here https://metacpan.org/module/DBI#Placeholders-and-Bind-Values

DBD::SQLite: Insert or Update - Question

While writing my question (howto: insert a new entry and if it exists already, update it) I found some answers in the Related Questions:
$sql = "INSERT OR REPLACE INTO $table ( Id, Name, Rating ) VALUES( ?, ?, ? )";
$sth_rating = $dbh->prepare( $sql );
$sth_rating->execute( $id, $name, $rating );
.
$sql = "INSERT OR IGNORE INTO $table ( Id, Name, Rating ) VALUES ( ?, ?, ? )";
$sth_rating = $dbh->prepare( $sql );
$sth_rating->execute( $id, $name, $rating );
$sql = "UPDATE $table SET Rating = ? WHERE Id = ?";
$sth_rating = $dbh->prepare( $sql );
$sth_rating->execute( $rating, $id );
Is the second method more safe then the first one?
The second method is less safe because it is non-atomic. In other words, it happens in more than one step. Consider two processes both updating the same data. Time is moving down.
Process 1 Process 2
INSERT OR IGNORE...
INSERT OR IGNORE...
UPDATE...
UPDATE...
Process 1 starts first, but Process 2 sneaks in and does its update between. Process 1 then blows right over Process 2's update.
This isn't so bad in this particular situation, both processes are going to blow over each other no matter what, but you can easily get yourself into trouble by extending your technique.
(Unless I misunderstood the question and what you want is an upsert)

How can I write a subroutine for DBI updates with varying number of arguments?

I'm writing a subroutine for DBI updates, and are having some trouble figuring out how to add placeholder and stuff...
I have this:
sub row_update {
my $table = shift;
my %updates = #_;
my $placeholders = ...
$dbh->do("UPDATE $table SET (foo) WHERE (bar)") etc...
}
Any ideas?
I just want a simple update function where I can send in x number of arguments (as a hash).
Thanks in advance!
Something like this might be Good Enough:
sub update {
my ($dbh, $args) = #_;
my $table = $args->{table} || die 'need table';
my $updates = $args->{updates} || die 'need updates';
my #cols = keys %$updates;
my $query = 'UPDATE $table SET '.
(join ', ', map { '$_ = ?' } #cols)
($args->{where} ? ' WHERE '. $args->{where} : '');
my $sth = $dbh->prepare($query);
$sth->execute(map { $updates->{$_} } #cols);
return $sth;
}
Use it like:
my $sth = update $dbh, {
table => 'foo',
updates => {
col1 => 'new_value',
col2 => 'another_value',
},
where => 'id=42',
};
Really, though, you want to look into using an ORM like
DBIx::Class. It will do a much better job of building queries than
string manipulation like this will.
(Rewriting the where clause to be parameterized is left as an exercise to the reader. You also need to quote the update keys and table name. See why people use ORMs?)
Edit: Thinking about this a bit more, you might like DBIx::Simple combined with SQL::Abstract. This will take less configuration effort than an ORM, but still give you many of the benefits.
If I understand the question correctly, it sounds like you're after SQL::Abstract. First, we create an SQL::Abstract object:
use SQL::Abstract;
my $sql = SQL::Abstract->new;
Now, as an example, we'll use it to insert some data into a table:
my %record = (
FirstName => 'Buffy',
LastName => 'Summers',
Address => '1630 Revello Drive',
City => 'Sunnydale',
State => 'California',
Occupation => 'Student',
Health => 'Alive',
);
my ($stmt, #bind) = $sql->insert(’staff’,\%record);
This results in:
$stmt = "INSERT INTO staff
(FirstName, LastName, Address, City,
State, Occupation, Health)
VALUES (?, ?, ?, ?, ?, ?, ?)";
#bind = ('Buffy','Summers','1630 Revello Drive',
'Sunnydale',’California','Student','Alive');
The nice thing about this is we can pass it directly to DBI:
$dbh->do($stmt, undef, #bind);
Of course, you want to be updating records, not just inserting them. Luckily, this is also quite easy:
my $table = 'People';
my %new_fields = (
Occupation => 'Slayer',
Health => 'Dead',
);
my %where = (
FirstName => 'Buffy',
LastName => 'Summers',
);
my ($stmt, #bind) = $sql->update($table, \%new_fields, \%where);
$dbh->do($stmt, undef, #bind);
This produces:
$stmt = 'UPDATE People SET Health = ?, Occupation = ?
WHERE ( FirstName = ? AND LastName = ? )';
#bind = ('Dead', 'Slayer', 'Buffy', 'Summers');
If you're after more information about SQL::Abstract, I recommend you look at its CPAN page. There's also a chapter in Perl Training Australia's Database Programming with Perl manual, which are freely available from our course notes page.
All the best,
Paul
Disclaimer: I'm managing director of Perl Training Australia, and therefore think that our course notes are pretty good.
Others suggested the usual "build a query with the right number of '?'s" approach.
For most queries like this the DBI->quote method is forgotten, but it can keep the code simpler and in most cases it's not any slower than the "proper" placeholder approach.
1) Use DBI->quote instead of place holders to build the query. For example for a simple select:
my $sql = "select foo from bar where baz in ("
. join(",", map { DBI->quote($_) } #bazs)
. ")";
my $data = $dbh->selectall_arrayref($sql);
2) As jrockway suggested - use an ORM to do this sort of low level stuff for you. DBIx::Class or Rose::DB::Object, for example.
The NUM_OF_PARAMS attribute may be of assistance:
"NUM_OF_PARAMS" (integer, read-only)
The number of parameters (placeholders) in the prepared
statement.
So for instance, I've got a script to run arbitrary SQL from the command line that uses this code:
my $s = $h->prepare($_);
for my $i (1..$s->{NUM_OF_PARAMS}){
my $param = shift #ARGV;
$s->bind_param($i, $param);
print LOG "Bind param $i using $param.\n"
or die "can't append to $opt{log}: $!";
}
$s->execute();
I can't say I've used the other modules that have been suggested, so they may do a better job.