I'm trying to write Perl code that accepts bindings for a SQL INSERT statement, and identifies problems that might cause the INSERT to be rejected for bad data, and fixes them. To do this, I need to get and interpret column metadata.
The $dbh->column_info method returns the info in a coded form. I've pored through the official CPAN documentation but am still confused.
my $sth_column_info
= $dbh->column_info( $catalog, $schema, $table, undef );
my $columns_aoh_ref = $sth_column_info->fetchall_arrayref(
{ COLUMN_NAME => 1,
DATA_TYPE => 1,
NULLABLE => 1,
ORDINAL_POSITION => 1,
COLUMN_DEF => 1,
}
);
say $table;
for my $href (#$columns_aoh_ref) {
my #list;
while ( my ( $k, $v ) = each %$href ) {
push #list, "$k=" . ( $v // 'undef' );
}
say join '|', #list;
}
Output is:
dw_phone
NULLABLE=0|COLUMN_DEF=undef|DATA_TYPE=4|ORDINAL_POSITION=1|COLUMN_NAME=phone_id
NULLABLE=0|COLUMN_DEF=undef|DATA_TYPE=4|ORDINAL_POSITION=2|COLUMN_NAME=phone_no
NULLABLE=1|COLUMN_DEF=undef|DATA_TYPE=4|ORDINAL_POSITION=3|COLUMN_NAME=phone_ext
NULLABLE=0|COLUMN_DEF=undef|DATA_TYPE=1|ORDINAL_POSITION=4|COLUMN_NAME=phone_type
NULLABLE=0|COLUMN_DEF=undef|DATA_TYPE=1|ORDINAL_POSITION=5|COLUMN_NAME=phone_location
NULLABLE=1|COLUMN_DEF=undef|DATA_TYPE=1|ORDINAL_POSITION=6|COLUMN_NAME=phone_status
NULLABLE=0|COLUMN_DEF=undef|DATA_TYPE=11|ORDINAL_POSITION=7|COLUMN_NAME=insert_date
NULLABLE=0|COLUMN_DEF=undef|DATA_TYPE=11|ORDINAL_POSITION=8|COLUMN_NAME=update_date
Where - for example - does one find a mapping of data type codes to strings? Should I be using DATA_TYPE, TYPE_NAME, or SQL_DATA_TYPE? Should I be using NULLABLE or IS_NULLABLE, and why the two flavors?
I can appreciate the difficulty of documenting (let alone implementing) a universal interface for databases. But I wonder if anyone knows of a reference manual for using the DBI that's specific to MySQL?
UPDATE 1:
Tried to shed more light by retrieving all info using an array rather than a hash:
my $sth_column_info
= $dbh->column_info( $catalog, $schema, $table, undef );
my $aoa_ref = $sth_column_info->fetchall_arrayref; # <- chg. to arrayref, no parms
say $table;
for my $aref (#$aoa_ref) {
my #list = map $_ // 'undef', #$aref;
say join '|', #list;
}
Now I can see lots of potentially useful information mixed in there.
dw_contact_source
undef|dwcust1|dw_contact_source|contact_id|4|BIGINT|20|undef|undef|10|0|undef|undef|4|undef|undef|1|NO|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|1|bigint(20)|undef|0
undef|dwcust1|dw_contact_source|company_id|4|SMALLINT|6|undef|undef|10|0|undef|undef|4|undef|undef|2|NO|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|1|smallint(6)|undef|0
undef|dwcust1|dw_contact_source|contact_type_id|4|TINYINT|4|undef|undef|10|0|undef|undef|4|undef|undef|3|NO|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef||tinyint(4)|undef|0
undef|dwcust1|dw_contact_source|insert_date|11|DATETIME|19|undef|0|undef|0|undef|undef|9|-79|undef|4|NO|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef||datetime|undef|0
undef|dwcust1|dw_contact_source|update_date|11|DATETIME|19|undef|0|undef|0|undef|undef|9|-79|undef|5|NO|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef||datetime|undef|0
So my question would be:
How do I get the corresponding names/descriptions of these metadata?
How do I fetchall_arrayref just what I need, using symbols rather than integers? (I tried fetchall_arrayref([qw/COLUMN_NAME DATA_TYPE/]) and got back all undefs; now I'm just flailing about guessing.)
UPDATE 2:
Now I'm digging around in DBD::mysql.pm and I found a very interesting array:
my #names = qw(
TABLE_CAT TABLE_SCHEM TABLE_NAME COLUMN_NAME
DATA_TYPE TYPE_NAME COLUMN_SIZE BUFFER_LENGTH DECIMAL_DIGITS
NUM_PREC_RADIX NULLABLE REMARKS COLUMN_DEF
SQL_DATA_TYPE SQL_DATETIME_SUB CHAR_OCTET_LENGTH
ORDINAL_POSITION IS_NULLABLE CHAR_SET_CAT
CHAR_SET_SCHEM CHAR_SET_NAME COLLATION_CAT COLLATION_SCHEM COLLATION_NAME
UDT_CAT UDT_SCHEM UDT_NAME DOMAIN_CAT DOMAIN_SCHEM DOMAIN_NAME
SCOPE_CAT SCOPE_SCHEM SCOPE_NAME MAX_CARDINALITY
DTD_IDENTIFIER IS_SELF_REF
mysql_is_pri_key mysql_type_name mysql_values
mysql_is_auto_increment
);
These correspond precisely to what is returned by fetchall_arrayref. Now I can see I have four choices for learning data type, so let's see if any of the codes are documented.
UPDATE 3:
DBI Recipes is a very nice adjunct to CPAN DBI documentation about retrieving info back into Perl (Especially the {select|fetch}{row|all}_{hash|array} methods.)
This will help you determine the values for the data_types. I normally use the data_type to determine how to handle a column based on its type.
You then need to look at the MySQL data type key below and get the hash value. Then look at the DBI table below and match up the data name to get the data type value. Example: a BIGINT is an INTEGER type which matches SQL_INTEGER so the DATA_TYPE value is 4,
DBD::MySQL
### ANSI datatype mapping to mSQL datatypes
%DBD::mysql::db::ANSI2db = ("CHAR" => "CHAR",
"VARCHAR" => "CHAR",
"LONGVARCHAR" => "CHAR",
"NUMERIC" => "INTEGER",
"DECIMAL" => "INTEGER",
"BIT" => "INTEGER",
"TINYINT" => "INTEGER",
"SMALLINT" => "INTEGER",
"INTEGER" => "INTEGER",
"BIGINT" => "INTEGER",
"REAL" => "REAL",
"FLOAT" => "REAL",
"DOUBLE" => "REAL",
"BINARY" => "CHAR",
"VARBINARY" => "CHAR",
"LONGVARBINARY" => "CHAR",
"DATE" => "CHAR",
"TIME" => "CHAR",
"TIMESTAMP" => "CHAR"
);
DBI.pm
TYPE
The TYPE attribute contains a reference to an array of integer values representing the international standard values for the respective datatypes. The array of integers has a length equal to the number of columns selected within the original statement, and can be referenced in a similar way to the NAME attribute example shown earlier.
The standard values for common types are:
SQL_CHAR 1
SQL_NUMERIC 2
SQL_DECIMAL 3
SQL_INTEGER 4
SQL_SMALLINT 5
SQL_FLOAT 6
SQL_REAL 7
SQL_DOUBLE 8
SQL_DATE 9
SQL_TIME 10
SQL_TIMESTAMP 11
SQL_VARCHAR 12
SQL_LONGVARCHAR -1
SQL_BINARY -2
SQL_VARBINARY -3
SQL_LONGVARBINARY -4
SQL_BIGINT -5
SQL_TINYINT -6
SQL_BIT -7
SQL_WCHAR -8
SQL_WVARCHAR -9
SQL_WLONGVARCHAR -10
While these numbers are fairly standard,[61] the way drivers map their native types to these standard types varies greatly. Native types that don't correspond well to one of these types may be mapped into the range officially reserved for use by the Perl DBI: -9999 to -9000.
Related
I have this small perl code which is going to add a record to a table but I am confused why DBIC is not able to see the primary key?
I am not able to find any answer anywhere. First the names of table and columns were camelCase, which I then changed to underscore but it just won't run :(
$ ./test.pl
DBIx::Class::ResultSource::unique_constraint_columns(): Unknown unique constraint node_id on 'node' at ./test.pl line 80
code:
sub addNode
{
my $node = shift; my $lcNode = lc($node);
my $id = $schema
->resultset('Node')
->find_or_create
(
{ node_name => $lcNode },
{ key => 'node_id' }
);
return $id;
}
table details:
mysql> desc node;
+------------+-----------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+-----------------------+------+-----+---------+----------------+
| node_id | mediumint(5) unsigned | NO | PRI | NULL | auto_increment |
| node_name | varchar(50) | NO | | NULL | |
| node_notes | varchar(1000) | YES | | NULL | |
+------------+-----------------------+------+-----+---------+----------------+
3 rows in set (0.00 sec)
DBIx::Class::Resultset:
$ cat Node.pm
use utf8;
package Testdb::Schema::Result::Node;
# Created by DBIx::Class::Schema::Loader
# DO NOT MODIFY THE FIRST PART OF THIS FILE
use strict;
use warnings;
use base 'DBIx::Class::Core';
__PACKAGE__->table("node");
__PACKAGE__->add_columns(
"node_id",
{
data_type => "mediumint",
extra => { unsigned => 1 },
is_auto_increment => 1,
is_nullable => 0,
},
"node_name",
{ data_type => "varchar", is_nullable => 0, size => 50 },
"node_notes",
{ data_type => "varchar", is_nullable => 1, size => 1000 },
);
__PACKAGE__->set_primary_key("node_id");
# Created by DBIx::Class::Schema::Loader v0.07045 # 2017-08-21 22:14:58
# DO NOT MODIFY THIS OR ANYTHING ABOVE! md5sum:bWXf98hpLJgNBU93aaRYkQ
# You can replace this text with custom code or comments, and it will be preserved on regeneration
1;
https://metacpan.org/pod/DBIx::Class::ResultSet#find:
To aid with preparing the correct query for the storage you may supply the key attribute, which is the name of a unique constraint (the unique constraint corresponding to the primary columns is always named primary).
(Emphasis mine.)
In other words, to use the primary key, you need to specify { key => 'primary' }. Any other key attribute is looked up as the name of an additional unique constraint.
You didn't make clear how addNode should work exactly. But if you want to lookup existing nodes by node_name, you should simply remove the key attribute:
my $id = $schema->resultset('Node')->find_or_create(
{ node_name => $lcNode }
);
But read the caveat in the DBIC documentation:
If no such constraint is found, find currently defaults to a simple search->(\%column_values) which may or may not do what you expect. Note that this fallback behavior may be deprecated in further versions. If you need to search with arbitrary conditions - use "search". If the query resulting from this fallback produces more than one row, a warning to the effect is issued, though only the first row is constructed and returned as $result_object.
You should probably consider adding a unique constraint to node_name.
The real ans which came in DBIx list from Henry, is, whatever key you use, the col it corresponds to, should be in the query. all of above are ambiguous, they are not wrong, but they do not clarify the precise fact.
Typically SQLite's collation sorts case-sensitive. All capital letters come before small letters. But it's possible to tell SQLite in the ORDER BY clause to ignore that, by doing this:
... ORDER BY foo COLLATE NOCASE ASC
But how do we do this with DBIx::Class?
Consider the following example, which deploys an SQLite database in memory with a table foo and one comlumn bar. The connection uses the quote_names setting. It fills in the values z Z b B a A and then gets them back out using all on the ResultSet. I'll be using this setup in all my following examples. You need DBIx::Class and DBD::SQLite to run this.
use strict;
use warnings;
package Foo::Schema::Result::Foo;
use base 'DBIx::Class::Core';
__PACKAGE__->table("foo");
__PACKAGE__->add_columns( "bar", { data_type => "text" }, );
package Foo::Schema;
use base 'DBIx::Class::Schema';
__PACKAGE__->register_class( 'Foo' => 'Foo::Schema::Result::Foo' );
package main;
my $schema = Foo::Schema->connect(
{
dsn => 'dbi:SQLite:dbname=:memory:',
quote_names => 1,
}
);
$schema->deploy;
$schema->resultset('Foo')->create( { bar => $_ } ) for qw(z Z b B a A);
my #all = $schema->resultset('Foo')->search(
{},
{
order_by => { -asc => 'me.bar' },
},
)->all;
# example code starts here
print join q{ }, map { $_->bar } #all;
The output of this is sorted case-sensitive.
A B Z a b z
Now of course I could sort it with Perl and make it case-insensitive, like this.
print join q{ }, sort { lc $a cmp lc $b } map { $_->bar } #all;
Now I get
A a B b Z z
But I can also use the COLLATE NOCASE if I query using the underlying DBI handle directly.
$schema->storage->dbh_do(
sub {
my ( $storage, $dbh, #args ) = #_;
my $res = $dbh->selectall_arrayref(
"SELECT * FROM foo ORDER BY bar COLLATE NOCASE ASC"
);
print "$_->[0] " for #$res;
}
This gives us
a A b B z Z
I want DBIC to use the COLLATE NOCASE, but without running any SQL. I do not want to do any expensive string conversions in the ORDER BY, or do them later in Perl.
How do I tell DBIx::Class to use the COLLATE NOCASE when ordering with SQLite?
The following does not work:
order_by => { '-collate nocase asc' => 'me.bar' },
And this only works if quote_names is not turned on.
order_by => { -asc => 'me.bar COLLATE NOCASE' },
It will produce this query and error message with the above example code.
SELECT "me"."bar" FROM "foo" "me" ORDER BY "me"."bar COLLATE NOCASE"
ASC: DBIx::Class::Storage::DBI::_prepare_sth(): DBI Exception:
DBD::SQLite::db prepare_cached failed: no such column: me.bar COLLATE
NOCASE [for Statement "SELECT "me"."bar" FROM "foo" "me" ORDER BY
"me"."bar COLLATE NOCASE" ASC"]
Or I could do it by converting to upper or lower in the ORDER BY clause using DBIC.
my #all = $schema->resultset('Foo')->search(
{},
{
order_by => { -asc => 'lower(me.bar)' },
},
)->all;
print join q{ }, map { $_->bar } #all;
This gives
a A b B z Z
without quote_names which is similar, but the other way around. (That's not my concern here), but also throws an error when quote_names is turned on.
SELECT "me"."bar" FROM "foo" "me" ORDER BY "lower(me"."bar)" ASC:
DBIx::Class::Storage::DBI::_prepare_sth(): DBI Exception:
DBD::SQLite::db prepare_cached failed: no such column: lower(me.bar)
[for Statement "SELECT "me"."bar" FROM "foo" "me" ORDER BY
"lower(me"."bar)" ASC"]
If you're comfortable using a very small amount of SQL, you can pass a scalar reference to denote literal SQL, and DBIC won't mess with it:
order_by => \'me.bar COLLATE NOCASE ASC'
Or, with just the bare minimum amount of literal SQL:
order_by => { -asc => \'me.bar COLLATE NOCASE' }
Note that this syntax is technically discouraged, but I don't know of any other way to achieve what you're after:
The old scalarref syntax (i.e. order_by => \'year DESC') is still
supported, although you are strongly encouraged to use the hashref
syntax as outlined above.
msa6#outlook.com
postArray ( [zip] => 81007 [st] => co [dist] => 13 [email] => msa6#outlook.com [pw] => msa12345 [sq] => hot [sa] => dog [comment] => 8/8 original )
INSERT INTO mbr (zip,st,dist,email,pw,sq,sa) VALUES (81007,co,13,msa6#outlook.com,msa12345,hot,dog)
Problem in recording your member data. Please try again later. You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '#outlook.com,msa12345,hot,dog)' at line 1
varchars or characters should be enclosed in single quotes
Example:
'msa6#outlook.com'
Database schema:
create table requests(
rid integer primary key autoincrement,
oid integer references orders (oid),
command varchar(5),
account varchar(50),
txn_id varchar(12),
md5 varchar(30),
txn_date varchar(14),
sum float,
regdt timestamp default current_timestamp,
constraint u1 unique (command,txn_id)
);
create table orders (
oid integer primary key autoincrement,
cid integer references customers (cid),
pid integer references providers (pid),
account varchar(50),
amount float
);
Mapped to code by DBIx::Class::Schema::Loader.
My controller code :
my $req= $schema->resultset('Request');
my $order= $schema->resultset('Order');
my $r= $req->find(
{ command => 'check',
txn_id => $txn_id,
},
{ key => 'command_txn_id_unique' }
);
my $oid=$r->oid;
$req->create(
{ command => $command,
account => $account,
txn_id => $txn_id,
md5 => $md5,
txn_date => $txn_date,
sum => $sum,
oid => $oid
}
);
my $o = $order->find($oid);
$o->sum($sum);
$o->update;
My tracert sqls with DBIC_TRACE=1
SELECT me.rid, me.oid, me.command, me.account, me.txn_id, me.md5, me.txn_date, me.sum, me.regdt FROM requests me
WHERE ( ( me.command = ? AND me.txn_id = ? ) ): 'check', '1358505665'
SELECT me.oid, me.cid, me.pid, me.account, me.amount FROM orders me
WHERE ( me.oid = ? ): '18'
INSERT INTO requests ( account, command, md5, oid, sum, txn_date, txn_id) VALUES ( ?, ?, ?, ?, ?, ?, ? ): '1', 'pay', '44F4BC73D17E3FA906F658BB5916B7DC', '18', '500', '20130118104122', '1358505665'
DBIx::Class::Storage::DBI::SQLite::_dbi_attrs_for_bind(): Non-integer value supplied for column 'me.oid' despite the integer datatype at /home/.../lib/TestPrj/Test.pm line 128
SELECT me.oid, me.cid, me.pid, me.account, me.amount FROM orders me WHERE ( me.oid = ? ): 'TestPrj::Model::Result::Order=HASH(0x3dea448)'
datatype mismatch: bind param (0) =HASH(0x3dea448) as integer at /usr/share/perl5/DBIx/Class/Storage/DBI.pm line 1765.
I don't understand :
Why in first select query $oid = 18 and its ok. Its really 18.
But in second select query this $oid is a HASH ? Its not redefined anywhere in my code.
UPD:
Using Data::Dumper by advice #akawhy I saw that $oid is a blessed HASH.
So, looks like there is different context
scalar in { ... oid => $oid .. .}
and hash in find(...).
I don't know why it not a hash in first case.
But, when I changed to $oid=$r->get_column($oid) all works fine.
$resultset->find returns a Result object.
You didn't paste your ResultSource classes but as you wrote that you generated them with Schema::Loader I assume the 'oid' column accessor method is the same as the 'oid' relationship accessor. In that case it returns different things depending on scalar vs. list context.
I suggest to rename your relationship so you have two different accessors for the column value and its relationship.
I prefix all relationships with 'rel_' but you might prefer a different naming standard.
I think my $oid=$r->oid; the $oid is a ref. you can use ref or Data::Dumper to see its type.
Maybe you should use $oid->oid
And pay attention to this error
DBIx::Class::Storage::DBI::SQLite::_dbi_attrs_for_bind(): Non-......
I have a hash of a hash (HoH) that I got from using select_allhashref on a mysql query.
The format is perfect for what I want and I have two other HoH by similar means.
I want to eventually merge these HoH together and the only problem (I think) I have is that there is a key in the 'sub-hash' of one HoH that has the same name as one in another HoH.
E.g.
my $statement1 = "select id, name, age, height from school_db";
my $school_hashref = $dbh->selectall_hashref($statement1, 1);
my $statement2 = "select id, name, address, post_code from address_db";
my $address_hashref = $dbh->selectall_hashref($statement2, 1);
So when I dump the data I get results like so:
$VAR1 = {
'57494' => {
'name' => 'John Smith',
'age' => '9',
'height' => '120'
}
}
};
$VAR1 = {
'57494' => {
'name' => 'Peter Smith',
'address' => '5 Cambridge Road',
'post_code' => 'CR5 0FS'
}
}
};
(this is an example so might seem illogical to have different names, but I need it :) )
So I would like to rename 'name' to 'address_name' or such. Is this possible? I know you can do
$hashref->{$newkey} = delete $hashref->{$oldkey};
(edit: this is an example that I found online, but haven't tested.)
but I don't know how I would represent the 'id' part. Any ideas?
Not knowing the way that you are merging them together, the easiest solution might be to simply change the select statement to rename the column in your results. Rather than trying to manipulate the hash afterward.
my $statement2 = "select id, name as address_name, address, post_code from address_db";
my $address_hashref = $dbh->selectall_hashref($statement2, 1);
If that is not a realistic option for you then a loop might be your best option
foreach (keys %{$address_hashref}) {
$address_hashref->{$_}{$newkey} = delete $address_hashref->{$_}{$oldkey}
}
You have do it with $hashref->{$newkey} = delete $hashref->{$oldkey}; because of the way hashes are implemented.
You can do it with hash of hashes too.
$hashref->{$key_id}{$newKey} = delete $hashref->{$key_id}{$oldKey};
The hash function is used to transform the key into the index (the hash) of an array element (the slot or bucket) where the corresponding value is to be sought.
Here's a simple example:
Our hash
{
'a' => "apples",
'b' => "oranges"
}
Let's define our hash function idx = h(key) and using the function on our keys above gives us:
h('a') = 02;
h('b') = 00;
How its stored in an array or bucket
idx | value
00 | 'oranges'
01 | ''
02 | 'apples'
03 | ''
... and so on
Say we want they key of 'apples' to be 'c'. We cannot simply change the key to 'c', since the hash function always returns 02 for 'a', and will return something different for 'c'. So, if we want to change the key, we also have to move the value to the correct idx in the array/bucket.
Well, it might not be different, but that is a collision. Collisions are a special case that to be handled when implementing a hash.
For more info about on hashes:
http://en.wikipedia.org/wiki/Hash_table
How are hash tables implemented internally in popular languages?