Perl DBI Oracle not preserving column order after SELECT - perl

I'm using Perl v5.12.3 built by ActiveState on Windows. DBD::Oracle version 1.27. DBI version 1.616. I'm selecting the data below in this particular order and wanting the resulting data to be retrieved in that same order. Below are the code samples and some examples.
SQL Snippet (contents of $report_sql below)
select student_number, lastfirst, counselor,
a.dateenrolled as "Date Enrolled 1", a.dateleft as "Date Left 1", a.termid as "Term ID 1", a.course_number as "Course Number 1",
a.expression as "Expression 1", b.dateenrolled as "Date Enrolled 2", b.dateleft as "Date Left 2",
b.termid as "Term ID 2", b.course_number as "Course Number 2", b.expression as "Expression 2"
Perl code snippet
## run the resulting query
my $report_result = $dbh->prepare( $report_sql );
$report_result->execute();
while( my $report_row = $report_result->fetchrow_hashref())
{
print Dumper(\$report_row); ## contents of this posted below
Contents of print Dumper for $report_row
$VAR1 = \{
'Expression 2' => 'x',
'LASTFIRST' => 'xx',
'Term ID 1' => 'xx',
'Date Enrolled 2' => 'xx',
'Course Number 1' => 'xx',
'Term ID 2' => 'xx',
'STUDENT_NUMBER' => 'xx',
'Date Left 2' => 'xx',
'Expression 1' => 'xx',
'COUNSELOR' => 'xx',
'Date Left 1' => 'xx',
'Course Number 2' => 'xx',
'Date Enrolled 1' => 'xx'
};
Order I EXPECTED it to be printed
$VAR1 = \{
'STUDENT_NUMBER' => 'xx',
'LASTFIRST' => 'xx',
'COUNSELOR' => 'xx',
'Date Enrolled 1' => 'xx',
'Date Left 1' => 'xx',
'Term ID 1' => 'xx',
'Course Number 1' => 'xx',
'Expression 1' => 'xx',
'Date Enrolled 2' => 'xx',
'Date Left 2' => 'xx',
'Term ID 2' => 'xx',
'Course Number 2' => 'xx',
'Expression 2' => 'x'
};
One thing to note is that this query being ran is one of many that are being ran. This particular script runs through a series of queries and generates reports based on the returned results. The queries are stored in files on the hd alongside the perl script. The queries are read in and then ran. It's not always the same columns being selected.

You used a hash. Hash elements have no controllable order*. The order of elememts in an arrays can be controlled. If you want to present the order in which the fields were received, use an array instead of hash.
If you actually need the names, you can get the ordered names of the fields using #{ $sth->{NAME} }. You should still use an array for efficiency reasons, but you could use a hash if you wanted to.
* — Just like array elements are returned in the order they are "physically" organised in the array, hash elements are returned in the order they are physically organised in the hash. You cannot control where an element is physically placed in a hash, and the position changes as the hash is changed. With an array, you decide the physical position of an element, and it will remain in that position.

When the order of columns in a DBI result matters you can get the column names and values as array references.
...
my $names = $report_result->{NAME}; # or NAME_lc or NAME_uc
while( my $report_row = $report_result->fetchrow_arrayref() ) {
for my $col_idx ( 0 .. $#{$names} ) {
print "$names->[$col_idx]: $report_row->[$col_idx]\n";
}
}
Back before I had to worry about internationalization I used this a lot to generate CSV reports, just pass the NAME array to Text::CSV before passing the result arrays and writing a report just becomes writing a query.

Related

Return multiple output from stored proc in perl DBI returns extra 0 value [duplicate]

I'm beginner in sql. I have created the procedure as follows
create procedure testprocedure2 as
select 'one'
select 'three'
select 'five'
When I execute query into the database It shows the three result one three five. sql query is exec TEST_ABC_DB.dbo.testprocedure2
When I run the same query into the Perl it gives only one record which is one
$sth = $dbh->prepare("exec TEST_ABC_DB.dbo.testprocedure2");
$sth->execute();
while (#row= $sth->fetchrow_array())
{
print $row[0]."\t";
print "\n";
}
I don't know what is the problem. How can I fix it? I hope this answer will help in yesterday's question
Through the driver (e.g. DBD::ODBC)
Since you're using DBD::ODBC, you can use more_results provided by that driver to get the results of multiple queries in one execute.
This is the example they show in the documentation.
do {
my #row;
while (#row = $sth->fetchrow_array()) {
# do stuff here
}
} while ($sth->{odbc_more_results});
If we want to do this with your example queries, it's pretty much the same. You run your stored procedure, and then proceed with the do {} while construct (note that this is not a block, you cannot next out of it!).
my $sth = $dbh->prepare("exec TEST_ABC_DB.dbo.testprocedure2");
$sth->execute;
do {
while (my #row = $sth->fetchrow_array()) {
print $row[0]."\t";
print "\n";
}
} while ($sth->{odbc_more_results});
This should print your expected result.
one
three
five
Some other drivers also provide this. If they do, you can call $sth->more_results instead of using the internals as described below.
Workaround if your driver doesn't support this
There is no way for DBI itself to return the result of multiple queries at once. You can run them, but you cannot get the results.
If you really need three separate queries in your procedure and want all of the results, the answers by Shakheer and Shahzad to use a UNION are spot on.
However, your example is probably contrived. You probably don't have the same amount of columns in each of those queries, and you need to distinguish the results of each of the queries.
We have to change SQL and Perl code for this.
To get that to work, you can insert additional rows that you can later use to map each stack of results to each query.
Let's say the procedure looks like this:
create procedure testprocedure3 as
select 'one'
select 'three', 'three', 'three'
select 'five', 'five', 'five', 'five', 'five'
This is still just one row per query, but it should do as an example. With the UNION approach, it first becomes this:
create procedure testprocedure3 as
select 'one'
union all
select 'three', 'three', 'three'
union all
select 'five', 'five', 'five', 'five', 'five'
If you run this, it might fail. In ANSI SQL a UNION needs to have the same number of columns in all its queries, so I assume SQLServer also wants this. We need to fill them up with NULLs. Add them to all the queries so they match the number of columns in the one with the largest number of columns.
create procedure testprocedure3 as
select 'one', NULL, NULL, NULL, NULL
union all
select 'three', 'three', 'three', NULL, NULL
union all
select 'five', 'five', 'five', 'five', 'five'
If we now loop over it in Perl with the following code, we'll get something back.
use Data::Dumper;
my $sth = $dbh->prepare("exec TEST_ABC_DB.dbo.testprocedure3");
$sth->execute;
while ( my $row = $sth->fetchrow_arrayref ) {
print Dumper $row;
}
We'll see output similar to this (I didn't run the code, but wrote the output manually):
$VAR1 = [ 'one', undef, undef, undef, undef ];
$VAR1 = [ 'three', 'three', 'three', undef, undef ];
$VAR1 = [ 'five', 'five', 'five', 'five', 'five' ];
We have no way of knowing which line belongs to which part of the query. So let's insert a delimiter.
create procedure testprocedure3 as
select 'one', NULL, NULL, NULL, NULL
union all
select '-', '-', '-', '-', '-'
union all
select 'three', 'three', 'three', NULL, NULL
union all
select '-', '-', '-', '-', '-'
union all
select 'five', 'five', 'five', 'five', 'five'
Now the result of the Perl code will look as follows:
$VAR1 = [ 'one', undef, undef, undef, undef ];
$VAR1 = [ '-', '-', '-', '-', '-' ];
$VAR1 = [ 'three', 'three', 'three', undef, undef ];
$VAR1 = [ '-', '-', '-', '-', '-' ];
$VAR1 = [ 'five', 'five', 'five', 'five', 'five' ];
This might not be the best choice of delimiter, but it nicely illustrates what I am planning to do. All we have to do now is split this into separate results.
use Data::Dumper;
my #query_results;
my $query_index = 0;
my $sth = $dbh->prepare("exec TEST_ABC_DB.dbo.testprocedure3");
$sth->execute;
while ( my $row = $sth->fetchrow_arrayref ) {
# move to the next query if we hit the delimiter
if ( join( q{}, #$row ) eq q{-----} ) {
$query_index++;
next;
}
push #{ $query_results[$query_index] }, $row;
}
print Dumper \#query_results;
I've defined two new variables. #query_results holds all the results, sorted by query number. $query_index is the index for that array. It starts with 0.
We iterate all the resulting rows. It's important that $row is lexical here. It must be created with my in the loop head. (You are using use strict, right?) If we see the delimiter, we increment the $query_index and move on. If we don't we have a regular result line, so we stick that into our #query_results array within the current query's index.
The overall result is an array with arrays of arrays in it.
$VAR1 = [
[
[ 'one', undef, undef, undef, undef ]
],
[
[ 'three', 'three', 'three', undef, undef ]
],
[
[ 'five', 'five', 'five', 'five', 'five' ]
],
];
If you have actual queries that return many rows this starts making a lot of sense.
Of course you don't have to store all the results. You can also just work with the results of each query directly in your loop.
Disclaimer: I've run none of the code in this answer as I don't have access to an SQLServer. It might contain syntax errors in the Perl as well as the SQL. But it does demonstrate the approach.
The procedure you created is returning 3 result sets. And you are capturing only 1 result. If you are not bother about sets, make them as single result with UNION ALL
create procedure testprocedure2 as
select 'one'
union all
select 'three'
union all
select 'five'
Edit:
If you want to capture multiple resultsets returned from stored procedure, here is a good example explained with MySQL database Multiple data sets in MySQL stored procedures
simple use union all like this then only one table is shown with data.

Python pandas group by aggregate statement print

I am new to Pandas and I want to use group by statement. it worked but I am enable to print after group by function.
I am using Eclipse IDE
here is my code
import pandas as pd
df = pd.DataFrame({'A' : ['foo', 'bar', 'foo', 'bar',
'foo', 'bar', 'foo', 'foo'],
'B' : ['one', 'one', 'two', 'three',
'two', 'two', 'one', 'three'],
'C' : [1,2,3,4,5,6,0,2]})
grouped = df.groupby('C')
print grouped
and i get this as ouptut: <pandas.core.groupby.DataFrameGroupBy object at 0x02FCA7D0>
My question is how can i print grouped variable with correct answer
I've always forced it to print using .head()
grouped.head()
What is happening is you are grouping, but not aggregating the result.
grouped = df.groupby('A')
grouped
grouped_sum = df.groupby('A').sum()
grouped_sum
C
A
bar 12
foo 11

Find statement with alias

I'm trying to group a query's response by day, month, or year based on user input.
To do this I'm extracting the day, month, or year from the log_time column AS when and then trying to GROUP BY when but CakePHP says:
SQLSTATE[42601]: Syntax error: 7 ERROR: syntax error at or near "when" LINE 1: ...2-31' GROUP BY "Passenger"."route_id", "Route"."name", when ^
Here's an example of my function call where the grouping is 'month'.
$fields = array(
"CONCAT(DATE_PART('year', \"Passenger\".\"log_time\"), '-', DATE_PART('month', \"Passenger\".\"log_time\")) AS when",
"Passenger.route_id",
"Route.name",
"SUM(Passenger.embarked) AS embarked"
);
$conditions = array(
"Passenger.log_time >=" => $start,
"Passenger.log_time <=" => $end
);
$group = array("Passenger.route_id", "Route.name", "\"when\"");
return $this->find('all', array('fields' => $fields, 'conditions' => $conditions, 'group' => $group));
This is using Postgres on Cake 2.3.5. Has anyone else figured out how to do this?
when is a keyword that is used with CASE. Looks like PostgreSQL can figure out that the when in your select is just an identifier but there isn't enough context to figure it out in the GROUP BY. I'd quote it in both places:
"CONCAT(DATE_PART('year', \"Passenger\".\"log_time\"), '-', DATE_PART('month', \"Passenger\".\"log_time\")) AS \"when\"",
and:
$group = array("\"when\"", "Passenger.route_id", "Route.name");

Interpreting Perl DBI MySQL column_info()

I'm trying to write Perl code that accepts bindings for a SQL INSERT statement, and identifies problems that might cause the INSERT to be rejected for bad data, and fixes them. To do this, I need to get and interpret column metadata.
The $dbh->column_info method returns the info in a coded form. I've pored through the official CPAN documentation but am still confused.
my $sth_column_info
= $dbh->column_info( $catalog, $schema, $table, undef );
my $columns_aoh_ref = $sth_column_info->fetchall_arrayref(
{ COLUMN_NAME => 1,
DATA_TYPE => 1,
NULLABLE => 1,
ORDINAL_POSITION => 1,
COLUMN_DEF => 1,
}
);
say $table;
for my $href (#$columns_aoh_ref) {
my #list;
while ( my ( $k, $v ) = each %$href ) {
push #list, "$k=" . ( $v // 'undef' );
}
say join '|', #list;
}
Output is:
dw_phone
NULLABLE=0|COLUMN_DEF=undef|DATA_TYPE=4|ORDINAL_POSITION=1|COLUMN_NAME=phone_id
NULLABLE=0|COLUMN_DEF=undef|DATA_TYPE=4|ORDINAL_POSITION=2|COLUMN_NAME=phone_no
NULLABLE=1|COLUMN_DEF=undef|DATA_TYPE=4|ORDINAL_POSITION=3|COLUMN_NAME=phone_ext
NULLABLE=0|COLUMN_DEF=undef|DATA_TYPE=1|ORDINAL_POSITION=4|COLUMN_NAME=phone_type
NULLABLE=0|COLUMN_DEF=undef|DATA_TYPE=1|ORDINAL_POSITION=5|COLUMN_NAME=phone_location
NULLABLE=1|COLUMN_DEF=undef|DATA_TYPE=1|ORDINAL_POSITION=6|COLUMN_NAME=phone_status
NULLABLE=0|COLUMN_DEF=undef|DATA_TYPE=11|ORDINAL_POSITION=7|COLUMN_NAME=insert_date
NULLABLE=0|COLUMN_DEF=undef|DATA_TYPE=11|ORDINAL_POSITION=8|COLUMN_NAME=update_date
Where - for example - does one find a mapping of data type codes to strings? Should I be using DATA_TYPE, TYPE_NAME, or SQL_DATA_TYPE? Should I be using NULLABLE or IS_NULLABLE, and why the two flavors?
I can appreciate the difficulty of documenting (let alone implementing) a universal interface for databases. But I wonder if anyone knows of a reference manual for using the DBI that's specific to MySQL?
UPDATE 1:
Tried to shed more light by retrieving all info using an array rather than a hash:
my $sth_column_info
= $dbh->column_info( $catalog, $schema, $table, undef );
my $aoa_ref = $sth_column_info->fetchall_arrayref; # <- chg. to arrayref, no parms
say $table;
for my $aref (#$aoa_ref) {
my #list = map $_ // 'undef', #$aref;
say join '|', #list;
}
Now I can see lots of potentially useful information mixed in there.
dw_contact_source
undef|dwcust1|dw_contact_source|contact_id|4|BIGINT|20|undef|undef|10|0|undef|undef|4|undef|undef|1|NO|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|1|bigint(20)|undef|0
undef|dwcust1|dw_contact_source|company_id|4|SMALLINT|6|undef|undef|10|0|undef|undef|4|undef|undef|2|NO|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|1|smallint(6)|undef|0
undef|dwcust1|dw_contact_source|contact_type_id|4|TINYINT|4|undef|undef|10|0|undef|undef|4|undef|undef|3|NO|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef||tinyint(4)|undef|0
undef|dwcust1|dw_contact_source|insert_date|11|DATETIME|19|undef|0|undef|0|undef|undef|9|-79|undef|4|NO|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef||datetime|undef|0
undef|dwcust1|dw_contact_source|update_date|11|DATETIME|19|undef|0|undef|0|undef|undef|9|-79|undef|5|NO|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef|undef||datetime|undef|0
So my question would be:
How do I get the corresponding names/descriptions of these metadata?
How do I fetchall_arrayref just what I need, using symbols rather than integers? (I tried fetchall_arrayref([qw/COLUMN_NAME DATA_TYPE/]) and got back all undefs; now I'm just flailing about guessing.)
UPDATE 2:
Now I'm digging around in DBD::mysql.pm and I found a very interesting array:
my #names = qw(
TABLE_CAT TABLE_SCHEM TABLE_NAME COLUMN_NAME
DATA_TYPE TYPE_NAME COLUMN_SIZE BUFFER_LENGTH DECIMAL_DIGITS
NUM_PREC_RADIX NULLABLE REMARKS COLUMN_DEF
SQL_DATA_TYPE SQL_DATETIME_SUB CHAR_OCTET_LENGTH
ORDINAL_POSITION IS_NULLABLE CHAR_SET_CAT
CHAR_SET_SCHEM CHAR_SET_NAME COLLATION_CAT COLLATION_SCHEM COLLATION_NAME
UDT_CAT UDT_SCHEM UDT_NAME DOMAIN_CAT DOMAIN_SCHEM DOMAIN_NAME
SCOPE_CAT SCOPE_SCHEM SCOPE_NAME MAX_CARDINALITY
DTD_IDENTIFIER IS_SELF_REF
mysql_is_pri_key mysql_type_name mysql_values
mysql_is_auto_increment
);
These correspond precisely to what is returned by fetchall_arrayref. Now I can see I have four choices for learning data type, so let's see if any of the codes are documented.
UPDATE 3:
DBI Recipes is a very nice adjunct to CPAN DBI documentation about retrieving info back into Perl (Especially the {select|fetch}{row|all}_{hash|array} methods.)
This will help you determine the values for the data_types. I normally use the data_type to determine how to handle a column based on its type.
You then need to look at the MySQL data type key below and get the hash value. Then look at the DBI table below and match up the data name to get the data type value. Example: a BIGINT is an INTEGER type which matches SQL_INTEGER so the DATA_TYPE value is 4,
DBD::MySQL
### ANSI datatype mapping to mSQL datatypes
%DBD::mysql::db::ANSI2db = ("CHAR" => "CHAR",
"VARCHAR" => "CHAR",
"LONGVARCHAR" => "CHAR",
"NUMERIC" => "INTEGER",
"DECIMAL" => "INTEGER",
"BIT" => "INTEGER",
"TINYINT" => "INTEGER",
"SMALLINT" => "INTEGER",
"INTEGER" => "INTEGER",
"BIGINT" => "INTEGER",
"REAL" => "REAL",
"FLOAT" => "REAL",
"DOUBLE" => "REAL",
"BINARY" => "CHAR",
"VARBINARY" => "CHAR",
"LONGVARBINARY" => "CHAR",
"DATE" => "CHAR",
"TIME" => "CHAR",
"TIMESTAMP" => "CHAR"
);
DBI.pm
TYPE
The TYPE attribute contains a reference to an array of integer values representing the international standard values for the respective datatypes. The array of integers has a length equal to the number of columns selected within the original statement, and can be referenced in a similar way to the NAME attribute example shown earlier.
The standard values for common types are:
SQL_CHAR 1
SQL_NUMERIC 2
SQL_DECIMAL 3
SQL_INTEGER 4
SQL_SMALLINT 5
SQL_FLOAT 6
SQL_REAL 7
SQL_DOUBLE 8
SQL_DATE 9
SQL_TIME 10
SQL_TIMESTAMP 11
SQL_VARCHAR 12
SQL_LONGVARCHAR -1
SQL_BINARY -2
SQL_VARBINARY -3
SQL_LONGVARBINARY -4
SQL_BIGINT -5
SQL_TINYINT -6
SQL_BIT -7
SQL_WCHAR -8
SQL_WVARCHAR -9
SQL_WLONGVARCHAR -10
While these numbers are fairly standard,[61] the way drivers map their native types to these standard types varies greatly. Native types that don't correspond well to one of these types may be mapped into the range officially reserved for use by the Perl DBI: -9999 to -9000.

Can I rename a hash key inside another hash in Perl?

I have a hash of a hash (HoH) that I got from using select_allhashref on a mysql query.
The format is perfect for what I want and I have two other HoH by similar means.
I want to eventually merge these HoH together and the only problem (I think) I have is that there is a key in the 'sub-hash' of one HoH that has the same name as one in another HoH.
E.g.
my $statement1 = "select id, name, age, height from school_db";
my $school_hashref = $dbh->selectall_hashref($statement1, 1);
my $statement2 = "select id, name, address, post_code from address_db";
my $address_hashref = $dbh->selectall_hashref($statement2, 1);
So when I dump the data I get results like so:
$VAR1 = {
'57494' => {
'name' => 'John Smith',
'age' => '9',
'height' => '120'
}
}
};
$VAR1 = {
'57494' => {
'name' => 'Peter Smith',
'address' => '5 Cambridge Road',
'post_code' => 'CR5 0FS'
}
}
};
(this is an example so might seem illogical to have different names, but I need it :) )
So I would like to rename 'name' to 'address_name' or such. Is this possible? I know you can do
$hashref->{$newkey} = delete $hashref->{$oldkey};
(edit: this is an example that I found online, but haven't tested.)
but I don't know how I would represent the 'id' part. Any ideas?
Not knowing the way that you are merging them together, the easiest solution might be to simply change the select statement to rename the column in your results. Rather than trying to manipulate the hash afterward.
my $statement2 = "select id, name as address_name, address, post_code from address_db";
my $address_hashref = $dbh->selectall_hashref($statement2, 1);
If that is not a realistic option for you then a loop might be your best option
foreach (keys %{$address_hashref}) {
$address_hashref->{$_}{$newkey} = delete $address_hashref->{$_}{$oldkey}
}
You have do it with $hashref->{$newkey} = delete $hashref->{$oldkey}; because of the way hashes are implemented.
You can do it with hash of hashes too.
$hashref->{$key_id}{$newKey} = delete $hashref->{$key_id}{$oldKey};
The hash function is used to transform the key into the index (the hash) of an array element (the slot or bucket) where the corresponding value is to be sought.
Here's a simple example:
Our hash
{
'a' => "apples",
'b' => "oranges"
}
Let's define our hash function idx = h(key) and using the function on our keys above gives us:
h('a') = 02;
h('b') = 00;
How its stored in an array or bucket
idx | value
00 | 'oranges'
01 | ''
02 | 'apples'
03 | ''
... and so on
Say we want they key of 'apples' to be 'c'. We cannot simply change the key to 'c', since the hash function always returns 02 for 'a', and will return something different for 'c'. So, if we want to change the key, we also have to move the value to the correct idx in the array/bucket.
Well, it might not be different, but that is a collision. Collisions are a special case that to be handled when implementing a hash.
For more info about on hashes:
http://en.wikipedia.org/wiki/Hash_table
How are hash tables implemented internally in popular languages?