I am using perl dbi and getting the sql results however it's coming as an array. The problem is , it only returns 1 array element. I tried to push to get the next element but there is nothing. I am sure (with other tool) I can see there are more then one values. It is (from other tool) I get multiple ddls. With my query I am only getting one of them. Here is my query:
my $rows = $dbh->selectall_arrayref
("show SELECT * FROM TABLE
INNER JOIN TABLE1
ON bla bla ;");
my $len = #$rows;
print $len;
foreach $h (#$rows) {
$st = $h->[0];
$st =~ s/(\d),(\d)/\1|\2/g;
#lines = split/,/,$st;
foreach $x (#lines) {
print FH $x . "\n";
}
}
My question is how do I shift the same array and get next result set from same row? Thanks!
The result of the above code is below:
PRIMARY INDEX ( ProcID ,CollectTimeStamp );ONE)NICODE NOT CASESPECIFIC,,OT NULL,C,
I don't think it brings back the entire content, somehow it is not complete
[This probably counts as "not an answer", but I'm trying to extract more useful information from the OP, and it's far easier to show him the code I want him to run in an answer]
We still really have no idea what you are trying to do here. And, to be honest, I'm not convinced that you do either :-)
Let's have a look at the raw data - without any of the processing that you're adding. Please try running the following code and then edit your question to add the output (if it returns a number of rows, you only need to include a couple).
my $rows = $dbh->selectall_arrayref
("show SELECT * FROM TABLE
INNER JOIN TABLE1
ON bla bla ;");
foreach my $row (#$rows) {
print join(' / ', #$row), "\n";
}
Then, please tell us (using those rows as an example) what output you want.
Related
I'm trying to make a simple select from a database, the thing is that I want the same script to be able to select any of the tables in it. I have gotten everything solved up until the point when I need to bind the columns to variables, since they must be generated dynamically I just don't know how to do it.
here's the code:
if($op eq "SELECT"){
if ($whr){
$query1 = "SELECT $colsf FROM $tab WHERE $whr";
}else{
$query1 = "SELECT $colsf FROM $tab";
}
$seth = $dbh->prepare($query1);
$seth->execute();
foreach $cajas(#columnas){
$seth->bind_col(*$dynamically_generated_var*);
}
print $q->br();
print $q->br();
print $q->br();
The variable #columans contains the name of the selected columns (which varies a lot), and I need a variable assigned for each of the columns on the $seth->bind_col().
How can I acheive this?
Using bind_col will not gain you anything here. As you have already figured out, that's used to bind a fixed number of results to a set of variables. But you do not have a fixed set.
Thinking in terms of oh, I can just create them dynamically is a very common mistake. It will get you into all kinds of trouble later. Perl has a data structure specifically for this use case: the hash.
DBI has a bunch of functions built in for retrieving data after execute. One of those is fetchrow_hashref. It will return the results as a hash reference, with one key per column, one row at a time.
while (my $res = $sth->fetchrow_hashref) {
p $res; # p is from Data::Printer
}
Let's assume the result looks like this:
$res = {
id => 1,
color => 'red',
}
You can access the color by saying $res->{color}. The perldocs on perlref and perlreftut have a lot of info about this.
Note that the best practice for naming statement handle variables is $sth.
In your case, you have a dynamic number of columns. Those have to be joined to be in the format of col1, col2, col3. I guess you have already done that in $colsf. The table is pretty obvious in $tab, so we only have the $whr left.
This part is tricky. It's important to always sanitize your input, especially in a CGI environment. With DBI this is best done by using placeholders. They will take care of all the escaping for you, and they are easy to use.
my $sth = $dbi->prepare('select cars from garage where color=?');
$sth->execute($color);
Now we don't need to care if the color is red, blue or ' and 1; --, which might have broken stuff. If it's all very dynamic, use $dbi->quote instead.
Let's put this together in your code.
use strict;
use warnings;
use DBI;
# ...
# the columns
my $colsf = join ',', #some_list_of_column_names; # also check those!
# the table name
my $table = $q->param('table');
die 'invalid table name' if $table =~ /[^a-zA-Z0-9_]/; # input checking
# where
# I'm skipping this part as I don't know where it is comming from
if ($op eq 'SELECT') {
my $sql = 'SELECT $colsf FROM $table';
$sql .= ' WHERE $whr' if $whr;
my $sth = $dbh->prepare($sql) or die $dbi->errstr;
$sth->execute;
my #headings = $sth->{NAME}; # see https://metacpan.org/pod/DBI#NAME1
while (my $res = $sth->fetchrow_hashref) {
# do stuff here
}
}
I am selecting from sql three values, hostname, place, message, assigning them to hashref and then in loop printing into file. Before I print it I have to clean the message as it contains in DB plenty of mess like html tags etc
while (my $result = $sth_get_message->fetchrow_hashref) {
my $message = $result->{MESSAGE};
$message =~ s/<[^>]*>//g;
$message =~ s/\n/ /g;
$message =~ s/ |"/ /g;
$message =~ s/\r//g;
my $message_out;
if (index($message, 'removed') != -1) {$message_out = $message;} else {$message_out= "";}
my $pc = $result->{PC};
my $place = $result->{PLACE};
print OUT "$pc|$message_out|$place\n";
}
the result is like this
pc1||place
pc2|message with word removed|place
pc3||place
pc3|message with word removed|place
pc4||place
pc4||place
pc4|message with word removed|place
some of them have more messages, but only one is relevant, so rest is cleaned but I have duplicates of pc's. What I want to get is to have unique pc list and if it has no message ok, show blank, if it has message with removed, show only this line
pc1||place
pc2|message with word removed|place
pc3|message with word removed|place
pc4|message with word removed|place
as far as I understood perl hashes this should be possible if I put pc as a key and then work with values but I have no clue how
thanks for help
When you have a list of values that you wish to find the unique values for you can populate a hash along these lines:
my $hashref = {}; ## initialise an empty hashref .. nb I prefer to work with refs but you could use has hash type variable
foreach my $row ( $rows )
{
$hashref->{$row} = 1;
}
now you have a hasref that is indexed by unique keys so you can do the following
print join(",", keys %$hashref );
or
foreach my $unique_key ( sort keys %$hashref )
{
print "$unique key in sorted order\n";
}
In your case you could use the string with the fields concatenated or you could just use the values that you wish to make unique and then assign something containing the entire row detail you are interested in place of the 1 value used above as a placeholder. You are then able to retrieve the full row detail with something like $hashref->{$unique_key}
The Problem:
I'm using DBI with Perl, and need to do a double nest loop through records in my records set.
In the past, I've used while statements like:
my $someQuery = "SELECT * FROM foo;";
my $sth = $dbh->prepare($someQuery);
$sth->execute();
while (my $ref = $sth->fetchrow_hashref()) {
my $someVariable = "$ref->{'dbFieldName'}";
# do stuff
}
What I've trying to achieve would be easy to do with a for loop instead of a while loop, but I'd need to know how to reference the results by row index as well as the total rowcount. Any idea on how to do this?
Bonus Points:
The above will help me solve the one problem, but I'd like to get better techniques to let me figure it out on my own. I'm using Perl EPIC in Eclipse, which doesn't give any sort of context look-ahead/autocomplete (or if it does I don't know how to enable it). Is there a way to enable this or a different add-on for Eclipse that would show the context look-ahead, so I could see what options are available?
Current row number
There are a couple of ways you can get the row number:
Perl variable
my $row_num = 0;
while (my $row = $sth->fetchrow_hashref) {
$row_num++;
# Do stuff
}
SQL variable
This depends on which DBMS you're using, but you can do something like this in MySQL:
$dbh->do('SET #row_num := 0');
my $statement = <<'SQL';
SELECT #row_num := #row_num + 1 AS row_num,
foo,
bar
FROM table
SQL
my $sth = $dbh->prepare($statement);
$sth->execute;
while (my $row = $sth->fetchrow_hashref) {
say $row->{row_num};
# Do stuff
}
Total rows returned
You can use $sth->rows, with the following caveat:
Generally, you can only rely on a row count after a non-SELECT execute (for some specific operations like UPDATE and DELETE), or after fetching all the rows of a SELECT statement.
For SELECT statements, it is generally not possible to know how many rows will be returned except by fetching them all. Some drivers will return the number of rows the application has fetched so far, but others may return -1 until all rows have been fetched. So use of the rows method or $DBI::rows with SELECT statements is not recommended.
One alternative method to get a row count for a SELECT is to execute a "SELECT COUNT(*) FROM ..." SQL statement with the same "..." as your query and then fetch the row count from that.
However, if you're already counting each row as you fetch it, you could simply use that instead.
Below is the hack I did to get the results into an array structure, which will allow me to do the nested looping easily. Hopefully someone else will post an answer if they have a better way.
my #results;
my $fieldFirstName = 0;
my $fieldLastName = 1;
my $fieldAddress = 2;
my $i = 0;
my $someQuery = "SELECT FirstName, LastName, Address FROM foo;";
my $sth = $dbh->prepare($someQuery);
$sth->execute();
while (my $ref = $sth->fetchrow_hashref()) {
$results[$i][$fieldFirstName] = "$ref->{'FirstName'}";
$results[$i][$fieldLastName] = "$ref->{'LastName'}";
$results[$i][$fieldAddress] = "$ref->{'Address'}";
$i++;
}
$sth->finish();
my $resultsRecordCount = #results;
open(INFILE1,"INPUT.txt");
my $modfile = 'Data.txt';
open MODIFIED,'>',$modfile or die "Could not open $modfile : $!";
for (;;) {
my $line1 = <INFILE1>;
last if not defined $line1;
my $line2 = <INFILE1>;
last if not defined $line2;
my ($tablename1, $colname1,$sql1) = split(/\t/, $line1);
my ($tablename2, $colname2,$sql2) = split(/\t/, $line2);
if ($tablename1 eq $tablename2)
{
my $sth1 = $dbh->prepare($sql1);
$sth1->execute;
my $hash_ref1 = $sth1->fetchall_hashref('KEY');
my $sth2 = $dbh->prepare($sql2);
$sth2->execute;
my $hash_ref2 = $sth2->fetchall_hashref('KEY');
my #fieldname = split(/,/, $colname1);
my $colcnt=0;
my $rowcnt=0;
foreach $key1 ( keys(%{$hash_ref1}) )
{
foreach (#fieldname)
{
$colname =$_;
my $strvalue1='';
#val1 = $hash_ref1->{$key1}->{$colname};
if (defined #val1)
{
my #filtered = grep /#val1/, #metadata;
my $strvalue1 = substr(#filtered[0],index(#filtered[0],'||') + 2);
}
my $strvalue2='';
#val2 = $hash_ref2->{$key1}->{$colname};
if (defined #val2)
{
my #filtered = grep /#val2/, #metadata2;
my $strvalue2 = substr(#filtered[0],index(#filtered[0],'||') + 2);
}
if ($strvalue1 ne $strvalue2 )
{
$colcnt = $colcnt + 1;
print MODIFIED "$tablename1\t$colname\t$strvalue1\t$strvalue2\n";
}
}
}
if ($colcnt>0)
{
print "modified count is $colcnt\n";
}
%$hash_ref1 = ();
%$hash_ref2 = ();
}
The program is Read input file in which every line contrain three strings seperated by tab. First is TableName, Second is ALL Column Name with commas in between and third contain the sql to be run. As this utlity is doing comparison of data, so there are two rows for every tablename. One for each DB. So data needs to be picked from each respective db's and then compared column by column.
SQL returns as ID in the result set and if the value is coming from db then it needs be translated to a string by reading from a array (that array contains 100K records with Key and value seperated by ||)
Now I ran this for one set of tables which contains 18K records in each db. There are 8 columns picked from db in each sql. So for every record out of 18K, and then for every field in that record i.e. 8, this script is taking a lot of time.
My question is if someone can look and see if it can be imporoved so that it takes less time.
File contents sample
INPUT.TXT
TABLENAME COL1,COL2 select COL1,COL2 from TABLENAME where ......
TABLENAMEB COL1,COL2 select COL1,COL2 from TABLENAMEB where ......
Metadata array contains something like this(there are two i.e. for each db)
111||Code 1
222||Code 2
Please suggest
Your code does look a bit unusual, and could gain clarity from using subroutines vs. just using loops and conditionals. Here are a few other suggestions.
The excerpt
for (;;) {
my $line1 = <INFILE1>;
last if not defined $line1;
my $line2 = <INFILE1>;
last if not defined $line2;
...;
}
is overly complicated: Not everyone knows the C-ish for(;;) idiom. You have lots of code duplication. And aren't you actually saying loop while I can read two lines?
while (defined(my $line1 = <INFILE1>) and defined(my $line2 = <INFILE1>)) {
...;
}
Yes, that line is longer, but I think it's a bit more self-documenting.
Instead of doing
if ($tablename1 eq $tablename2) { the rest of the loop }
you could say
next if $tablename1 eq $tablename2;
the rest of the loop;
and save a level of intendation. And better intendation equals better readability makes it easier to write good code. And better code might perform better.
What are you doing at foreach $key1 (keys ...) — something tells me you didn't use strict! (Just a hint: lexical variables with my can perform slightly better than global variables)
Also, doing $colname = $_ inside a for-loop is a dumb thing, for the same reason.
for my $key1 (keys ...) {
...;
for my $colname (#fieldname) { ... }
}
my $strvalue1='';
#val1 = $hash_ref1->{$key1}->{$colname};
if (defined #val1)
{
my #filtered = grep /#val1/, #metadata;
my $strvalue1 = substr(#filtered[0],index(#filtered[0],'||') + 2);
}
I don't think this does what you think it does.
From the $hash_ref1 you retrive a single element, then assign that element to an array (a collection of multiple values).
Then you called defined on this array. An array cannot be undefined, and what you are doing is quite deprecated. Calling defined function on a collection returns info about the memory management, but does not indicate ① whether the array is empty or ② whether the first element in that array is defined.
Interpolating an array into a regex isn't likely to be useful: The elements of the array are joined with the value of $", usually a whitespace, and the resulting string treated as a regex. This will wreak havoc if there are metacharacters present.
When you only need the first value of a list, you can force list context, but assign to a single scalar like
my ($filtered) = produce_a_list;
This frees you from weird subscripts you don't need and that only slow you down.
Then you assign to a $strvalue1 variable you just declared. This shadows the outer $strvalue1. They are not the same variable. So after the if branch, you still have the empty string in $strvalue1.
I would write this code like
my $val1 = $hash_ref1->{$key1}{$colname};
my $strvalue1 = defined $val1
? do {
my ($filtered) = grep /\Q$val1/, #metadata;
substr $filtered, 2 + index $filtered, '||'
} : '';
But this would be even cheaper if you pre-split #metadata into pairs and test for equality with the correct field. This would remove some of the bugs that are still lurking in that code.
$x = $x + 1 is commonly written $x++.
Emptying the hashrefs at the end of the iteration is unneccessary: The hashrefs are assigned to a new value at the next iteration of the loop. Also, it is unneccessary to assist Perls garbage collection for such simple tasks.
About the metadata: 100K records is a lot, so either put it in a database itself, or at the very least a hash. Especially for so many records, using a hash is a lot faster than looping through all entries and using slow regexes … aargh!
Create the hash from the file, once at the beginning of the program
my %metadata;
while (<METADATA>) {
chomp;
my ($key, $value) = split /\|\|/;
$metadata{$key} = $value; # assumes each key only has one value
}
Simply look up the key inside the loop
my $strvalue1 = defined $val1 ? $metadata{$val1} // '' : ''
That should be so much faster.
(Oh, and please consider using better names for variables. $strvalue1 doesn't tell me anything, except that it is a stringy value (d'oh). $val1 is even worse.)
This is not really an answer but it won't really fit well in a comment either so, until you provide some more information, here are some observations.
Inside you inner for loop, there is:
#val1 = $hash_ref1->{$key1}->{$colname};
Did you mean #val1 = #{ $hash_ref1->{$key1}->{$colname} };?
Later, you check if (defined #val1)? What did you really want to check? As perldoc -f defined points out:
Use of "defined" on aggregates (hashes and arrays) is
deprecated. It used to report whether memory for that aggregate
had ever been allocated. This behavior may disappear in future
versions of Perl. You should instead use a simple test for size:
In your case, if (defined #val1) will always be true.
Then, you have my #filtered = grep /#val1/, #metadata; Where did #metadata come from? What did you actually intend to check?
Then you have my $strvalue1 = substr(#filtered[0],index(#filtered[0],'||') + 2);
There is some interesting stuff going on in there.
You will need to verbalize what you are actually trying to do.
I strongly suspect there is a single SQL query you can run that will give you what you want but we first need to know what you want.
Effectively a duplicate of: How can I display data in table with Perl
The accepted answer there applies here. So do some of the alternatives.
I am trying to run raw database queries from Perl program and display results to the user. Something like select * from table. I want to display the information in a HTML table. The columns in the HTML table correspond with the returned columns.
I am having some issues. I can run describe table query to return the number of columns there are in the table. However, how will I store the information from the returned results into arrays?
So if I am storing results like this:
while (($f1, $t2, $n3, $k4, $d5, $e6) = $sth1->fetchrow_array)
In this case I only know that there are, say four columns (which I got from describe table). But this number four is dynamic and can change depending on the table name. I can not declare my variables based on this number. Any suggestions?
Try:
print "<table>\n";
# display HTML header
#cols = #{$sth->{NAMES_uc}};
print "<tr>".join("", map { "<th>${_}</th>" } #cols)."</tr>\n";
# display one HTML table row for each DB row
while (my #row = $sth->fetchrow_array) {
print "<tr>".join("", map { "<td>${_}</td>" } #row)."</tr>\n";
}
print "</table>\n";
while (my #row = $sth->fetchrow_array)
{
print "<tr>".join("", map{ "<td>${_}</td>" } #row)."</tr>" ;
}
Use the technique suggested in the answer(s) to the other question - use fetchrow_array to fetch into an array:
while (my #row = $sth->fetchrow_array())
{
...process array...
}
Or use an alternative to fetchrow_array(), such as fetchrow_hashref().