I'm trying to make a simple select from a database, the thing is that I want the same script to be able to select any of the tables in it. I have gotten everything solved up until the point when I need to bind the columns to variables, since they must be generated dynamically I just don't know how to do it.
here's the code:
if($op eq "SELECT"){
if ($whr){
$query1 = "SELECT $colsf FROM $tab WHERE $whr";
}else{
$query1 = "SELECT $colsf FROM $tab";
}
$seth = $dbh->prepare($query1);
$seth->execute();
foreach $cajas(#columnas){
$seth->bind_col(*$dynamically_generated_var*);
}
print $q->br();
print $q->br();
print $q->br();
The variable #columans contains the name of the selected columns (which varies a lot), and I need a variable assigned for each of the columns on the $seth->bind_col().
How can I acheive this?
Using bind_col will not gain you anything here. As you have already figured out, that's used to bind a fixed number of results to a set of variables. But you do not have a fixed set.
Thinking in terms of oh, I can just create them dynamically is a very common mistake. It will get you into all kinds of trouble later. Perl has a data structure specifically for this use case: the hash.
DBI has a bunch of functions built in for retrieving data after execute. One of those is fetchrow_hashref. It will return the results as a hash reference, with one key per column, one row at a time.
while (my $res = $sth->fetchrow_hashref) {
p $res; # p is from Data::Printer
}
Let's assume the result looks like this:
$res = {
id => 1,
color => 'red',
}
You can access the color by saying $res->{color}. The perldocs on perlref and perlreftut have a lot of info about this.
Note that the best practice for naming statement handle variables is $sth.
In your case, you have a dynamic number of columns. Those have to be joined to be in the format of col1, col2, col3. I guess you have already done that in $colsf. The table is pretty obvious in $tab, so we only have the $whr left.
This part is tricky. It's important to always sanitize your input, especially in a CGI environment. With DBI this is best done by using placeholders. They will take care of all the escaping for you, and they are easy to use.
my $sth = $dbi->prepare('select cars from garage where color=?');
$sth->execute($color);
Now we don't need to care if the color is red, blue or ' and 1; --, which might have broken stuff. If it's all very dynamic, use $dbi->quote instead.
Let's put this together in your code.
use strict;
use warnings;
use DBI;
# ...
# the columns
my $colsf = join ',', #some_list_of_column_names; # also check those!
# the table name
my $table = $q->param('table');
die 'invalid table name' if $table =~ /[^a-zA-Z0-9_]/; # input checking
# where
# I'm skipping this part as I don't know where it is comming from
if ($op eq 'SELECT') {
my $sql = 'SELECT $colsf FROM $table';
$sql .= ' WHERE $whr' if $whr;
my $sth = $dbh->prepare($sql) or die $dbi->errstr;
$sth->execute;
my #headings = $sth->{NAME}; # see https://metacpan.org/pod/DBI#NAME1
while (my $res = $sth->fetchrow_hashref) {
# do stuff here
}
}
Related
Holy cats man,
I'm maintaining some pretty awful legacy code and there is a part for adding some values to the database from a HTML form created by a loop and increments the names of the variables it submits to 14, I'm changing the loop to the amount of rows it selects before creating the HTML.
But the issue is with how it inserts the HTML form back into the database.
Here's an ad-hoc version of the way it handles the database inserts
while (my $count <= 14) {
if ($count == 1) {
$name = $name1;
$email = $email1;
}
# ...
if ($count == 14) {
$name = $name14;
$email = $email14;
}
my $sth = $dbh->prepare("INSERT INTO table SET name = ? AND email = ?");
$sth->execute($name, $email);
$count++;
}
While I'm probably just going to rewrite this entire section, I'm curious if you could add something like;
elsif ($count > 14) {
# Say count is 15 and we want to assign
# $name to $name15 using a string and the $count variable here.
$name = "name".$count;
$email = "email".$count;
}
Is that technically feasible?
What you're describing is called a "symbolic reference" in Perl lingo and it's generally frowned upon as a very, very, very... very bad practice because it only works with global variables (which are generally best avoided in their own right) and it's one of the easiest ways to create bugs which are nearly-impossible to find.
But it can be done. And, since you're asking for the sake of maintaining legacy code which is probably already just as bad, I'll show you how:
perl -e '$count14 = 42; $sref = "count14"; print $$sref . "\n"'
It's as simple as that.
But, really, don't do it if you can avoid it.
The general-case solution for avoiding symbolic references is to use a hash instead:
my %values = (name1 => 'Alice', name2 => 'Bob');
for my $count (1 .. 2) {
my $name = $values{'name' . $count};
print "$name\n";
}
In the specific case of a bunch of variables named foo1, foo2, etc., though, you probably want an array instead:
my #names = qw( . Alice Bob ); # '.' is a dummy to fill index 0 so the names start at 1
for my $count (1 .. 2) {
my $name = $names[$count];
print "$name\n";
}
I would strongly advise using one of these other techniques instead of symbolic references unless the existing code heavily depends on having $name1, $name2, etc. available. If you can afford the time to replace all of those with arrays and test that the array-based version still works, you'll have improved the quality of the code for future maintainers (which will probably include yourself).
I have to write rows generated after running a sql query to a file.
# Run the SQL script.
my $dbh = get_dbh($source);
my $qry = $dbh->prepare("$sql_data");
$qry->execute();
# Dump the data to file.
open(my $fh_write, ">", "$filename");
while (my #data = $qry->fetchrow_array())
{
print {$fh_write} join("\t", #data) . "\n";
}
close($fh_write);
Clearly i am doing two thing in a function:
Running the sql query.
Writing the data to file.
Is there a way to do this using SRP ?
There are lots of rows in the data so returning the array of rows from a seperate function might not be nice idea.
You could split it up into two different functions. One would query the database, and the other would write data to a file.
sub run_query {
my ( $sql, #args ) = #_;
# if you truly want separation of concerns,
# you need to connect $dbh somewhere else
my $sth = $dbh->prepare($sql);
$sth->execute(#args);
# this creates an iterator
return sub {
return $sth->fetchrow_arrayref;
};
}
This function takes a an SQL query and some arguments (remember to use placeholders!) and runs the query. It returns a code reference that closes over $sth. Every time that reference is invoked, one line of results will be fetched. When the statement handle $sth is empty, it will return undef, which is handed through, and you're done iterating. That might seem overkill, but stay with me for a moment.
Next, we make a function to write data to a file.
sub write_to_file {
my ( $filename, $iter ) = #_;
open my $fh, '>', $filename or die $!;
while ( my $data = $iter->() ) {
print $fh join( "\t", #{$data} ), "\n";
}
return;
}
This takes a filename and an iterator, which is a code reference. It opens the file, and then iterates until there is no more data left. Every line is written to the file. We don't need close $fh because it's a lexical filehandle that will be closed implicitly once $fh goes out of scope at the end of the function anyway.
What you've done now is define an interface. Your write_to_file function's interface is that it takes a file name and an iterator that always returns an array reference of fields.
Let's put this together.
my $iter = run_query('SELECT * FROM orders');
write_to_file( 'orders.csv', $iter );
Two lines of code. One runs the query, the other one writes the data. Looks pretty separated to me.
The good thing about this approach is that now you can also write other things to a file with the same code. The following code could for example talk to some API. The iterator that it returns again gives us one row of results per invocation.
sub api_query {
my ($customer_id) = #_;
my $api = API::Client->new;
my $res = $api->get_orders($customer_id); # returns [ {}, {}, {}, ... ]
my $i = 0;
return sub {
return if $i == $#{ $res };
return $res->[$i++];
}
}
You could drop this into the above example instead of run_query() and it would work, because this function returns something that adheres to the same interface. You could just as well make a write_to_api or write_to_slack_bot function that has the same partial interface. One of the parameters would be the same kind of iterator. Now those are exchangeable too.
Of course this whole example is very contrived. In reality it highly depends on the size and complexity of your program.
If it's a script that runs as a cronjob that does nothing but create this report once a day, you should not care about this separation of concerns. The pragmatic approach would likely be the better choice.
Once you have a lot of those, you'd start caring a bit more. Then my above approach might be viable. But only if you really need to have things flexible.
Not every concept is always applicable, and not every concept always makes sense.
Please keep in mind that there are tools that are better suited for those jobs. Instead of making your own CSV file you can use Text::CSV_XS. Or you could use an ORM like DBIx::Class and have ResultSet objects as your interface.
You should be using a seperate function for doing the job, in your situation using doing how you are doing things currently makes much more sense than sticking to SRP.
I'm new to using DBI for SQL queries in a perl script. The issue I'm having pertains to data in fields that have a forward slash. I'm wanting to use variables as input for my where clause, but it is doing what DBI intends a forward slash to do: stop the query. I tried numerous different work arounds from binds, quotes, etc. but none worked, is it even possible? Data in this is consistent. The line with the my $sql variable is where the trouble is.
#!/usr/bin/perl
# Modules
use DBI;
use DBD::Oracle;
use strict;
use warnings;
# Connection Info
$platform = "Oracle";
$database = "mydb";
$user = "user";
$pw = "pass";
# Data Source
$ds = "dbi:Oracle:$database";
my $dbh = DBI->connect($ds, $user, $pw);
# my $dbh = DBI->connect();
my $XCOD = $dbh->quote('cba');
my $a = $dbh->quote('abc');
my $b = $dbh->quote('123');
# tried this as well my $pid = $dbh->quote('$a/$b');
my $sql = "SELECT P_ID FROM MyTable WHERE P_ID=$a/$b AND XCOD=$XCOD";
my $sth = $dbh->prepare($sql);
$sth->execute();
my $outfile = 'superunique.txt';
open OUTFILE, '>', $outfile or die "Unable to open $outfile: $!";
while(my #re = $sth->fetchrow_array) {
print OUTFILE #re,"\n";
}
close OUTFILE;
$sth->finish();
$dbh->disconnect();
I don't like to see folks use variable interpolation in SQL queries. Try using placeholders:
[ snip ]
my $P_ID = "$a/$b"
my $sql = "SELECT P_ID FROM MyTable WHERE P_ID = ? AND XCOD = ?";
my $sth = $dbh->prepare($sql);
$sth->execute($P_ID, $XCOD);
[ snip ]
You have been given the correct solution to your problem (use placeholders) but you might be interested to see why what you are doing doesn't work.
The problem is that you seem to misunderstand the quote method. The documentation says this:
Quote a string literal for use as a literal value in an SQL statement,
by escaping any special characters (such as quotation marks) contained
within the string and adding the required type of outer quotation
marks.
You use quote in these three lines.
my $XCOD = $dbh->quote('cba');
my $a = $dbh->quote('abc');
my $b = $dbh->quote('123');
It would be instructive to print out the values of $XCOD, $a and $b (as an aside $a and $b are really bad names for variables - apart from their non-descriptive nature, they are also special variables used in sorting).
I suspect that you'll see "cba", "abd" and "123". The method has found no special characters to escape, so all it has done is to add quote marks around the strings.
You then interpolate these values into your SQL.
my $sql = "SELECT P_ID FROM MyTable WHERE P_ID=$a/$b AND XCOD=$XCOD";
Again, you should take a close look at what $sql contains after this statement has been executed. It will look something like this:
SELECT P_ID FROM MyTable WHERE P_ID="abc"/"123" AND XCOD="cba"
It's probably the first part of the WHERE clause that is a problem. Oracle is treating that as a division. And who knows what Oracle does when you divide one string by another. So you end up looking for a row where P_ID is some strange (perhaps undefined) value.
So this looks to be an example where the simplest of debugging techniques (a few print statements in the code) would have guided you in the right direction.
open(INFILE1,"INPUT.txt");
my $modfile = 'Data.txt';
open MODIFIED,'>',$modfile or die "Could not open $modfile : $!";
for (;;) {
my $line1 = <INFILE1>;
last if not defined $line1;
my $line2 = <INFILE1>;
last if not defined $line2;
my ($tablename1, $colname1,$sql1) = split(/\t/, $line1);
my ($tablename2, $colname2,$sql2) = split(/\t/, $line2);
if ($tablename1 eq $tablename2)
{
my $sth1 = $dbh->prepare($sql1);
$sth1->execute;
my $hash_ref1 = $sth1->fetchall_hashref('KEY');
my $sth2 = $dbh->prepare($sql2);
$sth2->execute;
my $hash_ref2 = $sth2->fetchall_hashref('KEY');
my #fieldname = split(/,/, $colname1);
my $colcnt=0;
my $rowcnt=0;
foreach $key1 ( keys(%{$hash_ref1}) )
{
foreach (#fieldname)
{
$colname =$_;
my $strvalue1='';
#val1 = $hash_ref1->{$key1}->{$colname};
if (defined #val1)
{
my #filtered = grep /#val1/, #metadata;
my $strvalue1 = substr(#filtered[0],index(#filtered[0],'||') + 2);
}
my $strvalue2='';
#val2 = $hash_ref2->{$key1}->{$colname};
if (defined #val2)
{
my #filtered = grep /#val2/, #metadata2;
my $strvalue2 = substr(#filtered[0],index(#filtered[0],'||') + 2);
}
if ($strvalue1 ne $strvalue2 )
{
$colcnt = $colcnt + 1;
print MODIFIED "$tablename1\t$colname\t$strvalue1\t$strvalue2\n";
}
}
}
if ($colcnt>0)
{
print "modified count is $colcnt\n";
}
%$hash_ref1 = ();
%$hash_ref2 = ();
}
The program is Read input file in which every line contrain three strings seperated by tab. First is TableName, Second is ALL Column Name with commas in between and third contain the sql to be run. As this utlity is doing comparison of data, so there are two rows for every tablename. One for each DB. So data needs to be picked from each respective db's and then compared column by column.
SQL returns as ID in the result set and if the value is coming from db then it needs be translated to a string by reading from a array (that array contains 100K records with Key and value seperated by ||)
Now I ran this for one set of tables which contains 18K records in each db. There are 8 columns picked from db in each sql. So for every record out of 18K, and then for every field in that record i.e. 8, this script is taking a lot of time.
My question is if someone can look and see if it can be imporoved so that it takes less time.
File contents sample
INPUT.TXT
TABLENAME COL1,COL2 select COL1,COL2 from TABLENAME where ......
TABLENAMEB COL1,COL2 select COL1,COL2 from TABLENAMEB where ......
Metadata array contains something like this(there are two i.e. for each db)
111||Code 1
222||Code 2
Please suggest
Your code does look a bit unusual, and could gain clarity from using subroutines vs. just using loops and conditionals. Here are a few other suggestions.
The excerpt
for (;;) {
my $line1 = <INFILE1>;
last if not defined $line1;
my $line2 = <INFILE1>;
last if not defined $line2;
...;
}
is overly complicated: Not everyone knows the C-ish for(;;) idiom. You have lots of code duplication. And aren't you actually saying loop while I can read two lines?
while (defined(my $line1 = <INFILE1>) and defined(my $line2 = <INFILE1>)) {
...;
}
Yes, that line is longer, but I think it's a bit more self-documenting.
Instead of doing
if ($tablename1 eq $tablename2) { the rest of the loop }
you could say
next if $tablename1 eq $tablename2;
the rest of the loop;
and save a level of intendation. And better intendation equals better readability makes it easier to write good code. And better code might perform better.
What are you doing at foreach $key1 (keys ...) — something tells me you didn't use strict! (Just a hint: lexical variables with my can perform slightly better than global variables)
Also, doing $colname = $_ inside a for-loop is a dumb thing, for the same reason.
for my $key1 (keys ...) {
...;
for my $colname (#fieldname) { ... }
}
my $strvalue1='';
#val1 = $hash_ref1->{$key1}->{$colname};
if (defined #val1)
{
my #filtered = grep /#val1/, #metadata;
my $strvalue1 = substr(#filtered[0],index(#filtered[0],'||') + 2);
}
I don't think this does what you think it does.
From the $hash_ref1 you retrive a single element, then assign that element to an array (a collection of multiple values).
Then you called defined on this array. An array cannot be undefined, and what you are doing is quite deprecated. Calling defined function on a collection returns info about the memory management, but does not indicate ① whether the array is empty or ② whether the first element in that array is defined.
Interpolating an array into a regex isn't likely to be useful: The elements of the array are joined with the value of $", usually a whitespace, and the resulting string treated as a regex. This will wreak havoc if there are metacharacters present.
When you only need the first value of a list, you can force list context, but assign to a single scalar like
my ($filtered) = produce_a_list;
This frees you from weird subscripts you don't need and that only slow you down.
Then you assign to a $strvalue1 variable you just declared. This shadows the outer $strvalue1. They are not the same variable. So after the if branch, you still have the empty string in $strvalue1.
I would write this code like
my $val1 = $hash_ref1->{$key1}{$colname};
my $strvalue1 = defined $val1
? do {
my ($filtered) = grep /\Q$val1/, #metadata;
substr $filtered, 2 + index $filtered, '||'
} : '';
But this would be even cheaper if you pre-split #metadata into pairs and test for equality with the correct field. This would remove some of the bugs that are still lurking in that code.
$x = $x + 1 is commonly written $x++.
Emptying the hashrefs at the end of the iteration is unneccessary: The hashrefs are assigned to a new value at the next iteration of the loop. Also, it is unneccessary to assist Perls garbage collection for such simple tasks.
About the metadata: 100K records is a lot, so either put it in a database itself, or at the very least a hash. Especially for so many records, using a hash is a lot faster than looping through all entries and using slow regexes … aargh!
Create the hash from the file, once at the beginning of the program
my %metadata;
while (<METADATA>) {
chomp;
my ($key, $value) = split /\|\|/;
$metadata{$key} = $value; # assumes each key only has one value
}
Simply look up the key inside the loop
my $strvalue1 = defined $val1 ? $metadata{$val1} // '' : ''
That should be so much faster.
(Oh, and please consider using better names for variables. $strvalue1 doesn't tell me anything, except that it is a stringy value (d'oh). $val1 is even worse.)
This is not really an answer but it won't really fit well in a comment either so, until you provide some more information, here are some observations.
Inside you inner for loop, there is:
#val1 = $hash_ref1->{$key1}->{$colname};
Did you mean #val1 = #{ $hash_ref1->{$key1}->{$colname} };?
Later, you check if (defined #val1)? What did you really want to check? As perldoc -f defined points out:
Use of "defined" on aggregates (hashes and arrays) is
deprecated. It used to report whether memory for that aggregate
had ever been allocated. This behavior may disappear in future
versions of Perl. You should instead use a simple test for size:
In your case, if (defined #val1) will always be true.
Then, you have my #filtered = grep /#val1/, #metadata; Where did #metadata come from? What did you actually intend to check?
Then you have my $strvalue1 = substr(#filtered[0],index(#filtered[0],'||') + 2);
There is some interesting stuff going on in there.
You will need to verbalize what you are actually trying to do.
I strongly suspect there is a single SQL query you can run that will give you what you want but we first need to know what you want.
good evening dear community,
first of all - i am very very happy that i have found this great place. I like this forum very very much, since it has a great and supportive community! I learn alot form you folks here! Each question has got some great reviewers and - each thread is a rich value and learning asset.
Well i am farily new to Perl - and fairly new to this board here: i am currently workin out a little parser: i want to parse a table
click here to see the target url- with the very simple table (some rows only)
This page has a table: well a table with vaules and lables. We need to provide something that uniquely identifies the table in question. This can be the content of its headers or the HTML attributes. In this case, there is only one table in the document, so we don't even need to do that. But, what about to provide anything to the constructor, I would provide the class of the table.
We do not want the columns of the table. The first column of this table consists of labels and the second column consists of values. To get the labels and values at the same time, we should process the table row-by-row. Well - can this be done like so:
#!/usr/bin/perl
use strict; use warnings;
use HTML::TableExtract;
use YAML;
my $te = HTML::TableExtract->new(
attribs => { class => 'bp_ergebnis_tab_info' },
);
$te->parse_file('t.html');
# here the file with the captured site is stored
foreach my $table ( $te->tables ) {
foreach my $row ($table->rows) {
print " ", join(',', #$row), "\n";
}
}
See the results:
martin#suse-linux:~/perl> perl parser_perl_nrw2.pl
Use of uninitialized value $row in join or string at parser_perl_nrw2.pl line 17.
Schuldaten,
Schule hat Schulbetrieb
Schulnummer,143960
Amtliche Bezeichnung,�Franziskusschule Kath. Hauptschule Ahaus - Sekundarstufe I -
Strasse,Hof zum Ahaus 6
Plz und Ort,48683 Ahaus
Telefon,02561 4291990
Fax,02561 42919920
E-Mail-Adresse,143960#schule.nrw.de
Internet,http://www.franziskusschule.de
,Schule in �ffentlicher Tr�gerschaft
WELL i want to get the data that are shown above - but if you see below - there are some more lines of text and code, ... talking like so. (/i want to´get rid of these following lines!!!)
Use of uninitialized value $row in join or string at parser_perl_nrw2.pl
line 17.
,Schülergesamtzahl,648
Use of uninitialized value $row in join or string at parser_perl_nrw2.pl
line 17.
,Ganztagsunterricht,Ja (erweiterter Ganztagsbetrieb)
Sonstiges,Teilnahme am Projekt 'Betrieb und Schule (BUS)'
Use of uninitialized value $row in join or string at parser_perl_nrw2.pl
line 17.
Unterrichtsangebote,
Use of uninitialized value $row in join or string at parser_perl_nrw2.pl
line 17.
Schule erteilt Unterricht in Fremdsprache(n)...,
,Englisch
Question: how do i get rid of the unsanitized data! All is nice - but i want to get rid of the unsanitized data... that is very very ugly - and since i want to store the data into a database - i do not need the unsanitized data...!
As allways: any and all help will be greatly appreciated - many thanks in advance!
regards
zero
You want to get rid of the uninitialized value warnings?
Some of the table cells are empty so you may want to test for them or filter them out. Like this for example:
foreach my $table ( $te->tables ) {
foreach my $row ($table->rows) {
my #values = grep {defined} #$row;
print " ", join(',', #values), "\n";
}
}
You could also outright disable warnings for that particular block with no warnings ' uninitialized', but it is generally not a good practice.