Export data from SQL database to CSV file, some rows of data get split into more than one row - export-to-csv

When I export data from my SQL database to CSV file, some rows of data (records) get split into more than one row, as if there is a CR. I know that one reason is the following: One of the columns of the data is "Notes" that contains text that sometimes does contain a CR; I understand why this causes a new row in the CSV, but I would like that not to happen, either. How can I strip the CR, but add a period+space to format the Note so it's readable even without the CR?
However, I also get the extra row even if there is no CR, meaning the CSV has a blank row after a record, or the Note is on an extra line. I've included a screenshot of a portion of the CSV file to illustrate this and also illustrate that not all records show the behavior.
Here is my code. I did not write this, I inherited it. Also, I am not very experienced writing code.
header('Content-Type: application/msexcel-tab');
header('Content-Disposition: attachment; filename="Invaders of Texas Data -- '.date("Y-m-d").'.xls"');
$whereclause = '';
$passclause = '';
$satellite = $_REQUEST['satellite'];
$collector = $_REQUEST['collector'];
$sn = $_REQUEST['sn'];
$cn = $_REQUEST['cn'];
if ($satellite){
$whereclause .= " AND `satellite_id` = ".$satellite." ";
$passclause .= "&satellite=".$satellite;
}
if ($collector){
$whereclause .= " AND `collector_id` = ".$collector." ";
$passclause .= "&collector=".$collector;
}
if ($sn){
$whereclause .= " AND `plant_id` LIKE '".$sn."' ";
$passclause .= "&sn=".$sn;
}
if ($cn){
$whereclause .= " AND `plant_id` LIKE '".$cn."' ";
$passclause .= "&cn=".$cn;
}
$count_sql = "
SELECT COUNT(*) AS `counttotal`
FROM `inv_sites`
WHERE 1
$whereclause
AND `valid` LIKE 'Yes'
;
";
//echo $count_sql;
$count_total = mysql_fetch_array(mysql_query($count_sql));
$sql = "
SELECT *
FROM `inv_sites`
WHERE 1
$whereclause
AND `valid` LIKE 'Yes'
ORDER BY `collection_date` ASC
;
";
$the_result = mysql_query($sql);
?>
Invaders of Texas
www.texasinvasives.org
Exported: <?= date("Y-m-d G:i"); ?>
Obs_ID Date USDA Species Time_Spent Satellite Collector Lat Long Location_Error Loc_Err_Units Disturbance Patch_Type Abundance Validated Valid_Name Valid_Date Notes
<?php
if ($this_row = mysql_fetch_array($the_result)){
do {
?>
<?=$this_row['site_id'];?> <?=$this_row['collection_date'];?> <?=$this_row['plant_id']?> <?=sn_from_usda($this_row['plant_id'])?> <?=$this_row['collection_time'];?> <?=satellite_from_id($this_row['satellite_id']);?> <?=$this_row['collector_id'];?> <?=$this_row['latitude'];?> <?=$this_row['longitude'];?> <?=$this_row['error'];?> <?=$this_row['error_unit'];?> <?=$this_row['disturbance'];?> <?=$this_row['patch_type'];?> <?=$this_row['abundance'];?> <?=$this_row['valid'];?> <?=$this_row['valid_name'];?> <?=$this_row['valid_date'];?> <?=$this_row['notes'];?>
<?php
} while ($this_row = mysql_fetch_array($the_result));
}
?>
I'd appreciate any help!! Thanks.

You could replace the newlines in the PHP or the SQL query.
You have the following line above.
<?=$this_row['site_id'];?> <?=$this_row['collection_date'];?> <?=$this_row['plant_id']?> <?=sn_from_usda($this_row['plant_id'])?> <?=$this_row['collection_time'];?> <?=satellite_from_id($this_row['satellite_id']);?> <?=$this_row['collector_id'];?> <?=$this_row['latitude'];?> <?=$this_row['longitude'];?> <?=$this_row['error'];?> <?=$this_row['error_unit'];?> <?=$this_row['disturbance'];?> <?=$this_row['patch_type'];?> <?=$this_row['abundance'];?> <?=$this_row['valid'];?> <?=$this_row['valid_name'];?> <?=$this_row['valid_date'];?> <?=$this_row['notes'];?>
Try replacing it with the below (the change is on the very end).
<?=$this_row['site_id'];?> <?=$this_row['collection_date'];?> <?=$this_row['plant_id']?> <?=sn_from_usda($this_row['plant_id'])?> <?=$this_row['collection_time'];?> <?=satellite_from_id($this_row['satellite_id']);?> <?=$this_row['collector_id'];?> <?=$this_row['latitude'];?> <?=$this_row['longitude'];?> <?=$this_row['error'];?> <?=$this_row['error_unit'];?> <?=$this_row['disturbance'];?> <?=$this_row['patch_type'];?> <?=$this_row['abundance'];?> <?=$this_row['valid'];?> <?=$this_row['valid_name'];?> <?=$this_row['valid_date'];?> <?=trim(preg_replace('/\s+/', ' ', $this_row['notes']));?>
The preg_replace allows you to use regular expressions in php to remove the newlines.
If this doesn't work you may need to alter your SQL query to remove the newline from the database query.
See this post
Pete

Related

How to speed up the search function of datatables?

here's my code to search in tables (datatables):
Total records: 333,213
Estimated time to search/appear the result: 5 to 10 seconds.
using : ajax: "sample.php", // json datasource
how to make it fast?
what should I fix the database or the code I'm using.
<?php
/* Database connection start */
include ('connectvl.php');
/* Database connection end */
// storing request (ie, get/post) global array to a variable
$requestData= $_REQUEST;
$columns = array(
// datatable column index => database column name
0=> 'id',
1=> 'FULLNAME',
2=> 'BrgyName',
3=> 'BDAY',
4=> 'RESSTREET'
);
// getting total number records without any search
$sql = "SELECT id";
$sql.=" FROM voterslist2012";
$query=mysqli_query($conn, $sql) or die("cswd_listofpendingprocessgrid.php: get employees");
$totalData = mysqli_num_rows($query);
$totalFiltered = $totalData; // when there is no search parameter then total number rows = total number filtered rows.
$sql = "SELECT id, FULLNAME, BrgyName, BDAY, RESSTREET";
$sql.=" FROM voterslist2012 WHERE 1=1";
if( !empty($requestData['search']['value']) ) { // if there is a search parameter, $requestData['search']['value'] contains search parameter
$sql.=" AND ( id LIKE '".$requestData['search']['value']."%' ";
$sql.=" OR FULLNAME LIKE '".$requestData['search']['value']."%' ";
$sql.=" OR BDAY LIKE '".$requestData['search']['value']."%' ";
$sql.=" OR BrgyName LIKE '".$requestData['search']['value']."%' )";
}
// If there is a search parameter
/*if( !empty($requestData['search']['value']) ) {
$search = mysqli_real_escape_string(
$conn,
// Match beginning of word boundary
"[[:<:]]".
// Replace space characters with regular expression
// to match one or more space characters in the target field
implode("[[.space.]]+",
preg_split("/\s+/",
// Quote regular expression characters
preg_quote(trim($requestData['search']['value']))
)
).
// Match end of word boundary
"[[:>:]]"
);
$sql.=" AND ( id REGEXP '$search' ";
$sql.=" OR FULLNAME REGEXP '$search' ";
$sql.=" OR BrgyName REGEXP '$search' ";
$sql.=" OR BDAY REGEXP '$search' ";
$sql.=" OR RESSTREET REGEXP '$search' )";
}*/
$query=mysqli_query($conn, $sql) or die("cswd_listofpendingprocessgrid.php: get employees");
$totalFiltered = mysqli_num_rows($query); // when there is a search parameter then we have to modify total number filtered rows as per search result.
$sql.=" ORDER BY ". $columns[$requestData['order'][0]['column']]." ".$requestData['order'][0]['dir']." LIMIT ".$requestData['start']." ,".$requestData['length']." ";
/* $requestData['order'][0]['column'] contains colmun index, $requestData['order'][0]['dir'] contains order such as asc/desc */
$query=mysqli_query($conn, $sql) or die("cswd_listofpendingprocessgrid.php: get employees");
$data = array();
while( $row=mysqli_fetch_array($query) ) { // preparing an array
$nestedData=array();
$nestedData[] = $row["id"];
$nestedData[] = $row["FULLNAME"];
$nestedData[] = $row["BDAY"];
$nestedData[] = $row["BrgyName"];
$nestedData[] = $row["RESSTREET"];
$data[] = $nestedData;
}
$json_data = array(
"draw" => intval( $requestData['draw'] ), // for every request/draw by clientside , they send a number as a parameter, when they recieve a response/data they first check the draw number, so we are sending same number in draw.
"recordsTotal" => intval( $totalData ), // total number of records
"recordsFiltered" => intval( $totalFiltered ), // total number of records after searching, if there is no searching then totalFiltered = totalData
"data" => $data // total data array
);
echo json_encode($json_data); // send data as json format
?>
Using LIKE in your WHERE clause is always going to be dead slow, since it bypasses any indexes present and uses a full table scan. I'm going to leave the issue of prepared statements vs. dynamic SQL for others to raise (You should be using prepared statements. ;) ), but if you make id your primary key and then do an exact match in your WHERE clause, the difference will bloody well astound you.
EDIT: If practicable, you should consider placing indexes on any fields you search frequently. Your BDAY column would be a good candidate. I'm not familiar enough with MySQL's optimizer to say if indexing FULLNAME or BRGYNAME be a good idea, but you could experiment.

Mysqli second query doesn't work

I want to use de result from my first query to insert in a second query, there's not error in the sintaxis.
I've been reading a lot but anything seems to work.
HELP!!!!
$sql = "CALL sp_producto(1);";
$rs = $mysqli->query($sql);
$idNuevo;
if ($fila = $rs->fetch_assoc()) {
$idNuevo = $fila['idNew']; //recupera el id del insertado
$talleres = explode(",", $_POST["proceso_prod"]);
foreach ($talleres as $t) {
//THIS DOESN'T WORK BUT IDONT KNOW WHY
$sql = "INSERT INTO proceso_produccion () VALUES (" . $idNuevo . ", ".$t.")";
echo $sql;
$mysqli->query($sql);
}
}
INSERT INTO proceso_produccion () VALUES (" . $idNuevo . ", ".$t.")
This line is the issue, if you are inserting values in all of your table columns no need to have this ()
,change your code to this :
INSERT INTO proceso_produccion VALUES (" . $idNuevo . ", ".$t.")
but if you are inserting to only certain column, you have to specify the column names,
like this :
INSERT INTO proceso_produccion ('firstColumn','anotherColumn') VALUES (" . $idNuevo . ", ".$t.")
Niang answered right, but also check if both values you are entering are integers. Otherwise, the should be in quotes like so:
INSERT INTO proceso_produccion ('firstColumn','anotherColumn') VALUES ('" . $idNuevo . "', '".$t."')
Also, this condition is not a real condition:
if ($fila = $rs->fetch_assoc()) {
You are only assigning value to $fila this way. It should be:
if ($fila == $rs->fetch_assoc()) {
Finally, if query still doesn't work can you please post echoed query, the result of echo $sql.

Add data to csv column based on row value

I have a CSV that I read in. The problem is that this CSV as columns that are not filled with data. I need to go through a column and when the ID number matches the ID of the content I want to add, add in the data. Currently I have,
$counter = 0
foreach($invoice in $allInvoices){
$knownName += $info | where-Object{$_.'DOCNUMBR'.trim().contains($cleanInvNum)}
$DocNumbr = $info | where-Object{$_.'DOCNUMBR'.trim() -eq $cleanInvNum}
$output = $ResultsTable.Rows.Add(" ", " ", " ", " ", $DocNumbr[$counter].'ORG', $DocNumbr[$counter].'CURNT', $knownName[$counter].'DOCNUMBR', " ")
$counter++
}
The problem with this code is that it just adds rows under the CSV and does not add the data to the row. How can I do a statement where I find ID and add the above content into that row?
I was able to resolve my issue by reworking my foreach loop and setting the values directly like so,
$output.'CheckNumber' = $DocNumbr.'ORTRXAMT'
$output.'CheckDate' = $DocNumbr.'CURTRXAM'
$output.'Invoice Number' = $knownName.'DOCNUMBR'

Quote raw sql in ZEND to avoid sql injection

Long story short, I have an admin section where the user can choose from multiple dropdown lists the tables and fields that must be queries in order to get some values. Therefore, the query in ZEND is performed by concatenating the strings
$query = "SELECT $fieldName1, $fieldName2 from $tableName where $fieldName1 = $value";
How can I escape the above using ZEND approach to avoid sql injection? I tried adding them all as ? and calling quoteinto but it seems this does not work on some of the variables (like table names or field names)
ZF has quoteIdentifier() specifically for this purpose:
$query = "SELECT ".$db->quoteIdentifier($fieldName1).","...
In your case you might (also) want to check against a white list of valid column names.
Use quoteInto() or Zend_db_Select::where() for the values, and for the table and column names, I would simply strip any non alpha characters and then wrap them in ` quotes prior to using them in your SQL.
Example:
// Strip non alpha and quote
$fieldName1 = '`' . preg_replace('/[^A-Za-z]/', '', $fieldName1) . '`';
$tableName = '`' . preg_replace('/[^A-Za-z]/', '', $tableName) . '`';
// ....
// Build the SQL using Zend Db Select
$db->select()->from($tableName, array($fieldName1, $fieldName2))
->where($fieldName1 . ' = ?', $value);
In SafeMysql you can make it as simple, as
$sql = "SELECT ?n, ?n from ?n where ?n = ?s";
$data = $db->getAll($sql,$fieldName1,$fieldName2, $tableName, $fieldName1, $value);
though I understand that you won't change your ZF to SafeMysql.
Nevertheless, there is one essential thing that is ought to be done manually:
I doubt you want to let users to browse users table or financial table or whatever.
So, you have to verify a passed table name against an allowed tables array.
like
$allowed = ('test1','test2');
if (!in_array($tableName, $allowed)) {
throw new _403();
}

Perl DBI and sql now()

I have been trying to use sql NOW() function while I update a table. But nothing happens to that date field, I think DBI just ignores that value. Code is :
dbUpdate($id, (notes => 'This was an update', filesize => '100.505', dateEnd => 'NOW()'));
and the function is :
sub dbUpdate {
my $id = shift;
my %val_hash = #_;
my $table = 'backupjobs';
my #fields = keys %val_hash;
my #values = values %val_hash;
my $update_stmt = "UPDATE $table SET ";
my $count = 1;
foreach ( #fields ) {
$update_stmt .= "$_ = ? ";
$update_stmt .= ', ' if ($count != scalar #fields);
$count++;
}
$update_stmt .= " WHERE ID = ?";
print "update_stmt is : $update_stmt\n";
my $dbo = dbConnect();
my $sth = $dbo->prepare( $update_stmt );
$sth->execute( #values, $id ) or die "DB Update failed : $!\n";
$dbo->disconnect || die "Failed to disconnect\n";
return 1;
}#dbUpdate
As you can see this is a "dynamic" generation of the sql query and hence I dont know where an sql date function(like now()) come.
In the given example the sql query will be
UPDATE backupjobs SET filesize = ? , notes = ? , dateEnd = ? WHERE ID = ?
with param values
100.55, This was an update, NOW(), 7
But the date column still shows 0000-00-........
Any ideas how to fix this ?
I have been trying to use sql NOW() function while I update a table. But
nothing happens to that date field, I think DBI just ignores that
value.
No, it doesn't. But the DB thinks you are supplying a datetime or timestamp data type when you are in fact trying to add the string NOW(). That of course doesn't work. Unfortunately DBI cannot find that out for you. All it does is change ? into an escaped (via $dbh->quote()) version of the string it got.
There are several things you could do:
Supply a current timestamp string in the appropriate format instead of the string NOW().
If you have it available, you can use DateTime::Format::MySQL like this:
use DateTime::Format::MySQL;
dbUpdate(
$id,
(
notes => 'This was an update',
filesize => '100.505',
dateEnd => DateTime::Format::MySQL->format_datetime(DateTime->now),
)
);
If not, just use DateTime or Time::Piece.
use DateTime;
my $now = DateTime->now->datetime;
$now =~ y/T/ /;
dbUpdate(
$id,
(notes => 'This was an update', filesize => '100.505', dateEnd => $now));
Tell your dbUpdate function to look for things like NOW() and react accordingly.
You can do something like this. But there are better ways to code this. You should also consider that there is e.g. CURRENT_TIMESTAMP() as well.
sub dbUpdate {
my $id = shift;
my %val_hash = #_;
my $table = 'backupjobs';
my $update_stmt = "UPDATE $table SET ";
# Check for the NOW() value
# This could be done with others too
foreach ( keys %val_hash ) {
if ($val_hash{$_} =~ m/^NOW\(\)$/i) {
$update_stmt .= "$_ = NOW()";
$update_stmt .= ', ' if scalar keys %val_hash > 1;
delete $val_hash{$_};
}
}
# Put everything together, join keeps track of the trailing comma
$update_stmt .= join(', ', map { "$_=?" } keys %val_hash );
$update_stmt .= " WHERE ID = ?";
say "update_stmt is : $update_stmt";
say "values are: ", join(', ', values %val_hash);
my $dbo = dbConnect();
my $sth = $dbo->prepare( $update_stmt );
$sth->execute( values %val_hash, $id ) or die "DB Update failed : $!\n";
$dbo->disconnect || die "Failed to disconnect\n";
return 1;
}
Write your queries yourself.
You're probably not going to do it and I'll not add an example since you know how to do it anyway.
Here's something else: Is this the only thing you do with your database while your program runs? It is not wise to connect and disconnect the database every time you make a query. It would be better for performance to connect the database once you need it (or at the beginning of the program, if you always use it) and just use this dbh/dbo everywhere. It saves a lot of time (and code).
Sorry but you can't use NOW() as an interpolated value like that.
when a ? is used in an SQL statement, the corresponding value is (via some mechanism or other) passed to the database escaped, so whatever value is interpreted as a string, not as SQL. So you are in effect attempting to use the string 'NOW()' as a date value, not the function NOW().
To use NOW() as a function, you will have to insert it into the SQL itself rather than pass it as a bound value. This means you will either have to use a hack of your dbupdate function or write a new one, or obtain the time as a string in perl and pass the resulting string the dbupdate.