i need to mark a batch of rows in the database as "processed".
In the olden days, when things were easier, i would create some SQL that said:
UPDATE Readings SET IsProcessed = 1 WHERE ReadingDateTime IN (
"2010-10-07 22:02:13.327",
"2010-10-07 22:02:14.213",
"2010-10-07 22:02:15.595",
...
"2010-10-07 23:03:36.981")
by looping through a list of dates:
sql = "UPDATE Readings SET IsProcessed = 1 WHERE ReadingDateTime IN (";
foreach (DateTime datetime in dates)
{
sql = sql + CRLF+ DateTimeToSql(datetime)+",";
}
sql = sql+")";
And issue the SQL:
ExecuteNonQuery(connection, sql);
And all was good.
Now i want to try to do things the hard way; i want to try to use parameters:
sql = ???;
command.CommandText = sql;
DbCommand command = connection.CreateCommand();
foreach (DateTime datetime in readings)
{
command.Parameters.Add(new SqlParameter(???, SqlDbType.DateTime) {Value = datetime});
}
using (DbTransaction transaction = connection.BeginTransaction())
{
command.Transaction = transaction;
command.ExecuteNonQuery();
transaction.Commit();
}
The only trick is what to put in the sql string, and what to put in each iteration of the parameters loop. They need some name.
What's the recommended way to name arbitrary number of parameters?
I think what you could do is something like this:-
sql = "UPDATE Readings SET IsProcessed = 1 WHERE ReadingDateTime IN (";
for (int count=0; count<dates.Length; count++)
{
sql = sql + CRLF + ":param" + count;
}
sql = sql+")";
for (int count=0; count<dates.Length; count++)
{
command.Parameters.Add(new SqlParameter(":param" + count, SqlDbType.DateTime) {Value = datetime});
}
However, i think that having parameters specifically in this scenario is kind of unnecessary.
Considering that your dynamic values are dates and not strings, you can instead validate the dates directly using a TryParse to ensure that they are the correct datatype before you append them in your original solution!!
I dont know if you are achieving anything extra over that by using parameters in this case.
You can go as minimal as you like: the provider will accept parameters named #0, #1, etc. But if you want to be more descriptive, just use a base name of #ReadingDateTime, with an integer suffix from 0..n. Easy peasy.
Related
again, I'm struggeling with Linq or EF6.
I have a table with columns (DateTime)[BeginOfWork] and (DateTime)[EndOfWork].
Now, I need to substract the BeginOfWork-value from the EndOfWork-value. From this result, I need to build a total sum.
In SQL, it looks like this:
SELECT Timesheets.EmployeeId, Sum(DateDiff("h",[Timesheets].[BeginOfWork],[Timesheets].[EndOfWork])) AS HoursOfWork
FROM Timesheets
GROUP BY Timesheets.EmployeeId
HAVING (((Timesheets.EmployeeId)=1));
How do I do this in EF6?
Actually, not working some variations of that:
int employeeId = 1;
var timesheetSum = (from ts in db.Timesheets
where (ts.EmployeeId == employeeId)
select new
{
hoursOfWork = DbFunctions.DiffHours(ts.BeginOfWork, ts.EndOfWork)
}).Sum(ts => ts.hoursOfWork);
The above results in integer rounded hoursOfWork, that don't include minutes.
So I tried that:
var timesheetSum = (from ts in db.Timesheets
where (ts.EmployeeId == employeeId)
select new
{
hoursOfWork = DbFunctions.DiffMinutes(ts.BeginOfWork, ts.EndOfWork) / 60
}).Sum(ts => ts.hoursOfWork);
But here, hoursOfWork seems to be rounded (integer), so 2,5 hours will result in 2. Perhaps a conversion of the result would work but I don't get it run.
`(double)hoursOfWork` results in an error.
Perhaps someone has a link to a complete guide, how to convert SQL to Linq-queries.
SOLUTION
var timesheetSum = (from ts in db.Timesheets
where (ts.EmployeeId == employeeId && ts.TimesheetDate <= DateTime.Today)
select new
{
hoursOfWork = DbFunctions.DiffMinutes(ts.BeginOfWork, ts.EndOfWork)/60m
}).Sum(ts => ts.hoursOfWork);
decimal result = 0;
if(timesheetSum!=null)
result = (decimal)timesheetSum;
Thanks a lot
Carsten
I am currently working on weather monitoring.
For example a record of temperature has a date and a location (coordinates).
All of the coordinates are already in the database, what I need to add is time and the value of the temperature. Values and metadata are in a CSV file.
Basically what I'm doing is:
Get time through the file's name
Insert time into DB, and keep the primary key
Reading file, get the value and coordinates
Select query to get the id of the coordinates
Insert weather value with foreign keys (time and coordinates)
The issue is that the
"SELECT id FROM location WHERE latitude = ... AND longitude = ..."
is too slow. I have got 230k files and currently one file takes more than 2 minutes to be processed... Edit: by changing the index, it now takes 25 seconds and is still too slow. Moreover, the PreparedStatement is also still slower and I cannot figure out why.
private static void putFileIntoDB(String variableName, ArrayList<String[]> matrix, File file, PreparedStatement prepWeather, PreparedStatement prepLoc, PreparedStatement prepTime, Connection conn){
try {
int col = matrix.size();
int row = matrix.get(0).length;
String ts = getTimestamp(file);
Time time = getTime(ts);
// INSERT INTO takes 14ms
prepTime.setInt(1, time.year);
prepTime.setInt(2, time.month);
prepTime.setInt(3, time.day);
prepTime.setInt(4, time.hour);
ResultSet rs = prepTime.executeQuery();
rs.next();
int id_time = rs.getInt(1);
//for each column (longitude)
for(int i = 1 ; i < col ; ++i){
// for each row (latitude)
for(int j = 1 ; j < row ; ++j){
try {
String lon = matrix.get(i)[0];
String lat = matrix.get(0)[j];
String var = matrix.get(i)[j];
lat = lat.substring(1, lat.length()-1);
lon = lon.substring(1, lon.length()-1);
double latitude = Double.parseDouble(lat);
double longitude = Double.parseDouble(lon);
double value = Double.parseDouble(var);
// With this prepared statement, instruction needs 16ms to be executed
prepLoc.setDouble(1, latitude);
prepLoc.setDouble(2, longitude);
ResultSet rsLoc = prepLoc.executeQuery();
rsLoc.next();
int id_loc = rsLoc.getInt(1);
// Whereas this block takes 1ms
Statement stm = conn.createStatement();
ResultSet rsLoc = stm.executeQuery("SELECT id from location WHERE latitude = " + latitude + " AND longitude =" + longitude + ";" );
rsLoc.next();
int id_loc = rsLoc.getInt(1);
// INSERT INTO takes 1ms
prepWeather.setObject(1, id_time);
prepWeather.setObject(2, id_loc);
prepWeather.setObject(3, value);
prepWeather.execute();
} catch (SQLException ex) {
Logger.getLogger(ECMWFHelper.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
} catch (SQLException ex) {
Logger.getLogger(ECMWFHelper.class.getName()).log(Level.SEVERE, null, ex);
}
}
What I already did:
Set two B-tree index on table location on columns latitude and longitude
Drop foreign keys constraints
PreparedStatements in parameters are :
// Prepare selection for weather_radar foreign key
PreparedStatement prepLoc = conn.prepareStatement("SELECT id from location WHERE latitude = ? AND longitude = ?;");
PreparedStatement prepTime = conn.prepareStatement("INSERT INTO time(dataSetID, year, month, day, hour) " +
"VALUES(" + dataSetID +", ?, ? , ?, ?)" +
" RETURNING id;");
// PrepareStatement for weather_radar table
PreparedStatement prepWeather = conn.prepareStatement("INSERT INTO weather_radar(dataSetID, id_1, id_2, " + variableName + ")"
+ "VALUES(" + dataSetID + ", ?, ?, ?)");
Any idea to get things go quicker?
Ubuntu 16.04 LTS 64-bits
15.5 Gio
Intel® Core™ i7-6500U CPU # 2.50GHz × 4
PostgreSQL 9.5.11 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609, 64-bit
Netbeans IDE 8.2
JDK 1.8
postgresql-42.2.0.jar
The key issue you have here is you miss ResultSet.close() and Statement.close() kind of calls.
As you resolve that (add relevant close calls) you might find that having SINGLE con.prepareStatement call (before both for loops) would improve the performance even further (of course, you will not need to close the statement in a loop, however you still would need to close resultsets in a loop).
Then you might apply batch SQL
Using EXPLAIN, the point where query becomes latent could be figured out.
One of the situation where I have encountered case alike being:
Compound queries e.g. parameterized similar date ranges, from different tables and then joining them on some indexed value. Even if the date in the above serve as index still the query produced in preparedStatement, could not hit the indexes and ended up doing a scan over the joining data.
I need to do a simple query, Select Statement
I want to search in Table all record with value "ValueA, ValueB".
If I use this code, not work well:
String255 valueToFilter;
valueToFilter = 'ValueA, ValueB';
select count (RecId) from MyTable
where MyTable.Field like valueToFilter ;
But not working, I need to keep all record with value "ValueA" or "ValueB", if in the file there is value like : "ValueA, ValueC" I want to get too.
I don't know the number of values (valueToFilter).
Thanks!
From my point of view the easiest way to accomplish this is to split your filter string:
String255 valueToFilterA = 'ValueA';
String255 valueToFilterB = 'ValueB';
;
select count (RecId) from MyTable
where MyTable.Field like valueToFilterA
|| MyTable.Field like valueToFilterB;
If you don't know the number of values you should use query object to add ranges dynamically:
Query query = new Query();
QueryRun queryRun;
QueryBuildDataSource qbds;
QueryBuildRange queryRange;
container conValues;
;
qbds = query.addDataSource(tableNum(MyTable));
for (i = 1; i <= conlen(conValues); i++)
{
queryRange = qbds.addRange(fieldNum(MyTable, Field));
queryRange.value(SysQuery::valueLike(conPeek(conValues, i)));
}
queryRun = new QueryRun(query);
info(strFmt("Records count %1", SysQuery::countTotal(queryRun)));
I am trying to do this simple query, but it does not work. Thanks.
SELECT * FROM TSimple where (start_date < '2012-04-20' and end_date is null) or
(end_date > '2012-04-20' and start_date < '2012-04-20')
class TSimple (models.Model):
start_date = models.DateTimeField()
end_date = models.DateTimeField(blank=True, null=True)
...
class TSimpleResource(ModelResource):
def dehydrate(self, bundle):
request_method = bundle.request.META['REQUEST_METHOD']
if request_method=='GET':
new_date = bundle.request.GET.get('new_date', '')
qs = TSimple.objects.raw(
'SELECT * FROM TSimple where (start_date<=\'' +
new_date + '\' and end_date>=\'' +
new_date + '\') or (start_date<=\'' + new_date +
'\' and end_date is null)')
ret_list = [row for row in qs]
// NOT WORK. Not able to get correct json data in javascript.
// It needs return bundle. HOW to replace bundle?
// Is this correct way to do it?
return ret_list
else:
// This is ok.
return bundle
I have following questions:
1) (raw sql method) If implementing in dehydrate method is correct way to do it? If it is, above does not work. It should return bundle object. How to construct new bundle?
If above method is ok, I noticed that bundle already constructed .data field with default query(?), which will be thrown away with new query. That raise the questions if this is right way to do it.
2) If there are other raw sql method to do it? Where to execute the sql?
3) How to do it in filter?
4) I know sql and not familiar with complex filter. That's why I am trying to use raw sql method to do quick prototype. What are the draw back? I noticed that using Tastypie has many unnecessary queries which I don't know how to get rid of it. Example, query on table with foreign key trigger query to another table's data, which I don't want to get.
I figure out the filter and it seems worked. But I am still interested in raw sql.
def apply_filters(self, request, applicable_filters):
base_filter = super(TSimpleResource, self).apply_filters(request,
applicable_filters)
new_date = request.GET.get('new_date', None)
if new_date:
qset = (
(
Q(start_date__lte=new_date) &
Q(end_date__gte=new_date)
) |
(
Q(start_date__lte=new_date) &
Q(end_date__isnull=True)
)
)
base_filter = base_filter.filter(qset)
return base_filter
I am using JDBC in JSP with PostGreSQL.
I want to read all values of a row with the given titel and interpret from a text field but the AND operator doesn't work.
// some code
stmt = conn.createStatement();
res = stmt.executeQuery(
"SELECT * " +
"FROM album " +
"WHERE interpret = ? AND titel = ? " +
"ORDER BY interpret, titel ASC "
);
//... closte statements, etc.
Not I get a syntax exception for AND.
Do you guys have any advices why?
You cannot use bind variables in a statement created with createStatement.
PreparedStatement is what you should be working with.
Use:
stmt = conn.prepareStatement();
stmt.setString(1, interpretValue); //set the value for the first parameter to interpretValue
stmt.setString(2, titleValue); //second parameter
A PreparedStatement is the preferred way of executing SQL statements because the statement is precompiled. This can be more efficient than a Statement, specially if the same query is executed multiple times.