I'm looking for a way to capture the raw SQL for all the queries that the Objection.js library executes with the bindings interpolated into the SQL string.
I realize that there's a Knex event handler that I can take advantage of but the second argument to the on('query', data) is an object containing an SQL template with the bindings separate.
e.g.
{
sql: "select \"accounts\".* from \"accounts\" where \"id\" = ?",
bindings: [1]
}
I'm wondering if the most elegant way to do this would be to use something like the .toString() method that exists on the QueryBuilder but I don't think a specific instance of a QueryBuilder is available in the callback. Ideally I don't reinvent the wheel and re-write Knex's interpolation method.
Any pointers would be greatly appreciated.
Thank you!
You can use the .toKnexQuery() function to pull out the underlying knex query builder and gain access to .toSQL() and .toQuery().
I tested and verified the following example using version 2 of Objection. I couldn't find .toKnexQuery() in the version 1 docs and therefore can't verify it will work with earlier versions of Objection.
// Users.js
const { Model } = require('objection')
class Users extends Model {
static get tableName() { return 'users' }
// Insert jsonSchema, relationMappings, etc. here
}
module.exports = Users
const Users = require('./path/to/Users')
const builder = Users.query()
.findById(1)
.toKnexQuery()
console.log(builder.toQuery())
// "select `users`.* from `users` where `users`.`id` = 1"
console.log(builder.toSQL())
// {
// method: 'select',
// bindings: [ 1 ],
// sql: 'select `users`.* from `users` where `users`.`id` = ?'
// }
It should probably be reiterated that in addition to .toString(), .toQuery() can also be vulnerable to SQL injection attacks (see here).
A more "responsible" way to modify the query might be something like this (with MySQL):
const { sql, bindings } = Users.query()
.insert({ id: 1 })
.toKnexQuery()
.toSQL()
.toNative()
Users.knex().raw(`${sql} ON DUPLICATE KEY UPDATE foo = ?`, [...bindings, 'bar'])
Knex / objection.js does not provide any methods that can securely do the interpolation. .toString() can produce invalid results in some cases and they can be vulnerable to sql injection attacks.
If it is only for debugging purposes looking how .toQuery() is implemented helps. https://github.com/knex/knex/blob/e37aeaa31c8ef9c1b07d2e4d3ec6607e557d800d/lib/interface.js#L12
knex.client._formatQuery(sql, bindings, tz)
It is not a public API though so it is not guaranteed to be the same even between patch versions of knex.
Related
What I have is a code like this one below
var innerQuery = knex('foo_table').select('bar_column')
var joinTable = knex('some_table').select('bar_column')
knex(innerQuery).innerJoin(joinTable,joinTable.bar_column,innerQuery.bar_column)
it gives me
routine: "errorMissingColumn"
I have tried giving tables allias but it was same mistake.
Knex does allow variables in .innerJoin, but the variable must be a function. It gets passed the query builder object (or, in the case of non-arrow functions, you can use this. as the query builder). So you could do something like this:
const innerQuery = knex('foo_table').select('bar_column')
const onBarColumn = qb => qb.on('foo_table.bar_column', '=', 'some_table.bar_column')
const joinTable = qb => qb('some_table').innerJoin('foo_table', onBarColumn)
innerQuery.innerJoin(joinTable)
You can build it up gradually like this if it works better for you. Obviously, the above helpers could be wrapped to make the arguments more flexible.
I'm not sure why this is causing me an issue, but I'm using Orient 2.1.19, found this in 2.1.12 as well. We are building some hooks to implement a method of encryption. I know 2.2 implements some encryption, but we had some further requirements.
Anyway, we have hooks for onRecordAfterRead, onRecordBeforeCreate and onRecordBeforeUpdate. It works for most statements fine, but with the hook in place, running a query that sets a link property using a subquery in an insert fails. Here's an example query:
create EDGE eThisEdge from (select from vVertex where thisproperty = 'this') to (select from vVertex where thatProperty = 'that' ) set current = (select from lookupCurrent where displayCurrentPast = 'Current');
Runnning this query gives me the error:
com.orientechnologies.orient.core.exception.OValidationException: The field 'eThisEdge.current' has been declared as LINK but the value is not a record or a record-id.
It's some issue with the way a subquery is ran during just an insert though, because if I run the insert without setting any properties, then run an update to set the properties, that works. I'd hate to have to rewrite all of our inserts for our base data and our coding just as a work around for this, and it seems like I'm just missing something here.
Has anyone seen this kind of issue with hooks as well?
The biggest issue seems to be surrounding the onRecordBeforeCreate code. We are trying to have a generic hook that encrypts strings in our database. Here's the basics of the onRecordBeforeCreate method:
public RESULT onRecordBeforeCreate( ODocument oDocument) {
RESULT changed = RESULT.RECORD_NOT_CHANGED;
try {
if(classIsCipherable(oDocument)) {
for (String field : oDocument.fieldNames()) {
if (oDocument.fieldType(field) != null && oDocument.fieldType(field) == OType.STRING && oDocument.field(field) != null) {
oDocument.field(field, crypto.encrypt(oDocument.field(field).toString()));
changed = RESULT.RECORD_CHANGED;
}
}
}
return changed;
} catch (Exception e) {
throw new RuntimeException( e );
}
Is there anything there that looks obvious that I'd have issues with running a create edge statement that sets properties with a property that is a link?
The query select from lookupCurrent where displayCurrentPast = "Current" return more than one element, you must use a LinkList or a LinkSet
This is the query I am trying to run in PostgreSQL:
SELECT * FROM message WHERE id IN (
SELECT unnest(message_ids) "mid"
FROM session_messages WHERE session_id = '?' ORDER BY "mid" ASC
);
However, I am not able do something:
create.selectFrom(Tables.MESSAGE).where(Tables.MESSAGE.ID.in(
create.select(DSL.unnest(..))
Because DSL.unnest is a Table<?>, which makes sense since it is trying to take a List-like object (mostly a literal) and convert it to table.
I have a feeling that I need to find a way to wrap the function around my field name, but I have no clue as to how to proceed.
NOTE. The field message_ids is of type bigint[].
EDIT
So, this is how I am doing it now, and it works exactly as expected, but I am not sure if this is the best way to do it:
Field<Long> unnestMessageIdField = DSL.field(
"unnest(" + SESSION_MESSAGES.MESSAGE_IDS.getName() + ")",
Long.class)
.as("mid");
Field<Long> messageIdField = DSL.field("mid", Long.class);
MESSAGE.ID.in(
ctx.select(messageIdField).from(
ctx.select(unnestMessageIdField)
.from(Tables.CHAT_SESSION_MESSAGES)
.where(Tables.CHAT_SESSION_MESSAGES.SESSION_ID.eq(sessionId))
)
.where(condition)
)
EDIT2
After going through the code on https://github.com/jOOQ/jOOQ/blob/master/jOOQ/src/main/java/org/jooq/impl/DSL.java I guess the right way to do this would be:
DSL.function("unnest", SQLDataTypes.BIGINT.getArrayType(), SESSION_MESSAGES.MESSAGE_IDS)
EDIT3
Since as always lukas is here for my jOOQ woes, I am going to capitalize on this :)
Trying to generalize this function, in a signature of sort
public <T> Field<T> unnest(Field<T[]> arrayField) {
return DSL.function("unnest", <??>, arrayField);
}
I don't know how I can fetch the datatype. There seems to be a way to get DataType<T[]> from DataType<T> using DataType::getArrayDataType(), but the reverse is not possible. There is this class I found ArrayDataType, but it seems to be package-private, so I cannot use it (and even if I could, it does not expose the field elementType).
Old PostgreSQL versions had this funky idea that it is OK to produce a table from within the SELECT clause, and expand it into the "outer" table, as if it were declared in the FROM clause. That is a very obscure PostgreSQL legacy, and this example is a good chance to get rid of it, and use LATERAL instead. Your query is equivalent to this one:
SELECT *
FROM message
WHERE id IN (
SELECT "mid"
FROM session_messages
CROSS JOIN LATERAL unnest(message_ids) AS t("mid")
WHERE session_id = '?'
);
This can be translated to jOOQ much more easily as:
DSL.using(configuration)
.select()
.from(MESSAGE)
.where(MESSAGE.ID).in(
select(field(name("mid"), MESSAGE.ID.getDataType()))
.from(SESSION_MESSAGES)
.crossJoin(lateral(unnest(SESSION_MESSAGES.MESSAGE_IDS)).as("t", "mid"))
.where(SESSION_MESSAGES.SESSION_ID.eq("'?'"))
)
The Edit3 in the question is quite close to a decent solution for this problem.
We can create a custom generic unnest method for jOOQ which accepts Field and use it in jOOQ query normally.
Helper method:
public static <T> Field<T> unnest(Field<T[]> field) {
var type = (Class<T>) field.getType().getComponentType();
return DSL.function("unnest", type, field);
}
Usage:
public void query(SessionId sessionId) {
var field = unnest(SESSION_MESSAGES.MESSAGE_IDS, UUID.class);
dsl.select().from(MESSAGE).where(
MESSAGE.ID.in(
dsl.select(field).from(SESSION_MESSAGES)
.where(SESSION_MESSAGES.SESSION_ID.eq(sessionId.id))
.orderBy(field)
)
);
}
Before we start: I have made a very small git repo with seed data and two REST endpoints to test this issue here: https://github.com/juanpasolano/sails-nested-test
So I have 3 models: Appointment which has many procedure which has one to one procedureItem.
Since there is not nested population in sails I am getting procedures with procedureItem by hand using something like:
Appointment.find(1).exec(function(err, appointments){
if(err) return res.negotiate(err);
async.eachSeries(appointments, function(appointment, cb){
Procedure.find({appointment: appointment.id}).populate('procedureItem').exec(function(errP, procedures){
if(errP) return cb(errP);
appointment.procedures = procedures;
appointment.proceduress = procedures;
cb()
})
}, function(errE){
if(errE) return cb(errE);
res.ok(appointments)
So the issue is when I want to replace the proceduresattribute with the new array procedures (which has the nested depth i need) it just doesn't get set, if you hit the endpoints of the demo, there is no procedures attribute available.
As a test I have attached the same new procedures array to proceduress (double s) attribute and this one gets properly set.
I have tried using the .toObject() method with no luck.
Have tried sails 0.10.5 and 0.11.0 with no luck.
Yes, there is no procedures attribute, as Appointment has MANY Procedure, so there is no field procedures in DB in Appointment table. So if you do not populate like:
Appointment.find(1).populate('procedures').exec(function(err, appointments){
.....
}
There will not be attribute procedures in appointment Object
I have the exactly same problem a while ago, I believe it is because the "procedure" attribute is in use by the model and not be able to set. I end up cloning each row and then the value was able to be set.
To change your code:
if(err) return res.negotiate(err);
var toreturn=_.map(appointments, function(a){
var b=_.clone(a); // so you clone it to a new obj, then you can set the procedure...
b.procedures = 'aaaa'
return b;
})
//res.json(toreturn); // Here you will get appointments with procedures='aaa'
async.eachSeries(toreturn, function(appointment, cb){
Procedure.find({appointment: appointment.id}).populate('procedureItem').exec(function(errP, procedures){
if(errP) return cb(errP);
appointment.proceduress = procedures;
appointment.procedures = procedures;
cb()
})
}, function(errE){
if(errE) return cb(errE);
res.ok(toreturn)
})
I do not know why this happened exactly, but this is just one solution.
On a side note, may I suggest you instead of doing async for each appoint, will takes a query per appoint, do two big find, and then loop to merge them?
Something like this:
var globalApps=[];
Appointment.find().then(function(apps){
globalApps=apps;
var appIDs=apps.map(function(a){return a.id}); //Get all appointment IDs
return Procedure.find({appointment:appIDs}).populate('procedureItem');
}).then(function(procs){ //This procs is all procedures match the ids
var toReturn =_.map(globalApps,function(a){
var b=_.clone(a);
b.procedure=_.find(procs,{appointment:b.id}); // This is O(n^2) complexity matching, but you can do something smart.
return b;
});
// Now you have toReturn as the appointment array.
return res.json(toReturn);
})
I have spent a lot of time on this problem before, glad someone had the same problem
#sgress454 from Balderdash here. The issue you're encountering is that the procedures attribute isn't a plain Javascript variable; it has a "getter" and a "setter", which are necessary in order to allow these to work:
appointment.procedures.add(<some Procedure obj or id>);
appointment.procedures.remove(<some Procedure id>);
So attempting to set the procedures attribute to a new value won't work, because of the setter. Cloning the appointment is an acceptable workaround (you can just do appointment.toObject()) as it transforms those special attributes into plain old Javascript variables. But in your case, it seems like you don't actually need to replace the procedures array--you just want each procedure's procedureItem to be filled out. And since only attributes representing many-to-many associations have getters and setters, you can accomplish this without any cloning at all:
Appointment
// Find the appointments
.find({...some criteria...})
// Populate each appointment's procedure
.populate('procedure')
.exec(function(err, appointments) {
if (err) {return res.negotiate(err);}
// Loop through the returned appointments
async.each(appointments, function(appointment, appointmentCb) {
// For each one, loop through its procedures
async.each(appointment.procedures, function(procedure, procedureCb) {
// Look up the related procedureItem
ProcedureItem.findOne(procedure.procedureItem).exec(function(err, procedureItem) {
if (err) {return procedureCb(err);}
// Attach the procedureItem object to the procedure
procedure.procedureItem = procedureItem;
return procedureCb();
});
}, appointmentCb);
}, function done(err) {
if (err) {return res.negotiate(err);}
res.ok(appointments);
});
});
Note that there's a lot of optimization that could be done here--you could pluck out all of the ProcedureItem IDs after retrieving appointments, look up all the ProcedureItem instances at once, and then map them onto the procedures afterwards, saving a ton of queries.
With dapper, I can do batch execute for Stored Procedures, something similar to:
connection.Execute(#"
exec sp1 #i = #one, #y = #two
exec sp2 #i = #three",
new { one = 1, two = 2, three = 3 });
However, the only means of retrieving data that I have seen till now is by using
results.Read<Type>()
What if the results don't map to an object? For instance, I am writing "generic" code to execute any SP with variable in/out parameters & result sets.
Thanks
What API do you want? If you can process the grids separately: do that:
using(var multi = connection.QueryMultiple(...))
{
while(!multi.IsConsumed) {
// ...
}
}
where ... has access to:
Read() for dynamic rows - noting that each row also implements IDictionary<string,object>
Read<T>() for typed rows via generics
Read(Type) for typed rows without generics
Read<DapperRow>() (actually, this is just the T that Read<T>() uses to implement Read(), but perhaps more convenient), which provides slightly more access to metadata
If you want to drop to a raw IDataReader, do that:
using(var reader = connection.ExecuteReader(...)) {
// whatever you want
}
With regards to parameters: the DynamicParameters class provides much richer access to parameter control, including parameter-direction etc.