I want to fetch data from a Postgres database recursively including all associated models using SequelizeJS.
GeoLocations table
id | isoCode | name | parentId | type | updatedAt | deletedAt
parentId holds the id of the parent GeoLocation.
Model Associations
GeoLocation.belongsTo(GeoLocation, {
foreignKey: 'parentId',
as: 'GeoLocation'
});
Data explanation and example
GeoLocation types and their relations:
city -> subdivision -> country -> continent
id: 552, name: Brooklyn, type: city, parentId: 551
id: 551, name: New York, type: subdivision, parentId: 28
id: 28, name: United States, type: country, parentId: 27
id: 27, name: North America, type: continent, parentId: NULL
Now, when querying a city, I want all relations to be included as long as parentId is set.
GeoLocation.findOne({
where: {
id: 552
},
include: [
{
model: GeoLocation,
as: 'GeoLocation',
include: [
{
model: GeoLocation,
as: 'GeoLocation',
include: [
{
model: GeoLocation,
as: 'GeoLocation',
}
],
}
],
}
],
});
Response in JSON
{
"GeoLocation":{
"id":552,
"type":"city",
"name":"Brooklyn",
"GeoLocation":{
"id":551,
"type":"subdivision",
"name":"New York",
"GeoLocation":{
"id":28,
"type":"country",
"name":"United States",
"GeoLocation":{
"id":27,
"type":"continent",
"name":"North America"
}
}
}
}
}
The solution above works, but I have the strong feeling, that there are better ways to do this, without having to include the model multiple times. I can't find anything related in the docs. Am I missing something?
I already moved on a different solution. I am not using sequelize native pseudo language to achieve this. Because apparently, recursive includes feature is not possible with sequelize.
Here is the recursive model I use :
I want this recursive info, which organisation is related to which, no matter how I get this info, I'll process it later to get the info into shape (tree, flat list, ... whatever)
I use a sequelize RAW query involving recursive SQL:
exports.getArborescenceByLienId = function (relationType, id) {
const query = 'With cte as (\n' +
' select id_organisationSource, \n' +
' relationType,\n' +
' id_organisationTarget\n' +
' from (\n' +
' select * from relation where relationType= :relationType\n' +
' ) relations_sorted,\n' +
' (\n' +
' select #pv := :orgId\n' +
' ) initialisation\n' +
' where find_in_set(id_organisationSource, #pv)\n' +
' and length(#pv := concat(#pv, \',\', id_organisationTarget))\n' +
')\n' +
'select source.id as `idSource`, cte.relationType, target.id as `idTarget`\n' +
'from cte \n' +
'left outer join organisation source on cte.id_organisationSource = source.id\n' +
'left outer join organisation target on cte.id_organisationTarget = target.id;';
return models.sequelize.query(
query,
{
type: models.sequelize.QueryTypes.SELECT,
replacements: { orgId: id, relationType: relationType}
}
);
};
Imagine I have an object model like this:
(Assuming each relation here have same type : say isManaging)
Result of query with ID = 1 would be :
Result of query with ID = 2 would be :
I then transform this flat list to a Tree, and VOILA :)
Let me know if your interested in this Flat list to tree transforming algorithm.
Hope this help. Cheers !
Related
Hello, I need your help please with 2 questions.
I have 2 Models
One to Many
(One) Customer{ id, names, dni} -> Invoice {id, date, ....customer_id} (Many)
1. How can I get this?
I need to consume the api "GET /api/invoices" and that the json return of this, in turn, returns an array
[{
id: 1,
date: '2022-01-01',
....invoice
customer: {
dni: 1,
names: 'Example'
}
},
{
id: 2,
date: '2022-01-02',
....invoice
customer: {
dni: 2,
names: 'Example 2'
}
},
]
So far what I have found in the sailsjs documentation are only examples with POPULATE, where they only show how to list the User model with its corresponding created ones (hasMany)
//var users = await User.find().populate('pets');
// The users object would look something like the following
// [{
// id: 123,
// firstName: 'Foo',
// lastName: 'Bar',
// pets: [{
// id: 1,
// breed: 'labrador',
// type: 'dog',
// name: 'fido',
// user: 123
// }]
// }]
//---This is not what I need.
Is there a function or configuration that I have not found?
Or would I do something like this?
Invoices.find().exec(async(err, invoices)=>{
if(invoices){
for(i = 0; i< invoices.length; i++){
const customer = await Customer.find({id: invoices[i].customer_id});
invoices[i].customer = customer;
}
});
The point is that this takes much longer than doing a query with join
const invoices = await sails.sendNativeQuery('SELECT * from INVOICE A A inner join CUSTOMER B on A.customer_id=B.id ', []);
But I don't know how to get a JSON with the previous structure if I do it by query
2. What is the best option that can solve my problem?
The populate method works in both directions: oneToMany, manyToMany, and manyToOne:
https://sailsjs.com/documentation/reference/waterline-orm/queries/populate
If any condition is required, you could check the details on the section Populating a collection association:
var usersNamedFinn = await User.find({ name:'Finn' })
.populate('currentSwords', {
where: {
color: 'purple'
},
limit: 3,
sort: 'hipness DESC'
});
I am using KeystoneJS with PostgreSQL as my backend and Apollo on the frontend for my app.
I have a schema that has a list that is linked to another list.
I want to be able to allow users to change the order of the second list.
This is a simplified version of my schema
keystone.createList(
'forms',
{
fields: {
name: {
type: Text,
isRequired: true,
},
buttons: {
type: Relationship,
ref: 'buttons.attached_forms',
many: true,
},
},
}
);
keystone.createList(
'buttons',
{
fields: {
name: {
type: Text,
isRequired: true,
},
attached_forms: {
type: Relationship,
ref: 'forms.buttons',
many: true,
},
},
}
);
So what I would like to do, is allow users to change the order of buttons so when I fetch them in the future from forms:
const QUERY = gql`
query getForms($formId: ID!) {
allforms(where: {
id: $formId,
}) {
id
name
buttons {
id
name
}
}
}
`;
The buttons should come back from the backend in a predefined order.
{
id: 1,
name: 'Form 1',
buttons: [
{
id: 1,
name: 'Button 1',
},
{
id: 3,
name: 'Button 3',
},
{
id: 2,
name: 'Button 2',
}
]
}
Or even just have some data on that returns with the query that will allow for sorting according to the user-defined sort order on the frontend.
The catch is that this relationship is many to many.
So it wouldn't be enough to add a column to the buttons schema as the ordering needs to be relationship-specific. In other words, if a user puts a particular button last on a particular form, it shouldn't change the order of that same button on other forms.
In a backend that I was creating myself, I would add something to the joining table, like a sortOrder field or similar and then change those values to change the order, or even order them on the frontend using that information.
Something like this answer here.
The many-to-many join table would have columns like formId, buttonId, sortOrder.
I have been diving into the docs for KeystoneJS and I can't figure out a way to make this work without getting into the weeds of overriding the KnexAdapter that we are using.
I am using:
{
"#keystonejs/adapter-knex": "^11.0.7",
"#keystonejs/app-admin-ui": "^7.3.11",
"#keystonejs/app-graphql": "^6.2.1",
"#keystonejs/fields": "^20.1.2",
"#keystonejs/keystone": "^17.1.2",
"#keystonejs/server-side-graphql-client": "^1.1.2",
}
Any thoughts on how I can achieve this?
One approach would be to have two "button" lists, one with a template for a button (buttonTemplate below) with common data such as name etc, and another (button below) which references one buttonTemplate and one form. This allows you to assign a formIndex property to each button, which dictates its position on the corresponding form.
(Untested) example code:
keystone.createList(
'Form',
{
fields: {
name: {
type: Text,
isRequired: true,
},
buttons: {
type: Relationship,
ref: 'Button.form',
many: true,
},
},
}
);
keystone.createList(
'Button',
{
fields: {
buttonTemplate: {
type: Relationship,
ref: 'ButtonTemplate.buttons',
many: false,
},
form: {
type: Relationship,
ref: 'Form.buttons',
many: false,
},
formIndex: {
type: Integer,
isRequired: true,
},
},
}
);
keystone.createList(
'ButtonTemplate',
{
fields: {
name: {
type: Text,
isRequired: true,
},
buttons: {
type: Relationship,
ref: 'Button.buttonTemplate',
many: true,
},
},
}
);
I think this is less likely to cause you headaches (which I'm sure you can see coming) down the line than your buttonOrder solution, e.g. users deleting buttons that are referenced by this field.
If you do decide to go with this approach, you can guard against such issues with the hook functionality in Keystone. E.g. before a button is deleted, go through all the forms and rewrite the buttonOrder field, removing any references to the deleted button.
I had a similar challenge once, so after some research and found this answer, I implemented a solution to a project using PostgreSQL TRIGGER.
So you can add a trigger where on an update, it should shift the buttonOrder.
Here is the SQL I had on me, this was the test code, I regex replaced the terms to fit your question :)
// Assign order
await knex.raw(`
do $$
DECLARE form_id text;
begin
CREATE SEQUENCE buttons_order_seq;
CREATE VIEW buttons_view AS SELECT * FROM "buttons" ORDER BY "createdAt" ASC, "formId";
CREATE RULE buttons_rule AS ON UPDATE TO buttons_view DO INSTEAD UPDATE buttons SET order = NEW.order WHERE id = NEW.id;
FOR form_id IN SELECT id FROM form LOOP
ALTER SEQUENCE buttons_order_seq RESTART;
UPDATE buttons_view SET order = nextval('buttons_order_seq') WHERE "formId" = form_id;
END LOOP;
DROP SEQUENCE buttons_order_seq;
DROP RULE buttons_rule ON buttons_view;
DROP VIEW buttons_view;
END; $$`);
// Create function that shifts orders
await knex.raw(`
CREATE FUNCTION shift_buttons_order()
RETURNS trigger AS
$$
BEGIN
IF NEW.order < OLD.order THEN
UPDATE buttons SET order = order + 1, "shiftOrderFlag" = NOT "shiftOrderFlag"
WHERE order >= NEW.order AND order < OLD.order AND "formId" = OLD."formId";
ELSE
UPDATE buttons SET order = order - 1, "shiftOrderFlag" = NOT "shiftOrderFlag"
WHERE order <= NEW.order AND order > OLD.order AND "formId" = OLD."formId";
END IF;
RETURN NEW;
END;
$$
LANGUAGE 'plpgsql'`);
// Create trigger to shift orders on update
await knex.raw(`
CREATE TRIGGER shift_buttons_order BEFORE UPDATE OF order ON buttons FOR EACH ROW
WHEN (OLD."shiftOrderFlag" = NEW."shiftOrderFlag" AND OLD.order <> NEW.order)
EXECUTE PROCEDURE shift_buttons_order()`);
One option that we came up with is to add the order to the form table.
keystone.createList(
'forms',
{
fields: {
name: {
type: Text,
isRequired: true,
},
buttonOrder: {
type: Text,
},
buttons: {
type: Relationship,
ref: 'buttons.attached_forms',
many: true,
},
},
}
);
This new field buttonOrder could contain a string representation of the order of the button Ids, like in a JSON stringified array.
The main issue with this is that it will be difficult to keep this field in-sync with the actual linked buttons.
could not find a way to get subqueries to work with sequelize so I used raw querying. I am trying to figure out how to get data from associated tables when I do a raw query. This is what I have tried, but it only returns the data from the primary table and nothing from the associated table:
const rawQuery = `select * from (
select distinct on ("patientId") *
from public."Billings"
order by "patientId","createdAt" desc) as "recentPatientBilling"
where balance > 0;`;
const debtors = await sequelize.query(
rawQuery,
{
model: Billing,
mapToModel: true,
nest: true,
raw: true,
include: [{
model: Patient, attributes: ['id']
}]
}
);
The association is:
Billing.associate = function(models) {
Billing.belongsTo(models.User, {
foreignKey: 'employeeId',
as: 'employee'
});
Billing.belongsTo(models.Patient, {
foreignKey: 'patientId',
as: 'patient'
});
};
Any ideas on how I can get this working? (edited)
Did you try putting {as:'patient'} in the include statement?
include: [{
model: Patient,
as:'patient',
attributes: ['id']
}]
I have a case when my data in in nested arrays of jsonb in order to find the value I have to do multiple JSONB_ARRAY_ELEMENTS which is costly and takes a lots of nested code.
The json file has the continents inside countries and inside cities.
I need to access a city value.
Is there a way to make this query simpler and faster?
I was trying to solve it using JSON_EXTRACT_PATH but in order to get in to a array but I need the indexes.
WITH mydata AS (
SELECT '
{
"continents":[
{
"name":"America",
"area":43316000,
"countries":[
{
"country_name":"Canada",
"capital":"Toronto",
"cities":[
{
"city_name":"Ontario",
"population":2393933
},
{
"city_name":"Quebec",
"population":12332
}
]
},
{
"country_name":"Brazil",
"capital":"Brasilia",
"cities":[
{
"city_name":"Sao Paolo",
"population":34534534
},
{
"city_name":"Rio",
"population":445345
}
]
}
]
},
{
"name":"Europa",
"area":10530751,
"countries":[
{
"country_name":"Switzerland",
"capital":"Zurich",
"cities":[
{
"city_name":"Ginebra",
"population":4564565
},
{
"city_name":"Basilea",
"population":4564533
}
]
},
{
"country_name":"Norway",
"capital":"Oslo",
"cities":[
{
"city_name":"Oslo",
"population":3243534
},
{
"city_name":"Steinkjer",
"population":4565465
}
]
}
]
}
]
}
'::JSONB AS data_column
)
SELECT cit.city->>'city_name' AS city,
(cit.city->>'population')::INTEGER AS population
FROM (SELECT JSONB_ARRAY_ELEMENTS(coun.country->'cities') AS city
FROM (SELECT JSONB_ARRAY_ELEMENTS(cont.continent->'countries') AS country
FROM (SELECT JSONB_ARRAY_ELEMENTS(data_column->'continents') AS continent
FROM mydata
) AS cont
WHERE cont.continent #> '{"name":"Europa"}'
) AS coun
WHERE coun.country #> '{"country_name" : "Norway"}'
) AS cit
WHERE cit.city #> '{"city_name": "Oslo"}'
See my nested queries? looks ugly, I can get the answer using: JSONB_EXTRACT_PATH( data_column->'continents', '1', 'countries', '1', 'cities', '0', 'population') but I had to hardcode the array indexes.
Hope you can help me out.
Thanks.
You don't need any nesting, you can do lateral queries:
SELECT
city->>'city_name' AS city,
(city->>'population')::INTEGER AS population
FROM
mydata,
JSONB_ARRAY_ELEMENTS(data_column->'continents') AS continent,
JSONB_ARRAY_ELEMENTS(continent->'countries') AS country,
JSONB_ARRAY_ELEMENTS(country->'cities') AS city
WHERE continent ->> 'name' = 'Europa'
AND country ->> 'country_name' = 'Norway'
AND city ->> 'city_name' = 'Oslo';
(online demo)
However, since you mentioned paths and having to specify indices in there, this is actually the perfect use case for Postgres 12 JSON paths:
SELECT jsonb_path_query(data_column, '$.continents[*]?(#.name == "Europa").countries[*]?(#.country_name=="Norway").cities[*]?(#.city_name=="Oslo")') FROM mydata
(online demo)
I have:
ThinkingSphinx::Index.define :new_post, with: :real_time do
indexes title, sortable: true
indexes text
has state, type: :string
has forum_hidden, type: :boolean
has created_at, type: :timestamp
has publish_at, type: :timestamp
has reminde_at, type: :timestamp
has deleted_at, type: :timestamp
has content_category_ids, type: :integer, multi: true
end
And let's i need to get all the records where #title=query with any value publish_at or #text=query with publish_at = 1.month.ago..Time.current
That is, I need to combine these two requests:
NewPost.search(conditions: { title: query })
NewPost.search(conditions: { text: query }, with: { publish_at: 1.month.ago..Time.current })
The result is needed with excerpts
UPDATE
published_at interval for the #title and #text fields is always different and depends on the user's rights. For example, there can be such a situation:
NewPost.search(conditions: { title: query }, with: { publish_at: 1.year.ago..Time.current })
NewPost.search(conditions: { text: query }, with: { publish_at: 6.month.ago..Time.current })
and all results that do not fall under these conditions should not be displayed at all
The main thing when wanting to 'combine' multiple criteria is deciding on a 'calculation' how to compute a weight that gives that effect. Sphinx computes a weight and orders by that. Rather than unions of multiple distinct queries
For example
https://freelancing-gods.com/thinking-sphinx/searching.html#sorting
ThinkingSphinx.search(
:select => '*, WEIGHT() + IF(publish_at>NOW()-2592000,1000,0) AS custom_weight',
:order => 'custom_weight DESC'
)
would use a the 'full-text' search weight, but add 1000 if in last month.
Uses the sphinx NOW() function
http://sphinxsearch.com/docs/current.html#expr-func-now
You might also want field weights
https://freelancing-gods.com/thinking-sphinx/searching.html#fieldweights
to boost title matches over 'text' matches
:field_weights => { :title=>10 }
Bringing all togehter (if got the ruby syntax right... )
NewPost.search( query ,
:select => '*, weight() + IF(publish_at>NOW()-2592000,1000,0) as custom_weight',
:order => 'custom_weight DESC',
:field_weights => { :title=>10 }
)
... in theory title matches will be first. And recent ones towards start too. Not EXACTLY what you asked for, but close.