Environment: DataJoint version (0.12.9) on Ubuntu with MySQL 5.7
The concept of FreeTable is nice. Provide a dj.conn() object and full_table_name and a table you can manipulate is created, independent of modules, virtual modules and schemas.
However, the behavior of this table is not like other user tables. For example, if you restrict with two dictionaries that should have no overlapping results, a normal table would return an empty set. However, a FreeTable return the results from just the second dictionary.
For example:
FreeTable & {'exists': 1} & {'exists': 0}
should return an empty table because and entry can't be both 1 and 0 simultaneously.
However, in this case the table that is returned is equivalent to:
FreeTable & {'exists': 0}
(Also querying with {'exists': 1} in the second position returns FreeTable & {'exists': 1})
The following works as expected and returns a table with no results:
FreeTable & dj.AndList([{'exists': 1}, {'exists': 0}])
I believe the reason this happens is because FreeTable inherits from Table and not UserTable, where the metaclass properties take effect.
My question is, what is the purpose of FreeTable and why couldn't it have inherited its properties from UserTable so it could be more useful?
The purpose of FreeTable is exactly as you described: to provide access to a table in the database without a corresponding class.
Consider the following code
q = dj.FreeTable(dj.conn(), '`ephys`.`session`')
For example, if the table ephys.session has the field exists, then you can restrict the query as
q & {'exists': 1}
This will restrict the query q to the rows where the field exists = 1. However, if the table contains no such field, then no restriction is applied.
What you probably saw in your code is that the table of interest has no field with the name exists. When DataJoint restricts a query by a dict, then the non-existing fields are ignored.
Related
Is it possible to do a greater than search across a jsonb field using hasura?
it looks to be possible in PostgreSQL itself, How can I do less than, greater than in JSON Postgres fields?
in postgres I'm storing a table
asset
name: string
version: int
metadata: jsonb
the metadata looks like this.
{'length': 5}
I am able to find asset that matches exactly using the _contains.
{
asset(where:{metadata : {_contains : {length: 5}}}){
name
metadata
}
}
I would like to be able to find asset with a length over 10.
I tried:
{
asset(where:{metadata : {_gt : {length: 10}}}){
name
metadata
}
}
A. Possibility to do on graphql level directly
Hasura documentation: JSONB operators (_contains, _has_key, etc.) mentions only 4 operators:
The _contains, _contained_in, _has_key, _has_keys_any and _has_keys_all operators are used to filter based on JSONB columns.
So direct answer for your question: No. It's not possible to do on graphql level in hasura.
(At least it's not possible yet. Who knows: maybe in future releases more operators will be implemented.
)
B. Using derived views
But there is another way, the one explained in https://hasura.io/blog/postgres-json-and-jsonb-type-support-on-graphql-41f586e47536/#derived-data
This recomendation is repeated in: https://github.com/hasura/graphql-engine/issues/6331
We don't have operators like that for JSONB (might be solved by something like #5211) but you can use a view or computed field to flatten the text field from the JSONB column into a new column and then do a like on that.
Recipe is:
1. Create a view
CREATE VIEW assets -- note plural here. Name view accordingly to your style guide
AS
SELECT
name,
version,
metadata,
(metadata->>'length')::int as meta_len -- cast to other number type if needed
FROM asset
2. Register this view
3. Use it in graphql queries as usual table
E.g.
query{
assets(where: {meta_len: {_gt:10}}){
name
metadata
}
C. Using SETOF-functions
1. Create SETOF-function
CREATE FUNCTION get_assets(min_length int DEFAULT 0)
RETURNS SETOF asset
LANGUAGE SQL
STABLE
AS $$
SELECT * FROM asset
WHERE
(metadata->>'length')::int > min_length;
$$;
2. Register in hasura
3. Use in queries
query{
get_assets(args: {min_length: 10}){
name
metadata
}
I think that was the last possible option.
It will not gives you full "schemaless freedom" that maybe you're looking but IDK know about other ways.
Using FireDAC's Array DML feature, it doesn't seem possible to utilise a RETURNING clause (in my case PostgeSQL).
If I run a simple insert query such as:
With FDQuery Do
begin
SQL.Text := 'INSERT INTO temptab(email, name) '
+'VALUES (''email1'', ''name1''), '
+'(''email2'', ''name2'') '
+'RETURNING id';
Open;
end;
The query returns two records containing the id for the newly inserted records.
For larger inserts I would prefer to use Array DML, but in some cases I also need to be able to get returned data.
The Open function does not have an ATimes parameter. Whilst you can call Open with Array DML, it results in the insertion and return of just the first record.
I cannot find any other properties, methods which would seem to facilitate this. I have posted on Praxis to see if anyone there has any ideas, but I have had no response. I have also posted this as a new feature request on Quality Central.
If anyone knows of a way of achieving this using Array DML, I would be grateful to hear, but my principal question is what is the most efficient route for retrieving the inserted data (principally IDs) from the DB if I persist with Array DML?
A couple of ideas occur to me, neither of which seem tremendously attractive:
Within StartTransaction and Commit and following the insertion retrieve the id of the last inserted record and then grab backwards the requisite number. This seems to be to be a bit risky, although as within a transaction, should probably be okay.
Add an integer field to the relevant table and populate each inserted record with a unique identifier and following insert retrieve the records with that identifier. Whilst this would ensure the return of the inserted records, it would be relatively inefficient unless I index the field being used to store the identifier.
Both the above would be dependent on records being inserted into the DB in the order they are supplied to the Array DML, but I assume/hope that is a given.
I would appreciate views on the best (ie most efficient and reliable) of the above options and any suggestions as to alternative even better options even if those entail abandoning Array DML where a Returning clause is needed.
You actually can get all returned ID's. You can tell Firedac to store the result values in paramters with {INTO }. See for example the following code:
FDQuery.SQL.Text := 'INSERT into tablename (fieldname) values (:p1) returning id {into :p2}';
FDQuery.Params.ArraySize := 2;
FDQuery.Params[0].AsStrings[0] := 'one';
FDQuery.Params[0].AsStrings[1] := 'two';
FDQuery.Params[1].ParamType := ptInputOutput;
FDQuery.Params[1].DataType := ftLargeInt;
FDQuery.Execute(2,0);
ID1 := FDQuery.Params[1].AsLargeInts[0];
ID2 := FDQuery.Params[1].AsLargeInts[1];
This works when 1 row is returned per arraydml element. I think it will not work for >1 row, but I've not tested it. If it does, you would have to know which result corresponds with your arraydml element.
Note that Firedac throws an AV when 0 rows are returned for one or more elements in the arraydml. For example when you UPDATE a row that was deleted in the meantime. The AV has nothing to do with the array DML itself. When FDQuery.Execute; is called, you'll get an AV as well.
I've suggested another option earlier on the delphipraxis forum, but that is a suboptimal solution as that uses a temp table to store the ID's:
https://en.delphipraxis.net/topic/4693-firedac-array-dml-returning-values-from-inserted-records/
I'm converting a one-to-one relationship into a one-to-many relationship. The old relationship was just a foreign key on the parent record. The new relationship will be an array of foreign keys on the parent record.
(Using Postgres dialect, BTW.)
First I'll add a new JSONB column, which will hold an array of UUIDs.
Then I'll run a query to update all existing rows such that the value from the old column is now stored in the new column (as the first element in an array).
Finally, I'll remove the old column.
I'm looking for help with step 2: writing the update statement that will update all rows, setting the value of the new column based on the value of the old column. Basically, I'm trying to figure out how to express this SQL query using Sequelize:
UPDATE "myTable"
SET "newColumn" = json_build_array("oldColumn")
-- ^^ this really works, btw
Where:
newColumn is type JSONB, and should hold an array (of UUIDs)
oldColumn is type UUID
names are double-quoted because they're mixed case in the DB (shrug)
Expressed using Sequelize sugar, that might be something like:
const { models } = require('../sequelize')
await models.MyModel.update({ newColumn: [ 'oldColumn' ] })
...except that would result in saving an array that contains the string "oldColumn" rather than an array whose first element is the value in that row's oldColumn column.
My experience, and the Sequelize documentation, is focused on working with individual rows via the standard instance methods. I could do that here, but it'd be a lot better to have the database engine do the work internally instead of forcing it to transfer every row to Node and then back again.
Looking for whatever is the most Sequelize-idiomatic way of doing this, if there is one.
Any help is appreciated.
I am trying to set an index to a tstzrange[] column in PostgreSQL 10. I created the column via the pgAdmin 4 GUI, set its name and data type as tstzrange[] and set it as not null, nothing more.
I then did a CREATE EXTENSION btree_gist; for the database and it worked.
Then I saw in the documentation that I should index the range and I do:
CREATE INDEX era_ac_range_idx ON era_ac USING GIST (era_ac_range);
...but then I get:
ERROR: data type tstzrange[] has no default operator class for
access method "gist"
which, frankly, I don't know what it actually means, or how to solve it. What should I do ?
PS, that column is currently empty, has no data yet.
Ps2, This table describes chronological eras, there is an id, the era name (eg the sixties) and the timezone range (eg 1960-1969).
A date is inserted by the user and I want to check in which era it belongs.
Well, you have an array of timestamp-ranges as a single column. You can index an array with a GIN index and a range with (iirc) GIN or GiST. However, I'm not sure how an index on a column that is both would operate. I guess you could model it as an N-dimensional r-tree or some such.
I'm assuming you want to check for overlapping ranges.Could you normalise the data and have a linked table with one range in each row?
Env: Rails 4.2.4, Postgres 9.4.1.0
Is there a guarantee that ActiveRecord#first method will always return a record with minimal ID and ActiveRecord#last - with maximum ID?
I can see from Rails console that for these 2 methods appropriate ORDER ASC/DESC is added to generated SQL. But an author of another SO thread Rails with Postgres data is returned out of order tells that first method returned NOT first record...
ActiveRecord first:
2.2.3 :001 > Account.first
Account Load (1.3ms) SELECT "accounts".* FROM "accounts" ORDER BY "accounts"."id" ASC LIMIT 1
ActiveRecord last:
2.2.3 :002 > Account.last
Account Load (0.8ms) SELECT "accounts".* FROM "accounts" ORDER BY "accounts"."id" DESC LIMIT 1
==========
ADDED LATER:
So, I did my own investigation (based on D-side answer) and the Answer is NO. Generally speaking the only guarantee is that first method will return first record from a collection. It may as a side effect add ORDER BY PRIMARY_KEY condition to SQL, but it depends on either records were already loaded into cache/memory or not.
Here's methods extraction from Rails 4.2.4:
/activerecord/lib/active_record/relation/finder_methods.rb
# Find the first record (or first N records if a parameter is supplied).
# If no order is defined it will order by primary key.
# ---> NO, IT IS NOT. <--- This comment is WRONG.
def first(limit = nil)
if limit
find_nth_with_limit(offset_index, limit)
else
find_nth(0, offset_index) # <---- When we get there - `find_nth_with_limit` method will be triggered (and will add `ORDER BY`) only when its `loaded?` is false
end
end
def find_nth(index, offset)
if loaded?
#records[index] # <--- Here's the `problem` where record is just returned by index, no `ORDER BY` is applied to SQL
else
offset += index
#offsets[offset] ||= find_nth_with_limit(offset, 1).first
end
end
Here's a few examples to be clear:
Account.first # True, records are ordered by ID
a = Account.where('free_days > 1') # False, No ordering
a.first # False, no ordering, record simply returned by #records[index]
Account.where('free_days > 1').first # True, Ordered by ID
a = Account.all # False, No ordering
a.first # False, no ordering, record simply returned by #records[index]
Account.all.first # True, Ordered by ID
Now examples with has-many relationship:
Account has_many AccountStatuses, AccountStatus belongs_to Account
a = Account.first
a.account_statuses # No ordering
a.account_statuses.first
# Here is a tricky part: sometimes it returns #record[index] entry, sometimes it may add ORDER BY ID (if records were not loaded before)
Here is my conclusion:
Treat method first as returning a first record from already loaded collection (which may be loaded in any order, i.e. unordered). And if I want to be sure that first method will return record with minimal ID - then a collection upon which I apply first method should be appropriately ordered before.
And Rails documentation about first method is just wrong and need to be rewritten.
http://guides.rubyonrails.org/active_record_querying.html
1.1.3 first
The first method finds the first record ordered by the primary key. <--- No, it is not!
If sorting is not chosen, the rows will be returned in an unspecified
order. The actual order in that case will depend on the scan and join
plan types and the order on disk, but it must not be relied on. A
particular output ordering can only be guaranteed if the sort step is
explicitly chosen.
http://www.postgresql.org/docs/9.4/static/queries-order.html (emphasis mine)
So ActiveRecord actually adds ordering by primary key, whichever that is, to keep the result deterministic. Relevant source code is easy to find using pry, but here are extracts from Rails 4.2.4:
# show-source Thing.all.first
def first(limit = nil)
if limit
find_nth_with_limit(offset_index, limit)
else
find_nth(0, offset_index)
end
end
# show-source Thing.all.find_nth
def find_nth(index, offset)
if loaded?
#records[index]
else
offset += index
#offsets[offset] ||= find_nth_with_limit(offset, 1).first
end
end
# show-source Thing.all.find_nth_with_limit
def find_nth_with_limit(offset, limit)
relation = if order_values.empty? && primary_key
order(arel_table[primary_key].asc) # <-- ATTENTION
else
self
end
relation = relation.offset(offset) unless offset.zero?
relation.limit(limit).to_a
end
it may change depending of your Database engine, it returns always the minimal ID in mysql with first method but it does not works the same for postgresql, I had several issues with this when I was a nobai, my app was working as expected in local with mysql, but everything was messed up when deployed to heroku with postgresql, so for avoid issues with postgresql always order your records by id before the query:
Account.order(:id).first
The above ensures minimal ID for mysql, postgresql and any other database engine as you can see in the query:
SELECT `accounts`.* FROM `accounts` ORDER BY `accounts`.`id` ASC LIMIT 1
I don't think that answer you reference is relevant (even to the question it is on), as it refers to non-ordered querying, whereas first and last do apply an order based on id.
In some cases, where you are applying your own group on the query, you cannot use first or last because an order by cannot be applied if the grouping does not include id, but you can use take instead to just get the first row returned.
There have been versions where first and/or last did not apply the order (one of the late Rails 3 on PostgreSQL as I recall), but they were errors.