DynamoDB PartiQL Request Parameter In Select - select

I am attempting to use PartiQL to query a set of data that looks like this:
{
"userId": {
"S": "someuserID"
},
"mapWithData": {
"M": {
"1": {
"M": {
"neededVal": {
"S": "A Needed Value1"
},
"name": {
"S": "A Name1"
}
}
},
"2": {
"M": {
"neededVal": {
"S": ""A Needed Value12"
},
"name": {
"S": "A Name2"
}
}
}
}
},
"userName": {
"S": "someuserName"
}
}
I am developing this query using the NoSQL Workbench. I want to use a Request Parameter in the query to get a specific object from the mapWithData based on its key value. For instance, if I wanted to get the n-th value from the map, I could use this query:
SELECT "mapWithData"."N"."neededVal"
FROM "some-table"
WHERE "userId" = 'someuserID'
But I would like to be able to make the "N" and the 'someruserID' into Request Parameters to prevent any PartiQL injections (assuming the Request Parameters are actually cleansed). So what I'm trying to do is this:
SELECT "mapWithData".?."neededVal"
FROM "some-table"
WHERE "userId" = ?
This does not work unfortunately and I get this error:
Execute PartiQL statement failed: Validation Error: Statement wasn't well formed, can't be processed: Invalid path dot component
So then I tried to use a different format like this:
SELECT "mapWithData"[ ? ]["neededVal"]
FROM "some-table"
WHERE "userId" = ?
But when I do that, I get this error:
Execute PartiQL statement failed: Validation Error: Unexpected path component at 1:23:1
Is it possible to include a ? for inserting request parameters in the SELECT part of this statement? Is there another way to do this that I'm missing?

#Francis, sure you can do that: if you use the PythonSDK to implement it you can write:
table_name = 'mytable'
someruserID =
stmt = f"SELECT * FROM {table_name} WHERE userId=?"
pmt =[{
"S": userId
}]
resp = dynamodb.execute_statement(
Statement=stmt , Parameters= pmt
)
Hope that helps. A

Related

DynamoDB PartiQL Query For Specific Map Element From List

I have a DynamoDB with data that looks like this:
{
"userId": {
"S": "someuserID"
},
"listOfMaps": {
"L": [
{
"M": {
"neededVal": {
"S": "ThisIsNeeded1"
},
"id": {
"S": "1"
}
}
},
{
"M": {
"neededVal": {
"S": "ThisIsNeeded2"
},
"id": {
"S": "2"
}
}
},
...
]
},
"userName": {
"S": "someuserName"
}
}
The listOfMaps can contain more than just 2 maps, as is denoted by the ellipsis. Is there a PartiQL query I can put together that will let me get the neededVal based on the userId and the id of the item in the map itself?
I know that I can query for the n-th item's neededVal like this:
SELECT "listOfMaps"[N]."neededVal"
FROM "table-name"
WHERE "userId" = 'someuserID'
But is it possible to make it do something like this:
SELECT "listOfMaps"."neededVal"
FROM "table-name"
WHERE "userId" = 'someuserID' AND "listOfMaps"."id" = '4'
It looks like you're modeling the one-to-many relationship using a complex attribute (e.g. a list of objects). This is a completely valid approach to modeling one-to-many relationships and is best used when 1) the results data doesn't change (or don't change often) and 2) you don't have any access patterns around the data within the complex attribute.
However, since you do want to perform searches based on data within the complex attribute, you'd be better off modeling the data differently.
For example, you might consider modeling results in the user partition with a PK=user_id SK=neededVal#. This would allow you to fetch items by User ID (QUERY where PK=USER#user_id SK begins_with neededVal#).
I don't know the specifics around your access patterns, but can say that you'll need to move results into their own items if you want to support access patterns around the data within your complex attribute.

mongodb $ causing error The positional operator did not find the match needed from the query

I’ve been trying to update the data in my mongoDB.
I want to update all products with a new productName field.
my data looks something like:
{
"id": "12345",
"products": [{
"id": 0
"productCode": "test",
"status": "PENDING",
},
{
"id": 1
"productCode": "test",
"status": "COMPLETE",
}],
}
When I try the following. I get this error The positional operator did not find the match needed from the query.
db.customers.updateMany(
{ id: "12345" },
{ $set: {
"products.$.productName": "Name here" }
}
)
If I do account.0.productName then it’s fine and updates. I’m not sure why $ is not working for me
db.customers.updateMany(
{ id: "12345" },
{ $set: {
"products.0.productName": "Name here" }
}
)
Positional operator is not working because you are not using the array into the find (first object)
If you try this query it will work as expected because you have the position finding by products.id.
Otherwise, if you don't have the position into array where update, yo can't use $ operator in this way. You need this query:
db.collection.update({
"id": "12345",
},
{
"$set": {
"products.$[].newField": "test2"
}
},
{
"multi": true
})
Mongo playground example here
Using $[] you can reference the array and add the value into each object.
$[] docs here
It says:
The all positional operator $[] indicates that the update operator should modify all elements in the specified array field.
That's exactly we want :)

Elasticsearch java high level client group by and max

I am using Scala 2.12 and Elasticsearch 6.5. Using the high level java client to query the ES.
Required Data is as E.g. Simple example of Documents has 2 sets of data (published 2 times) with different id and timestamp.
id: id_123 and id_234 (Theese are 2 different ids of required documents) and timestamp(representation only) 10 AM (for id_123) and 11 AM (for id_234).
So I just need those documents which are latest among these i.e. 11 AM one.
I have some filter conditions and then need to group on field1 and take the max of field2 (which is timestamp).
val searchRequest = new SearchRequest("index_name")
val searchSourceBuilder = new SearchSourceBuilder()
val qb = QueryBuilders.boolQuery()
.must(QueryBuilders.matchQuery("myfield.date", "2019-07-02"))
.must(QueryBuilders.matchQuery("myfield.data", "1111"))
.must(QueryBuilders.boolQuery()
.should(QueryBuilders.regexpQuery("myOtherFieldId", "myregex1"))
.should(QueryBuilders.regexpQuery("myOtherFieldId", "myregex2"))
)
val myAgg = AggregationBuilders.terms("group_by_Id").field("field1.Id").subAggregation(AggregationBuilders.max("timestamp").field("field1.timeStamp"))
searchSourceBuilder.query(qb)
searchSourceBuilder.aggregation(myAgg)
searchSourceBuilder.size(1000)
searchRequest.source(searchSourceBuilder)
val searchResponse = client.search(searchRequest, RequestOptions.DEFAULT)
Basically, all works good if I do not use the aggregation.
When I use the aggregation, I am getting the following error:
ElasticsearchException[Elasticsearch exception [type=illegal_argument_exception, reason=Expected numeric type on field [field1.timeStamp], but got [keyword]]]
So what am I missing here?
I am basically looking for SQL-like query, which has fileter (where, AND/OR clause) and then group by a field (Id) and take documents only where timeStamp is max.
UPDATE:
I tried the above query in cURL via command prompt and get the same error when using "max" on aggregaation.
{
"query": {
"bool": {
"must": [
{
"match": { "myfield.date" : "2019-07-02" }
},
{
"match": { "myfield.data" : "1111" }
},
{
"bool": {
"should": [
{
"regexp": { "myOtherFieldId": "myregex1" }
},
{
"regexp": { "myOtherFieldId": "myregex2" }
}
]
}
}
]
}
},
"aggs": {
"NAME" : {
"terms": {
"field": "field1.Id"
},
"aggs": {
"NAME": {
"max" : {
"field": "field1.timeStamp"
}
}
}
}
},
"size": "10000"
}
I am getting the same error.
I tried to check the mappings of the index.
It is showing as keyword. So how to do max on such fields?
Adding the relevant mappings:
{"index_name":{"mappings":{"data":{"dynamic_templates":[{"boolean_as_keyword":{"match":"*","match_mapping_type":"boolean","mapping":{"ignore_above":256,"type":"keyword"}}},{"double_as_keyword":{"match":"*","match_mapping_type":"double","mapping":{"ignore_above":256,"type":"keyword"}}},{"long_as_keyword":{"match":"*","match_mapping_type":"long","mapping":{"ignore_above":256,"type":"keyword"}}},{"string_as_keyword":{"match":"*","match_mapping_type":"string","mapping":{"ignore_above":256,"type":"keyword"}}}],"date_detection":false,"properties":{"header":{"properties":{"Id":{"type":"keyword","ignore_above":256},"otherId":{"type":"keyword","ignore_above":256},"someKey":{"type":"keyword","ignore_above":256},"dataType":{"type":"keyword","ignore_above":256},"processing":{"type":"keyword","ignore_above":256},"otherKey":{"type":"keyword","ignore_above":256},"sender":{"type":"keyword","ignore_above":256},"receiver":{"type":"keyword","ignore_above":256},"system":{"type":"keyword","ignore_above":256},"timeStamp":{"type":"keyword","ignore_above":256}}}}}}}}
UPDATE2:
I think I need to aggregate (timeStamp) on keyword.
Please note that timeStamp is a subfield i.e. under field1. So below syntax for keyword doesn't seem to work or I am missing something else.
"aggs": {
"NAME" : {
"terms": {
"field": "field1.Id"
},
"aggs": {
"NAME": {
"max" : {
"field": "field1.timeStamp.keyword"
}
}
}
}
}
It fails now saying:
"Invalid aggregator order path [field1.timeStamp]. Unknown aggregation [field1]"

How to do PostgreSQL '||' operator with knex.update

I am working with the following data in table my_table:
[
{
"item": {
"id": 1,
"data": {
"name": "ABC",
"status": "Active"
}
}
},
{
"item": {
"id": 2,
"data": {
"name": "DEF",
"status": "Active"
}
}
}
]
I would like to update name property of data, keeping the rest of data intact. A PostgreSQL query for that purpose would look like this:
UPDATE my_table SET data = data || '{"name":"GHI"}' WHERE id = 1;
However, I am struggling to achieve this with knex, as I've tried:
knex('my_table')
.update({ data: knex.raw('data || ?', [{ name: 'GHI' }]) })
.where('id', 1);
and many other similar queries, but in vain. If you might have any ideas about this, please share them below. Thanks in advance!
Using knex you indeed have to use raw expressions to update single field inside JSONB column.
However in objection.js which is an ORM, built on top of knex, there are some additional support for JSONB operations.
With objection.js your update would look like
await MyTableModel.query(knex).update({'data:name', 'GHI'}).where('id', 1);
Which outputs SQL like this (with bindings ["GHI", 1]):
update "my_table" set "data" = jsonb_set("data", '{name}', ?, true) where "id" = ?
Runkit example https://runkit.com/embed/0dai0bybplxv

Using MERGE with properties via REST

According to the sample code at http://docs.neo4j.org/chunked/2.0.0-M03/rest-api-transactional.html I'm trying to use the MERGE statement.
But when I apply the following statement:
{
"statements": [
{
"statement": "MERGE (p:PERSON { identification }) ON CREATE p SET { properties } ON MATCH p SET { properties } RETURN p",
"parameters": {
"identification": {
"guid": "abc123xyz"
},
"properties": {
"lastName": "Doe",
"firstName": "John"
}
}
}
]
}
it gets back with the following 2 errors:
{ identification }
code: 42000,
status: STATEMENT_EXECUTION_ERROR,
message: Tried to set a property to a collection of mixed types. List(Map(guid -> abc123xyz))
SET { properties }
code: 42001,
status: STATEMENT_SYNTAX_ERROR",
message: =' expected butO' found\n\nThink we should have …
Can this not be done this way (yet) or am I missing something?
Thanks for your help
Daniel
It seems you've discovered a bug. I've reported the issue here:
https://github.com/neo4j/neo4j/issues/975
The issue is that MERGE needs to know the keys you will search on in advance. Passing a map of parameters hides this.
To achieve the same, list each key explicitly. If you still want to pass them all in a single map, you can probably do something like: MERGE (p:Person {name: {merge_map}.name, email: {merge_map}.email}).
Daniel,
I think you have to use SET differently, something like this:
MERGE (p:PERSON { identification })
ON CREATE p SET p={ properties }
ON MATCH p SET p={ properties }
RETURN p
But I'm not sure if that SET overrides all your properties. So it might be that you have to specify them one by one.
{
"statements": [
{
"statement": "MERGE (p:PERSON { guid : {guid} })
ON CREATE p SET p.lastName={lastName},p.firstName={ firstName }
ON MATCH p SET p.lastName={lastName},p.firstName={ firstName }
RETURN p",
"parameters": {
"guid": "abc123xyz",
"lastName": "Doe",
"firstName": "John"
}
}
]
}