Query DynamoDB with Two Matching Fileds - mongodb

I am new into DynamoDB. I am trying to query a collection with two matching field. I have written a code in mongoDB, I am trying to migrate to DocumentDB. I am facing issue.
MongoDB Code
This works well
getUser= async(req,res)=>{
let user = await user.findOne({phone:123456789, otp:2345});
}
DynamoDB Code
getUser= async(req,res)=>{
const params = {
KeyConditionExpression: 'phone = :phone and #otp = :otp',
ExpressionAttributeValues: {
':phone': 919600923917,
":otp":2387
},
TableName: "users",
};
const user= await documentClient.query(params).promise();
}
Issue: Invalid KeyConditionExpression: An expression attribute name
used in the document path is not defined; attribute name: #otp

As your error is shown
Issue: Invalid KeyConditionExpression: An expression attribute name
used in the document path is not defined; attribute name: #otp
It simply means you add #otp = :otp in KeyConditionExpression, it should be not there in KeyConditionExpression. otp = :otp do something like in KeyConditionExpression.
Updated Answer:
As mentioned, the attribute included in "KeyConditionExpression" should be your hash key only, matching your base table schema (in this case 'phone' maybe). If you want to query on both 'phone' and 'otp', you need to create the table with hash and range key and specify 'otp' as your range key.
Global secondary indexes are completely separate from the base table, so in this case you can query the indexes separately (use 'otp' in key condition when querying OtpIndex).
Sample Code:
var params = {
TableName: 'users',
IndexName: "OtpIndex",
KeyConditionExpression: "phone = :phone and otp = :otp",
ExpressionAttributeValues: {
':phone': 919600923917,
':otp': 2387
},
};
Please find more details on querying global secondary indexes Doc

You can use Two matching field. to do this you need to define 2 keys
Partition Key
Sort Key
Mark one field as Partition key and another as Sort Key
Example- Partition Key - phone
Sort Key - otp.
if you have already made other attribute as key and not able to make otp/phone as key then you can create Secondary index and mark phone and otp as key into that.
you can mark different attributes (columns) as partition key or sort key other than already mentioned primary key or sort key in base table ( Main table).
also you can use --filter-expression
something like this
aws dynamodb query \
--table-name users \
--key-condition-expression "phone = :phone \
--filter-expression "#otp = :otp" \
--expression-attribute-names '{"#otp": "otp"}' \
--expression-attribute-values file://values.json

Related

DynamoDB - How to upsert nested objects with updateItem

Hi I am newbie to dynamoDB. Below is the schema of the dynamo table
{
"user_id":1, // partition key
"dob":"1991-09-12", // sort key
"movies_watched":{
"1":{
"movie_name":"twilight",
"movie_released_year":"1990",
"movie_genre":"action"
},
"2":{
"movie_name":"harry potter",
"movie_released_year":"1996",
"movie_genre":"action"
},
"3":{
"movie_name":"lalaland",
"movie_released_year":"1998",
"movie_genre":"action"
},
"4":{
"movie_name":"serendipity",
"movie_released_year":"1999",
"movie_genre":"action"
}
}
..... 6 more attributes
}
I want to insert a new item if the item(that user id with dob) did not exist, otherwise add the movies to existing movies_watched map by checking if the movie is not already available the movies_watched map .
Currently, I am trying to use update(params) method.
Below is my approach:
function getInsertQuery (item) {
const exp = {
UpdateExpression: 'set',
ExpressionAttributeNames: {},
ExpressionAttributeValues: {}
}
Object.entries(item).forEach(([key, item]) => {
if (key !== 'user_id' && key !== 'dob' && key !== 'movies_watched') {
exp.UpdateExpression += ` #${key} = :${key},`
exp.ExpressionAttributeNames[`#${key}`] = key
exp.ExpressionAttributeValues[`:${key}`] = item
}
})
let i = 0
Object.entries(item. movies_watched).forEach(([key, item]) => {
exp.UpdateExpression += ` movies_watched.#uniqueID${i} = :uniqueID${i},`
exp.ExpressionAttributeNames[`#uniqueID${i}`] = key
exp.ExpressionAttributeValues[`:uniqueID${i}`] = item
i++
})
exp.UpdateExpression = exp.UpdateExpression.slice(0, -1)
return exp
}
The above method just creates update expression with expression names and values for all top level attributes as well as nested attributes (with document path).
It works well if the item is already available by updating movies_watched map. But throws exception if the item is not available and while inserting. Below is exception:
The document path provided in the update expression is invalid for update
However, I am still not sure how to check for duplicate movies in movies_watched map
Could someone guide me in right direction, any help is highly appreciated!
Thanks in advance
There is no way to do this, given your model, without reading an item from DDB before an update (at that point the process is trivial). If you don't want to impose this additional read capacity on your table for update, then you would need to re-design your data model:
You can change movies_watched to be a Set and hold references to movies. Caveat is that Set can contain only Numbers or Strings, thus you would have movie id or name or keep the data but as JSON Strings in your Set and then parse it back into JSON on read. With SET you can perform ADD operation on the movies_watched attribute. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.UpdateExpressions.html#Expressions.UpdateExpressions.ADD
You can go with single table design approach and have these movies watched as separate items with (PK:userId and SK:movie_id). To get a user you would perform a query and specify only PK=userId -> you will get a collection where one item is your user record and others are movies_watched. If you are new to DynamoDB and are learning the ropes, then I would suggest go with this approach. https://www.alexdebrie.com/posts/dynamodb-single-table/

Nested Hasura GraphQL Upsert mutation is there a way to stop nesting on conflict?

I use Hasura and I have a social-network like situation.
In which I have a "User" object and a "Feed" object.
Every user has a feed.
I have a relationship from user.id to feed.id.
The relevant mutation is UpsertUserDetails as follows:
mutation UserDetailsUpsert(
$email: String!
$picture: String
) {
insert_users_one(
object: {
email: $email
feed: { data: {} }
picture: $picture
}
on_conflict: { constraint: users_tid_email_key, update_columns: [picture] }
) {
id
}
}
So when I create a new user it also creates a feed for it.
But when I only update user details I don't want it to create a new feed.
I would like to stop the upsert from going through to relationships instead of the above default behavior.
and according to this manual I don't see if its even possible: https://hasura.io/docs/latest/graphql/core/databases/postgres/mutations/upsert.html#upsert-in-nested-mutations
To allow upserting in nested cases, set update_columns: []. By doing this, in case of a conflict, the conflicted column/s will be updated with the new value (which is the same values as they had before and hence will effectively leave them unchanged) and will allow the upsert to go through.
Thanks!
I'd recommend that you design your schema such that bad data cannot be entered in the first place. You can put partial unique indices on the feed table in order to prevent duplicate feeds from ever being created. Since you have both users and groups you can implement it with 2 partial indices.
CREATE UNIQUE INDEX unique_feed_per_user ON feed (user_id)
WHERE user_id IS NOT NULL;
CREATE UNIQUE INDEX unique_feed_per_group ON feed (group_id)
WHERE group_id IS NOT NULL;

MongoDB Complex Query with Java

We have following structure in MongoDB documents.
{
"id":"1111",
"keys":[
{
"name":"Country",
"value":"USA"
},
{
"name":"City",
"value":"LongIsland"
},
{
"name":"State",
"value":"NewYork"
}
]
}
Now using Springframework Query object, I figured out a way to pull the details using below syntax
query.addCriteria(
Criteria.where("keys.value").is(countryparam).
andOperator(
Criteria.where("keys.value").is(stateparam)
)
);
Two issue with this query model.
First issue is it is irrelevant if countryparam and stateparam are actually meant to match Country key name and State key name respectively. If just the values matches, the query returns the document. Means, if I have Country and City params, this just works if user passes Country and City values, even if they are swapped. So how can I exactly compare City to cityparam and State to Stateparam?
More complexity is if I have to extract the document basing on multiple key value pairs, I should be correspondingly able to match key name with respective value and query the document. How can I do this?
Thanks in advance!

how to convert map<anydata> to json

In my CRUD Rest Service I do an insert into a DB and want to respond to the caller with the created new record. I am looking for a nice way to convert the map to json.
I am running on ballerina 0.991.0 and using a postgreSQL.
The return of the Update ("INSERT ...") is a map.
I tried with convert and stamp but i did not work for me.
import ballerinax/jdbc;
...
jdbc:Client certificateDB = new({
url: "jdbc:postgresql://localhost:5432/certificatedb",
username: "USER",
password: "PASS",
poolOptions: { maximumPoolSize: 5 },
dbOptions: { useSSL: false }
}); ...
var ret = certificateDB->update("INSERT INTO certificates(certificate, typ, scope_) VALUES (?, ?, ?)", certificate, typ, scope_);
// here is the data, it is map<anydata>
ret.generatedKeys
map should know which data type it is, right?
then it should be easy to convert it to json like this:
{"certificate":"{certificate:
"-----BEGIN
CERTIFICATE-----\nMIIFJjCCA...tox36A7HFmlYDQ1ozh+tLI=\n-----END
CERTIFICATE-----", typ: "mqttCertificate", scope_: "QARC", id_:
223}"}
Right now i do a foreach an build the json manually. Quite ugly. Maybe somebody has some tips to do this in a nice way.
It cannot be excluded that it is due to my lack of programming skills :-)
The return value of JDBC update remote function is sql:UpdateResult|error.
The sql:UpdateResult is a record with two fields. (Refer https://ballerina.io/learn/api-docs/ballerina/sql.html#UpdateResult)
UpdatedRowCount of type int- The number of rows which got affected/updated due to the given statement execution
generatedKeys of type map - This contains a map of auto generated column values due to the update operation (only if the corresponding table has auto generated columns). The data is given as key value pairs of column name and column value. So this map contains only the auto generated column values.
But your requirement is to get the entire row which is inserted by the given update function. It can’t be returned with the update operation if self. To get that you have to execute the jdbc select operation with the matching criteria. The select operation will return a table or an error. That table can be converted to a json easily using convert() function.
For example: Lets say the certificates table has a auto generated primary key column name ‘cert_id’. Then you can retrieve that id value using below code.
int generatedID = <int>updateRet.generatedKeys.CERT_ID;
Then use that generated id to query the data.
var ret = certificateDB->select(“SELECT certificate, typ, scope_ FROM certificates where id = ?”, (), generatedID);
json convertedJson = {};
if (ret is table<record {}>) {
var jsonConversionResult = json.convert(ret);
if (jsonConversionResult is json) {
convertedJson = jsonConversionResult;
}
}
Refer the example https://ballerina.io/learn/by-example/jdbc-client-crud-operations.html for more details.?

Multi level MongoDB object querying by key

If you only know the key name (say "nickname"), but not the exact path to that key in the object.
e.g. nickname may be at the first level like:
{"nickname":"Howie"}
or at the second level:
{"user":{"nickname":"Howie"}}
Is it possible to query for nickname equal "Howie" that would return both documents?
Unfortunately there is no wild card that allows you to search for a field at any level in the db. If the position is not relevant and you can modify the document structure you have 2 choices here. You can store your document as
{ fieldname:"nickname", value : "Howie" }
{ fieldname:"user/nickname", value: "Howie" }
You can then query using
db.so.find({fieldname:/nickname/, value:"Howie"})
Alternatively you can store as
db.so.insert({value:"Howie", fieldpath:["nickname"]})
db.so.insert({value:"Howie", fieldpath:["user", "nickname"]})
The advantage with the second approach is that you can now index {fieldpath:1, value:1} and a query on it such as
db.so.find({fieldpath:"nickname", value:"Howie"})
will be an indexed query.