I have not seen examples of DynamoDB search using a partition key and an index (python, boto way). Would it be possible to direct to any links/examples?
#import Key, Attr
#dynamodb=boto3.resource(....)
#table = dynamodb.resource('tablename')...
#usual stuff commented above
response = table.query(
KeyConditionExpression=Key('hash_key').eq(hash_value),
FilterExpression = Attr('column-name-that-has-an-index-NOT-THE-INDEX-NAME').eq(indexvalue)
)
Related
I'm trying to query dynamoDB through withFilterExpression. I get an error as the argument is a composite key
Filter Expression can only contain non-primary key attributes: Primary key attribute: question_id
and also as it uses OR operator in the query and it cannot be passed to withKeyConditionExpression.
The query that was passed to withFilterExpression is similar to this question_id = 1 OR question_id = 2. The entire code is like follows
def getQuestionItems(conceptCode : String) = {
val qIds = List("1","2","3")
val hash_map : java.util.Map[String, Object] = new java.util.HashMap[String, Object]()
var queries = ArrayBuffer[String]()
hash_map.put(":c_id", conceptCode)
for ((qId, index) <- qIds.zipWithIndex) {
val placeholder = ":qId" + index
hash_map.put(placeholder, qId)
queries += "question_id = " + placeholder
}
val query = queries.mkString(" or ")
val querySpec = new QuerySpec()
.withKeyConditionExpression("concept_id = :c_id")
.withFilterExpression(query)
.withValueMap(hash_map)
questionsTable.query(querySpec)
}
Apart from withFilterExpression and withConditionExpression is there any other methods that I can use which is a part of QuerySpec ?
Let's raise things up a level. With a Query (as opposed to a GetItem or Scan) you provide a single PK value and optionally an SK condition. That's what a Query requires. You can't provide multiple PK values. If you want multiple PK values, you can do multiple Query calls. Or possibly you may consider a Scan across all PK values.
You can also consider having a GSI that presents the data in a format more suitable to efficient lookup.
Side note: With PartiQL you can actually specify multiple PK values, up to a limit. So if you really truly want this, that's a possibility. The downside is it raises things up to a new level of abstraction and can make inefficiencies hard to spot.
I am new into DynamoDB. I am trying to query a collection with two matching field. I have written a code in mongoDB, I am trying to migrate to DocumentDB. I am facing issue.
MongoDB Code
This works well
getUser= async(req,res)=>{
let user = await user.findOne({phone:123456789, otp:2345});
}
DynamoDB Code
getUser= async(req,res)=>{
const params = {
KeyConditionExpression: 'phone = :phone and #otp = :otp',
ExpressionAttributeValues: {
':phone': 919600923917,
":otp":2387
},
TableName: "users",
};
const user= await documentClient.query(params).promise();
}
Issue: Invalid KeyConditionExpression: An expression attribute name
used in the document path is not defined; attribute name: #otp
As your error is shown
Issue: Invalid KeyConditionExpression: An expression attribute name
used in the document path is not defined; attribute name: #otp
It simply means you add #otp = :otp in KeyConditionExpression, it should be not there in KeyConditionExpression. otp = :otp do something like in KeyConditionExpression.
Updated Answer:
As mentioned, the attribute included in "KeyConditionExpression" should be your hash key only, matching your base table schema (in this case 'phone' maybe). If you want to query on both 'phone' and 'otp', you need to create the table with hash and range key and specify 'otp' as your range key.
Global secondary indexes are completely separate from the base table, so in this case you can query the indexes separately (use 'otp' in key condition when querying OtpIndex).
Sample Code:
var params = {
TableName: 'users',
IndexName: "OtpIndex",
KeyConditionExpression: "phone = :phone and otp = :otp",
ExpressionAttributeValues: {
':phone': 919600923917,
':otp': 2387
},
};
Please find more details on querying global secondary indexes Doc
You can use Two matching field. to do this you need to define 2 keys
Partition Key
Sort Key
Mark one field as Partition key and another as Sort Key
Example- Partition Key - phone
Sort Key - otp.
if you have already made other attribute as key and not able to make otp/phone as key then you can create Secondary index and mark phone and otp as key into that.
you can mark different attributes (columns) as partition key or sort key other than already mentioned primary key or sort key in base table ( Main table).
also you can use --filter-expression
something like this
aws dynamodb query \
--table-name users \
--key-condition-expression "phone = :phone \
--filter-expression "#otp = :otp" \
--expression-attribute-names '{"#otp": "otp"}' \
--expression-attribute-values file://values.json
I am trying to make a complex query in swift to get data from DynamoDB.
I am able to get all information by using the userID. However there are times that I may not know the entirety of the userID and need to make a more complex query.
For instance, if I know the first name and the last name, and the user id format is "firstname:lastname:email", I need to be able to query all userID's that include the first and last name, then add a where for another column.
I am very new to dynamo and want to accomplish something like the sql query below.
SQL example:
SELECT * FROM mytable
WHERE column2 LIKE '%OtherInformation%'
AND (column1 LIKE '%lastname%' OR column1 LIKE '%firstname%')
Here is the code I have in swift4 for getting the userID if I know it exaclty, not entirely sure how to modify this for complex queries.
func queryDBForUser(Fname: String, Lname: String) {
let userId = Fname + "." + Lname + ":" + (UIDevice.current.identifierForVendor?.uuidString)!
self.UserId = userId
let objectMapper = AWSDynamoDBObjectMapper.default()
let queryExpression = AWSDynamoDBQueryExpression()
queryExpression.keyConditionExpression = "#userId = :userId"
queryExpression.expressionAttributeNames = ["#userId": "userId",]
queryExpression.expressionAttributeValues = [":userId": userId,]
objectMapper.query(CheckaraUsers.self, expression: queryExpression, completionHandler: {(response: AWSDynamoDBPaginatedOutput? ,error: Error?) -> Void in
if let error = error {
print("Amazon DynamoDB Error: \(error)")
return
}
I have also tried many variations along the lines of the following code, with no luck:
queryExpression.keyConditionExpression = "#FirstName = :firstName and #LastName = :lastName,"
queryExpression.expressionAttributeNames = ["#FirstName": "FirstName" , "#LastName": "LastName"]
queryExpression.expressionAttributeValues = [":FirstName": Fname,":LastName": Lname]
Any help would be greatly appreciated, thanks in advance!
You won't be able to do this with a DynamoDB query. When you query a table (or index) in DynamoDB you must always specify the complete primary key. In your case that would mean the full value of "firstname:lastname:email".
You could sort of do this with a DynamoDB scan and a filter expression, but that will look at every item in your table, so it could be slow and expensive. Amazon will charge you for the read capacity necessary to look at every item in the table.
So if you really wanted to, the filter expression for the scan operation would be something like:
"contains (#FirstName, :firstName) and contains (#LastName, : lastName)"
Note that contains looks for an exact substring match, so if you want case insensitive matches (like ILIKE in SQL) it won't work.
If you need to do these types of queries then you need to evaluate whether or not DynamoDB is the right choice for you. DynamoDB is a NoSQL key/value store basically. It trades limited querying functionality for scalability and performance. If you are coming at DynamoDB from a SQL background and are expecting to be able to do freeform queries of anything in your table, you will be disappointed.
Got the query working by adding a secondary index to my DynamoDB table, although this is not what I initially wanted, it still works as now I can query for a value that exists in both columns I needed, without doing a table scan and filtering after.
query code:
queryExpression.indexName = "Index-Name" queryExpression.keyConditionExpression = "#LastName = :LastName and #otherValue = :otherValue"
queryExpression.expressionAttributeNames = ["#LastName": "LastName" , "#otherValue": "otherValue"]
queryExpression.expressionAttributeValues = [":LastName": Lname,":otherValue": self.otherValue!]
How insert item on top table in PostgreSQL? That it is possible? In the table I have only two fields as text. First is primary key.
CREATE TABLE news_table (
title text not null primary key,
url text not null
);
I need a simple query for the program in java.
OK, this is my code:
get("/getnews", (request, response) -> {
List<News> getNews = newsService.getNews();
List<News> getAllNews = newsService.getAllNews();
try (Connection connection = DB.sql2o.open()) {
String sql = "INSERT INTO news_table(title, url) VALUES (:title, :url)";
for (News news : getNews) {
if (!getAllNews.contains(news)) {
connection.createQuery(sql, true)
.addParameter("title", news.getTitle())
.addParameter("url", news.getUrl())
.executeUpdate()
.getKey();
}
}
}
return newsService.getNews();
}, json());
The problem is that as it calls getnews method for the second time this new news adds at the end of the table, and there is no extant hronologi news. How this resolve? I use Sql2o + sparkjava.
Probably already I know. I need to reverse the List getnews before I will must contains object getnews and getallnews?
There is no start or end in a table. If you want to sort your data, just use an ORDER BY in your SELECT statements. Without ORDER BY, there is no order.
Relational theory, the mathematical foundation of relational databases, lays down certain conditions that relations (represented in real databases as tables) must obey. One of them is that they have no ordering (i.e., the rows will neither be stored nor retrieved in any particular order, since they are treated as a mathematical set). It's therefore completely under the control of the RDBMS where a new row is entered into a table.
Hence there is no way to ensure a particular ordering of the data without using an ORDER BY clause when you retrieve the data.
use annotation #Options(useGeneratedKeys = true, keyProperty = "id") to get the generated primary key for single insert is fine for me , but when I use annotation #InsertProvider to make a batch insert , I have no idea how to get the generated primary keys , any comments will be appreciated . thx in advance
Now Mybatis 3.3.1 had supported it. Please see https://github.com/mybatis/mybatis-3/pull/547
import org.apache.ibatis.annotations.Insert;
import org.apache.ibatis.annotations.Options;
import org.apache.ibatis.annotations.Param;
public interface TestMapper {
...
#Options(useGeneratedKeys = true, keyProperty = "id", keyColumn = "id")
#Insert({
"<script>",
"INSERT INTO test_table",
"(column_one, column_two)",
"VALUES" +
"<foreach item='item' collection='list' open='' separator=',' close=''>" +
"(" +
"#{item.columnOne,jdbcType=VARCHAR},",
"#{item.columnTwo,jdbcType=VARCHAR}" +
")" +
"</foreach>",
"</script>"})
void insertBatchTestTable(#Param("list") List<TestObject> testObjs);
}
ps.:
Set keyColumn and keyProperty
Use #Param("list")
MyBatis will set objects keyProperty by reflection
I haven't used annotations with mybatis, only xml. But, I have used both useGeneratedKeys and batch insert, together.
With Mybatis, you have to execute the same query ( One with useGeneratedKeys ) and call the insert repeatedly for each object in your collection. This will map the generated key to your pojo. Flush the session after every N records, and commit.
That's it. I hope you are not using Oracle as your DB. As, with Oracle you'll have to flush after call to insert, which beats to purpose of batch.