val queryReq: com.amazonaws.services.dynamodbv2.model.QueryRequest = new com.amazonaws.services.dynamodbv2.model.QueryRequest(tableName)
.withIndexName(...)
.addKeyConditionsEntry(..., new com.amazonaws.services.dynamodbv2.model.Condition()
.withAttributeValueList(Seq(new com.amazonaws.services.dynamodbv2.model.AttributeValue(...)):_*)
.withComparisonOperator(...))
.addKeyConditionsEntry(..., new com.amazonaws.services.dynamodbv2.model.Condition()
.withAttributeValueList(Seq(new com.amazonaws.services.dynamodbv2.model.AttributeValue().withN(...)):_*)
.withComparisonOperator(...))
.withScanIndexForward(scanDirection)
.withLimit(5000000)
This query is having withLimit = 5000000, but only returned about 3000 results. Is there a limit in how much data being returned by dynamodb in each query? If so, is there a clean way to overcome this limit?
I just found the solution. So I will answer my own question:
I can just create a while loop that appends to the result set when getLastEvaluatedKey() is not null nor empty.
Related
I'm trying to execute two queries against the database, the first one inserts the data then I do another query to get some data from a different collection then combine the result, but the zipWith is not being executed.
Mono<String> orderMono = orderDtoMono
.map(EntityDtoUtil::toEntity)
.flatMap(this.repository::insert)
.zipWhen(order -> this.clientRepository.findById(order. getOrderId()))
.map(i -> {
System.out.println("here");
return i.getT1().getOrderId() +" : "+i.getT2().getSuccessUrl();});
return orderMono;
The insert query works, however, the zipWhen is not executed and I get API result as 200 empty body.
The desired result would be the string created inside map operator at the end.
Zip will work only when both publishers return some data. Otherwise it will be empty.
In your case, you had just inserted a new record via repository .flatMap(this.repository::insert) - then you immediately expect this order id to be present via this.clientRepository.findById(order. getOrderId()). I believe it will be empty - so in your case you get empty response.
Do you have any DB trigger to insert records in other tables based on the new order id? Can you explain this - this.clientRepository.findById(order. getOrderId()) ?
I created a quill query, which should find some data in database by given parameter:
val toFind = "SomeName"
val query = query.find(value => infix"$value = ${lift(toFind)}".as[Boolean])
It works fine when for example I have data in database "SomeName", but if I want to have same results by passing there "somename" I found nothing. The problem is with data case-sensitive.
Is it possible to always find values with case-insensitive way? In quill docs I have not found anything about it.
Ok, I found a solution. It is enough to add LOWER() sql function to infix:
val query = query.find(value => infix"LOWER($value) = ${lift(toFind.toLowerCase)}".as[Boolean])
That is meant to be read as a dual upsert operation, upsert the document then the array element.
So MongoDB is a denormalized store for me (we're event sourced) and one of the things I'm trying to deal with is the concurrent nature of that. The problem is this:
Events can come in out of order, so each update to the database need to be an upsert.
I need to be able to not only upsert the parent document but an element in an array property of that document.
For example:
If the document doesn't exist, create it. All events in this stream have the document's ID but only part of the information depending on the event.
If the document does exist, then update it. This is the easy part. The update command is just written as UpdateOneAsync and as an upsert.
If the event is actually to update a list, then that list element needs to be upserted. So if the document doesn't exist, it needs to be created and the list item will be upserted (resulting in an insert); if the document does exist, then we need to find the element and update it as an upsert, so if the element exists then it is updated otherwise it is inserted.
If at all possible, having it execute as a single atomic operation would be ideal, but if it can only be done in multiple steps, then so be it. I'm getting a number of mixed examples on the net due to the large change in the 2.x driver. Not sure what I'm looking for beyond the UpdateOneAsync. Currently using 2.4.x. Explained examples would be appreciated. TIA
Note:
Reiterating that this is a question regarding the MongoDB C# driver 2.4.x
Took some tinkering, but I got it.
var notificationData = new NotificationData
{
ReferenceId = e.ReferenceId,
NotificationId = e.NotificationId,
DeliveredDateUtc = e.SentDate.DateTime
};
var matchDocument = Builders<SurveyData>.Filter.Eq(s => s.SurveyId, e.EntityId);
// first upsert the document to make sure that you have a collection to write to
var surveyUpsert = new UpdateOneModel<SurveyData>(
matchDocument,
Builders<SurveyData>.Update
.SetOnInsert(f => f.SurveyId, e.EntityId)
.SetOnInsert(f => f.Notifications, new List<NotificationData>())){ IsUpsert = true};
// then push a new element if none of the existing elements match
var noMatchReferenceId = Builders<SurveyData>.Filter
.Not(Builders<SurveyData>.Filter.ElemMatch(s => s.Notifications, n => n.ReferenceId.Equals(e.ReferenceId)));
var insertNewNotification = new UpdateOneModel<SurveyData>(
matchDocument & noMatchReferenceId,
Builders<SurveyData>.Update
.Push(s => s.Notifications, notificationData));
// then update the element that does match the reference ID (if any)
var matchReferenceId = Builders<SurveyData>.Filter
.ElemMatch(s => s.Notifications, Builders<NotificationData>.Filter.Eq(n => n.ReferenceId, notificationData.ReferenceId));
var updateExistingNotification = new UpdateOneModel<SurveyData>(
matchDocument & matchReferenceId,
Builders<SurveyData>.Update
// apparently the mongo C# driver will convert any negative index into an index symbol ('$')
.Set(s => s.Notifications[-1].NotificationId, e.NotificationId)
.Set(s => s.Notifications[-1].DeliveredDateUtc, notificationData.DeliveredDateUtc));
// execute these as a batch and in order
var result = await _surveyRepository.DatabaseCollection
.BulkWriteAsync(
new []{ surveyUpsert, insertNewNotification, updateExistingNotification },
new BulkWriteOptions { IsOrdered = true })
.ConfigureAwait(false);
The post linked as being a dupe was absolutely helpful, but it was not the answer. There were a few things that needed to be discovered.
The 'second statement' in the linked example didn't work
correctly, at least when translated literally. To get it to work, I had to match on the
element and then invert the logic by wrapping it in the Not() filter.
In order to use 'this index' on the match, you have to use a
negative index on the array. As it turns out, the C# driver will
convert any negative index to the '$' character when the query is
rendered.
In order to ensure they are run in order, you must include bulk write
options with IsOrdered set to true.
I've probably missed something obvious, but within the ReactiveMongo API (v0.8) how do you set a limit on the number of documents returned by a query?
I'd like to return the single most recent document added to a collection. This is my code so far:
def getLatest()(implicit reader: reactivemongo.bson.handlers.RawBSONReader[T]): Future[Option[T]] = {
collection.find (QueryBuilder(
queryDoc = Some(BSONDocument()),
sortDoc = Some(BSONDocument("_id" -> BSONInteger(-1)))
)).headOption().mapTo[Option[T]]
}
headOption() works to retrieve a single result, but I'm not explicitly using any kind of Mongo limit clause so I'm worried about this query's impact on the DB. Please help me improve this code. Thanks in advance.
In 0.8 you have to set the batchSize option to 1 in order to tell MongoDB to close the database cursor automatically:
val maybedoc = collection.find(BSONDocument(), QueryOpts().batchSize(1)).headOption
// or using QueryBuilder like you do
val maybedoc2 = collection.find (QueryBuilder(
queryDoc = Some(BSONDocument()),
sortDoc = Some(BSONDocument("_id" -> BSONInteger(-1)))
), QueryOpts().batchSize(1)).headOption()
In 0.9 collections have been refactored and greatly simplified. Now you can do this:
val maybedoc = collection.
find(BSONDocument()).
sort(BSONDocument("_id" -> -1)).
one[BSONDocument]
The one[T] method in 0.9 sets the batchSize flag for you and returns an Option[T].
Yes, the headOption() function limits the query to just one result:
def headOption()(implicit ec: ExecutionContext) :Future[Option[T]] = {
collect[Iterable](1).map(_.headOption)
}
https://github.com/zenexity/ReactiveMongo/blob/0.8/src/main/scala/api/cursor.scala#L180
I'm trying to get a feel for how fast MongoDB is compared to traditional RDBMSs. To this end, I'm using Java to try and get the result of a traditional SQL join by defining a MongoDB function that will return an object after embedding another object into it:
s_with_user = function(rows){
var result=[]
db.subscriptions.find().limit(rows).forEach( function(s) {
s.user= db.users.find({id: s.user_id});
result.push( s );
});
return result;
}
Then, I use:
DB db = new Mongo("localhost", 27017).getDB("test");
Object result = db.eval("s_with_user(1000)");
Measuring the time taken for the last statement, I'm confident that MongoDB is running the command and evaluating the data I want. However, the result object is always null.
How can I do this in such a way that I can inspect the results?
First of all, Stennie is right about eval() not being the right way to do joins in MongoDB.
However, to try to answer your question, it would work this way:
String f = "function(rows) { ... }";
DB db = new Mongo("localhost", 27017).getDB("test");
Object result = db.eval( f, 1000 );
Also, your "join" function needs a small correction: it should use findOne instead of find:
s.user = db.users.findOne({id: s.user_id});