I'm new to Couchbase and am struggling to get a composite index to do what I want it to. The use-case is this:
I have a set of "Enumerations" being stored as documents
Each has a "last_updated" field which -- as you may have guessed -- stores the last time that the field was updated
I want to be able to show only those enumerations which have been updated since some given date but still sort the list by the name of the enumeration
I've created a Couchbase View like this:
function (doc, meta) {
var time_array;
if (doc.doc_type === "enum") {
if (doc.last_updated) {
time_array = doc.last_updated.split(/[- :]/);
} else {
time_array = [0,0,0,0,0,0];
}
for(var i=0; i<time_array.length; i++) { time_array[i] = parseInt(time_array[i], 10); }
time_array.unshift(meta.id);
emit(time_array, null);
}
}
I have one record that doesn't have the last_updated field set and therefore has it's time fields are all set to zero. I thought as a first test I could filter out that result and I put in the following:
startkey = ["a",2012,0,0,0,0,0]
endkey = ["Z",2014,0,0,0,0,0]
While the list is sorted by the 'id' it isn't filtering anything! Can anyone tell me what I'm doing wrong? Is there a better composite view to achieve these results?
In couchbase when you query view by startkey - endkey you're unable to filter results by 2 or more properties. Couchbase has only one index, so it will filter your results only by first param. So your query will be identical to query with:
startkey = ["a"]
endkey = ["Z"]
Here is a link to complete answer by Filipe Manana why it can't be filtered by those dates.
Here is a quote from it:
For composite keys (arrays), elements are compared from left to right and comparison finishes as soon as a element is different from the corresponding element in the other key (same as what happens when comparing strings à la memcmp() or strcmp()).
So if you want to have a view that filters by date, date array should go first in composite key.
Related
I am working on implementing grouping w/ the Server Side Row Model. I need to generate an appropriate ID for the intermediate group rows. For example, if I group by Status then I would have intermediate rows representing each Status (NEW, IN PROGRESS, COMPLETE, etc). I need to come up with a unique ID for these rows (but preferable something deterministic if they need to be accessed/updated later).
The getRowId function is passed an object that contains things like the row's data, the previous parent group values, a reference to the api, etc.
What I would ideally like to know is the current list of group fields... I have all of the values readily accessible, but I don't know what field the current row is being grouped by - else I could just go grab that field from the row's data to use as part of the row id...
Is there any good way to acquire this information?
The columnApi exposes the 'getRowGroupColumns' function from which the field property can be deduced:
getRowId: ({ columnApi, data, level, parentKeys = [] }) => {
const groupColumns = columnApi.getRowGroupColumns();
if (groupColumns.length > level) {
const field = groupColumns[level].getColDef().field;
return [...parentKeys, data[field]].join('-');
}
return [...parentKeys, data.athlete, data.year];
},
Hi I am newbie to dynamoDB. Below is the schema of the dynamo table
{
"user_id":1, // partition key
"dob":"1991-09-12", // sort key
"movies_watched":{
"1":{
"movie_name":"twilight",
"movie_released_year":"1990",
"movie_genre":"action"
},
"2":{
"movie_name":"harry potter",
"movie_released_year":"1996",
"movie_genre":"action"
},
"3":{
"movie_name":"lalaland",
"movie_released_year":"1998",
"movie_genre":"action"
},
"4":{
"movie_name":"serendipity",
"movie_released_year":"1999",
"movie_genre":"action"
}
}
..... 6 more attributes
}
I want to insert a new item if the item(that user id with dob) did not exist, otherwise add the movies to existing movies_watched map by checking if the movie is not already available the movies_watched map .
Currently, I am trying to use update(params) method.
Below is my approach:
function getInsertQuery (item) {
const exp = {
UpdateExpression: 'set',
ExpressionAttributeNames: {},
ExpressionAttributeValues: {}
}
Object.entries(item).forEach(([key, item]) => {
if (key !== 'user_id' && key !== 'dob' && key !== 'movies_watched') {
exp.UpdateExpression += ` #${key} = :${key},`
exp.ExpressionAttributeNames[`#${key}`] = key
exp.ExpressionAttributeValues[`:${key}`] = item
}
})
let i = 0
Object.entries(item. movies_watched).forEach(([key, item]) => {
exp.UpdateExpression += ` movies_watched.#uniqueID${i} = :uniqueID${i},`
exp.ExpressionAttributeNames[`#uniqueID${i}`] = key
exp.ExpressionAttributeValues[`:uniqueID${i}`] = item
i++
})
exp.UpdateExpression = exp.UpdateExpression.slice(0, -1)
return exp
}
The above method just creates update expression with expression names and values for all top level attributes as well as nested attributes (with document path).
It works well if the item is already available by updating movies_watched map. But throws exception if the item is not available and while inserting. Below is exception:
The document path provided in the update expression is invalid for update
However, I am still not sure how to check for duplicate movies in movies_watched map
Could someone guide me in right direction, any help is highly appreciated!
Thanks in advance
There is no way to do this, given your model, without reading an item from DDB before an update (at that point the process is trivial). If you don't want to impose this additional read capacity on your table for update, then you would need to re-design your data model:
You can change movies_watched to be a Set and hold references to movies. Caveat is that Set can contain only Numbers or Strings, thus you would have movie id or name or keep the data but as JSON Strings in your Set and then parse it back into JSON on read. With SET you can perform ADD operation on the movies_watched attribute. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.UpdateExpressions.html#Expressions.UpdateExpressions.ADD
You can go with single table design approach and have these movies watched as separate items with (PK:userId and SK:movie_id). To get a user you would perform a query and specify only PK=userId -> you will get a collection where one item is your user record and others are movies_watched. If you are new to DynamoDB and are learning the ropes, then I would suggest go with this approach. https://www.alexdebrie.com/posts/dynamodb-single-table/
We have following structure in MongoDB documents.
{
"id":"1111",
"keys":[
{
"name":"Country",
"value":"USA"
},
{
"name":"City",
"value":"LongIsland"
},
{
"name":"State",
"value":"NewYork"
}
]
}
Now using Springframework Query object, I figured out a way to pull the details using below syntax
query.addCriteria(
Criteria.where("keys.value").is(countryparam).
andOperator(
Criteria.where("keys.value").is(stateparam)
)
);
Two issue with this query model.
First issue is it is irrelevant if countryparam and stateparam are actually meant to match Country key name and State key name respectively. If just the values matches, the query returns the document. Means, if I have Country and City params, this just works if user passes Country and City values, even if they are swapped. So how can I exactly compare City to cityparam and State to Stateparam?
More complexity is if I have to extract the document basing on multiple key value pairs, I should be correspondingly able to match key name with respective value and query the document. How can I do this?
Thanks in advance!
I have a simple collection populated with student data and I need to remove some records based on some parameters. I executed the following from mongoshell
for(i=0;i<200;i++) {
var rec = db.grades.find({student_id:i,type:'homework'}).sort({score:1}).limit(1)
db.grades.remove(rec)
}
Ideally it should remove lowest score of type homework for all student_ids. Apparently, only the last 2 records (student_id: 199) from the find parameter was purged and the rest still exists.
db.grades.find({student_id:10,type:'homework'}).sort({score:1}).limit(1)
{ "_id" : ObjectId("50906d7fa3c412bb040eb5a1"), "student_id" : 10, "type" : "homework", "score" : 6.094174990746648 }
Is it because of the aysnchoronous nature of JS / Mongo ? What are the other alternatives for solving the same?
rec is not a document, it is a database cursor. You need to actually get a document from it:
for(i=0;i<200;i++) {
var cur = db.grades.find({student_id:i,type:'homework'}).sort({score:1}).limit(1);
var actualDoc = cur.next();
db.grades.remove(actualDoc);
}
Otherwise, you're trying to remove documents based on cursor properties, which is not what you want. See also http://docs.mongodb.org/manual/core/read-operations/#cursors.
You need to query the collection and return all of the documents in the collection first before iterating through it e.g.
var collection = grades.find({'type':'homework'}).sort({'student_id',1, 'score':1})
Then iterate through the records in the variable 'collection' removing documents with the lowest score. You also have an issue assigning i as a value to student_id without assigning the documents in the collection. And according to your code you're iterating through the collection based on student id. You don't need to do this to iterate through the collection. Just query all records of type homework then remove based on parameters. If you need to assign the value of student_id to a variable (hint: as a parameter to remove records), just assign student_id to a variable like so:
var id = ['student_id']
Alternatively (and this is the way I did it), you could sort all the records first by student_id and then by score. The score should be sorted in descending order.
Then iterate through the collection using a for loop, and when the student_id changes remove the record. To recognise the change in student_id store that value in a variable outside the loop and inside the loop (as 2 separate variables) then update them as you loop through the collection. Then compare the variables and remove the record if the values of the variables are not equal.
var oldid=-1;
var cursor=db.grades.find({'type':'homework'}).sort({'student_id':1,'score':1});
while (cursor.hasNext()) {
var doc = cursor.next();
var id = doc['student_id'];
if (oldid!=id)
{
db.grades.remove(doc);
oldid=id;
}
}
I have not clearly issue during query using two criterials like Id and Other. I use a Repository storing some data like id,iso,value. I have created an index("_id","Iso") to performs queries but queries are only returning my cursor if i use only one criterial like _id, but is returning nothing if a use two (_id, Iso) (commented code).
Are the index affecting the response or the query method are failing?
use :v1.6.5 and C# official.
Sample.
//Getting Data
public List<BsonObject> Get_object(string ID, string Iso)
{
using (var helper = BsonHelper.Create())
{
//helper.Db.Repository.EnsureIndex("_Id","Iso");
var query = Query.EQ("_Id", ID);
//if (!String.IsNullOrEmpty(Iso))
// query = Query.And(query, Query.EQ("Iso", Iso));
var cursor = helper.Db.Repository.FindAs<BsonObject>(query);
return cursor.ToList();
}
}
Data:
{
"_id": "2345019",
"Iso": "UK",
"Data": "Some data"
}
After that I have Updated my data using Update.Set() methods. I can see the changed data using MongoView. The new data are correct but the query is always returning the sames olds values. To see these values i use a page that can eventually cached, but if add a timestamp at end are not changing anything, page is always returning the same olds data. Your comments are welcome, thanks.
I do not recall offhand how the C# driver creates indexes, but the shell command for creating an index is like this:
db.things.ensureIndex({j:1});
Notice the '1' which is like saying 'true'.
In your code, you have:
helper.Db.Repository.EnsureIndex("_Id","Iso");
Perhaps it should be:
helper.Db.Repository.EnsureIndex("_Id", 1);
helper.Db.Repository.EnsureIndex("Iso", 1);
It could also be related to the fact that you are creating indexes on "_Id" and the actual id field is called "_id" ... MongoDB is case sensitive.
Have a quick look through the index documentation: http://www.mongodb.org/display/DOCS/Indexes