I need to output a list of sketchid values for which tdata birthdate is not present/null, considering only the first birthdate in each tdata.
There are many fields in each collection, but I am only interested in two of them:
{
"sketchid": NumberLong(51148),
"tdata": "[
{ "birthdate": ["529718400"], ...},
{ "birthdate": null, ... }
]
}
I need to print the sketchid for each tdata in which the first birthdate is empty.
Note: A single tdata contains many birthdate fields, I need to check only the first birthdate field and if it is empty I need to output the sketchid for that set.
I have tried this, but I couldn't get the solution.
db.timeline.find({tdata:{birthdate:{$exists: true, “$ne”: null }}},{sketchid})
To illustrate the data schema, the find() command (db.timeline.find().pretty()) gives the following output:
{
"_id" : ObjectId("56ea7b9490e2a5b17c143b56"),
"sketchid" : NumberLong(51147),
"tdata" : "[{\\\"type\\\":\\\"dob\\\",\\\"title\\\":\\\"Vishakha Singh\\\",\\\"year\\\":null,\\\"color\\\":null,\\\"content\\\":\\\"\\\",\\\"birthdate\\\":[\\\"831254400\\\"],\\\"birthplace\\\":[\\\"Abu Dhabi, UAE\\\"],\\\"zodiac\\\":\\\"Taurus\\\"},{\\\"type\\\":\\\"timeline\\\",\\\"title\\\":\\\"Cameo Role\\\",\\\"year\\\":[\\\"2014\\\"],\\\"color\\\":[\\\"#85330B\\\"],\\\"content\\\":\\\"<p><span data-sheets-value="[null,2,"Tamil movie Oru Oorla Rendu Raja"]" data-sheets-userformat="[null,null,513,[null,0],null,null,null,null,null,null,null,null,0]">Tamil movie Oru Oorla Rendu Raja<\\\\/span><\\\\/p>\\\\n\\\",\\\"birthdate\\\":null,\\\"birthplace\\\":null,\\\"zodiac\\\":\\\"Capricorn\\\"},{\\\"type\\\":\\\"timeline\\\",\\\"title\\\":\\\"Noticable Flick\\\",\\\"year\\\":[\\\"2013\\\"],\\\"color\\\":[\\\"#21C8F5\\\"],\\\"content\\\":\\\"<p><span data-sheets-value="[null,2,"Neetu in Fukrey"]" data-sheets-userformat="[null,null,513,[null,0],null,null,null,null,null,null,null,null,0]">Neetu in Fukrey<\\\\/span><\\\\/p>\\\\n\\\",\\\"birthdate\\\":null,\\\"birthplace\\\":null,\\\"zodiac\\\":\\\"Capricorn\\\"},{\\\"type\\\":\\\"timeline\\\",\\\"title\\\":\\\"Gained Recognition\\\",\\\"year\\\":[\\\"2010\\\"],\\\"color\\\":[\\\"#C90688\\\"],\\\"content\\\":\\\"<p><span data-sheets-value="[null,2,"Ashutosh Gowarikar\\'s Khelein Hum Jee Jaan Sey"]" data-sheets-userformat="[null,null,47617,[null,0],null,null,null,null,null,null,null,null,0,null,[null,2,721024],"sans-serif",11,null,1]">Ashutosh Gowarikar’s Khelein Hum Jee Jaan Sey<\\\\/span><\\\\/p>\\\\n\\\",\\\"birthdate\\\":null,\\\"birthplace\\\":null,\\\"zodiac\\\":\\\"Capricorn\\\"},{\\\"type\\\":\\\"timeline\\\",\\\"title\\\":\\\"Movie Debut\\\",\\\"year\\\":[\\\"2008\\\"],\\\"color\\\":[\\\"#FFCF00\\\"],\\\"content\\\":\\\"<p><span data-sheets-value="[null,2,"Hum Se Jahan"]" data-sheets-userformat="[null,null,513,[null,0],null,null,null,null,null,null,null,null,0]">Hum Se Jahan<\\\\/span><\\\\/p>\\\\n\\\",\\\"birthdate\\\":null,\\\"birthplace\\\":null,\\\"zodiac\\\":\\\"Capricorn\\\"},{\\\"type\\\":\\\"quote-card\\\",\\\"title\\\":\\\"vishakha singh\\\",\\\"year\\\":null,\\\"color\\\":null,\\\"content\\\":\\\"<p><span data-sheets-value="[null,2,"I am not really sophisticated but happy and chilled out"]" data-sheets-userformat="[null,null,14915,[null,0],[null,2,16777215],null,null,null,null,0,null,null,0,null,[null,2,0],"helvetica",11]">I am not really sophisticated but happy and chilled out<\\\\/span><\\\\/p>\\\\n\\\",\\\"birthdate\\\":null,\\\"birthplace\\\":null,\\\"zodiac\\\":\\\"Capricorn\\\"}]"
}
{
"_id" : ObjectId("56ea7b9490e2a5b17c143b57"),
"sketchid" : NumberLong(51148),
"tdata" : "[{\\\"type\\\":\\\"dob\\\",\\\"title\\\":\\\"Ali Fazal\\\",\\\"year\\\":null,\\\"color\\\":null,\\\"content\\\":\\\"\\\",\\\"birthdate\\\":[\\\"529718400\\\"],\\\"birthplace\\\":[\\\"Lucknow, Uttar Pradesh\\\"],\\\"zodiac\\\":\\\"Libra\\\"},{\\\"type\\\":\\\"timeline\\\",\\\"title\\\":\\\"Bollywood Appearance\\\",\\\"year\\\":[\\\"2016\\\"],\\\"color\\\":[\\\"#C90688\\\"],\\\"content\\\":\\\"<p><span data-sheets-value="[null,2,"Anand L. Rai\\'s Happy Bhaag Jayegi"]" data-sheets-userformat="[null,null,47617,[null,0],null,null,null,null,null,null,null,null,0,null,[null,2,721024],"sans-serif",11,null,1]">Anand L. Rai’s Happy Bhaag Jayegi<\\\\/span><\\\\/p>\\\\n\\\",\\\"birthdate\\\":null,\\\"birthplace\\\":null,\\\"zodiac\\\":\\\"Capricorn\\\"},{\\\"type\\\":\\\"timeline\\\",\\\"title\\\":\\\"English Movie\\\",\\\"year\\\":[\\\"2015\\\"],\\\"color\\\":[\\\"#FFCF00\\\"],\\\"content\\\":\\\"<p><span data-sheets-value="[null,2,"Special Appearance in Furious 7"]" data-sheets-userformat="[null,null,513,[null,0],null,null,null,null,null,null,null,null,0]">Special Appearance in Furious 7<\\\\/span><\\\\/p>\\\\n\\\",\\\"birthdate\\\":null,\\\"birthplace\\\":null,\\\"zodiac\\\":\\\"Capricorn\\\"},{\\\"type\\\":\\\"timeline\\\",\\\"title\\\":\\\"First Lead Role\\\",\\\"year\\\":[\\\"2015\\\"],\\\"color\\\":[\\\"#B6D402\\\"],\\\"content\\\":\\\"<p><span data-sheets-value="[null,2,"Khamoshiyan"]" data-sheets-userformat="[null,null,513,[null,0],null,null,null,null,null,null,null,null,0]">Khamoshiyan<\\\\/span><\\\\/p>\\\\n\\\",\\\"birthdate\\\":null,\\\"birthplace\\\":null,\\\"zodiac\\\":\\\"Capricorn\\\"},{\\\"type\\\":\\\"timeline\\\",\\\"title\\\":\\\"Supporting Role\\\",\\\"year\\\":[\\\"2014\\\"],\\\"color\\\":[\\\"#FF2305\\\"],\\\"content\\\":\\\"<p><span data-sheets-value="[null,2,"Bobby Jasoos with Vidya Balan"]" data-sheets-userformat="[null,null,513,[null,0],null,null,null,null,null,null,null,null,0]">Bobby Jasoos with Vidya Balan<\\\\/span><\\\\/p>\\\\n\\\",\\\"birthdate\\\":null,\\\"birthplace\\\":null,\\\"zodiac\\\":\\\"Capricorn\\\"},{\\\"type\\\":\\\"timeline\\\",\\\"title\\\":\\\"First Success\\\",\\\"year\\\":[\\\"2013\\\"],\\\"color\\\":[\\\"#85330B\\\"],\\\"content\\\":\\\"<p><span data-sheets-value="[null,2,"Fukrey"]" data-sheets-userformat="[null,null,513,[null,0],null,null,null,null,null,null,null,null,0]">Fukrey<\\\\/span><\\\\/p>\\\\n\\\",\\\"birthdate\\\":null,\\\"birthplace\\\":null,\\\"zodiac\\\":\\\"Capricorn\\\"},{\\\"type\\\":\\\"timeline\\\",\\\"title\\\":\\\"Bollywood Debut\\\",\\\"year\\\":[\\\"2009\\\"],\\\"color\\\":[\\\"#21C8F5\\\"],\\\"content\\\":\\\"<p><span data-sheets-value="[null,2,"Special Appearance in 3 Idiots"]" data-sheets-userformat="[null,null,513,[null,0],null,null,null,null,null,null,null,null,0]">Special Appearance in 3 Idiots<\\\\/span><\\\\/p>\\\\n\\\",\\\"birthdate\\\":null,\\\"birthplace\\\":null,\\\"zodiac\\\":\\\"Capricorn\\\"},{\\\"type\\\":\\\"timeline\\\",\\\"title\\\":\\\"American Miniseries\\\",\\\"year\\\":[\\\"2009\\\"],\\\"color\\\":[\\\"#C90688\\\"],\\\"content\\\":\\\"<p><span data-sheets-value="[null,2,"Bollywood Hero"]" data-sheets-userformat="[null,null,513,[null,0],null,null,null,null,null,null,null,null,0]">Bollywood Hero<\\\\/span><\\\\/p>\\\\n\\\",\\\"birthdate\\\":null,\\\"birthplace\\\":null,\\\"zodiac\\\":\\\"Capricorn\\\"},{\\\"type\\\":\\\"timeline\\\",\\\"title\\\":\\\"Screen Debut\\\",\\\"year\\\":[\\\"2008\\\"],\\\"color\\\":[\\\"#FFCF00\\\"],\\\"content\\\":\\\"<p><span data-sheets-value="[null,2,"Small role in the English language film The Other End of the Line"]" data-sheets-userformat="[null,null,309761,[null,0],null,null,null,null,null,null,null,null,0,null,[null,2,721024],"sans-serif",11,null,1,null,null,1]">Small role in the English language film The Other End of the Line<\\\\/span><\\\\/p>\\\\n\\\",\\\"birthdate\\\":null,\\\"birthplace\\\":null,\\\"zodiac\\\":\\\"Capricorn\\\"},{\\\"type\\\":\\\"quote-card\\\",\\\"title\\\":\\\"Ali Fazal\\\",\\\"year\\\":null,\\\"color\\\":null,\\\"content\\\":\\\"<p><span data-sheets-value="[null,2,"I love being in love."]" data-sheets-userformat="[null,null,14915,[null,0],[null,2,16777215],null,null,null,null,0,null,null,0,null,[null,2,4144959],"georgia",11]">I love being in love.<\\\\/span><\\\\/p>\\\\n\\\",\\\"birthdate\\\":null,\\\"birthplace\\\":null,\\\"zodiac\\\":\\\"Capricorn\\\"}]"
}
When I tried following commands I get the following errors
db.timeline.find({tdata:{"birthdate":{$exists: true, “$ne”: null }}},{sketchid})
2016-08-19T12:02:14.665+0530 E QUERY [thread1] SyntaxError: illegal character #(shell):1:53
db.timeline.find({tdata:{"birthdate\\":{$exists: true, “$ne”: null }}},{sketchid})
2016-08-19T12:02:53.357+0530 E QUERY [thread1] SyntaxError: unterminated string literal #(shell):1:25
db.timeline.find({tdata:{\\"birthdate\\":{$exists: true, “$ne”: null }}},{sketchid})
2016-08-19T12:03:06.572+0530 E QUERY [thread1] SyntaxError: illegal character #(shell):1:25
You need to use the positional operator if you want to test only the first element of the tdata array. Also depending on your version {$ne:null} doesn't always give what you'd expect. You can check against the null type 10 instead:
db.timeline.find({"tdata.0.birthdate": {$exists:true,$not : { $type : 10 }} },{sketchid:1})
db.movies.find({"original_title" : {$regex: input_data, $options:'i'}}, function (err, datares){
if (err || datares == false) {
db.movies.find({"release_date" : {$regex: input_data + ".*", $options:'i'}}, function (err, datares){
if(err || datares == false){
db.movies.find({"cast" : {$regex: input_data, $options:'i'}}, function (err, datares){
if(err || datares == false){
db.movies.find({"writers" : {$regex: input_data, $options:'i'}}, function (err, datares){
if(err || datares == false){
db.movies.find({"genres.name" : {$regex: input_data, $options:'i'}}, function (err, datares){
if(err || datares == false){
db.movies.find({"directors" : {$regex: input_data, $options:'i'}}, function (err, datares){
if(err || datares == false){
res.status(451);
res.json({
"status" : 451,
"error code": "dataNotFound",
"description" : "Invalid Data Entry."
});
return;
} else{
res.json(datares);
return;
}
});
} else {
res.json(datares);
return;
}
});
} else {
res.json(datares);
return;
}
});
} else {
res.json(datares);
return;
}
});
} else {
res.json(datares);
return;
}
});
} else {
res.json(datares);
return;
}
});
I am trying to implement a so called "all-in-one" search so that whenever a user types in any kind of movie related information, my application tries to return all relevant information. However I have noticed that this transaction might be expensive on the backend and sometimes the host is really slow.
How do I smoothly close the db connection and where should I use it?
I read here that it is best not to close a mongodb connection in node.js >>Why is it recommended not to close a MongoDB connection anywhere in Node.js code?
Is the a proper way to implement a all-in-one search kind of a thing by using nested find commands?
Your current approach is full of problems and is not necessary to do this way. All you are trying to do is search for what a can gather is a plain string within a number of fields in the same collection. It may possibly be a regular expression construct but I'm basing two possibilities on a plain text search that is case insensitive.
Now I am not sure if you came to running one query dependant on the results of another because you didn't know another way or though it would be better. Trust me on this, that is not a better approach than anything listed here nor is it really required as will be shown:
Regex query all at once
The first basic option here is to continue your $regex search but just in a singular query with the $or operator:
db.movies.find(
{
"$or": [
{ "original_title" : { "$regex": input_data, "$options":"i"} },
{ "release_date" : { "$regex": input_data, "$options":"i"} },
{ "cast" : { "$regex": input_data, "$options":"i"} },
{ "writers" : { "$regex": input_data, "$options":"i"} },
{ "genres.name" : { "$regex": input_data, "$options":"i"} },
{ "directors" : { "$regex": input_data, "$options":"i"} }
]
},
function(err,result) {
if(err) {
// respond error
} else {
// respond with data or empty
}
}
);
The $or condition here effectively works like "combining queries" as each argument is treated as a query in itself as far as document selection goes. Since it is one query than all the results are naturally together.
Full text Query, multiple fields
If you are not really using a "regular expression" built from regular expression operations i.e ^(\d+)\bword$, then you are probably better off using the "text search" capabilities of MongoDB. This approach is fine as long as you are not looking for things that would be generally excluded, but your data structure and subject actually suggests this is the best option for what you are likely doing here.
In order to be able to perform a text search, you first need to create a "text index", specifically here you want the index to span multiple fields in your document. Dropping into the shell for this is probably easiest:
db.movies.createIndex({
"original_title": "text",
"release_date": "text",
"cast" : "text",
"writers" : "text",
"genres.name" : "text",
"directors" : "text"
})
There is also an option to assign a "weight" to fields within the index as you can read in the documentation. Assigning a weight give "priority" to the terms listed in the search for the field that match in. For example "directors" might be assigned more "weight" than "cast" and matches for "Quentin Tarantino" would therefore "rank higher" in the results where he was a director ( and also a cast member ) of the movie and not just a cast member ( as in most Robert Rodriguez films ).
But with this in place, performing the query itself is very simple:
db.movies.find(
{ "$text": { "$search": input_data } },
function(err,result) {
if(err) {
// respond error
} else {
// respond with data or empty
}
}
);
Almost too simple really, but that is all there is to it. The $text query operator knows to use the required index ( there can only be one text index per collection ) and it will just then look through all of the defined fields.
This is why I think this is the best fit for your use case here.
Parallel Queries
The final alternate I'll give here is you still want to demand that you need to run separate queries. I still deny that you do need to only query if the previous query does not return results, and I also re-assert that the above options should be considered "first", with preference to text search.
Writing dependant or chained asynchronous functions is a pain, and very messy. Therefore I suggest leaning a little help from another library dependency and using the node-async module here.
This provides an aync.map.() method, which is perfectly suited to "combining" results by running things in parallel:
var fields = [
"original_title",
"release_date",
"cast",
"writers",
"genres.name",
"directors"
];
async.map(
fields,
function(field,callback) {
var search = {},
cond = { "$regex": input_data, "$options": "i" };
search[field] = cond; // assigns the field to search
db.movies.find(search,callback);
},
function(err,result) {
if(err) {
// respond error
} else {
// respond with data or empty
}
}
);
And again, that is it. The .map() operator takes each field and transposes that into the query which in turn returns it's results. Those results are then accessible after all queries are run in the final section, "combined" as if they were a single result set, just as the other alternates do here.
There is also a .mapSeries() variant that runs each query in series, or .mapLimit() if you are otherwise worried about using database connections and concurrent tasks, but for this small size this should not be a problem.
I really don't think that this option is necessary, however if the Case 1 regular expression statements still apply, this "may" possibly provide a little performance benefit due to running queries in parallel, but at the cost of increased memory and resource consumption in your application.
Anyhow, the round up here is "Don't do what you are doing", you don't need to and there are better ways to handle the task you want to achieve. And all of them are mean cleaner and easier to code.