Mongodb - How to find string in multiple fields? - mongodb

Using Pymongo for this scenario.
I have User that has email, first_name, last_name.
I am using this Pymongo snippet:
user_found = users.find({'$or':[
{'email':{'$regex':searchString, '$options':'i'}},
{'first_name':{'$regex':searchString, '$options':'i'}},
{'last_name':{'$regex':searchString, '$options':'i'}}]})
this example works, if I want to find searchString in:
email, or
first_name, or
last_name
now I need to also find searchString in first_name + last_name combined.
how can I do that?
Is there a way in mongo, through the query, to combine the two into a "fullname" then search the fullname?

Easiest way is to add an array field and populate it with all of the variants that you want to search on. Index that array field.
That way you only need one index and your search across all fields is simple and doesn't change when you want to search on some new search variant. You can also normalize the text you put into the search array, for example, lower casing it, removing punctuation etc.
See https://stackoverflow.com/q/8206188/224370
Edit: MongoDB's documentation now covers keyword search and the new full-text search feature.

I had the same problem. I already used regex string search, so my solution was:
generate a helper collection. Here I combine all relevant string, like:
{
search_field: email + " " + first_name + " " + last_name,
ref_id: (id to real object)
}
I then use a regexp to creat what i allow to be looked for:
// logic found here: http://stackoverflow.com/questions/10870372/regex-match-if-string-contain-all-the-words-or-a-condition
var words = query.split(/[ ,]+/);
var regstr = "";
for (var i = 0; i < words.length; ++i) {
var word = words[i];
regstr += "(?=.*?\\b" + word + ")";
}
regstr += "^.*$";
regex = new RegExp(regstr, "i");
This then also gives some flexibility about the order.
Searching is not the fastest, since it still uses regex on all elements, but it is ok for me. (I also index the collection on search_field.
Getting results also becomes a nested call, since first you need to get the _ids you really want, and then you can query for them like so:
connection.find({ "search_field" : regex }, { _id: 0, ref_id: 1 }, { limit: limit, skip: start }).toArray(function (err, docs) {
if (err) throw err;
// map array of documents into simple array of ids
var ids = [];
for (var i = 0; i < docs.length; ++i)
{
var doc = docs[i];
ids.push(doc.ref_id);
}
if (ids.length > 0)
MongooseEmails.find({ "_id": { $in: ids } }, function (err, docres) {
if (err) throw err;
res.send(JSON.stringify(docsres));
});
else
res.send("");
});
This is edited code.. perhaps there is a syntax error, generally, it is working for me.

Related

Conditional Searching in mongodb

I have three fields to be searched inside a mongodb database. Sometimes I have to search based on only one field, sometimes two and sometimes three as well. It is something like-if there is only one field value present, then search on the basis of only one field, if two are there then find using like $and and if three all of the three,find using like $and. Is there any single query to make the search working?
the below is the image where i should make selection from
You need to make a query string like below,
let queryString = {};
if (field1 !== undefined) {
queryString.field1 = field1;
}
if (field2 !== undefined) {
queryString.field2 = field2;
}
if (field3 !== undefined) {
queryString.field3 = field3;
}
let documents = await db.find({ $and: [queryString] }).exec();

Mongodb error: data.includes is not a function

I am running a js script for identifying and writing field values with certain date formats (YYYY-mm-dd in this case) in a txt file. All was going well with single fields and even with one subfield, such as 'Status.dateTime', to which I wrote: var data = ({'Status.dateTime':{$exists:true}) and worked just fine.
The problem started in the following case, in which I have a field, an array '0' and a subfield of this array. How can I declare the variables and proceed in this case?
db.ato.find({'Area.0.Date':{$exists:true})
.projection({})
.sort({_id:-1})
//.limit(1000)
.forEach(function(doc) {
const fs = require('fs')
var data = ({'Area.0.Date':{$exists:true})
if(data != null && !data.includes("/") && data != "" && data.indexOf("-") == 4) {
//print(doc._id + " " + data)
fs.appendFile('C:/Users/victo/Desktop/output_query_YYYY-mm-dd.txt', doc._id + " " + data + "\n", (err) => {
if (err) throw err;
})
}
})
Inside the forEach MongoDb uses JavaScript. So the doc inside the function is your mongodb document.
In order to get the date from the first element of the Area array inside your doc you would grab it using the JavaScript.
var data = doc.Area[0].Date
To make things more intuitive you can use
print(doc)
inside the forEach and see the JavaScript Object

mapreduce between consecutive documents

Setup:
I got a large collection with the following entries
Name - String
Begin - time stamp
End - time stamp
Problem:
I want to get the gaps between documents, Using the map-reduce paradigm.
Approach:
I'm trying to set a new collection of pairs mid, after that I can compute differences from it using $unwind and Pair[1].Begin - Pair[0].End
function map(){
emit(0, this)
}
function reduce(){
var i = 0;
var pairs = [];
while ( i < values.length -1){
pairs.push([values[i], values[i+1]]);
i = i + 1;
}
return {"pairs":pairs};
}
db.collection.mapReduce(map, reduce, sort:{begin:1}, out:{replace:"mid"})
This works with limited number of document because of the 16MB document cap. I'm not sure if I need to get the collection into memory and doing it there, How else can I approach this problem?
The mapReduce function of MongoDB has a different way of handling what you propose than the method you are using to solve it. The key factor here is "keeping" the "previous" document in order to make the comparison to the next.
The actual mechanism that supports this is the "scope" functionality, which allows a sort of "global" variable approach to use in the overall code. As you will see, what you are asking when that is considered takes no "reduction" at all as there is no "grouping", just emission of document "pair" data:
db.collection.mapReduce(
function() {
if ( last == null ) {
last = this;
} else {
emit(
{
"start_id": last._id,
"end_id": this._id
},
this.Begin - last.End
);
last = this;
}
},
function() {}, // no reduction required
{
"out": { "inline": 1 },
"scope": { "last": null }
}
)
Out with a collection as the output as required to your size.
But this way by using a "global" to keep the last document then the code is both simple and efficient.

mongodb $not _id

I need a way to search but not include an _id which is already on the screen in front of the user. For example, I have 3 pet profiles one which the user is already viewing.
On that page I have a heading called My Family. I then run this search:
public function fetch_family($owner)
{
$collection = static::db()->mypet;
$cursor = $collection->find(array('owner' => new MongoId($owner)));
if ($cursor->count() > 0)
{
$family = array();
// iterate through the results
while( $cursor->hasNext() ) {
$family[] = ($cursor->getNext());
}
return $family;
}
}
And it returns all the pets in my family even knowing I am already showing one. So I want to exclude that one _id from the search.
I thought something like this.
$cursor = $collection->find(array('owner' => new MongoId($owner), '$not'=>array('_id'=>new MongoId(INSERT ID HERE))));
However, that just stops the whole thing from running.
You need to do a $ne (not equal) to make sure the current pet you are viewing is excluded from the search by owner.
Example in the mongo shell:
var viewingPetId = ObjectId("515535b6760fe8735f5f6899");
var ownerId = ObjectId("515535ba760fe8735f5f689a");
db.mypet.find(
{
_id: { $ne: viewingPetId },
owner: ownerId
}
)
Use $ne as (notice no need to use ObjectId(), string will autocast to ObjectId):
db.organizations.find({"_id" : {$ne:"563c50e05cdb2be30391e873"}})

Looking for help with reading from MongoDB in Node.JS

I have a number of records stored in a MongoDB I'm trying to output them to the browser window by way of a Node.JS http server. I think I'm a good portion of the way along but I'm missing a few little things that are keeping it from actually working.
The code below uses node-mongo-native to connect to the database.
If there is anyone around who can help me make those last few connections with working in node I'd really appreciate it. To be fair, I'm sure this is just the start.
var sys = require("sys");
var test = require("assert");
var http = require('http');
var Db = require('../lib/mongodb').Db,
Connection = require('../lib/mongodb').Connection,
Server = require('../lib/mongodb').Server,
//BSON = require('../lib/mongodb').BSONPure;
BSON = require('../lib/mongodb').BSONNative;
var host = process.env['MONGO_NODE_DRIVER_HOST'] != null ? process.env['MONGO_NODE_DRIVER_HOST'] : 'localhost';
var port = process.env['MONGO_NODE_DRIVER_PORT'] != null ? process.env['MONGO_NODE_DRIVER_PORT'] : Connection.DEFAULT_PORT;
sys.puts("Connecting to " + host + ":" + port);
function PutItem(err, item){
var result = "";
if(item != null) {
for (key in item) {
result += key + '=' + item[key];
}
}
// sys.puts(sys.inspect(item)) // debug output
return result;
}
function ReadTest(){
var db = new Db('mydb', new Server(host, port, {}), {native_parser:true});
var result = "";
db.open(function (err, db) {
db.collection('test', function(err, collection) {
collection.find(function (err, cursor){
cursor.each( function (err, item) {
result += PutItem(err, item);
});
});
});
});
return result;
}
http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.end("foo"+ReadTest());
}).listen(8124);
console.log('Server running on 8124');
Sources:
- mongo connectivity code:
https://github.com/christkv/node-mongodb-native/blob/master/examples/simple.js
- node. http code: nodejs.org
EDIT CORRECTED CODE
Thanks to Mic below who got me rolling in the right direction. For anyone interested, the corrected solution is here:
function ReadTest(res){
var db = new Db('mydb', new Server(host, port, {}), {native_parser:true});
var result = "";
res.write("in readtest\n");
db.open(function (err, db) {
res.write("now open\n");
db.collection('test', function(err, collection) {
res.write("in collection\n");
collection.find(function (err, cursor){
res.write("found\n");
cursor.each( function (err, item) {
res.write("now open\n");
var x = PutItem(err, item);
sys.puts(x);
res.write(x);
if (item == null) {
res.end('foo');
}
});
});
});
});
}
http.createServer(function (req, res) {
res.writeHead(200, {'Content-Type': 'text/plain'});
res.write("start\n");
ReadTest(res);
}).listen(8124);
console.log('Server running on 8124');
My guess is that you are returning result, writing the response, and closing the connection before anything is fetched from the db.
One solution would be to pass the response object to where you actually need it, something like:
function readTest(res) {
db.open(function (err, db) {
db.collection('test', function(err, collection) {
collection.find(function (err, cursor) {
res.writeHead(200, {'Content-type' : 'text/plain'});
cursor.each( function (err, item) { res.write(item); });
res.end();
...
Of course, you should also handle errors and try to avoid nesting too many levels, but that's a different discussion.
Instead of writing all the low-level Mongodb access code, you might want to try a simple library like mongous so that you can focus on your data, not on MongoDB quirks.
You might want to try mongoskin too.
Reading documents
To apply specific value filters, we can pass specific values to the find() command. Here is a SQL query:
SELECT * FROM Table1 WHERE name = 'ABC'
which is equivalent to the following in MongoDB (notice Collection1 for Table1):
db.Collection1.find({name: 'ABC'})
We can chain count() to get the number of results, pretty() to get a readable result. The results can be further narrowed by adding additional parameters:
db.Collection1.find({name: 'ABC', rollNo: 5})
It's important to notice that these filters are ANDed together, by default. To apply an OR filter, we need to use $or. These filters will be specified depending upon the structure of the document. Ex: for object attribute name for an object school, we need to specify filter like "school.name" = 'AUHS'
We're using here the DOT notation, by trying to access a nested field name of a field school. Also notice that the filters are quoted, without which we'll get syntax errors.
Equality matches on arrays can be performed:
on the entire arrays
based on any element
based on a specific element
more complex matches using operators
In the below query:
db.Collection1.find({name: ['ABC','XYZ']})
MongoDB is going to identify documents by an exact match to an array of one or more values. Now for these types of queries, the order of elements matters, meaning that we will only match documents that have ABC followed by XYZ and those are the only 2 elements of the array name
{name:["ABC","GHI","XYZ"]},
{name:["DEF","ABC","XYZ"]}
In the above document, let's say that we need to get all the documnts where ABC is the first element. So, we'll use the below filter:
db.Schools.find({'name.0': 'ABC' })