Waterline find model where RegExp - sails.js

I'm trying to create a simple search-function in SailsJS (beta), using the waterline ORM for data handling. My function in my controller does the search using the predefined find()-method of waterline.
I wonder, is it possible to send an SQL-regexp pattern to the waterline find()-function to make it fetch all users having a matching REGEXP?
My find-method in my controller contains this (except the regexp functionality, which I'm asking for):
// Replace any spaces with SQL regex wildcards
var regexp = req.body.search.replace(' ', '*');
// Show found users
User.find()
.where({
or: [
{username: { contains: req.body.search }},
{email: { contains: req.body.search }},
{fullName: { regexp: regexp }}
]
}).exec(function(err, users){
if (err){
return res.send(403, 'Failed to find users' + JSON.stringify(err));
}
res.view('userlist', {
users: users
});
});
The fullName attribute is defined as an instance method within the attributes, and is probably not searchable. However, is there a way to make a decent search for names?
This is an extract from my User model:
module.exports = {
attributes: {
firstName: {
type: 'string'
},
lastName: {
type: 'string'
},
fullName: function(){
return this.firstName + ' ' + this.lastName;
},
(...some more attributes, as well as email and username...)
}
}
How do i make the find()-method work with SQL regex? My goal is to make a search for "Hans Eikerol" display "Hans Martin Eikerol" in the result set.

Author Edit:
Thanks for the answer:) However, I managed to find a suitable solution to my problem without having to use regex pattern search. This is my new controller method:
var split = req.body.search.split(' ');
// Assume last piece of search is the lastName, ignore middlenames
var firstName = split[0];
var lastName = split.pop();
// Show found users
User.find()
.where({
or: [
{username: { contains: req.body.search }},
{email: { contains: req.body.search }},
{firstName: { contains: firstName }},
{lastName: { contains: lastName }},
{firstName: {startsWith: firstName}, lastName: lastName}
]
}).exec(function(err, users){
if (err){
return res.send(403, 'Failed to find users' + JSON.stringify(err));
}
res.view('admin/users', {
users: users
});
});

Waterline maps .find() to the underlying find syntax for the database you're connected to. It doesn't support regular expressions because regex is not supported by all of the database adapters. One could argue that since regular expression search comes standard with most mainstream databases these days, that the feature might have a place in Waterline, but in the meantime you can use .query to do a native SQL query. Then you coerce the results back to Waterline models. For example, with sails-mysql:
User.query("[some sql query using RLIKE]", function(err, rows) {
if (err){
return res.send(403, 'Failed to find users' + JSON.stringify(err));
}
res.view('userlist', {
// Map the rows returned by the query to Waterline instances
users: rows.map(function(row){return new User._model(row);});
});
});

thanks a lot for your answer, I have implemented something like this, using waterline on sails:
.find({ $and:[{"userType":"basic"},{"name":{contains:req.params.name}} ] })
.exec(function(err,seller){
if(err){
res.send(err);
}else{
console.log(seller);
res.send(seller)
}
})

Related

Prisma.io + Full Text Search + PostgreSQL: Search is only working with exact match

I have enabled full text search for prisma and I would like to search the email field returning all entries that match.
I got the following code:
const data = await this.prismaService.merchant.findMany({
where: {
email: {
search: '12rwqg13tr222vqfgedvqrw22#someprovider.de',
},
},
});
This is working when I enter the exact email address. However, when I try to search for a part of it, i.e. 12rwqg13tr222vqfgedvqrw22#someprovider, I get no results.
Do I have to create indexes to accomplish this? In the docs it is mentioned that I only need indexes for PostgreSQL if I want to speed up the queries. Am I missing something here?
apparently I was looking at the wrong feature. contains is what I was looking for:
const res = await prisma.post.findMany({
where: {
author: {
email: {
contains: 'prisma.io',
},
},
},
})
Edit: If you need case insensitive search, have a look at the prisma docs for case sensitivity
const res = await prisma.post.findMany({
where: {
author: {
email: {
contains: 'prisma.io',
mode: 'insensitive',
},
},
},
})
This would match Prisma.IO as well.

How to use azure cognitive search ? Can we use for global filteron table with pagination and sorting?

I am trying to return all the records that match my searchtext.So far I have only seen examples where we need to specify field name but I want return records if any of the field contains the searchtext, without specifying any field name. And I got to see $text , but unfortunatly it's not supported in cosmosdb API mongodb.
Can someone please help me to resolve this issue ?
Here is what I tried but failed
let querySpec = {
entity: "project",
$text: { $search: "\"test\"" } ,
$or: [{
accessType: "Private",
userName: userName
}, {
accessType: "Public"
}]
}
dbHandler.findandOrder(querySpec, sortfilter, "project").then(function (response) {
res.status(200).json({
status: 'success',
data: utils.unescapeList(response),
totalRecords:response.length
});
exports.findandOrder = function (filterObject, sortfilter, collectionname) {
return new Promise((resolve, reject) => {
return getConnection().then((db) => {
if (db == null) {
console.log("db in findandOrder() is undefined");
} else {
db.db(config.mongodb.dbname).collection(collectionname).find(filterObject).sort(sortfilter).toArray((err, res) => {
if (db) {
//db.close();
}
if (err) {
reject(err);
} else {
resolve(res);
}
});
}
});
});
};
Error:
{"message":{"ok":0,"code":115,"codeName":"CommandNotSupported","name":"MongoError"}}
I am using $regex as temperory solution as $text is not supported.
Please suggest ...
From the MongoDB manual, Create a Wildcard Index on All Fields:
db.collection.createIndex( { "$**" : 1 } )
but this is far from an ideal way to index your data. On the same page is this admonition:
Wildcard indexes are not designed to replace workload-based index planning.
In other words, know your data/schema and tailor indices accordingly. There are many caveats to wildcard indices so read the manual page linked above.
Alternately you can concatenate all the text you want to search into a single field (while also maintaining the individual fields) and create a text index on that field.

MongoDB Shell: Is it possible do an update query using a function?

I have a collection with data like this:
[{
_id: 1,
address: '1/23 Fake Street'
},
{
_id: 2,
address: '5/20 Whatever Lane'
},
{
_id: 3,
address: '10 Foo Avenue'
}]
I'd like to perform a Mongo bulk update query, which does the following:
Transforms the address field to lowercase
Creates a new field, 'buildingAddress', which splits an address at the slash (if present, as with the first two items) and uses the text after it to populate the new field
In Node, I'd do something like this:
const cursor = db.items.find({});
for await (const item of cursor) {
try {
await pageMapper(item);
} catch (e) {
console.error(e);
}
}
async function pageMapper(item){
const newAddress = item.address.toLowerCase()
const buildingAddress = newAddress.split('/ ')[1];
return db.items.updateOne(item._id, {
$set: {
address: newAddress,
buildingAddress
}
})
}
I'm wondering if there's a way to do this in the MongoDB shell itself, passing in a function to db.collection.update? Or should I stick to the node driver for doing more complex update operations?
If you are using MongoDB 4.2+, you can use aggregation or the pipeline form of update to accomplish that.
$toLower converts a string to lower case
$split to split the field
$slice or $arrayElemAt to pick the element(s) to keep
One possible way to do that with update:
db.items.updateMany({},[
{$addFields:{
address:{$toLower:"$address"}
}},
{$addFields:{
buildingAddress:{
$arrayElemAt:[
{$split:["$address","/"]},
-1
]
}
}}
])

MongoDB Full and Partial Text Search

Env:
MongoDB (3.2.0) with Mongoose
Collection:
users
Text Index creation:
BasicDBObject keys = new BasicDBObject();
keys.put("name","text");
BasicDBObject options = new BasicDBObject();
options.put("name", "userTextSearch");
options.put("unique", Boolean.FALSE);
options.put("background", Boolean.TRUE);
userCollection.createIndex(keys, options); // using MongoTemplate
Document:
{"name":"LEONEL"}
Queries:
db.users.find( { "$text" : { "$search" : "LEONEL" } } ) => FOUND
db.users.find( { "$text" : { "$search" : "leonel" } } ) => FOUND (search caseSensitive is false)
db.users.find( { "$text" : { "$search" : "LEONÉL" } } ) => FOUND (search with diacriticSensitive is false)
db.users.find( { "$text" : { "$search" : "LEONE" } } ) => FOUND (Partial search)
db.users.find( { "$text" : { "$search" : "LEO" } } ) => NOT FOUND (Partial search)
db.users.find( { "$text" : { "$search" : "L" } } ) => NOT FOUND (Partial search)
Any idea why I get 0 results using as query "LEO" or "L"?
Regex with Text Index Search is not allowed.
db.getCollection('users')
.find( { "$text" : { "$search" : "/LEO/i",
"$caseSensitive": false,
"$diacriticSensitive": false }} )
.count() // 0 results
db.getCollection('users')
.find( { "$text" : { "$search" : "LEO",
"$caseSensitive": false,
"$diacriticSensitive": false }} )
.count() // 0 results
MongoDB Documentation:
Text Search
$text
Text Indexes
Improve Text Indexes to support partial word match
As at MongoDB 3.4, the text search feature is designed to support case-insensitive searches on text content with language-specific rules for stopwords and stemming. Stemming rules for supported languages are based on standard algorithms which generally handle common verbs and nouns but are unaware of proper nouns.
There is no explicit support for partial or fuzzy matches, but terms that stem to a similar result may appear to be working as such. For example: "taste", "tastes", and tasteful" all stem to "tast". Try the Snowball Stemming Demo page to experiment with more words and stemming algorithms.
Your results that match are all variations on the same word "LEONEL", and vary only by case and diacritic. Unless "LEONEL" can be stemmed to something shorter by the rules of your selected language, these are the only type of variations that will match.
If you want to do efficient partial matches you'll need to take a different approach. For some helpful ideas see:
Efficient Techniques for Fuzzy and Partial matching in MongoDB by John Page
Efficient Partial Keyword Searches by James Tan
There is a relevant improvement request you can watch/upvote in the MongoDB issue tracker: SERVER-15090: Improve Text Indexes to support partial word match.
As Mongo currently does not supports partial search by default...
I created a simple static method.
import mongoose from 'mongoose'
const PostSchema = new mongoose.Schema({
title: { type: String, default: '', trim: true },
body: { type: String, default: '', trim: true },
});
PostSchema.index({ title: "text", body: "text",},
{ weights: { title: 5, body: 3, } })
PostSchema.statics = {
searchPartial: function(q, callback) {
return this.find({
$or: [
{ "title": new RegExp(q, "gi") },
{ "body": new RegExp(q, "gi") },
]
}, callback);
},
searchFull: function (q, callback) {
return this.find({
$text: { $search: q, $caseSensitive: false }
}, callback)
},
search: function(q, callback) {
this.searchFull(q, (err, data) => {
if (err) return callback(err, data);
if (!err && data.length) return callback(err, data);
if (!err && data.length === 0) return this.searchPartial(q, callback);
});
},
}
export default mongoose.models.Post || mongoose.model('Post', PostSchema)
How to use:
import Post from '../models/post'
Post.search('Firs', function(err, data) {
console.log(data);
})
Without creating index, we could simply use:
db.users.find({ name: /<full_or_partial_text>/i}) (case insensitive)
If you want to use all the benefits of MongoDB's full-text search AND want partial matches (maybe for auto-complete), the n-gram based approach mentioned by Shrikant Prabhu was the right solution for me. Obviously your mileage may vary, and this might not be practical when indexing huge documents.
In my case I mainly needed the partial matches to work for just the title field (and a few other short fields) of my documents.
I used an edge n-gram approach. What does that mean? In short, you turn a string like "Mississippi River" into a string like "Mis Miss Missi Missis Mississ Mississi Mississip Mississipp Mississippi Riv Rive River".
Inspired by this code by Liu Gen, I came up with this method:
function createEdgeNGrams(str) {
if (str && str.length > 3) {
const minGram = 3
const maxGram = str.length
return str.split(" ").reduce((ngrams, token) => {
if (token.length > minGram) {
for (let i = minGram; i <= maxGram && i <= token.length; ++i) {
ngrams = [...ngrams, token.substr(0, i)]
}
} else {
ngrams = [...ngrams, token]
}
return ngrams
}, []).join(" ")
}
return str
}
let res = createEdgeNGrams("Mississippi River")
console.log(res)
Now to make use of this in Mongo, I add a searchTitle field to my documents and set its value by converting the actual title field into edge n-grams with the above function. I also create a "text" index for the searchTitle field.
I then exclude the searchTitle field from my search results by using a projection:
db.collection('my-collection')
.find({ $text: { $search: mySearchTerm } }, { projection: { searchTitle: 0 } })
I wrapped #Ricardo Canelas' answer in a mongoose plugin here on npm
Two changes made:
- Uses promises
- Search on any field with type String
Here's the important source code:
// mongoose-partial-full-search
module.exports = exports = function addPartialFullSearch(schema, options) {
schema.statics = {
...schema.statics,
makePartialSearchQueries: function (q) {
if (!q) return {};
const $or = Object.entries(this.schema.paths).reduce((queries, [path, val]) => {
val.instance == "String" &&
queries.push({
[path]: new RegExp(q, "gi")
});
return queries;
}, []);
return { $or }
},
searchPartial: function (q, opts) {
return this.find(this.makePartialSearchQueries(q), opts);
},
searchFull: function (q, opts) {
return this.find({
$text: {
$search: q
}
}, opts);
},
search: function (q, opts) {
return this.searchFull(q, opts).then(data => {
return data.length ? data : this.searchPartial(q, opts);
});
}
}
}
exports.version = require('../package').version;
Usage
// PostSchema.js
import addPartialFullSearch from 'mongoose-partial-full-search';
PostSchema.plugin(addPartialFullSearch);
// some other file.js
import Post from '../wherever/models/post'
Post.search('Firs').then(data => console.log(data);)
If you are using a variable to store the string or value to be searched:
It will work with the Regex, as:
{ collection.find({ name of Mongodb field: new RegExp(variable_name, 'i') }
Here, the I is for the ignore-case option
The quick and dirty solution, that worked for me: use text search first, if nothing is found, then make another query with a regexp. In case you don't want to make two queries - $or works too, but requires all fields in query to be indexed.
Also, you'd better not to use case-insensitive rx, because it can't rely on indexes. In my case I've made lowercase copies of used fields.
Good n-gram based approach for fuzzy matching is explained here
(Also explains how to score higher for Results using prefix Matching)
https://medium.com/xeneta/fuzzy-search-with-mongodb-and-python-57103928ee5d
Note : n-gram based approaches can be storage extensive and mongodb collection size will increase.
I create an additional field which combines all the fields within a document that I want to search. Then I just use regex:
user = {
firstName: 'Bob',
lastName: 'Smith',
address: {
street: 'First Ave',
city: 'New York City',
}
notes: 'Bob knows Mary'
}
// add combined search field with '+' separator to preserve spaces
user.searchString = `${user.firstName}+${user.lastName}+${user.address.street}+${user.address.city}+${user.notes}`
db.users.find({searchString: {$regex: 'mar', $options: 'i'}})
// returns Bob because 'mar' matches his notes field
// TODO write a client-side function to highlight the matching fragments
full/partial search in MongodB for a "pure" Meteor-project
I adpated flash's code to use it with Meteor-Collections and simpleSchema but without mongoose (means: remove the use of .plugin()-method and schema.path (altough that looks to be a simpleSchema-attribute in flash's code, it did not resolve for me)) and returing the result array instead of a cursor.
Thought that this might help someone, so I share it.
export function partialFullTextSearch(meteorCollection, searchString) {
// builds an "or"-mongoDB-query for all fields with type "String" with a regEx as search parameter
const makePartialSearchQueries = () => {
if (!searchString) return {};
const $or = Object.entries(meteorCollection.simpleSchema().schema())
.reduce((queries, [name, def]) => {
def.type.definitions.some(t => t.type === String) &&
queries.push({[name]: new RegExp(searchString, "gi")});
return queries
}, []);
return {$or}
};
// returns a promise with result as array
const searchPartial = () => meteorCollection.rawCollection()
.find(makePartialSearchQueries(searchString)).toArray();
// returns a promise with result as array
const searchFull = () => meteorCollection.rawCollection()
.find({$text: {$search: searchString}}).toArray();
return searchFull().then(result => {
if (result.length === 0) throw null
else return result
}).catch(() => searchPartial());
}
This returns a Promise, so call it like this (i.e. as a return of a async Meteor-Method searchContact on serverside).
It implies that you attached a simpleSchema to your collection before calling this method.
return partialFullTextSearch(Contacts, searchString).then(result => result);
import re
db.collection.find({"$or": [{"your field name": re.compile(text, re.IGNORECASE)},{"your field name": re.compile(text, re.IGNORECASE)}]})

Is it possible to extract only embedded documents from a Model in Mongoose?

I'd like to run a query on a Model, but only return embedded documents where the query matches. Consider the following...
var EventSchema = new mongoose.Schema({
typ : { type: String },
meta : { type: String }
});
var DaySchema = new mongoose.Schema({
uid: mongoose.Schema.ObjectId,
events: [EventSchema],
dateR: { type: Date, 'default': Date.now }
});
function getem() {
DayModel.find({events.typ : 'magic'}, function(err, days) {
// magic. ideally this would return a list of events rather then days
});
}
That find operation will return a list of DayModel documents. But what I'd really like is a list of EventSchemas alone. Is this possible?
It's not possible to fetch the Event objects directly, but you can restrict which fields your query returns like this:
DayModel.find({events.typ : 'magic'}, ['events'], function(err, days) {
...
});
You will still need to loop through the results to extract the actual embedded fields from the documents returned by the query, however.