Converting Document fields before returning - rest

I'm building a flask project with mongoengine, and I'm struggling to find a good way to convert some fields before returning a document.
I have this as a document:
class Effects(Document):
name = StringField(unique=True, required=True,
min_length=2, max_length=20, validation=name_is_not_taken)
parameters = EmbeddedDocumentListField(Parameter)
parameterKeys = ListField(StringField(required=True, null=True))
created_at = DateTimeField(default=datetime.now)
created_by = StringField(required=True)
effect_link = StringField(required=True)
description = StringField(required=False)
file = FileField(required=False)
When calling a route to get all of those documents (Effect.objects().all), here is what it looks like
{
"_id": {
"$oid": "62138365ae75aceb4c4f3466"
},
"created_at": {
"$date": 1645449589917
},
"effect_link": "put link here",
"name": "gain",
"parameters": {
"Gain": {
"default_value": 0.5,
"range_end": 1.0,
"range_start": 0.0,
"type": "range",
"unit": "float"
}
}
},
My question is, is there a way to consistently convert the _id and created_at from objects to string everytime I return one or more Effects document so the result looks like this:
{
"id": "62138365ae75aceb4c4f3466",
"created_at": "62138365ae75aceb4c4f3466",
"effect_link": "put link here",
"name": "gain",
"parameters": {
"Gain": {
"default_value": 0.5,
"range_end": 1.0,
"range_start": 0.0,
"type": "range",
"unit": "float"
}
}
}
I've seen that you can use aggregate in order to do operations like this but I'm not sure if it is the best way.
Thanks a lot !

If I am not mistaken mongoengine does not supports serialization. I had few applications written in mongoengine. One of them used django so I was using django-rest-framework-mongoengine, it may not be relevant to your project, because you are using flask. Alternative would be to use mongoengine_serialize, it only supports ObjectIds and datetimes, so I think it will be useful for your project.

Related

Which analyzer to use on specific strings?

I have a document in my collection with a property name like this:
name: [{Value: "steel 0.8x1000x2000mm"}]
Now I'm trying to create a search index for it, so far looks like this:
...
"name": {
"fields": {
"Value": [
{
"analyzer": "lucene.finnish",
"searchAnalyzer": "lucene.finnish",
"type": "string"
},
{
"dynamic": true,
"type": "document"
}
]
},
"type": "document"
},
...
And it works pretty fine except for such documents. The issue is that the query 0.8x1000x2000 doesn't match anything, though 0.8x1000x2000mm works fine.
I guess I'm using the wrong analyzer, but can't really figure out which one should I. Or I should make a custom one?

Elastic Search Fuzzy Search root and nested fields

I am new to Elastic Search and facing a couple of issues when querying. I have a simple Mongodb database with collections of cities and places of interest. Each collection has a cityName and other details like website etc, and also a places object array. This is my mapping;
{
"mappings": {
"properties": {
"cityName": {
"type": "text"
},
"phone": {
"type": "keyword"
},
"email": {
"type": "keyword"
},
"website": {
"type": "keyword"
},
"notes": {
"type": "keyword"
},
"status": {
"type": "keyword"
},
"places": {
"type": "nested",
"properties": {
"name": {
"type": "text"
},
"status": {
"type": "keyword"
},
"category": {
"type": "keyword"
},
"reviews": {
"properties": {
"rating": {
"type": "long"
},
"comment": {
"type": "keyword"
},
"user": {
"type": "nested"
}
}
}
}
}
}
}
}
I need a fuzzy query where user can search both cityName and places.name, however I get results when I search a single word, adding multiple words return 0 hits. I am sure I am missing something here because I started learning elastic search 2 days ago. The following query returns results because I have a document with cityName: Islamabad and places array having objects that have the keyword Islamabad in their name, in some places the keyword Islamabad is at the beginning of the place.name and in some places objects it might be in the middle or end
This is what I am using : Returns results when only one word
{
"query": {
"bool": {
"should": [
{
"fuzzy": {
"cityName": "Islamabad"
}
},
{
"nested": {
"path": "places",
"query": {
"fuzzy": {
"places.name": "Islamabad"
}
}
}
}
]
}
}
}
Adding another word, say, club, to the above query returns 0 hits when I actually do have places having names Islamabad club and Islamabad Golf club
Problem
The search query is sent from an app and so it is dynamic, so the term to search is same for both cityName and places.name AND places.name doesn't always have the cityName in it.
What do I need exactly??
I need a query where I can search cityName and the array of places (only searching places.name). The query should be of Fuzzy type so that it still returns results if the word Islamabad is spelled like Islambad or even return results for Islam or Abad. And the query should also return results for multiple words, I am sure am I doing something wrong there. Any help would be appreciated.
**P.S : ** I am actually using MongoDB as my database but migrating to Elastic Search ONLY for improving our search feature. I tried different ways with MongoDB, used the mongoose-fuzzy-searching npm module but that didn't work, so if there's a simpler solution for MongoDB please share that too.
Thanks.
EDIT 1:
I had to change the structure (mapping) of my data. Now I have 2 separate indices, one for cities with city details and a cityId and another index for all places, each place has a cityId which will be used for joining later if needed. Each place also has a cityName key so I will only be searching the places index because it has all the details (place name and city name).
I have a city including the word Welder's in it's name and also the some places inside the same location have the word Welder's in their name, which have a type:text. However when searched for welder both of the following queries don't return these documents, a search for welders OR welder's does return these documents. I am not sure why welder won't match with Welder's*. I didn't specify any analyzer during the creation of both the indices and neither am I explicitly defining it in the query can anyone help me out with this query so it behaves as expected:
Query 1 :
{
"query": {
"bool": {
"should": [
{
"match": {
"name": {
"query": "welder",
"fuzziness": 20
}
}
},
{
"match": {
"cityName": {
"query": "welder",
"fuzziness": 20
}
}
}
]
}
}
}
Query 2 :
{
"query": {
"match": {
"name": {
"query": "welder",
"fuzziness": 20
}
}
}
}
the fuzzy query is meant to be used to find approximations of your complete query within a certain distance :
To find similar terms, the fuzzy query creates a set of all possible
variations, or expansions, of the search term within a specified edit
distance. The query then returns exact matches for each expansion.
If you you cant to allow fuzzy matching of individual terms in your query your need to use a match query with the fuzziness activated.
POST <your_index>/_search
{
"query": {
"bool": {
"should": [
{
"match": {
"cityName": {
"query": "Islamabad golf",
"fuzziness": "AUTO"
}
}
},
{
"nested": {
"path": "places",
"query": {
"match": {
"places.name": {
"query": "Islamabad golf",
"fuzziness": "AUTO"
}
}
}
}
}
]
}
}
}
Reminder: Fuzziness in elasticsearch allow at max 2 corrections per term. SO you will never be able to match Islam with Islamabad since there are 4 changes between those terms.
For more information on distance and fuzziness parameters please refer to this documentation page fuzziness parameters

How to insert server timestamp in Firestore document using REST?

I want to insert a document using a REST call to Firestore createDocument method. One of the fields is a timestamp field that should be set on the server. With Android SDK it's as simple as annotating a Date field with #ServerTimestamp and keeping it null — now how do I do it in REST?
{
"fields": {
"timezoneId": {
"stringValue": "Europe\/London"
},
"city": {
"stringValue": "London"
},
"timestamp": {
"timestampValue": "???"
}
}
}
I tried using null, 0, empty string, timestamp — everything fails with an error requiring the standard RFC3339 format (e.g. 2018-01-31T13:50:30.325631Z). Is there any placeholder value I can use, or any way to obtain that timestamp?
The Android SDK doesn't execute a createDocument request when creating the document. Instead it uses the write request to issue an update and a transform request at the same time. If you are wanting to only use createDocument, then the answer is no.
Your payload would look something like this:
{
"writes": [
{
"update": {
"name": "projects/{projectId}/databases/{databaseId}/documents/{document_path}",
"fields": {
"timezoneId": {
"stringValue": "Europe\/London"
},
"city": {
"stringValue": "London"
}
}
},
// ensure the document doesn't exist
"currentDocument": {
"exists": false
}
},
{
"transform": {
"document": "projects/{projectId}/databases/{databaseId}/documents/{document_path}",
"fieldTransforms": [
{
"fieldPath": "timestamp",
"setToServerValue": "REQUEST_TIME"
}
]
}
}
]
}
The only downside to adding documents this way is you would need to generate the Document ID yourself (the SDK's generate them). I hope this helps.

How to update the Embedded Data which is inside of another Embedded Data?

I have document like below in MongoDB:
{
"_id": "test",
"tasks": [
{
"Name": "Task1",
"Parameter": [
{
"Name": "para1",
"Type": "String",
"Value": "*****"
},
{
"Name": "para2",
"Type": "String",
"Value": "*****"
}
]
},
{
"Name": "Task2",
"Parameter": [
{
"Name": "para1",
"Type": "String",
"Value": "*****"
},
{
"Name": "para2",
"Type": "String",
"Value": "*****"
}
]
}
]
}
There is Embedded Data Structure (Parameter) inside of another Embedded Data Structure (Tasks). Now I want to update the para1 in Task1's Parameter.
I have tried many ways but I can only use query tasks.Parameter.name to find the para1 but cannot update it. the example in the doc are using .$. to update the value in a Embedded Data Structure but it doesn't work in my case.
Anyone have any ideas ?
MongoDB currently only supports the positional operator once, and only for the top level array. There is a ticket SERVER-831 to change this behavior for your use case. You can follow the issue there and up vote it.
However, you might be able to change your approach to accomplish what you want to do. One way is to change your schema. Collapse the tasks name into the array so the document looks like this:
{
_id:test,
tasks:
[
{
Task:1
Name:para1,
Type:String,
Value:*****
},
{
Task:1
Name:para2,
Type:String,
Value:*****
},
{
Task:2
Name:para1,
Type:String,
Value:*****
},
{
Task:2
Name:para2,
Type:String,
Value:*****
}
]
}
Another approach that may work for you is to use $pull and $push. For instance something like this to replace a task (this assumes that tasks.Parameter.Name is unique to an array of Parameters):
db.test2.update({$and: [{"tasks.Name": "Task3"}, {"tasks.Parameter.Name":"para1"}]}, {$pull: {"tasks.$.Parameter": {"Name": "para1"}}})
db.test2.update({"tasks.Name": "Task3"}, {$push: {"tasks.$.Parameter": {"Name": "para3", Type: "String", Value: 1}}})
With this solution you need to be careful with regard to concurrency, as there will be a brief moment where the document doesn't exist.

Remove entry in an array of a MongoDB document

Say I have a document that looks something like this:
{
"_id": ObjectId("50b6a7416cb035b629000001"),
"businesses": [{
"name": "Biz1",
"id": ObjectId("50b6bc953e47dc923e000001")
}, {
"name": "Biz2",
"id": ObjectId("50b6ccebae0513bf52000001")
}, {
"name": "Biz3",
"id": ObjectId("50b6d015c58b414156000001")
}, {
"name": "Biz4",
"id": ObjectId("50b6d0c8a4cdd5e356000001")
}]
}
I want to remove
{
"name": "Biz3",
"id": ObjectId("50b6d015c58b414156000001")
}
from the array of businesses. I tried this (using business name instead of id for clarity):
db.users.update({'businesses.name':'Biz3'},{$pull:{'businesses.name':'Biz3'}})
but of course it didn't work. I know that the query part is correct because I get the document back when I do this:
db.users.find({'businesses.name' : 'Biz3'})
So the problem is with the update part.
Just ran a quick lil test and this works
I think trying db.users.update({'businesses.name':'Biz3'},{$pull:{'businesses':{'name':'Biz3'}}}) should do it