In my MongoDB database, I have this data :
In a User collection :
{
"_id" : ObjectId("5a2c725621691170792fbc3e"),
"username" : "username",
"email" : "test#gmail.com"
}
In a Content Collection :
{
"_id" : ObjectId("5a2c7857b4f9e44a4d618bfd"),
"title" : "Article title",
"author" : ObjectId("5a2c725621691170792fbc3e")
}
So I've made 2 models in Loopback with a relation hasMany/belongsTo :
{
"name": "user",
...
"properties": {
"username": {
"type": "string",
"required": true
},
"email": {
"type": "string",
"required": true
}
},
"relations": {
"contents": {
"type": "hasMany",
"model": "content",
"foreignKey": "author"
}
}
}
Content has an author :
{
"name": "content",
"properties": {
"title": {
"type": "string",
"required": true
}
},
"relations": {
"author": {
"type": "belongsTo",
"model": "user",
"foreignKey": "author"
}
},
}
So that's great, Loopback give me those URLS :
/users/{id}/contents to get Contents of an user, it's working fine
/contents/{id}/author
to get the Author of a content, but this one give me an error :
400 Key mismatch: content.author: function () { [native code] },
user.id: 5a2c725621691170792fbc3e
I can't figure out what's wrong, thanks !
I set up an empty project and added models and relations like described and it works to get contents through user and user through content.
In you example data I see that author in content has a different value than id in user. Is that a typo in the question here or in the actual data?
"_id" : ObjectId("5a2c725621691170792fbc3e")
"author" : ObjectId("5a2c7af721691170792fbfd3")
Related
Hi folks I need to create AVRO schema for the following example ;
{ "Car" : { "Make" : "Ford" , "Year": 1990 , "Engine" : "V8" , "VIN" : "123123123" , "Plate" : "XXTT9O",
"Accident" : { "Date" :"2020/02/02" , "Location" : "NJ" , "Driver" : "Joe" } ,
"Owner" : { "Name" : "Joe" , "LastName" : "Doe" } }
Accident and Owner is optional objects and created schema also needs to validate following subset message;
{ "Car" : { "Make" : "Tesla" , "Year": 2020 , "Engine" : "4ELEC" , "VIN" : "54545426" , "Plate" : "TESLA" }
I read the AVRO specs and see a lot of optional attribute and array examples but none of them worked for the record. How can I define a record as optional ? Thanks.
Following schema without any optional attribute is working.
{
"name": "MyClass", "type": "record", "namespace": "com.acme.avro", "fields": [
{
"name": "Car", "type": {
"name": "Car","type": "record","fields": [
{ "name": "Make", "type": "string" },
{ "name": "Year", "type": "int" },
{ "name": "Engine", "type": "string" },
{ "name": "VIN", "type": "string" },
{ "name": "Plate", "type": "string" },
{ "name": "Accident",
"type":
{ "name": "Accident",
"type": "record",
"fields": [
{ "name": "Date","type": "string" },
{ "name": "Location","type": "string" },
{ "name": "Driver", "type": "string" }
]
}
},
{ "name": "Owner",
"type":
{"name": "Owner",
"type": "record",
"fields": [
{"name": "Name", "type": "string" },
{"name": "LastName", "type": "string" }
]
}
}
]
}
}
]
}
when I change the Owner object as suggested avro-tool is returning error.
{ "name": "Owner",
"type": [
"null",
"record" : {
"name": "Owner",
"fields": [
{"name": "Name", "type": "string" },
{"name": "LastName", "type": "string" }
]
}
] , "default": null }
]
}
}
]
}
Test:
Projects/avro_test$ java -jar avro-tools-1.8.2.jar fromjson --schema-file CarStackOver.avsc Car.json > o2
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" org.apache.avro.SchemaParseException: org.codehaus.jackson.JsonParseException: Unexpected character (':' (code 58)): was expecting comma to separate ARRAY entries
at [Source: org.apache.hadoop.fs.ChecksumFileSystem$FSDataBoundedInputStream#4034c28c; line: 26, column: 13]
at org.apache.avro.Schema$Parser.parse(Schema.java:1034)
at org.apache.avro.Schema$Parser.parse(Schema.java:1004)
at org.apache.avro.tool.Util.parseSchemaFromFS(Util.java:165)
at org.apache.avro.tool.DataFileWriteTool.run(DataFileWriteTool.java:83)
at org.apache.avro.tool.Main.run(Main.java:87)
at org.apache.avro.tool.Main.main(Main.java:76)
Caused by: org.codehaus.jackson.JsonParseException: Unexpected character (':' (code 58)): was expecting comma to separate ARRAY entries
at [Source: org.apache.hadoop.fs.ChecksumFileSystem$FSDataBoundedInputStream#4034c28c; line: 26, column: 13]
at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433)
at org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521)
at org.codehaus.jackson.impl.JsonParserMinimalBase._reportUnexpectedChar(JsonParserMinimalBase.java:442)
at org.codehaus.jackson.impl.Utf8StreamParser.nextToken(Utf8StreamParser.java:482)
at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeArray(JsonNodeDeserializer.java:222)
at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:200)
at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeArray(JsonNodeDeserializer.java:224)
at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:200)
at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:197)
at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeArray(JsonNodeDeserializer.java:224)
at org.codehaus.jackson.map.deser.std.BaseNodeDeserializer.deserializeObject(JsonNodeDeserializer.java:200)
at org.codehaus.jackson.map.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:58)
at org.codehaus.jackson.map.deser.std.JsonNodeDeserializer.deserialize(JsonNodeDeserializer.java:15)
at org.codehaus.jackson.map.ObjectMapper._readValue(ObjectMapper.java:2704)
at org.codehaus.jackson.map.ObjectMapper.readTree(ObjectMapper.java:1344)
at org.apache.avro.Schema$Parser.parse(Schema.java:1032)
You can make it possible for records to be optional by doing a union with null.
Like this:
{
"name": "Owner",
"type": [
"null",
{
"name": "Owner",
"type": "record",
"fields": [
{ "name": "Name", type": "string" },
{ "name": "LastName", type": "string" },
]
}
],
"default": null
},
I am trying to implement a database using MongoDB for my REST API that would be able to store and retrieve documents with specific fields.
I can easily use mongoose but I wish to use MongoDB's native driver, as I want to learn MongoDB rather than mongoose.
{
"$jsonSchema": {
"bsonType": "object",
"required": [
"name",
"email"
],
"properties": {
"name": {
"bsonType": "string"
},
"email": {
"bsonType": "string"
},
"profileImagePath": {
"bsonType": "string"
},
"blogs": {
"bsonType": ["object"]
}
}
}
}
I am expecting to only be able to insert data such as
"name" : "john",
"email" : "john#gmail.com"
Or
"name" : "john",
"email" : "john#gmail.com",
"profileImagePath" : "somePath"
but not
"name" : "john",
"email" : "john#gmail.com",
"height" : "5'11"
As height has not been specified in the properties.
so I've been going through MongoDB's documentation and one can specify whether or not additional properties can be inserted into the database or not by changing the default value of additionalProperties property.
The documentation states
If true, additional fields are allowed. If false, they are not. If a valid JSON Schema object is specified, additional fields must validate against the schema.
Defaults are true.
To avoid any additional properties from being inserted you'd simply have to add the additionalProperties property like so,
{
"$jsonSchema": {
"bsonType": "object",
"required": [
"name",
"email"
],
"properties": {
"name": {
"bsonType": "string"
},
"email": {
"bsonType": "string"
},
"profileImagePath": {
"bsonType": "string"
},
"blogs": {
"bsonType": ["object"]
}
},
"additionalProperties": false
}
}
I am a jolt newbie.
I have the following question, I have a JSON document structure of which may vary based on the type property. See my example below.
{
"recipientId": "xxx",
"messages": [
{
"type": "text",
"text": "hi there!"
},
{
"type": "image",
"url": "http://example.com/image.jpg",
"preview": "http://example.com/thumbnail.jpg"
}
]
}
After transformation I would like to receive the following output:
{
"messages" : [ {
"text" : "hi there!",
"type" : "text"
}, {
"type" : "image",
"url" : "http://example.com/image.jpg",
"preview": "http://example.com/thumbnail.jpg"
} ],
"to" : "xxx"
}
Here is the spec that I came up with:
[
{
"operation": "shift",
"spec": {
"recipientId": "to",
"messages": {
"*": {
"type": "messages[&1].type",
"text": "messages[&1].text",
"url": "messages[&1].url",
"preview": "messages[&1].previewImageUrl"
}
}
}
}
]
The problem with this approach is that if I have "type": "text" and if I also throw "preview" property with the value, it will not make sense as the type text should not have "preview" property set.
So, I would like jolt to either ignore some properties based on the value of "type" property or avoid transforming such payloads.
Is there a way to do such "validations" in JOLT? The other option that I see would be validating it with Jackson type hierarchy.
What you can do is match down to the value of "type", and then jump back up the tree some, and process the message as a "text" type or an "image" type.
Input
{
"recipientId": "xxx",
"messages": [
{
"type": "text",
"text": "hi there!",
"preview": "SHOULD NOT PASS THRU"
},
{
"type": "image",
"url": "http://example.com/image.jpg",
"preview": "http://example.com/thumbnail.jpg"
}
]
}
Spec
[
{
"operation": "shift",
"spec": {
"recipientId": "to",
"messages": {
"*": { // the array index of the messages array, referenced below
// as [&2] or [&4] depending how far down they have gone
"type": {
// always pass the value of type thru
"#": "messages[&2].type",
"text": {
// if the value of type was "text", then
// go back up the tree 3 levels (0,1,2)
// and process the whole message as a "text" type
"#2": {
"text": "messages[&4].text"
}
},
"image": {
"#2": {
// if the value of type was "image", then
// go back up the tree 3 levels (0,1,2)
// and process the whole message as a "image" type
"url": "messages[&4].url",
"preview": "messages[&4].previewImageUrl"
}
}
}
}
}
}
}
]
I've 3 collections in my database under mongo: shows - venues - dropdowns
shows are mapped like below
"show": {
"properties" : {
"description": {
"type": "string"
},
"image": {
"type": "string"
},
"site": {
"type": "string"
},
"title" : {
"type" : "multi_field",
"fields" : {
"title" : {"type" : "string", "index" : "analyzed"},
"raw_title" : {"type" : "string", "index" : "not_analyzed", "store": "no"}
}
}
}
}
venues like this
"venues": {
"properties" : {
"name" : {
"type" : "multi_field",
"fields" : {
"name" : {"type" : "string", "index" : "analyzed"},
"raw_name" : {"type" : "string", "index" : "not_analyzed", "store": "no"}
}
},
"city" : {
"type" : "multi_field",
"fields" : {
"city" : {"type" : "string", "index" : "analyzed"},
"raw_city" : {"type" : "string", "index" : "not_analyzed", "store": "no"}
}
},
"region" : {
"type" : "multi_field",
"fields" : {
"region" : {"type" : "string", "index" : "analyzed"},
"raw_region" : {"type" : "string", "index" : "not_analyzed", "store": "no"}
}
},
"state" : {
"type": "boolean"
}
}
}
and I've this model in mongo for dropdowns:
{
created: {
type: Date,
default: Date.now
},
analytics: {
type: String,
default: '',
trim: true
},
state: {
type: Boolean,
default: false,
index: true
},
show: {
type: Schema.ObjectId,
ref: 'Show'
},
venues:[{
venue:{
type: Schema.ObjectId,
ref: 'Venue',
index: true
},
site: {
type: String,
trim: true,
index: true
}
}]
}
I'd map dropdowns with parent/child schema into my index, but I can't understand if is possibile with ObjectId because I've tried with this mapping:
"dropdown": {
"properties" : {
"state": {
"type": "boolean"
},
"analytics": {
"type": "string"
},
"_parent":{
"type" : "show"
},
"venues" : {
"properties" : {
"venue" : {
"_parent": {
"type" : "venues"
}
}
},
"site" : {"type" : "string"}
}
}
}
But I received this error:
MapperParsingException[No type specified for property [show]]
There is anyway to setting up correctly my index?
Issue is that you're specifying _parent incorrectly. You have to set it not in properties field, but next to it. Please see documentation and example from it:
PUT /company
{
"mappings": {
"branch": {},
"employee": {
"_parent": {
"type": "branch"
}
}
}
}
So following that logic, I've taken your mappings, simplified it a bit and made it work:
PUT /test
{
"mappings": {
"show": {
"properties": {
"description": {
"type": "string"
},
"image": {
"type": "string"
},
"site": {
"type": "string"
},
"title": {
"type": "multi_field",
"fields": {
"title": {
"type": "string",
"index": "analyzed"
},
"raw_title": {
"type": "string",
"index": "not_analyzed",
"store": "no"
}
}
}
}
},
"venues": {
"properties": {
"name": {
"type": "multi_field",
"fields": {
"name": {
"type": "string",
"index": "analyzed"
},
"raw_name": {
"type": "string",
"index": "not_analyzed",
"store": "no"
}
}
},
"city": {
"type": "multi_field",
"fields": {
"city": {
"type": "string",
"index": "analyzed"
},
"raw_city": {
"type": "string",
"index": "not_analyzed",
"store": "no"
}
}
},
"region": {
"type": "multi_field",
"fields": {
"region": {
"type": "string",
"index": "analyzed"
},
"raw_region": {
"type": "string",
"index": "not_analyzed",
"store": "no"
}
}
},
"state": {
"type": "boolean"
}
}
},
"dropdown": {
"_parent": {
"type": "show"
},
"properties": {
"state": {
"type": "boolean"
},
"analytics": {
"type": "string"
},
"venues": {
"type": "object",
"_parent": {
"type": "venues"
},
"site": {
"type": "string"
}
}
}
}
}
}
I've tried this by myself on Elasticsearch 1.7.1 and it worked fine.
However, I'm not sure if you can declare _parent relationship inside nested documents as you did for venues. My mapping query didn't throw an error and accepted it. However, looking on how it got parsed in head plugin - _parent was eliminated and only object part remained as seen in screenshot:
If I tried to index it without specifying type - this error is thrown:
"MapperParsingException[mapping [dropdown]]; nested:
MapperParsingException[No type specified for property [venues]];
I am trying to build an autocomplete feature for our database running on MongoDB. We need to provide autocomplete which lets users complete their queries by offering suggestions while they are typing in the search box.
I have a collection of articles from various sources, which is having the following fields :
{
"title" : "Its the title of a random article",
"cont" : { "paragraphs" : [ .... ] },
and so on..
}
I went through a video by Clinton Gormley. From 37:00 through 42:00 minute, Gormley describes an autocomplete using edgeNGram. Also, I referred to this question to recognize that both are almost the same things, just the mappings differ.
So based on these experiences, I built almost identical settings and mapping and then restored articles collection to ensure that it is indexed by ElasticSearch
The indexing scheme is as follows:
POST /title_autocomplete/title
{
"settings": {
"analysis": {
"filter": {
"autocomplete": {
"type": "edgeNGram",
"min_gram": 2,
"max_gram": 50
}
},
"analyzer": {
"title" : {
"type" : "standard",
"stopwords":[]
},
"autocomplete": {
"type" : "autocomplete",
"tokenizer": "standard",
"filter": ["lowercase", "autocomplete"]
}
}
}
},
"mappings": {
"title": {
"type": "multi_field",
"fields" : {
"title" : {
"type": "string",
"analyzer": "title"
},
"autocomplete" : {
"type": "string",
"index_analyzer": "autocomplete",
"search_analyzer" : "title"
}
}
}
}
}
But when I run the search query, I am unable to get any hits!
GET /title_autocomplete/title/_search
{
"query": {
"bool" : {
"must" : {
"match" : {
"title.autocomplete" : "Its the titl"
}
},
"should" : {
"match" : {
"title" : "Its the titl"
}
}
}
}
}
Can anybody please explain what's wrong with the mapping query or settings? I have been reading ElasticSearch docs for over 7 days now but seem to get nowhere more than full text searches!
ElastiSearch version : 0.90.10
MongoDB version : v2.4.9
using _river
Ubuntu 12.04 64bit
UPDATE
I realised that mapping is screwed after applying previous settings:
GET /title_autocomplete/_mapping
{
"title_autocomplete": {
"title": {
"properties": {
"analysis": {
"properties": {
"analyzer": {
"properties": {
"autocomplete": {
"properties": {
"filter": {
"type": "string"
},
"tokenizer": {
"type": "string"
},
"type": {
"type": "string"
}
}
},
"title": {
"properties": {
"type": {
"type": "string"
}
}
}
}
},
"filter": {
"properties": {
"autocomplete": {
"properties": {
"max_gram": {
"type": "long"
},
"min_gram": {
"type": "long"
},
"type": {
"type": "string"
}
}
}
}
}
}
},
"content": {
... paras and all ...
}
"title": {
"type": "string"
},
"url": {
"type": "string"
}
}
}
}
}
Analyzers and filters are actually mapped into the document after the settings are applied whereas original title field is not affected at all! Is this normal??
I guess this explains why the query is not matching. There is no title.autocomplete field or title.title field at all.
So how should I proceed now?
For those facing this problem, its better to delete the index and start again instead of wasting time with the _river just as DrTech pointed out in the comment.
This saves time but is not a solution. (Therefore not marking it as answer.)
The key is to set up the mappings and index before you initiate the river.
We had an existing setup with a mongodb river and an index called coresearch that we wanted to add autocomplete capacity to, this is the set of commands we used to delete the existing index and river and start again.
Stack is:
ElasticSearch 1.1.1
MongoDB 2.4.9
ElasticSearchMapperAttachments v2.0.0
ElasticSearchRiverMongoDb/2.0.0
Ubuntu 12.04.2 LTS
curl -XDELETE "localhost:9200/_river/node"
curl -XDELETE "localhost:9200/coresearch"
curl -XPUT "localhost:9200/coresearch" -d '
{
"settings": {
"analysis": {
"filter": {
"autocomplete_filter": {
"type": "edge_ngram",
"min_gram": 1,
"max_gram": 20
}
},
"analyzer": {
"autocomplete": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"autocomplete_filter"
]
}
}
}
}
}'
curl -XPUT "localhost:9200/coresearch/_mapping/users" -d '{
"users": {
"properties": {
"firstname": {
"type": "string",
"search_analyzer": "standard",
"index_analyzer": "autocomplete"
},
"lastname": {
"type": "string",
"search_analyzer": "standard",
"index_analyzer": "autocomplete"
},
"username": {
"type": "string",
"search_analyzer": "standard",
"index_analyzer": "autocomplete"
},
"email": {
"type": "string",
"search_analyzer": "standard",
"index_analyzer": "autocomplete"
}
}
}
}'
curl -XPUT "localhost:9200/_river/node/_meta" -d '
{
"type": "mongodb",
"mongodb": {
"servers": [
{ "host": "127.0.0.1", "port": 27017 }
],
"options":{
"exclude_fields": ["time"]
},
"db": "users",
"gridfs": false,
"options": {
"import_all_collections": true
}
},
"index": {
"name": "coresearch",
"type": "documents"
}
}'