Elasticsearch mongodb and river indexing fail - mongodb

Dear my (I hope) saviors,
I have a very annoying problem with elasticsearch and mondogdb datas indexing.
The situation is like:
I have 2 mongodb collections, records and pois, that I need to index on elasticsearch, using river plugin (deprecated, I know).
(Record has a reference (DBRef) to poi and other collections, called otherref# here.)
Now, when I execute the curl call, it happens that....Sometimes all records document are indexed, sometimes just 200 (of 140k). Sometimes 900 pois documents are indexed, sometimes just 200 (never all, about 70k).
So, it seems that the script doesn't work properly.
I've monitored the /var/log/elasticsearch log , but no error has been logged.
Here the indexing script:
curl -XPUT "localhost:9200/lw_index_poi" -d '
{
"mappings": {
"lw_mapping_poi" : {
"properties" : {
"position" : {
"type" : "geo_shape"
},
"poi_id" : {
"type" : "string",
"index" : "not_analyzed"
}
}
}
}
}'
curl -XPUT "localhost:9200/lw_index_record" -d '
{
"mappings": {
"lw_mapping_record" : {
"date_detection": false,
"properties" : {
"other_ref1" : {
"type" : "string",
"index" : "not_analyzed"
},
"other_ref2" : {
"type" : "string",
"index" : "not_analyzed"
},
"poi_ref" : {
"type" : "string",
"index" : "not_analyzed"
}
}
}
}
}'
curl -XPUT "localhost:9200/_river/lw_index_poi/_meta" -d '
{
"type": "mongodb",
"mongodb": {
"servers":
[
{ "host": "mongodb", "port": 27017 }
],
"options": {
"secondary_read_preference" : false
},
"db": "lifewatch",
"collection": "poi",
"script": "if (ctx.document.decimalLatitude && ctx.document.decimalLongitude) { ctx.document.position = {}; ctx.document.position.type=\"Point\"; ctx.document.position.coordinates = [ctx.document.decimalLongitude, ctx.document.decimalLatitude]; } ctx.document.poi_id = ctx.document._id; delete ctx.document.decimalLatitude; delete ctx.document.decimalLongitude;"
},
"index": {
"name": "lw_index_poi",
"type": "lw_mapping_poi"
}
}'
curl -XPUT "localhost:9200/_river/lw_index_record/_meta" -d '
{
"type": "mongodb",
"mongodb": {
"servers":
[
{ "host": "mongodb", "port": 27017 }
],
"options": {
"secondary_read_preference" : false
},
"db": "lifewatch",
"collection": "record",
"script": "if (ctx.document.ref1) { ctx.document.ref1 = ctx.document.ref1.id; delete ctx.document.ref1;};if (ctx.document.poi) { ctx.document.poi_ref = ctx.document.poi.id; delete ctx.document.poi;};if (ctx.document.ref2) { ctx.document.ref2 = ctx.document.ref2.id; delete ctx.document.ref2;};"
},
"index": {
"name": "lw_index_record",
"type": "lw_mapping_record"
}
}'
What's wrong?
Thanks in advance

Related

Failed loading the Index Template in Elasticsearch

I am trying to install ELK for logs centralization. I am following steps described in digital ocean website: https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04
I fail to download the filebeat index template using curl from this link:
https://gist.githubusercontent.com/thisismitch/3429023e8438cc25b86c/raw/d8c479e2a1adcea8b1fe86570e42abab0f10f364/filebeat-index-template.json
Can someone give me another source to get it ?
Here is the content of that link:
{
"mappings": {
"_default_": {
"_all": {
"enabled": true,
"norms": {
"enabled": false
}
},
"dynamic_templates": [
{
"template1": {
"mapping": {
"doc_values": true,
"ignore_above": 1024,
"index": "not_analyzed",
"type": "{dynamic_type}"
},
"match": "*"
}
}
],
"properties": {
"#timestamp": {
"type": "date"
},
"message": {
"type": "string",
"index": "analyzed"
},
"offset": {
"type": "long",
"doc_values": "true"
},
"geoip" : {
"type" : "object",
"dynamic": true,
"properties" : {
"location" : { "type" : "geo_point" }
}
}
}
}
},
"settings": {
"index.refresh_interval": "5s"
},
"template": "filebeat-*"
}
Just make a file called filebeat-index-template.json with this command:
touch filebeat-index-template.json
Then open the file in an editor like this:
nano filebeat-index-template.json
Then copy and paste the contents from the link above and save the file (ctrl + x).
After that you should be able to continue through the digital ocean walkthrough at the "load the template" part.

Loopback belongsTo relation with MongoDB giving Error 400

In my MongoDB database, I have this data :
In a User collection :
{
"_id" : ObjectId("5a2c725621691170792fbc3e"),
"username" : "username",
"email" : "test#gmail.com"
}
In a Content Collection :
{
"_id" : ObjectId("5a2c7857b4f9e44a4d618bfd"),
"title" : "Article title",
"author" : ObjectId("5a2c725621691170792fbc3e")
}
So I've made 2 models in Loopback with a relation hasMany/belongsTo :
{
"name": "user",
...
"properties": {
"username": {
"type": "string",
"required": true
},
"email": {
"type": "string",
"required": true
}
},
"relations": {
"contents": {
"type": "hasMany",
"model": "content",
"foreignKey": "author"
}
}
}
Content has an author :
{
"name": "content",
"properties": {
"title": {
"type": "string",
"required": true
}
},
"relations": {
"author": {
"type": "belongsTo",
"model": "user",
"foreignKey": "author"
}
},
}
So that's great, Loopback give me those URLS :
/users/{id}/contents to get Contents of an user, it's working fine
/contents/{id}/author
to get the Author of a content, but this one give me an error :
400 Key mismatch: content.author: function () { [native code] },
user.id: 5a2c725621691170792fbc3e
I can't figure out what's wrong, thanks !
I set up an empty project and added models and relations like described and it works to get contents through user and user through content.
In you example data I see that author in content has a different value than id in user. Is that a typo in the question here or in the actual data?
"_id" : ObjectId("5a2c725621691170792fbc3e")
"author" : ObjectId("5a2c7af721691170792fbfd3")

ElasticSearch river from Mongo messing up field mappings

I'm using Mongo, Elastic Search and this river plugin: https://github.com/richardwilly98/elasticsearch-river-mongodb
I have successfully set everything up in that the river keeps the ES data updated when Mongo is updated, but the river is straight up copying all the properties from the Mongo documents into ES, but I only want a small sub-set of those records. E.g. if a Mongo doc has 30 properties all of them are getting put into ES instead of only the 5 that I want. I assume the issue is with the mappings, and I've followed several docs and another Stack Overflow thread (curl -X POST -d #mapping.json + mapping not created) but it still is not working for me. Here is what I'm doing:
I'm creating my index with:
curl -XPOST "http://localhost:9200/mongoindex" -d #index.json
index.json:
{
"settings" : {
"number_of_shards" : 1
},
"analysis" : {
"analyzer" : {
"str_search_analyzer" : {
"tokenizer" : "keyword",
"filter" : ["lowercase"]
},
"str_index_analyzer" : {
"tokenizer" : "keyword",
"filter" : ["lowercase", "ngram"]
}
},
"filter" : {
"ngram" : {
"type" : "ngram",
"min_gram" : 2,
"max_gram" : 20
}
}
}
}
Then running:
curl -XPOST "http://localhost:9200/mongoindex/listing/_mapping" -d #mapping.json
With this data:
{
"listing":{
"properties":{
"_all": {
"enabled": false
},
"title": {
"type": "string",
"store": false,
"index": "not_analyzed"
},
"bathrooms": {
"type": "integer",
"store": true,
"index": "analyzed"
},
"bedrooms": {
"type": "integer",
"store": true,
"index": "analyzed"
},
"address": {
"type": "nested",
"include_in_parent": true,
"store": true,
"properties": {
"counrty": {
"type":"string"
},
"city": {
"type":"string"
},
"stateOrProvince": {
"type":"string"
},
"fullStreetAddress": {
"type":"string"
},
"postalCode": {
"type":"string"
}
}
},
"location": {
"type": "geo_point",
"full_name": "geometry.coordiantes",
"store": true
}
}
}
}
Then finally creating the river with:
curl -XPUT "http://localhost:9200/_river/mongoindex/_meta" -d #river.json
river.json:
{
"type": "mongodb",
"mongodb": {
"db": "blueprint",
"collection": "Listing",
"options": {
"secondary_read_preference": true,
"drop_collection": true
}
},
"index": {
"name": "mongoindex",
"type": "listing"
}
}
After all that the river works in that ES is populated, but its a verbatim copy of Mongo right now, and I need to modify the mappings, but it just is not taking effect. What am I missing?
This is what my mapping looks like after the river runs.... nothing like what I want it to look like.
I would set dynamic mapping to false:
The dynamic creation of mappings for unmapped types can be completely
disabled by setting index.mapper.dynamic to false.
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-dynamic-mapping.html
Others have had similar issues to yours and it looks like the best solution so far has been to prevent the MongoDB River from dynamically mapping at all:
https://github.com/richardwilly98/elasticsearch-river-mongodb/issues/75
Turns out the issue was that the dynamic property was left out of the mappings config. It should be in 2 places, on the index.json as shown above, and in the mappings.json:
{
"listing":{
"_source": {
"enabled": false
},
"dynamic": false, // <--- Need to add this
"properties":{
"_all": {
"enabled": false
},
"title": {
"type": "string",
"store": false,
"index": "str_index_analyzer"
},
"bathrooms": {
"type": "integer",
"store": true,
"index": "analyzed"
},
"bedrooms": {
"type": "integer",
"store": true,
"index": "analyzed"
},
"address": {
"type": "nested",
"include_in_parent": true,
"store": true,
"properties": {
"counrty": {
"type":"string",
"index": "str_index_analyzer"
},
"city": {
"type":"string",
"index": "str_index_analyzer"
},
"stateOrProvince": {
"type":"string",
"index": "str_index_analyzer"
},
"fullStreetAddress": {
"type":"string",
"index": "str_index_analyzer"
},
"postalCode": {
"type":"string"
}
}
},
"location": {
"type": "geo_point",
"full_name": "geometry.coordiantes",
"store": true
}
}
}
}
The 902 docs vs 451, I think that is an bug in the ElasticSearch Head plugin I'm using to browse documents. It doesn't have duplicates, but a couple of spots show 902 docs as a summary of sorts.

elasticsearch: special behaviour of _id field?

I have some twitter data I want to work with. I want to be able to search for a name. When trying to generate ngrams of the 'name' and '_id' I run into some troubles.
first, I created the analyzers:
curl -XPUT 'localhost:9200/twitter_users' -d '
{
"settings": {
"analysis": {
"analyzer": {
"str_search_analyzer": {
"tokenizer": "keyword",
"filter": [
"lowercase"
]
},
"str_index_analyzer": {
"tokenizer": "keyword",
"filter": [
"lowercase",
"ngram"
]
}
},
"filter": {
"ngram": {
"type": "ngram",
"min_gram": 3,
"max_gram": 20
}
}
}
}
}'
then I defined my mappings:
curl -XPUT 'http://localhost:9200/twitter_users/users/_mapping' -d '
{
"users": {
"type" : "object",
"properties": {
"_id": {
"type": "string",
"copy_to": "id"
},
"id": {
"type": "string",
"search_analyzer": "str_search_analyzer",
"index_analyzer": "str_index_analyzer",
"index": "analyzed"
},
"name": {
"type": "multi_field",
"fields": {
"name": {
"type": "string",
"index": "not_analyzed"
},
"ngrams": {
"type": "string",
"search_analyzer": "str_search_analyzer",
"index_analyzer": "str_index_analyzer",
"index": "analyzed"
}
}
}
}
}
}'
and inserted some test data:
curl -XPUT "localhost:9200/twitter_users/users/johndoe" -d '{
"_id" : "johndoe",
"name" : "John Doe"
}'
curl -XPUT "localhost:9200/twitter_users/users/janedoe" -d '{
"_id" : "janedoe",
"name" : "Jane Doe"
}'
querying by name gets me the expected results:
curl -XPOST "http://localhost:9200/twitter_users/users/_search" -d '{
"query": {
"match": {
"name.ngrams": "doe"
}
}
}'
but querying on the id gives me no results:
curl -XPOST "http://localhost:9200/twitter_users/users/_search" -d '{
"query": {
"match": {
"id": "doe"
}
}
}'
I also tested to make _id a multi field like I did with name. But that didn't work either.
is _id behaving differently than other fields? Or am I doing something wrong here?
edit: using elasticsearch v1.1.2 and pulling the data from mongodb with a river plugin.
Thanks for your Help
Mirko
Looks like the 'copy_to' is the issue, but why not insert the 'id' values into the 'id' fields directly?
curl -XPUT "localhost:9200/twitter_users/users/johndoe" -d '{
"id" : "johndoe",
"name" : "John Doe"
}'
curl -XPUT "localhost:9200/twitter_users/users/janedoe" -d '{
"id" : "janedoe",
"name" : "Jane Doe"
}'

ElasticSearch autocomplete returning 0 hits

I am trying to build an autocomplete feature for our database running on MongoDB. We need to provide autocomplete which lets users complete their queries by offering suggestions while they are typing in the search box.
I have a collection of articles from various sources, which is having the following fields :
{
"title" : "Its the title of a random article",
"cont" : { "paragraphs" : [ .... ] },
and so on..
}
I went through a video by Clinton Gormley. From 37:00 through 42:00 minute, Gormley describes an autocomplete using edgeNGram. Also, I referred to this question to recognize that both are almost the same things, just the mappings differ.
So based on these experiences, I built almost identical settings and mapping and then restored articles collection to ensure that it is indexed by ElasticSearch
The indexing scheme is as follows:
POST /title_autocomplete/title
{
"settings": {
"analysis": {
"filter": {
"autocomplete": {
"type": "edgeNGram",
"min_gram": 2,
"max_gram": 50
}
},
"analyzer": {
"title" : {
"type" : "standard",
"stopwords":[]
},
"autocomplete": {
"type" : "autocomplete",
"tokenizer": "standard",
"filter": ["lowercase", "autocomplete"]
}
}
}
},
"mappings": {
"title": {
"type": "multi_field",
"fields" : {
"title" : {
"type": "string",
"analyzer": "title"
},
"autocomplete" : {
"type": "string",
"index_analyzer": "autocomplete",
"search_analyzer" : "title"
}
}
}
}
}
But when I run the search query, I am unable to get any hits!
GET /title_autocomplete/title/_search
{
"query": {
"bool" : {
"must" : {
"match" : {
"title.autocomplete" : "Its the titl"
}
},
"should" : {
"match" : {
"title" : "Its the titl"
}
}
}
}
}
Can anybody please explain what's wrong with the mapping query or settings? I have been reading ElasticSearch docs for over 7 days now but seem to get nowhere more than full text searches!
ElastiSearch version : 0.90.10
MongoDB version : v2.4.9
using _river
Ubuntu 12.04 64bit
UPDATE
I realised that mapping is screwed after applying previous settings:
GET /title_autocomplete/_mapping
{
"title_autocomplete": {
"title": {
"properties": {
"analysis": {
"properties": {
"analyzer": {
"properties": {
"autocomplete": {
"properties": {
"filter": {
"type": "string"
},
"tokenizer": {
"type": "string"
},
"type": {
"type": "string"
}
}
},
"title": {
"properties": {
"type": {
"type": "string"
}
}
}
}
},
"filter": {
"properties": {
"autocomplete": {
"properties": {
"max_gram": {
"type": "long"
},
"min_gram": {
"type": "long"
},
"type": {
"type": "string"
}
}
}
}
}
}
},
"content": {
... paras and all ...
}
"title": {
"type": "string"
},
"url": {
"type": "string"
}
}
}
}
}
Analyzers and filters are actually mapped into the document after the settings are applied whereas original title field is not affected at all! Is this normal??
I guess this explains why the query is not matching. There is no title.autocomplete field or title.title field at all.
So how should I proceed now?
For those facing this problem, its better to delete the index and start again instead of wasting time with the _river just as DrTech pointed out in the comment.
This saves time but is not a solution. (Therefore not marking it as answer.)
The key is to set up the mappings and index before you initiate the river.
We had an existing setup with a mongodb river and an index called coresearch that we wanted to add autocomplete capacity to, this is the set of commands we used to delete the existing index and river and start again.
Stack is:
ElasticSearch 1.1.1
MongoDB 2.4.9
ElasticSearchMapperAttachments v2.0.0
ElasticSearchRiverMongoDb/2.0.0
Ubuntu 12.04.2 LTS
curl -XDELETE "localhost:9200/_river/node"
curl -XDELETE "localhost:9200/coresearch"
curl -XPUT "localhost:9200/coresearch" -d '
{
"settings": {
"analysis": {
"filter": {
"autocomplete_filter": {
"type": "edge_ngram",
"min_gram": 1,
"max_gram": 20
}
},
"analyzer": {
"autocomplete": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"autocomplete_filter"
]
}
}
}
}
}'
curl -XPUT "localhost:9200/coresearch/_mapping/users" -d '{
"users": {
"properties": {
"firstname": {
"type": "string",
"search_analyzer": "standard",
"index_analyzer": "autocomplete"
},
"lastname": {
"type": "string",
"search_analyzer": "standard",
"index_analyzer": "autocomplete"
},
"username": {
"type": "string",
"search_analyzer": "standard",
"index_analyzer": "autocomplete"
},
"email": {
"type": "string",
"search_analyzer": "standard",
"index_analyzer": "autocomplete"
}
}
}
}'
curl -XPUT "localhost:9200/_river/node/_meta" -d '
{
"type": "mongodb",
"mongodb": {
"servers": [
{ "host": "127.0.0.1", "port": 27017 }
],
"options":{
"exclude_fields": ["time"]
},
"db": "users",
"gridfs": false,
"options": {
"import_all_collections": true
}
},
"index": {
"name": "coresearch",
"type": "documents"
}
}'