ElasticsearchRestTemplate can't save multiple documents to different index - spring-data

I have a test domain class
public class TestDocument {
private final String id;
private final String strField;
private final Integer intField;
public TestDocument(final String id, final String strField, final Integer intField) {
this.id = id;
this.strField = strField;
this.intField = intField;
}
}
now I invoke ElasticsearchRestTemplate.save method with 3 documents and want to save into 3 different indices.
#Service
public class TestEsService {
#Autowired
private ElasticsearchRestTemplate elasticsearchRestTemplate;
#PostConstruct
public void testSave() {
final TestDocument d1 = new TestDocument("id_1", "str1", 1);
final TestDocument d2 = new TestDocument("id_2", "str2", 2);
final TestDocument d3 = new TestDocument("id_3", "str3", 3);
this.save(List.of(d1, d2, d3));
}
public void save(final List<TestDocument> documents) {
final IndexCoordinates indexCoordinates = IndexCoordinates.of("index_1", "index_2", "index_3");
this.elasticsearchRestTemplate.save(documents, indexCoordinates);
}
}
After executed above code. I check my local elasticsearch.
curl -H 'Content-Type: application/json' 'http://localhost:9200/_cat/indices?pretty' -s
I got only one index in my ES.
yellow open index_1 17ppJ9vJRUGIVHYBKKxXog 1 1 3 0 5.5kb 5.5kb
and check the data of this index_1 index:
curl 'http://localhost:9200/index_1/_search?pretty'
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "index_1",
"_type" : "_doc",
"_id" : "id_1",
"_score" : 1.0,
"_source" : {
"_class" : "com.test.entity.TestDocument",
"id" : "id_1",
"strField" : "str1",
"intField" : 1
}
},
{
"_index" : "index_1",
"_type" : "_doc",
"_id" : "id_2",
"_score" : 1.0,
"_source" : {
"_class" : "com.test.entity.TestDocument",
"id" : "id_2",
"strField" : "str2",
"intField" : 2
}
},
{
"_index" : "index_1",
"_type" : "_doc",
"_id" : "id_3",
"_score" : 1.0,
"_source" : {
"_class" : "com.test.entity.TestDocument",
"id" : "id_3",
"strField" : "str3",
"intField" : 3
}
}
]
}
}
after dive into the code:
I found a clue within RequestFactory.bulkRequest:
queries.forEach(query -> {
if (query instanceof IndexQuery) {
bulkRequest.add(indexRequest((IndexQuery) query, index));
} else if (query instanceof UpdateQuery) {
bulkRequest.add(updateRequest((UpdateQuery) query, index));
}
});
actually IndexRequest() gets index name via index.getIndexName(); method:
public IndexRequest indexRequest(IndexQuery query, IndexCoordinates index) {
String indexName = index.getIndexName();
IndexRequest indexRequest;
where IndexCoordinates.getIndexName() return the first index name only.
public String getIndexName() {
return indexNames[0];
}
Is it a bug? Should I report to spring-data-elasticsearch Github issue?

Multiple names in IndexCoordinates are used when accessing an Elasticsearch API that uses multiple index names, for example when searching data in multiple indices, but not for writing access.
If you want to save the 3 entities to 3 indices, you need 3 calls with different IndexCoordinates - each of these having one index name.

Related

Get only nested Object from MongoDB by Spring Data mongoDb or mongo template

{
"_id" : "a9582f59-f52b-4fc8-84ab-cdd0bfb8dead",
"_class" : "com.db.Category",
"name" : "Cricket",
"subCategories" : [
{
"name" : "Gloves",
"creationDate" : NumberLong("1527404341099"),
"modificationDate" : NumberLong("1527404341099")
},
{
"name" : "Stumps",
"creationDate" : NumberLong("1527404369882"),
"modificationDate" : NumberLong("1527404369882")
},
{
"name" : "Bat",
"brandList" : [
{
"name" : "MRF",
"productDetails" : [
{
"name" : "Bat 111",
"price" : "1224",
"imageUrlList" : [
"https://s3.us-east-1.amazonaws.com/gasports/1527792222680-Bat_111",
"https://s3.us-east-1.amazonaws.com/gasports/1527792228375-Bat_111"
]
}
]
}
],
"creationDate" : NumberLong("1527424021629"),
"modificationDate" : NumberLong("1527424021629")
}
],
"creationDate" : NumberLong("1527404340938"),
"modificationDate" : NumberLong("1527404340938")
}
This is Category Document. Category has subcategory,Subcategory has brands and Brand has products.
#Document(collection="productInfo")
public class Category extends BaseProductInfo<Category> {
#Id
private String uid;
private String name;
private List<SubCategory> subCategories;
//Getters ans setters..
Now I have to get only for specific subcategory or Brand or Product. For this currently I am doing iteration to get object.
Is there any way in Spring Data Mongo Repository to get nested Object ?
You can write a method like this in your Category Repository class
Category findBySubCategories_BrandList_ProductDetails_Name(String name)
Just pass the product name you want to fetch. spring-data-mongodb will formulate the query based on your meethod name and fetch the documents matching the query.

Project BsonDocument without querying a collection

How can I project one BsonDocument to new instance without querying a collection?
inputs: document: BsonDocument, fields: new string[] { "_id", "meta.name", "type" }
output: BsonDocument with only the above elements populated
Itch scratched
input
{
"_id" : ObjectId("58b454f40960a1788ef48ebc"),
"schema" : {
"id" : "asset",
"version" : {
"major" : 1,
"minor" : 0
}
},
"type" : "asset",
"meta" : {
"name" : "Most Amazing Product",
"type" : null,
"legacy" : {
"url" : "https://s3.amazonaws.com/bucket_name/guid"
}
},
"content" : {
"is_s3" : true,
"s3_bucket" : "bucket_name",
"s3_key" : "guid.doc",
"url" : "https://s3.amazonaws.com/guid.doc"
},
"modified-date" : ISODate("2017-08-09T15:25:57.972Z"),
"modified-by" : "api"
}
code
nuget: MongoDB.Driver 2.4.4
using MongoDB.Bson;
using MongoDB.Bson.Serialization;
using MongoDB.Bson.IO;
BsonDocument original = BsonDocument.Parse(#"{ ""_id"" : ObjectId(""58b454f40960a1788ef48ebc""), ""schema"" : { ""id"" : ""asset"", ""version"" : { ""major"" : 1, ""minor"" : 0 } }, ""type"" : ""asset"", ""meta"" : { ""name"" : ""Most Amazing Product"", ""type"" : null, ""legacy"" : { ""url"" : ""https://s3.amazonaws.com/bucket_name/guid"" } }, ""content"" : { ""is_s3"" : true, ""s3_bucket"" : ""bucket_name"", ""s3_key"" : ""guid.doc"", ""url"" : ""https://s3.amazonaws.com/guid.doc"" }, ""modified-date"" : ISODate(""2017-08-09T15:25:57.972Z""), ""modified-by"" : ""api"" }");
string[] fields = new[] { "_id", "meta.name", "type" };
BsonDocument projection = new BsonDocument();
foreach (var fieldName in fields)
{
BsonDocument source = original;
BsonDocument target = projection;
string[] parts = fieldName.Split(new[] { "." }, StringSplitOptions.RemoveEmptyEntries);
for (int i = 0; i < parts.Length; i++)
{
string currentName = parts[i];
if (i == parts.Length - 1)
{
if(source.Contains(currentName))
target[currentName] = source[currentName];
}
else
{
// Does the source have a current property at this level
if (source.Contains(currentName))
{
// first time this has been visited on target
if (target.Contains(currentName) == false)
{
target.Add(currentName, new BsonDocument());
}
source = source[currentName] as BsonDocument;
target = target[currentName] as BsonDocument;
}
else
{
// no need to go any further if the source doesn't have the property specified
break;
}
}
}
}
result
{
"_id" : ObjectId("58b454f40960a1788ef48ebc"),
"meta" : {
"name" : "Most Amazing Product"
},
"type" : "asset"
}

Spring data and mongoDB - aggregation with java list

I have the following document (#Document):
#Id
private String id;
private String fileName;
private String projectId;
private List<DocumentFileVersion> documentFileVersions;
private List<String> userIdBlackList; // here userIds are included
and this is my current aggregation:
final String userId = "5589929b887dc1fdb501cdbb";
final Aggregation aggregate = newAggregation(match(Criteria.where("projectId").in(projectId)),
group("fileName").count().as("amountOfDocumentFiles"));
final AggregationResults<DocumentFileAmount> groupDocumentFiles = mongoTemplate.aggregate(aggregate, DocumentFile.class,
DocumentFileAmount.class);
final List<DocumentFileAmount> documentFileAmounts = groupDocumentFiles.getMappedResults();
final int amountOfDocumentFiles = documentFileAmounts.size();
Now I will extend the aggreagation in that way that I only will have the DocumentFiles where userId (in this case "1234") is not in userIdBlackList.
Is there a possibility to do that, like in pseudocode:
final Aggregation aggregate = newAggregation(match(Criteria.where("projectId").in(projectId).and(userId).notInList("userIdBlackList")),
group("fileName").count().as("amountOfDocumentFiles"));
I would need something like this: ... .and(userId).notInList("userIdBlackList") ...
[EDIT]
I have tried this query:
final Aggregation aggregate = newAggregation(
match(Criteria.where("projectId").in(projectId).and(userId).and("‌​userIdBlackList").ne(userId)),
group("fileName").count().as("amountOfDocumentFiles"));
A Database entry can look like this:
{
"_id" : ObjectId("587e7cabafdaff28743f3034"),
"_class" : "com.smartinnotec.legalprojectmanagement.dao.domain.DocumentFile",
"fileName" : "Hydrangeas.jpg",
"projectId" : "587e7c95afdaff28743f302e",
"userIdBlackList" : [
"5589929b887dc1fdb501cdbb"
]
}
but .and(userId).and("‌​userIdBlackList").ne(userId) has no effect.
[EDIT2]
I have tried to simulate it in the mongo console too.
I have listed all documentfiles with the command db.DocumentFile.find().pretty():
db.DocumentFile.find().pretty()
{
"_id" : ObjectId("587f0d61473c92b933a68efa"),
"_class" : "com.smartinnotec.legalprojectmanagement.dao.domain.DocumentFile",
"fileName" : "DocumentFile1",
"ending" : "jpg",
"projectId" : "587f0d61473c92b933a68ef9",
"active" : true,
"userIdBlackList" : [
"587f0d61473c92b933a68ef8"
]}
and my query looks like this:
db.DocumentFile.aggregate({ "$match" : { "projectId" : { "$in" : [ "587f0d61473c92b933a68ef9"]} , "‌​userIdBlackList" : { "$ne" : "587f0d61473c92b933a68ef8"}}}).pretty();
{
"_id" : ObjectId("587f0d61473c92b933a68efa"),
"_class" : "com.smartinnotec.legalprojectmanagement.dao.domain.DocumentFile",
"fileName" : "DocumentFile1",
"ending" : "jpg",
"projectId" : "587f0d61473c92b933a68ef9",
"active" : true,
"userIdBlackList" : [
"587f0d61473c92b933a68ef8"
]}
I have expected that I do not get a documentfile because of this expression "‌​userIdBlackList" : { "$ne" : "587f0d61473c92b933a68ef8"}
Does anyone know what I am doing wrong?
[EDIT3]
I have this two documents and with the aggegate:
final Aggregation aggregate = newAggregation(
match(Criteria.where("projectId").in(projectId).and("‌​userIdBlackList").nin(userId)),
group("fileName").count().as("amountOfDocumentFiles"));
I get the amount of 2 but it should 1. I don't know what I am doing wrong?
db.DocumentFile.find().pretty()
{
"_id" : ObjectId("587f2228e232342f74b166f9"),
"_class" : "com.smartinnotec.legalprojectmanagement.dao.domain.DocumentFile",
"fileName" : "DocumentFile1",
"ending" : "jpg",
"projectId" : "587f2228e232342f74b166f8",
"active" : true,
"userIdBlackList" : [
"587f2228e232342f74b166f7"
]}
{
"_id" : ObjectId("587f2228e232342f74b166fa"),
"_class" : "com.smartinnotec.legalprojectmanagement.dao.domain.DocumentFile",
"fileName" : "DocumentFile2",
"ending" : "jpg",
"projectId" : "587f2228e232342f74b166f8",
"active" : true,
"userIdBlackList" : [ ]
}
Have you tried using .nin
final Aggregation aggregate = newAggregation(
match(Criteria.where("projectId").in(projectId).and("‌​userIdBlackList").nin(userId)),
group("fileName").count().as("amountOfDocumentFiles"));

Morphia mapping with my JAVA Class

I am trying to do a simple findOne() using morphia. my code goes as follows:
public static void main(String[] args)
{
MongoClient client = new MongoClient();
Morphia morphia = new Morphia();
morphia.map(Restaurant_M.class);
Datastore ds = morphia.createDatastore(client, "test");
System.out.println(ds.find(Restaurant_M.class).get());
client.close();
}
I get a null printed out. I am unable to find whats going wrong. Can someone point me in the right direction? Thanks.
EDIT
Collection format
{
"_id" : ObjectId("572eb5df1d739cc73c21f953"),
"address" : {
"building" : "469",
"coord" : [
-73.961704,
40.662942
],
"street" : "Flatbush Avenue",
"zipcode" : "11225"
},
"borough" : "Brooklyn",
"cuisine" : "Hamburgers",
"grades" : [
{
"date" : ISODate("2014-12-30T00:00:00Z"),
"grade" : "A",
"score" : 8
},
{
"date" : ISODate("2014-07-01T00:00:00Z"),
"grade" : "B",
"score" : 23
},
{
"date" : ISODate("2013-04-30T00:00:00Z"),
"grade" : "A",
"score" : 12
},
{
"date" : ISODate("2012-05-08T00:00:00Z"),
"grade" : "A",
"score" : 12
}
],
"name" : "Wendy'S",
"restaurant_id" : "30112340"
}
#Entity class
#Entity("restaurants")
public class Restaurant_M
{
#Id
public ObjectId _id;
#Property("borough")
public String town;
public String cuisine;
public String name;
#Property("restaurant_id")
public String r_id;
The Problem is that you don't give all information needed about the entity
You Can Just Use to map to entity :
dt.getCollection(Restaurant_M.class); then use DBObject
Or
dt.createQuery(Restaurant_M.class).field("").equal("to specify").get;
i made a DBO implementation as follows:
public Restaurant_M getByID (String id)
{
Query<Restaurant_M> query = createQuery().field("cuisine").equal(id);
return query.get();
}
and updated the main method as follows and it worked
public static void main(String[] args)
{
MongoClient client = new MongoClient("127.0.0.1:27017");
Morphia morphia = new Morphia();
Datastore ds = morphia.createDatastore(client, "test");
RestaurantDAO rdao = new RestaurantDAOImpl(Restaurant_M.class, ds);
Restaurant_M r = rdao.getByID("Hamburgers");
System.out.println(r);
r = ds.find(Restaurant_M.class).get();
System.out.println (r);
}

MongoDB: how to find latest comment in collection for items particular user commented on

I have a MongoDB of gocomics comments. Sample comment (from
"db.comments.find().pretty()"):
{
"_id" : ObjectId("518f14e5394594efbe18068c"),
"body" : "plan for it",
"commentid" : "2525923",
"commentor" : "Chocoloop",
"commentorid" : "769338",
"da" : "25",
"filename" : "/mnt/sshfs/gocomics/comments/100.out.bz2",
"mo" : "11",
"strip" : "luann",
"stripname" : "Luann",
"time" : "1 day ago",
"yy" : "2011"
}
This shows that "Chocoloop" made the comment "plan for it" on the
2011-11-25 "Luann" strip. The commentid was 2525923, and is unique to
this comment. The other fields aren't relevant for this question.
A single person can make multiple comments on the same strip. For
example, "Chocoloop" may've made a later comment on the 2011-11-25
"Luann" strip. The later comment would have the same strip, da, mo,
yr, commentor fields, but a higher commentid.
I want to find the latest comment I've made on each strip. This is easy:
db.comments.aggregate(
{$match: {commentor:"barrycarter"}},
{$group: {_id: {strip: "$strip", yy: "$yy", da:"$da", mo:"$mo"},
mid: {$max:"$commentid"}}}
)
Here is one of many results:
{
"_id" : {
"strip" : "pearlsbeforeswine",
"yy" : "2007",
"da" : "28",
"mo" : "11"
},
"mid" : "2462203"
}
This says I made at least one comment (perhaps several) on the
pearlsbeforeswine strip dated 2007-11-28. Of the comments I made, the
latest one (the one with the highest commentid) had commentid 2462203
(mid = "max id").
Now, for each result, I want to know: has someone made a comment after
I made my last comment?
For the selected result above, this means: are there any comments for
the pearlsbeforeswine strip dated 2007-11-28 whose commentid exceeds
2462203?
Of course, I can write a query for that one special case:
db.comments.find(
{strip:"pearlsbeforeswine",yy:"2007",da:"28",mo:"11",
commentid: {$gt: "2462203"}}
).pretty()
but how do I do it for all the results in the resultset without
creating an individual query for each one (even automated, that seems
ugly).
Is this a poor use case for MongoDB? I have a similar (not identical)
SQLite3 database where this query is:
SELECT * FROM (SELECT strip,month,date,year,MAX(id) AS mid FROM
comments WHERE commentorid=801127 GROUP BY strip,month,date,year) AS t
JOIN comments c ON (t.strip=c.strip AND t.month=c.month AND
t.date=c.date AND t.year=c.year AND c.id > t.mid)
(where 801127 is my commentorid [the SQLite3 version doesn't include
"commentor" name field]).
NOTE: My MongoDB commentid's are strings, not ints. That's bad, but I
don't think it impacts this question.
You can do it using the aggregation framework and there are multiple approaches towards this. The simplest one is somewhat brute-force and long - it may not have the best performance but I think it's simplest to understand:
proj={"$project": {
"strip" : {"$concat" : ["$strip","-","$yy","/","$mo","/","$da"]},
"commentor" : 1,
"commentid" : 1
}
};
group={"$group": {
"_id" : "$strip",
"comms" : {
"$push" : {
"c" : "$commentor",
"i" : "$commentid"
}
},
"max" : {
"$max" : "$commentid"
}
}};
match = { "$match" : { "comms.c" : "<commentorname>" } };
unwind = { "$unwind" : "$comms" };
proj2 = { "$project" : {
"meLast" : {"$cond" : [
{"$eq" : [
"$max",
"$comms.i"
]
},
1,
0
] }
}
};
group2 = {"$group" : {
"_id" : "$_id",
"afterMe" : {
"$max" : "$meLast"
}
} };
match2 = { "$match" : { "afterMe" : 0 } };
db.comments.aggregate( proj, group, match, unwind, match, proj2, group2, match2 );
Basically, whichever way you do it, there are two {$group} steps in the pipeline that you must have, one to find max commentid for this particular commentor and one for over max commentid for that strip. So it could have been project, group, group, unwind, project with matches as appropriate. Hope you get the idea.
By the way, if you had a unique identifier of each strip (say "comicId") you could then get the list of comics a particular person commented on much simpler, and then you don't need aggregation as much you could just use:
db.comments.distinct("comicId",{commentor:"name"})
which would significantly reduce the number of comments that need to be aggregated. A simpler way to track conversations/replies may be to have comments have "in-reply-to" but then I'm not sure if you are tracking threaded conversations or just straight comments.
I think this is an excellent question and answer to the problem so I decided to solve this using Spring Data with MongoDB in java. To convert Asya's answer into java mongodb code, I did the following:
public void commentTest() {
BasicDBObject o1 = new BasicDBObject();
o1.append("c", "$commentor");
o1.append("i", "$commentid");
Aggregation aggCount = newAggregation(
project("commentid", "commentor")
.andExpression("concat(\"$strip\",\"-\",\"$yy\",\"/\",\"$mo\",\"/\",\"$da\")").as("strip"),
group("strip").push(o1).as("comms").max("commentid").as("max"),
match(Criteria.where("comms.c").is("Simon")),
unwind("comms"),
match(Criteria.where("comms.c").is("Simon")));
logger.info(aggCount.toString());
AggregationResults<CommentTest> groupCount = mongoTemplate.aggregate(aggCount, "commenttest", CommentTest.class);
List<CommentTest> resultCount = groupCount.getMappedResults();
ObjectMapper mapper = new ObjectMapper();
try {
logger.info(mapper.writeValueAsString(resultCount));
} catch (IOException e) {
e.printStackTrace();
}
}
Then to get mongotemplate to parse the results successfully into the CommentTest class, I had to create the class that minicks the results:
Document(collection = "commenttest")
#JsonInclude(JsonInclude.Include.NON_NULL)
public class CommentTest {
private String id, body, commentid, commentor, commentorid, da, filename, mo, strip, stripname, time, yy, max;
#JsonProperty
private comms comms;
public CommentTest.comms getComms() {
return comms;
}
public void setComms(CommentTest.comms comms) {
this.comms = comms;
}
public static class comms implements Serializable {
private String c,i;
public String getC() {
return c;
}
public void setC(String c) {
this.c = c;
}
public String getI() {
return i;
}
public void setI(String i) {
this.i = i;
}
}
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public String getBody() {
return body;
}
public void setBody(String body) {
this.body = body;
}
public String getCommentid() {
return commentid;
}
public void setCommentid(String commentid) {
this.commentid = commentid;
}
public String getCommentor() {
return commentor;
}
public void setCommentor(String commentor) {
this.commentor = commentor;
}
public String getCommentorid() {
return commentorid;
}
public void setCommentorid(String commentorid) {
this.commentorid = commentorid;
}
public String getDa() {
return da;
}
public void setDa(String da) {
this.da = da;
}
public String getFilename() {
return filename;
}
public void setFilename(String filename) {
this.filename = filename;
}
public String getMo() {
return mo;
}
public void setMo(String mo) {
this.mo = mo;
}
public String getStrip() {
return strip;
}
public void setStrip(String strip) {
this.strip = strip;
}
public String getStripname() {
return stripname;
}
public void setStripname(String stripname) {
this.stripname = stripname;
}
public String getTime() {
return time;
}
public void setTime(String time) {
this.time = time;
}
public String getYy() {
return yy;
}
public void setYy(String yy) {
this.yy = yy;
}
public String getMax() {
return max;
}
public void setMax(String max) {
this.max = max;
}
}
I then created some test data in mongodb by inserting these 4 mock entries:
{ "_id" : ObjectId("518f14e5394594efbe18068c"), "body" : "1", "commentid" : "2525923", "commentor" : "Simon", "commentorid" : "769338", "da" : "25", "filename" : "/mnt/sshfs/gocomics/comments/100.out.bz2", "mo" : "11", "strip" : "luann", "stripname" : "Luann", "time" : "1 day ago", "yy" : "2011" }
{ "_id" : ObjectId("518f14e5394594efbe18068d"), "body" : "2", "commentid" : "2525924", "commentor" : "Josh", "commentorid" : "769339", "da" : "25", "filename" : "/mnt/sshfs/gocomics/comments/100.out.bz2", "mo" : "11", "strip" : "luann", "stripname" : "Luann", "time" : "1 day ago", "yy" : "2011" }
{ "_id" : ObjectId("518f14e5394594efbe18068e"), "body" : "3", "commentid" : "2525925", "commentor" : "Peter", "commentorid" : "769340", "da" : "25", "filename" : "/mnt/sshfs/gocomics/comments/100.out.bz2", "mo" : "11", "strip" : "luann", "stripname" : "Luann", "time" : "1 day ago", "yy" : "2011" }
{ "_id" : ObjectId("518f14e5394594efbe18068f"), "body" : "old1", "commentid" : "2525905", "commentor" : "Peter", "commentorid" : "769340", "da" : "24", "filename" : "/mnt/sshfs/gocomics/comments/100.out.bz2", "mo" : "11", "strip" : "luann", "stripname" : "Luann", "time" : "1 day ago", "yy" : "2011" }
I then ran the code and here are the results:
[{"id":"luann-2011/11/25","max":"2525925","comms":{"c":"Simon","i":"2525923"}}]
The result can be interpreted as post luann-2011/11/25 has the maximum comment number (or mongo id) as 2525925 while your comment has an id of 2525923. Therefore there is a later comment after you had comment hence you will need to fetch that new comment. You will need to write your logic for it programmatically.