I have a test domain class
public class TestDocument {
private final String id;
private final String strField;
private final Integer intField;
public TestDocument(final String id, final String strField, final Integer intField) {
this.id = id;
this.strField = strField;
this.intField = intField;
}
}
now I invoke ElasticsearchRestTemplate.save method with 3 documents and want to save into 3 different indices.
#Service
public class TestEsService {
#Autowired
private ElasticsearchRestTemplate elasticsearchRestTemplate;
#PostConstruct
public void testSave() {
final TestDocument d1 = new TestDocument("id_1", "str1", 1);
final TestDocument d2 = new TestDocument("id_2", "str2", 2);
final TestDocument d3 = new TestDocument("id_3", "str3", 3);
this.save(List.of(d1, d2, d3));
}
public void save(final List<TestDocument> documents) {
final IndexCoordinates indexCoordinates = IndexCoordinates.of("index_1", "index_2", "index_3");
this.elasticsearchRestTemplate.save(documents, indexCoordinates);
}
}
After executed above code. I check my local elasticsearch.
curl -H 'Content-Type: application/json' 'http://localhost:9200/_cat/indices?pretty' -s
I got only one index in my ES.
yellow open index_1 17ppJ9vJRUGIVHYBKKxXog 1 1 3 0 5.5kb 5.5kb
and check the data of this index_1 index:
curl 'http://localhost:9200/index_1/_search?pretty'
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "index_1",
"_type" : "_doc",
"_id" : "id_1",
"_score" : 1.0,
"_source" : {
"_class" : "com.test.entity.TestDocument",
"id" : "id_1",
"strField" : "str1",
"intField" : 1
}
},
{
"_index" : "index_1",
"_type" : "_doc",
"_id" : "id_2",
"_score" : 1.0,
"_source" : {
"_class" : "com.test.entity.TestDocument",
"id" : "id_2",
"strField" : "str2",
"intField" : 2
}
},
{
"_index" : "index_1",
"_type" : "_doc",
"_id" : "id_3",
"_score" : 1.0,
"_source" : {
"_class" : "com.test.entity.TestDocument",
"id" : "id_3",
"strField" : "str3",
"intField" : 3
}
}
]
}
}
after dive into the code:
I found a clue within RequestFactory.bulkRequest:
queries.forEach(query -> {
if (query instanceof IndexQuery) {
bulkRequest.add(indexRequest((IndexQuery) query, index));
} else if (query instanceof UpdateQuery) {
bulkRequest.add(updateRequest((UpdateQuery) query, index));
}
});
actually IndexRequest() gets index name via index.getIndexName(); method:
public IndexRequest indexRequest(IndexQuery query, IndexCoordinates index) {
String indexName = index.getIndexName();
IndexRequest indexRequest;
where IndexCoordinates.getIndexName() return the first index name only.
public String getIndexName() {
return indexNames[0];
}
Is it a bug? Should I report to spring-data-elasticsearch Github issue?
Multiple names in IndexCoordinates are used when accessing an Elasticsearch API that uses multiple index names, for example when searching data in multiple indices, but not for writing access.
If you want to save the 3 entities to 3 indices, you need 3 calls with different IndexCoordinates - each of these having one index name.
I have the following problem - how to query by nested property with #Query?
My Product class (document in Mongo):
#Document(collection = "products")
public class Product {
#Id
private String id;
#DBRef
private ProductProperties properties;
How does it look in Mongo:
{
"_id" : ObjectId("5d5e78d20e8e3d0006079a84"),
"companyId" : "1234",
"properties" : {
"$ref" : "properties",
"$id" : ObjectId("5df8dd2331ea7b4a9384335b")
},
"calendar" : [
{
"startDate" : ISODate("2019-09-04T22:00:00.000Z"),
"endDate" : ISODate("2019-09-09T22:00:00.000Z")
}
],
"_class" : "org.abc.def"
}
ProductProperties class (document in Mongo):
#Document(collection = "product_properties")
public class ProductProperties {
#Id
private String id;
(...)
How does it look in Mongo:
{
"_id" : ObjectId("5df8dd2331ea7b4a9384335b"),
"brand" : "offer Brand_1",
"model" : "offer model_1",
"modelNumber" : "offer model number_1",
"size" : {
...
}
My Spring repository:
public interface ProductRepository extends MongoRepository<Product, String> {
#Query("{'properties.id': ?0 }")
List<Product> findByPropertiesId(String propertiesId);
I've tried also:
List<Product> findByProperties_id(String propertiesId)
or
#Query("{'properties.$id': ?0 }")
List<Product> findByPropertiesId(ObjectId propertiesId);
but WITHOUT success. DO you know what's wrong?
When I invoke:
public List<Product> findProductsByPropertiesId(String properties) {
if (properties == null) {
throw new IllegalArgumentException("onFind: propertiesId should not be null.");
}
return productRepository.findByProperties_Id(properties);
}
I get empty list:(
Maybe there is impossible to do that via Query?
#Query("{'properties.$id': ?0 }")
List<Product> findByPropertiesId(ObjectId propertiesId);
public List<Product> findProductsByPropertiesId(String properties) {
if (properties == null) {
throw new IllegalArgumentException("onFind: propertiesId should not be null.");
}
return productRepository.findByPropertiesId(new ObjectId(properties));
}
{
"_id" : "a9582f59-f52b-4fc8-84ab-cdd0bfb8dead",
"_class" : "com.db.Category",
"name" : "Cricket",
"subCategories" : [
{
"name" : "Gloves",
"creationDate" : NumberLong("1527404341099"),
"modificationDate" : NumberLong("1527404341099")
},
{
"name" : "Stumps",
"creationDate" : NumberLong("1527404369882"),
"modificationDate" : NumberLong("1527404369882")
},
{
"name" : "Bat",
"brandList" : [
{
"name" : "MRF",
"productDetails" : [
{
"name" : "Bat 111",
"price" : "1224",
"imageUrlList" : [
"https://s3.us-east-1.amazonaws.com/gasports/1527792222680-Bat_111",
"https://s3.us-east-1.amazonaws.com/gasports/1527792228375-Bat_111"
]
}
]
}
],
"creationDate" : NumberLong("1527424021629"),
"modificationDate" : NumberLong("1527424021629")
}
],
"creationDate" : NumberLong("1527404340938"),
"modificationDate" : NumberLong("1527404340938")
}
This is Category Document. Category has subcategory,Subcategory has brands and Brand has products.
#Document(collection="productInfo")
public class Category extends BaseProductInfo<Category> {
#Id
private String uid;
private String name;
private List<SubCategory> subCategories;
//Getters ans setters..
Now I have to get only for specific subcategory or Brand or Product. For this currently I am doing iteration to get object.
Is there any way in Spring Data Mongo Repository to get nested Object ?
You can write a method like this in your Category Repository class
Category findBySubCategories_BrandList_ProductDetails_Name(String name)
Just pass the product name you want to fetch. spring-data-mongodb will formulate the query based on your meethod name and fetch the documents matching the query.
I store documents from cars and want to get the temperature of all Mercedes cars as an array, how should the query be in Mongodb?
{ "_id" : { "$oid" : "5880ff305d15f416c89457b7" },
"car" : "mercedes",
"engine" : {
"sensor" : {
"temperatur" : "20",
"speed" : "100",
"hue" : "40"
}
},
"motor" : {
"Power" : "155",
"Topspeed" : "400"
}
}
{ "_id" : { "$oid" : "5880ff305d15f416c89457b7" },
"car" : "mercedes",
"engine" : {
"sensor" : {
"temperatur" : "50",
"speed" : "100",
"hue" : "40"
}
},
"motor" : {
"Power" : "155",
"Topspeed" : "400"
}
}
I would like to select the temperature for all Mercedes cars and receive it.
result should be like [20,50]
EDIT:
My code lookls like the following iam using JAVA:
MongoClient mongoClient = new MongoClient();
MongoDatabase database = mongoClient.getDatabase("test");
MongoCollection<Document> coll = database.getCollection("myTestCollection");
You can try something like this with regular query if you're okay with distinct values.
db.cars.distinct("engine.sensor.temperatur", {"car" : "mercedes"});
This will give you
[ "20", "50" ]
Update - Java equivalent:
List<String> temps = coll.distinct("engine.sensor.temperatur", new Document("car", "mercedes"), String.class).into(new ArrayList<>());
Update - Aggregation Option
Bson match = new Document("$match", new Document("car", "mercedes"));
Bson group = new Document("$group", new Document("_id", "$car").append("temps", new Document("$push", "$engine.sensor.temperatur")));
List<String> temps = (List<String>) coll.aggregate(Arrays.asList(match, group)).map(document -> document.get("temps")).first();
I have a MongoDB of gocomics comments. Sample comment (from
"db.comments.find().pretty()"):
{
"_id" : ObjectId("518f14e5394594efbe18068c"),
"body" : "plan for it",
"commentid" : "2525923",
"commentor" : "Chocoloop",
"commentorid" : "769338",
"da" : "25",
"filename" : "/mnt/sshfs/gocomics/comments/100.out.bz2",
"mo" : "11",
"strip" : "luann",
"stripname" : "Luann",
"time" : "1 day ago",
"yy" : "2011"
}
This shows that "Chocoloop" made the comment "plan for it" on the
2011-11-25 "Luann" strip. The commentid was 2525923, and is unique to
this comment. The other fields aren't relevant for this question.
A single person can make multiple comments on the same strip. For
example, "Chocoloop" may've made a later comment on the 2011-11-25
"Luann" strip. The later comment would have the same strip, da, mo,
yr, commentor fields, but a higher commentid.
I want to find the latest comment I've made on each strip. This is easy:
db.comments.aggregate(
{$match: {commentor:"barrycarter"}},
{$group: {_id: {strip: "$strip", yy: "$yy", da:"$da", mo:"$mo"},
mid: {$max:"$commentid"}}}
)
Here is one of many results:
{
"_id" : {
"strip" : "pearlsbeforeswine",
"yy" : "2007",
"da" : "28",
"mo" : "11"
},
"mid" : "2462203"
}
This says I made at least one comment (perhaps several) on the
pearlsbeforeswine strip dated 2007-11-28. Of the comments I made, the
latest one (the one with the highest commentid) had commentid 2462203
(mid = "max id").
Now, for each result, I want to know: has someone made a comment after
I made my last comment?
For the selected result above, this means: are there any comments for
the pearlsbeforeswine strip dated 2007-11-28 whose commentid exceeds
2462203?
Of course, I can write a query for that one special case:
db.comments.find(
{strip:"pearlsbeforeswine",yy:"2007",da:"28",mo:"11",
commentid: {$gt: "2462203"}}
).pretty()
but how do I do it for all the results in the resultset without
creating an individual query for each one (even automated, that seems
ugly).
Is this a poor use case for MongoDB? I have a similar (not identical)
SQLite3 database where this query is:
SELECT * FROM (SELECT strip,month,date,year,MAX(id) AS mid FROM
comments WHERE commentorid=801127 GROUP BY strip,month,date,year) AS t
JOIN comments c ON (t.strip=c.strip AND t.month=c.month AND
t.date=c.date AND t.year=c.year AND c.id > t.mid)
(where 801127 is my commentorid [the SQLite3 version doesn't include
"commentor" name field]).
NOTE: My MongoDB commentid's are strings, not ints. That's bad, but I
don't think it impacts this question.
You can do it using the aggregation framework and there are multiple approaches towards this. The simplest one is somewhat brute-force and long - it may not have the best performance but I think it's simplest to understand:
proj={"$project": {
"strip" : {"$concat" : ["$strip","-","$yy","/","$mo","/","$da"]},
"commentor" : 1,
"commentid" : 1
}
};
group={"$group": {
"_id" : "$strip",
"comms" : {
"$push" : {
"c" : "$commentor",
"i" : "$commentid"
}
},
"max" : {
"$max" : "$commentid"
}
}};
match = { "$match" : { "comms.c" : "<commentorname>" } };
unwind = { "$unwind" : "$comms" };
proj2 = { "$project" : {
"meLast" : {"$cond" : [
{"$eq" : [
"$max",
"$comms.i"
]
},
1,
0
] }
}
};
group2 = {"$group" : {
"_id" : "$_id",
"afterMe" : {
"$max" : "$meLast"
}
} };
match2 = { "$match" : { "afterMe" : 0 } };
db.comments.aggregate( proj, group, match, unwind, match, proj2, group2, match2 );
Basically, whichever way you do it, there are two {$group} steps in the pipeline that you must have, one to find max commentid for this particular commentor and one for over max commentid for that strip. So it could have been project, group, group, unwind, project with matches as appropriate. Hope you get the idea.
By the way, if you had a unique identifier of each strip (say "comicId") you could then get the list of comics a particular person commented on much simpler, and then you don't need aggregation as much you could just use:
db.comments.distinct("comicId",{commentor:"name"})
which would significantly reduce the number of comments that need to be aggregated. A simpler way to track conversations/replies may be to have comments have "in-reply-to" but then I'm not sure if you are tracking threaded conversations or just straight comments.
I think this is an excellent question and answer to the problem so I decided to solve this using Spring Data with MongoDB in java. To convert Asya's answer into java mongodb code, I did the following:
public void commentTest() {
BasicDBObject o1 = new BasicDBObject();
o1.append("c", "$commentor");
o1.append("i", "$commentid");
Aggregation aggCount = newAggregation(
project("commentid", "commentor")
.andExpression("concat(\"$strip\",\"-\",\"$yy\",\"/\",\"$mo\",\"/\",\"$da\")").as("strip"),
group("strip").push(o1).as("comms").max("commentid").as("max"),
match(Criteria.where("comms.c").is("Simon")),
unwind("comms"),
match(Criteria.where("comms.c").is("Simon")));
logger.info(aggCount.toString());
AggregationResults<CommentTest> groupCount = mongoTemplate.aggregate(aggCount, "commenttest", CommentTest.class);
List<CommentTest> resultCount = groupCount.getMappedResults();
ObjectMapper mapper = new ObjectMapper();
try {
logger.info(mapper.writeValueAsString(resultCount));
} catch (IOException e) {
e.printStackTrace();
}
}
Then to get mongotemplate to parse the results successfully into the CommentTest class, I had to create the class that minicks the results:
Document(collection = "commenttest")
#JsonInclude(JsonInclude.Include.NON_NULL)
public class CommentTest {
private String id, body, commentid, commentor, commentorid, da, filename, mo, strip, stripname, time, yy, max;
#JsonProperty
private comms comms;
public CommentTest.comms getComms() {
return comms;
}
public void setComms(CommentTest.comms comms) {
this.comms = comms;
}
public static class comms implements Serializable {
private String c,i;
public String getC() {
return c;
}
public void setC(String c) {
this.c = c;
}
public String getI() {
return i;
}
public void setI(String i) {
this.i = i;
}
}
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public String getBody() {
return body;
}
public void setBody(String body) {
this.body = body;
}
public String getCommentid() {
return commentid;
}
public void setCommentid(String commentid) {
this.commentid = commentid;
}
public String getCommentor() {
return commentor;
}
public void setCommentor(String commentor) {
this.commentor = commentor;
}
public String getCommentorid() {
return commentorid;
}
public void setCommentorid(String commentorid) {
this.commentorid = commentorid;
}
public String getDa() {
return da;
}
public void setDa(String da) {
this.da = da;
}
public String getFilename() {
return filename;
}
public void setFilename(String filename) {
this.filename = filename;
}
public String getMo() {
return mo;
}
public void setMo(String mo) {
this.mo = mo;
}
public String getStrip() {
return strip;
}
public void setStrip(String strip) {
this.strip = strip;
}
public String getStripname() {
return stripname;
}
public void setStripname(String stripname) {
this.stripname = stripname;
}
public String getTime() {
return time;
}
public void setTime(String time) {
this.time = time;
}
public String getYy() {
return yy;
}
public void setYy(String yy) {
this.yy = yy;
}
public String getMax() {
return max;
}
public void setMax(String max) {
this.max = max;
}
}
I then created some test data in mongodb by inserting these 4 mock entries:
{ "_id" : ObjectId("518f14e5394594efbe18068c"), "body" : "1", "commentid" : "2525923", "commentor" : "Simon", "commentorid" : "769338", "da" : "25", "filename" : "/mnt/sshfs/gocomics/comments/100.out.bz2", "mo" : "11", "strip" : "luann", "stripname" : "Luann", "time" : "1 day ago", "yy" : "2011" }
{ "_id" : ObjectId("518f14e5394594efbe18068d"), "body" : "2", "commentid" : "2525924", "commentor" : "Josh", "commentorid" : "769339", "da" : "25", "filename" : "/mnt/sshfs/gocomics/comments/100.out.bz2", "mo" : "11", "strip" : "luann", "stripname" : "Luann", "time" : "1 day ago", "yy" : "2011" }
{ "_id" : ObjectId("518f14e5394594efbe18068e"), "body" : "3", "commentid" : "2525925", "commentor" : "Peter", "commentorid" : "769340", "da" : "25", "filename" : "/mnt/sshfs/gocomics/comments/100.out.bz2", "mo" : "11", "strip" : "luann", "stripname" : "Luann", "time" : "1 day ago", "yy" : "2011" }
{ "_id" : ObjectId("518f14e5394594efbe18068f"), "body" : "old1", "commentid" : "2525905", "commentor" : "Peter", "commentorid" : "769340", "da" : "24", "filename" : "/mnt/sshfs/gocomics/comments/100.out.bz2", "mo" : "11", "strip" : "luann", "stripname" : "Luann", "time" : "1 day ago", "yy" : "2011" }
I then ran the code and here are the results:
[{"id":"luann-2011/11/25","max":"2525925","comms":{"c":"Simon","i":"2525923"}}]
The result can be interpreted as post luann-2011/11/25 has the maximum comment number (or mongo id) as 2525925 while your comment has an id of 2525923. Therefore there is a later comment after you had comment hence you will need to fetch that new comment. You will need to write your logic for it programmatically.