I am trying to create an elastic search index using the following Java code
public class MappingCreator {
static Logger log = Logger.getLogger(MappingCreator.class.getName());
final static String indexName = "indxyz";
final static String typeName = "indtype";
final static String mappingFileName = "customMapping.json";
final static String clusterName = "elasticsearch";
final static String hostName = "localhost";
public static void main(String args[]) throws IOException
{
MappingCreator mapCreator = new MappingCreator();
Client myESclient = getClient();
IndicesExistsResponse res = myESclient.admin().indices().prepareExists(indexName).execute().actionGet();
if (res.isExists()) {
log.warn("Index "+indexName +" already exists. Will be deleted");
final DeleteIndexRequestBuilder deleteIndexBuilder = myESclient.admin().indices().prepareDelete(indexName);
deleteIndexBuilder.execute().actionGet();
}
final CreateIndexRequestBuilder createIndexBuilder = myESclient.admin().indices().prepareCreate(indexName)
.addMapping(typeName, mapCreator.getIndexFieldMapping());
CreateIndexResponse createIndexResponse = createIndexBuilder.execute().actionGet();
log.debug("Created mapping "+createIndexResponse.toString());
myESclient.close();
}
private String getIndexFieldMapping() throws IOException {
return IOUtils.toString(getClass().getClassLoader().getResourceAsStream(mappingFileName));
}
private static Client getClient() {
TransportClient transportClient = null;
try
{
Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name", clusterName).build();
transportClient = new TransportClient(settings);
transportClient = transportClient.addTransportAddress(new InetSocketTransportAddress(hostName, 9300));
}
catch (Exception e)
{
log.error("Error in MappingCreator creating Elastic Search Client\n"
+ "Message "+e.getMessage()+"\n"
+ "StackTrace "+e.getStackTrace()
);
}
return (Client) transportClient;
}
}
customMapping.json looks like this
{
"properties": {
"popkDate": {
"type":"date",
"format": "yyyy-MM-dd 'T' HH:mm:ss.SSZ"
},
"popk": {
"type": "string"
}
}
}
now, after creating the mapping, if i run the following in the elasticsearch marvel plugin
get indxyz/
i get
{
"indxyz": {
"aliases": {},
"mappings": {
"indtype": {
"properties": {
"popkDate": {
"type": "date",
"format": "yyyy-MM-dd 'T' HH:mm:ss.SSZ"
},
"popk": {
"type": "string"
}
}
}
}
which is accurate.
now when i try to ingest this data
{
"popkDate":"2001-01-01T235959.00-0400",
"popk":"cuinbcwenvubn"
}
for some reason, the popkDate field is indexed as String, instead of Date. Now, if i again look at
get indxyz/
this time i get,
{
"indxyz": {
"aliases": {},
"mappings": {
"indtype": {
"properties": {
"popkDate": {
"type": "date",
"format": "yyyy-MM-dd 'T' HH:mm:ss.SSZ"
},
"popk": {
"type": "string"
}
}
},
"indtype": {
"properties": {
"popkDate": {
"type": "string"
},
"popk": {
"type": "string"
}
}
}
}
why was this duplicate mapping created ? for the same type ? Note that if i create the mapping without the JAVA api, i.e. through curl commands, then this behavior is not observed and the popkDate field is indexed as expected. Why would this be ?
Please advise.
Related
I'm starting learn spring data mongodb. Currently, I try to implement multiple condition query with Spring Data MongoDB.
First I have a sample collection
{
"_id": "df05532f-8893-4802-8710-ab92056c9c77",
"objectId": {
"$numberLong": "1"
},
"creator": {
"_id": {
"$numberLong": "3"
},
"username": "magic"
},
"content": "7878787887",
"attachments": [],
"pinned": false,
"histories": [
{
"content": "new new comment 2222",
"createdAt": {
"$date": {
"$numberLong": "1664970753576"
}
}
},
{
"content": "update me",
"createdAt": {
"$date": {
"$numberLong": "1664970753691"
}
}
},
{
"content": "update me3333",
"createdAt": {
"$date": {
"$numberLong": "1664970753734"
}
}
},
{
"content": "44444455666",
"createdAt": {
"$date": {
"$numberLong": "1664970753740"
}
}
}
],
"contentUpdateAt": {
"$date": {
"$numberLong": "1664970753745"
}
},
"createdAt": {
"$date": {
"$numberLong": "1664970753576"
}
},
"lastModifiedAt": {
"$date": {
"$numberLong": "1664970753772"
}
}
}
I'm trying to find a partial document when the array histories has one element that matched with content equals 44444455666
With MongoDB CLI, I completed build the query like
db.ticket_detail_comments.find(
{
histories: {
$elemMatch: {
content: "44444455666"
}
},
_id : "df05532f-8893-4802-8710-ab92056c9c77"
},
{ "histories.$": 1, name: 1 }
)
The result was exactly what I expected
{
"_id": "df05532f-8893-4802-8710-ab92056c9c77",
"histories": [
{
"content": "44444455666",
"createdAt": {
"$date": {
"$numberLong": "1664970753740"
}
}
}
]
}
When I tried to build query with Spring Data MongoDB (using MongoDbTemplate) I can't find any way to build query look like in console. I tried addOperation but seem not work,
I tried to build 2 separated Criteria (one, two) that match with 2 condition I expected, then combine them with andOperation like new Criteria().andOperator(one).andOperator(two);. The unhappy part is the query generated is so different
db.ticket_detail_comments.find(
{
"$and": [
{
histories: {
$elemMatch: {
content: "44444455666"
}
},
_id : "df05532f-8893-4802-8710-ab92056c9c77"
},
{ "histories.$": 1, name: 1 }
]
}
)
With this query, I always get a null value. Can anyone help me with that? What is correct Criteria syntax I should use?
Thanks for reading.
UPDATE
I created a small demo, let's see
Data in Mongo Collection look like
{
"_id": "dad9db98-241b-40fa-aab4-af1671f97631",
"data": "demo",
"children": [
{
"name": "tim",
"age": 19
},
{
"name": "alex",
"age": 18
}
],
"_class": "com.example.demo.Data"
}
MONGOCLI
db.data.find(
{
children: {
$elemMatch: {
name: "alex"
}
},
_id : "dad9db98-241b-40fa-aab4-af1671f97631"
},
{ "children.$": 1}
)
=> Result:
{ _id: 'dad9db98-241b-40fa-aab4-af1671f97631',
children: [ { name: 'alex', age: 18 } ] }
JAVA CODE
package com.example.demo;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.NoArgsConstructor;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.data.mongodb.core.MongoTemplate;
import org.springframework.data.mongodb.core.query.Criteria;
import org.springframework.data.mongodb.core.query.Query;
import javax.annotation.PostConstruct;
import java.util.List;
import java.util.UUID;
#SpringBootApplication
public class DemoApplication {
private final MongoTemplate template;
public DemoApplication(MongoTemplate template) {
this.template = template;
}
public static void main(String[] args) {
SpringApplication.run(DemoApplication.class, args);
}
#PostConstruct
public void init() {
Data.Child child1 = Data.Child.builder()
.name("alex")
.age(18)
.build();
Data.Child child2 = Data.Child.builder()
.name("tim")
.age(19)
.build();
final String id = UUID.randomUUID().toString();
Data data = Data.builder()
.id(id)
.data("demo")
.children(List.of(child2, child1))
.build();
template.save(data, "data");
Query query = Query.query(
Criteria.where("_id").is(id).and("children")
.elemMatch(Criteria.where("name").is("alex"))
);
query.fields().position("children", 1);
var r = template.findOne(query, Data.class, "data");
System.out.printf("DEBUG");
}
}
#lombok.Data
#AllArgsConstructor
#NoArgsConstructor
#Builder
class Data {
private String id;
private String data;
private List<Child> children;
#lombok.Data
#AllArgsConstructor
#NoArgsConstructor
#Builder
static class Child {
private String name;
private Integer age;
}
}
=> Result: Data Object with 2 elements in children list.
But what I expected is Data Object has one element in children list like resut from CLI
I have a lot of problem, but this time my problem is GraphQL and its payload, my schema.gql and resolvers.js files looks like these:
# schema.gql
scalar GraphQLDateTime
scalar GraphQLDate
scalar GraphQLTime
type Query {
magicQuery(
idMagic: Int
magicDate: GraphQLDateTime
): DamnWow
}
type DamnWow {
_id: String
magicDateResult: GraphQLDateTime
}
# resolvers.js
const { GraphQLDate, GraphQLTime, GraphQLDateTime } = require('graphql-iso-date');
const resolvers = {
Query: {
magicQuery: async (root, { idMagic, magicDate }, context) => {
// It's never enter here
console.warn(idMagic, magicDate);
const result = { _id: 'wow1', magicDate: new Date() };
return result;
}
},
GraphQLDateTime: GraphQLDateTime,
GraphQLDate: GraphQLDate,
GraphQLTime: GraphQLTime
};
module.exports = resolvers;
When I try to use my payload with a query like that:
# query
query magicQuery($idMagic: Int, $magicDate: DateTime) {
magicQuery(idMagic: $idMagic, magicDate: $magicDate) {
_id
magicDateResult
}
}
# variables
# {
# "idMagic": 1,
# "magicDate": "2007-12-03T10:15:30Z"
# }
it's return me an error:
{
"error": {
"errors": [
{
"message": "Unknown type \"DateTime\".",
"locations": [
{
"line": 1,
"column": 45
}
],
"extensions": {
"code": "GRAPHQL_VALIDATION_FAILED",
"exception": {
"stacktrace": [
"GraphQLError: Unknown type \"DateTime\".",
" at Object.NamedType (/home/alex/workspace/team/node_modules/graphql/validation/rules/KnownTypeNamesRule.js:57:29)",
" at Object.enter (/home/alex/workspace/team/node_modules/graphql/language/visitor.js:323:29)",
" at Object.enter (/home/alex/workspace/team/node_modules/graphql/utilities/TypeInfo.js:370:25)",
" at visit (/home/alex/workspace/team/node_modules/graphql/language/visitor.js:243:26)",
" at Object.validate (/home/alex/workspace/team/node_modules/graphql/validation/validate.js:69:24)",
" at validate (/home/alex/workspace/team/node_modules/apollo-server-core/dist/requestPipeline.js:221:34)",
" at Object.<anonymous> (/home/alex/workspace/team/node_modules/apollo-server-core/dist/requestPipeline.js:118:42)",
" at Generator.next (<anonymous>)",
" at fulfilled (/home/alex/workspace/team/node_modules/apollo-server-core/dist/requestPipeline.js:5:58)"
]
}
}
}
]
}
}
If I use same parameter directy in my code it works fine.
Does someone know the cause of this error?
Your scalar type name in the schema is GraphQLDateTime but in the request you are using DateTime.
Try changing the request to:
query magicQuery($idMagic: Int, $magicDate: GraphQLDateTime) {
magicQuery(idMagic: $idMagic, magicDate: $magicDate) {
_id
magicDateResult
}
}
I am trying to get the list of issues by their ids from Github using graphql, but looks like I am missing something or its not possible.
query ($ids:['517','510']!) {
repository(owner:"owner", name:"repo") {
issues(last:20, states:CLOSED) {
edges {
node {
title
url
body
author{
login
}
labels(first:5) {
edges {
node {
name
}
}
}
}
}
}
}
}
The above query is giving me response as below,
{
"errors": [
{
"message": "Parse error on \"'\" (error) at [1, 14]",
"locations": [
{
"line": 1,
"column": 14
}
]
}
]
}
Kindly help me identify if its possible or that I am doing something wrong here.
You can use aliases in order to build a single request requesting multiple issue object :
{
repository(name: "material-ui", owner: "mui-org") {
issue1: issue(number: 2) {
title
createdAt
}
issue2: issue(number: 3) {
title
createdAt
}
issue3: issue(number: 10) {
title
createdAt
}
}
}
Try it in the explorer
which gives :
{
"data": {
"repository": {
"issue1": {
"title": "Support for ref's on Input component",
"createdAt": "2014-10-15T15:49:13Z"
},
"issue2": {
"title": "Unable to pass onChange event to Input component",
"createdAt": "2014-10-15T16:23:28Z"
},
"issue3": {
"title": "Is it possible for me to make this work if I'm using React version 0.12.0?",
"createdAt": "2014-10-30T14:11:59Z"
}
}
}
}
This request can also be simplified using fragments to prevent repetition:
{
repository(name: "material-ui", owner: "mui-org") {
issue1: issue(number: 2) {
...IssueFragment
}
issue2: issue(number: 3) {
...IssueFragment
}
issue3: issue(number: 10) {
...IssueFragment
}
}
}
fragment IssueFragment on Issue {
title
createdAt
}
The request can be built programmatically, such as in this example python script :
import requests
token = "YOUR_TOKEN"
issueIds = [2,3,10]
repoName = "material-ui"
repoOwner = "mui-org"
query = """
query($name: String!, $owner: String!) {
repository(name: $name, owner: $owner) {
%s
}
}
fragment IssueFragment on Issue {
title
createdAt
}
"""
issueFragments = "".join([
"""
issue%d: issue(number: %d) {
...IssueFragment
}""" % (t,t) for t in issueIds
])
r = requests.post("https://api.github.com/graphql",
headers = {
"Authorization": f"Bearer {token}"
},
json = {
"query": query % issueFragments,
"variables": {
"name": repoName,
"owner": repoOwner
}
}
)
print(r.json()["data"]["repository"])
I don't think you can fetch for issues and pass in an array of integers for their ids.
But you can search for a single issue by id like so (this works for me)
query ($n: Int!) {
repository(owner:"owner", name:"repo-name") {
issue (number: $n) {
state
title
author {
url
}
}
}
}
where $n is {"n": <your_number>} defined.
If you have an array of ids, then you can just make multiple queries to GitHub.
Sadly, with this approach, you cannot specify what the state of the issue to be. But I think the logic is that once you know the issue Id, you shouldn't care what state it is, since you have that exact id.
I'm trying to query MongoDB for Just idProfil and hashtags list from profileTwitter and bellow is the object:
{
"idProfil": "5ded2abae1692808b799b239",
"tweets": [
{
"idTweet": "5ded2dffe1692808b799b241",
"datePublication": "2019-12-08T16:58:59.702+0000",
"hashtags": [
{
"idHashtag": "5ded2c64e1692808b799b23c",
"label": "recipes "
},
{
"idHashtag": "5ded2c71e1692808b799b23d",
"label": "delicious "
},
{
"idHashtag": "5ded2c7ce1692808b799b23e",
"label": "foodrecipes"
},
{
"idHashtag": "5ded2c84e1692808b799b23f",
"label": "canada "
},
{
"idHashtag": "5ded2c8de1692808b799b240",
"label": "Usa"
},
{
"idHashtag": "5dee65d7e39e962d44a31c40",
"label": "food"
},
{
"idHashtag": "5dee65c8e39e962d44a31c3f",
"label": "cooking"
}
]
}
}
So my question is how to do this using spring boot ?
This example based on data you have provided. Your question is not clear. Can you please provide more details on it.
List<AggregationOperation> stages = new ArrayList<>();
ProjectOperation projectOperation = project("idProfil").and("$tweets.hashtags").as("hashtags");
stages.add(projectOperation);
AggregationResults<ResultDTO> result = mongoOperation.aggregate(newAggregation(stages),
"profileTwitter", ResultDTO.class);
public class ResultDTO {
private String idProfil;
private List<HashtagDTO> hashtags;
//getter setter
}
public class HashtagDTO {
private String idHashtag;
private String label;
//getter setter
}
Are nested DTOs in JPQL queries not allowed in Spring Data ? :
#Query("SELECT new test.customresult.CategoryCounter(c.name, "
+ "new test.customresult.CarResult(COUNT(e.category), e.category)"
+ "FROM error e JOIN e.car c "
+ "GROUP BY c.name,e.category")
List<CategoryCounter>countErrors();
Because, I get the following error message when trying to use the above mentioned JPQL query :
Caused by: org.hibernate.hql.internal.ast.QuerySyntaxException: unexpected token: , near line 1,
I want to return such a JSON file from the JPQL query :
[{
"Car 1":
[{
"count": 1,
"category": "A"
}, {
"count": 2,
"category": "B"
}, {
"count": 0,
"category": "C"
}, {
"count": 0,
"category": "D"
}
]
}, {
"Car 2":
[{
"count": 0,
"category": "A"
}, {
"count": 0,
"category": "B"
}, {
"count": 4,
"category": "C"
}, {
"count": 5,
"category": "D"
}
]
}
]
There is a car table and an error table which contains the categories and a foreign key to the car table.
I wanted to use the nested DTO to represent the desired output :
The "wrapper" DTO
public class CategoryCounter {
private String name;
private CarResult carResult ;
public CategoryCounter (String name, CarResult carResult ) {
this.name= name;
this.carResult = carResult ;
}
public CarResult getCarResult () {
return carResult ;
}
public void setCarResult(CarResult carResult ) {
this.carResult = carResult ;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name= name;
}
}
and the "nested" DTO :
public class CarResult {
private Long count;
private String category;
public CarResult (Long count, String category) {
this.count = count;
this.category= category;
}
public Long getCount() {
return count;
}
public void setCount(Long count) {
this.count = count;
}
public String getCategory() {
return category;
}
public void setCategory(String category) {
this.category= category;
}
}
I'm not sure if this could be the problem... but you have your FROM clause starting immediately after the ) it says:
+ "new test.customresult.CarResult(COUNT(e.category), e.category)"
+ "FROM error e JOIN e.car c "
that is the same to: e.category)FROM so put a space in that part and try again...
+ "new test.customresult.CarResult(COUNT(e.category), e.category)"
+ " FROM error e JOIN e.car c "
the thing is that the error (Caused by: org.hibernate.hql.internal.ast.QuerySyntaxException) is a syntax error