using Rest from classic asp I am able to retrieve data from an index when I perform a simple search such as:
Set xml = Server.CreateObject("Microsoft.XMLHTTP")
xml.Open "GET", "http://elastic:changeme#10.128.128.109:9200/myindex/_search?q='fred blogs',_source='path.real'", False
I now want to retrive data from a much more complex query that if I was using Kibana would look something like this:
GET newsindex/_search
{
"query": {
"function_score": {
"query": { "bool" : {"should": [
{ "term": { "headline": "brown" } },
{ "term": { "bodytext": "brown" } },
{ "term": { "headline": "fox" } },
{ "term": { "bodytext": "fox" } }
]}},
"functions": [
{
"filter": {"match" : { "headline": "brown fox" }},
"weight": 2
},
{
"filter": {"match" : { "bodytext": "brown fox" }},
"weight": 1
},
{
"filter": {"match" : { "bodytext": "fox" }},
"weight": 3
},
{
"filter": {"match" : { "bodytext": "brown" }},
"weight": 3
},
{
"filter": {"match" : { "headline": "brown" }},
"weight": 4
},
{
"filter": {"match" : { "headline": "fox" }},
"weight": 4
}
],
"score_mode": "sum"
}
},"sort": [
{
"_score": {
"order": "asc"
}
}
],
"_source":[ "headline", "bodytext" ]
}
How can I pass this query using Rest? I'll be building queries like this on the fly to pass via rest. But I need to get the construct right.
By using curl command, you can do something like this:
curl -XGET "http://elastic:changeme#10.128.128.109:9200/myindex/_search" -H 'Content-Type: application/json' -d'
{
"query": {
"function_score": {
"query": {
"bool": {
"should": [
{
"term": {
"headline": "brown"
}
},
{
"term": {
"bodytext": "brown"
}
},
{
"term": {
"headline": "fox"
}
},
{
"term": {
"bodytext": "fox"
}
}
]
}
},
"functions": [
{
"filter": {
"match": {
"headline": "brown fox"
}
},
"weight": 2
},
{
"filter": {
"match": {
"bodytext": "brown fox"
}
},
"weight": 1
},
{
"filter": {
"match": {
"bodytext": "fox"
}
},
"weight": 3
},
{
"filter": {
"match": {
"bodytext": "brown"
}
},
"weight": 3
},
{
"filter": {
"match": {
"headline": "brown"
}
},
"weight": 4
},
{
"filter": {
"match": {
"headline": "fox"
}
},
"weight": 4
}
],
"score_mode": "sum"
}
},
"sort": [
{
"_score": {
"order": "asc"
}
}
],
"_source": [
"headline",
"bodytext"
]
}'
You can send your query in data by using curl. I'm sure the same can also be achieved by your Rest client. Just check how to send data using your Rest client.
This post might help. Replace the DataToSend variable in the example with your query and see if this works.
Related
OBS! Noob question probably :)
Given the following data, how can I query and return a summary for each index?
[
{
"title": "test",
"indexes":[
{ "id":1, "value": 0.5764860139860139860139860140 },
{ "id":2, "value": 0.3083479020979020979020979020 },
{ "id":3, "value": 0.1151660839160839160839160838 }
]
},
{
"title": "test",
"indexes":[
{ "id":1, "value": 0.5764860139860139860139860140 },
{ "id":2, "value": 0.3083479020979020979020979020 },
{ "id":3, "value": 0.1151660839160839160839160838 }
]
},
{
"title": "test",
"indexes":[
{ "id":1, "value": 0.5764860139860139860139860140 },
{ "id":2, "value": 0.3083479020979020979020979020 },
{ "id":3, "value": 0.1151660839160839160839160838 }
]
},
{
"title": "test",
"indexes":[
{ "id":1, "value": 0.5764860139860139860139860140 },
{ "id":2, "value": 0.3083479020979020979020979020 },
{ "id":3, "value": 0.1151660839160839160839160838 }
]
}
]
I.e. I want to produce something like this:
index.id:1, total: 2.305...
index.id:2, total: 1.233...
etc
db.collection.aggregate([
{
"$unwind": "$indexes"
},
{
$group: {
_id: "$indexes.id",
total: {
$sum: "$indexes.value"
}
}
}
])
try this query
you will get like this
[
{
"_id": 2,
"total": 1.2333916083916083
},
{
"_id": 1,
"total": 2.305944055944056
},
{
"_id": 3,
"total": 0.4606643356643357
}
]
db.collection.aggregate([
{
$unwind: "$indexes"
},
{
$group: {
_id: "$indexes.id",
total: {
$sum: "$indexes.value"
}
}
}
])
Working Mongo playground
I want to pass an event_id to Kibana/Elastic Search and find the min and max dates from the #timestamp field for this event_id. Then I want to set the date range to these dates and show all the results. I assume this is doable.
I can get the min and max with this aggregation:
GET /filebeat-*/_search
{
"query": {
"match": {
"event_id": 1234
}
},
"aggs" : {
"min_date": {"min": {"field": "#timestamp" }},
"max_date": {"max": {"field": "#timestamp" }}
}
}
and I can get the results by searching for the specific date range:
GET /filebeat-*/_search
{
"query": {
"bool": {
"filter": {
"range": {"#timestamp": {"gte": "2020-09-11T13:35:35.000Z", "lte": "2020-09-24T20:35:07.000Z"}}
}
}
}
}
how can I combine the two so that I can just change the event_id and have an auto date range type feature?
EDIT:
I can do this:
GET /filebeat-*/_search
{
"query": {
"bool": {
"must": {
"match": {
"event_id": 1234
}
},
"filter": {
"range": {
"#timestamp": {
"lte": "2020-09-25",
"gte": "2020-09-24"
}
}
}
}
},
"aggs": {
"min_date": {
"min": {
"field": "#timestamp"
}
},
"max_date": {
"max": {
"field": "#timestamp"
}
}
}
}
But what I would like to do is something like:
GET /filebeat-*/_search
{
"query": {
"bool": {
"must": {
"match": {
"event_id": 1234
}
},
"filter": {
"range": {
"#timestamp": {
"lte": "max_date",
"gte": "min_date"
}
}
}
}
},
"aggs": {
"min_date": {
"min": {
"field": "#timestamp"
}
},
"max_date": {
"max": {
"field": "#timestamp"
}
}
}
}
But this causes the error: "failed to parse date field [min_date]"
Is it possible to use the aggregated min and max values to define the date range?
Since you have not provided any sample index data, so applying range query on date type field
Adding a working example with index mapping, data, search query, and search result
Index Mapping:
{
"mappings": {
"properties": {
"date": {
"type": "date"
}
}
}
}
Index Data:
{
"date": "2015-02-10",
"event_id":"1234"
}
{
"date": "2015-01-01",
"event_id":"1235"
}
{
"date": "2015-02-01",
"event_id":"1234"
}
{
"date": "2015-02-01",
"event_id":"1235"
}
{
"date": "2015-01-20",
"event_id":"1234"
}
Search Query:
{
"query": {
"bool": {
"must": {
"match": {
"event_id": 1234
}
},
"filter": {
"range": {
"date": {
"lte": "2015-02-15",
"gte": "2015-01-11"
}
}
}
}
},
"aggs": {
"min_date": {
"min": {
"field": "date"
}
},
"max_date": {
"max": {
"field": "date"
}
}
}
}
Search Result:
"hits": {
"total": {
"value": 3,
"relation": "eq"
},
"max_score": 0.44183272,
"hits": [
{
"_index": "stof_64127765",
"_type": "_doc",
"_id": "3",
"_score": 0.44183272,
"_source": {
"date": "2015-02-01",
"event_id": "1234"
}
},
{
"_index": "stof_64127765",
"_type": "_doc",
"_id": "1",
"_score": 0.44183272,
"_source": {
"date": "2015-02-10",
"event_id": "1234"
}
},
{
"_index": "stof_64127765",
"_type": "_doc",
"_id": "5",
"_score": 0.44183272,
"_source": {
"date": "2015-01-20",
"event_id": "1234"
}
}
]
},
"aggregations": {
"max_date": {
"value": 1.4235264E12,
"value_as_string": "2015-02-10T00:00:00.000Z"
},
"min_date": {
"value": 1.421712E12,
"value_as_string": "2015-01-20T00:00:00.000Z"
}
}
I have a collection that stores history, i.e. a new document is created every time a change is made to the data, I need to extract fields based on the max value of a date field, however my query keeps returning either all of the dates or requires me to push the fields into an array which make the data hard to analyze for an end-user.
Expected output as CSV:
MAX(DATE), docID, url, type
1579719200216, 12371, www.foodnetwork.com, food
1579719200216, 12371, www.cnn.com, news,
1579719200216, 12371, www.wikipedia.com, info
Sample Doc:
{
"document": {
"revenueGroup": "fn",
"metaDescription": "",
"metaData": {
"audit": {
"lastModified": 1312414124,
"clientId": ""
},
"entities": [],
"docId": 1313943,
"url": ""
},
"rootUrl": "",
"taggedImages": {
"totalSize": 1,
"list": [
{
"image": {
"objectId": "woman-reaching-for-basket",
"caption": "",
"url": "",
"height": 3840,
"width": 5760,
"owner": "Facebook",
"alt": "Woman reaching for basket"
},
"tags": {
"totalSize": 4,
"list": []
}
}
]
},
"title": "The 8 Best Food Items of 2020",
"socialTitle": "The 8 Best Food Items of 2020",
"primaryImage": {
"objectId": "woman-reaching-for-basket.jpg",
"caption": "",
"url": "",
"height": 3840,
"width": 5760,
"owner": "Hero Images / Getty Images",
"alt": "Woman reaching for basket in laundry room"
},
"subheading": "Reduce your footprint with these top-performing diets",
"citations": {
"list": []
},
"docId": 1313943,
"revisionId": "1313943_1579719200216",
"templateType": "LIST",
"documentState": {
"activeDate": 579719200166,
"state": "ACTIVE"
}
},
"url": "",
"items": {
"totalSize": "",
"list": [
{
"type": "recipe",
"data": {
"comInfo": {
"list": [
{
"type": "food",
"id": "https://www.foodnetwork.com"
}
]
},
"type": ""
},
"id": 4,
"uuid": "1313ida-qdad3-42c3-b41d-223q2eq2j"
},
{
"type": "recipe",
"data": {
"comInfo": {
"list": [
{
"type": "news",
"id": "https://www.cnn.com"
},
{
"type": "info",
"id": "https://www.wikipedia.com"
}
]
},
"type": "PRODUCT"
},
"id": 11,
"uuid": "318231jc-da12-4475-8994-283u130d32"
}
]
},
"vertical": "food"
}
Below query:
db.collection.aggregate([
{
$match: {
vertical: "food",
"document.documentState.state": "ACTIVE",
"document.templateType": "LIST"
}
},
{
$unwind: "$document.items"
},
{
$unwind: "$document.items.list"
},
{
$unwind: "$document.items.list.contents"
},
{
$unwind: "$document.items.list.contents.list"
},
{
$match: {
"document.items.list.contents.list.type": "recipe",
"document.revenueGroup": "fn"
}
},
{
$sort: {
"document.revisionId": -1
}
},
{
$group: {
_id: {
_id: {
docId: "$document.docId",
date: {$max: "$document.revisionId"}
},
url: "$document.items.list.contents.list.data.comInfo.list.id",
type: "$document.items.list.contents.list.data.comInfo.list.type"
}
}
},
{
$project: {
_id: 1
}
},
{
$sort: {
"document.items.list.contents.list.id": 1, "document.revisionId": -1
}
}
], {
allowDiskUse: true
})
First of all, you need to go through the documentation of the $group aggregation here.
you should be doing this instead:
{
$group: {
"_id": "$document.docId"
"date": {
$max: "$document.revisionId"
},
"url": {
$first: "$document.items.list.contents.list.data.comInfo.list.id"
},
"type": {
$first:"$document.items.list.contents.list.data.comInfo.list.type"
}
}
}
This will give you the required output.
I am new to JOLT transformation. I am trying to create a transform spec.
I have an list of categories in the object where I need to only transform few details.
My sample code and spec re shown below.
In "0/SYS_CATALOG_DESCRIPTION" list, I need to convert it to a String based on the lang, i.e for en_US, I need to get AA Products
end result will be "_description" : "AA Products"
The "subCategories" should give me the following result:
"subCategories": [
{
"_id": "ce_155584",
"_parentIds": ["ce_128375"],
"_description": "Filters" //based on lang = en_US
}
]
Sample JSON:
{
"total": 16,
"max_score": 2.2809339,
"hits": [
{
"_index": "bosch-dms-frontend-service_en_us_1558584002",
"_type": "categories",
"_id": "ce_128375",
"_score": 2.2809339,
"_source": {
"_parentIds": [
"1234"
],
"0/SYS_CATALOG_DESCRIPTION": [
{
"lang": "de_DE",
"value": "AA Produkte"
},
{
"lang": "en_US",
"value": "AA Products"
}
],
"subCategories": [
{
"_index": "bosch-dms-frontend-service_en_us_1558584002",
"_type": "categories",
"_id": "ce_155584",
"_score": 2.2809339,
"_source": {
"_parentIds": [
"ce_128375"
],
"0/SYS_CATALOG_DESCRIPTION": [
{
"lang": "en_US",
"value": "Filters"
},
{
"lang": "zh_CN",
"value": "AA Filters (CN)"
}
],
"0/SYS_SYSTEMNAME": "AA_Filters"
}
}
]
}
}
]
}
SPEC:
[
{
"operation": "shift", // shift operation
"spec": {
"hits": {
"*": {
"_id": "_id",
"_source": {
"_parentIds": "_parentIds",
"0/SYS_CATALOG_DESCRIPTION": "_description",
}
}
}
}
}
]
The end result will be
{
"_id" : "ce_128375",
"_parentIds" : [ "1234" ],
"_description" : "AA Products (BR)",
"subCategories": [
{
"_id": "ce_155584",
"_score": 2.2809339,
"_parentIds": ["ce_128375"],
"_description" : "Filters"
}
]
}
I tried several ways but could not achieve the result.
Thank you.
Check if this spec is what you need:
[
{
"operation": "shift", // shift operation
"spec": {
"hits": {
"*": {
"_id": ["&",
"subCategories.[]._parentIds[]"],
"_source": {
"_parentIds": "&",
"0/SYS_CATALOG_DESCRIPTION": {
"*": {
"lang": {
"en_US": {
"#(2,value)": "_description"
}
}
}
},
"subCategories": {
"*": {
"_id": "subCategories.[&1].&",
"_score": "subCategories.[&1].&",
"_source": {
"0/SYS_CATALOG_DESCRIPTION": {
"*": {
"lang": {
"en_US": {
"#(2,value)": "subCategories.[&6]._description"
}
}
}
}
}
}
}
}
}
}
}
}
]
I have these 4 elements in my collection:
/* 1 */
{
"demographics": [
{
"key": "country",
"value": "ES"
},
{
"key": "city",
"value": "Sevilla"
},
{
"key": "region",
"value": "Andalucía"
}
]
}
/* 2 */
{
"demographics": [
{
"key": "city",
"value": "Cádiz"
},
{
"key": "country",
"value": "ES"
},
{
"key": "region",
"value": "Andalucía"
}
]
}
/* 3 */
{
"demographics": [
{
"key": "country",
"value": "GB"
},
{
"key": "region",
"value": "Greater London"
},
{
"key": "city",
"value": "London"
}
]
}
/* 4 */
{
"demographics": [
{
"key": "country",
"value": "ES"
},
{
"key": "region",
"value": "Andalucía"
},
{
"key": "city",
"value": "Sevilla"
}
]
}
I would like to group them by:
demographic.value when demographic.key = "country"
demographic.value when demographic.key = "region"
demographic.value when demographic.key = "city"
Having a result like this:
{ "values": ["ES", "Andalucía", "Sevilla"], "count": 2 }
{ "values": ["ES", "Andalucía", "Cádiz"], "count": 1 }
{ "values": ["GB", "Greater London", "London"], "count": 1 }
Attention: beware the order of the demographics array elements might be not always the same.
I have tried
db.getCollection('test').aggregate(
[
{ "$unwind": "$demographics" },
{
"$project" :{
"_id": 0,
"demographics.key": 1,
"demographics.value": 1
}
},
{
"$group" : {
"_id": {
"key": "$demographics.key",
"value": "$demographics.value"
},
"count": { "$sum": 1 }
}
},
{
"$group" : {
"_id": "$_id.key",
"values": { "$push": { "value": "$_id.value", "count": "$count" } }
}
}
]
)
This gives me this result:
/* 1 */
{
"_id": "country",
"values": [
{
"value": "GB",
"count": 1.0
},
{
"value": "ES",
"count": 3.0
}
]
}
/* 2 */
{
"_id": "region",
"values": [
{
"value": "Greater London",
"count": 1.0
},
{
"value": "Andalucía",
"count": 3.0
}
]
}
/* 3 */
{
"_id": "city",
"values": [
{
"value": "London",
"count": 1.0
},
{
"value": "Cádiz",
"count": 1.0
},
{
"value": "Sevilla",
"count": 2.0
}
]
}
But this is not the groups I am looking for
You can try running the following pipeline:
db.test.aggregate([
{ "$unwind": "$demographics" },
{ "$sort": { "demographics.key": 1, "demographics.value": 1 } },
{
"$group": {
"_id": "$_id",
"values": { "$push": "$demographics.value" }
}
},
{
"$group": {
"_id": "$values",
"count": { "$sum": 1 }
}
},
{
"$project": {
"_id": 0, "values": "$_id", "count": 1
}
}
])
Sample Output
/* 1 */
{
"count" : 2,
"values" : [
"Sevilla",
"ES",
"Andalucía"
]
}
/* 2 */
{
"count" : 1,
"values" : [
"London",
"GB",
"Greater London"
]
}
/* 3 */
{
"count" : 1,
"values" : [
"Cádiz",
"ES",
"Andalucía"
]
}