I try to familiarize myself with the Reindexing API of ElasticSearch and the use of Painless scripts.
I have the following model:
"mappings": {
"customer": {
"properties": {
"firstName": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"lastName": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"dateOfBirth": {
"type": "date"
}
}
}
}
I would like to reindex all documents from test-v1 to test-v2 and apply a few transformations on them (for example extract the year part of dateOfBirth, convert a date value to a timestamp, etc) and save the result as a new field. But I got an issue when I tried to access it.
When I made the following call, I got an error:
POST /_reindex?pretty=true&human=true&wait_for_completion=true HTTP/1.1
Host: localhost:9200
Content-Type: application/json
{
"source": {
"index": "test-v1"
},
"dest": {
"index": "test-v2"
},
"script": {
"lang": "painless",
"inline": "ctx._source.yearOfBirth = ctx._source.dateOfBirth.getYear();"
}
}
And the response:
{
"error": {
"root_cause": [
{
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"ctx._source.yearOfBirth = ctx._source.dateOfBirth.getYear();",
" ^---- HERE"
],
"script": "ctx._source.yearOfBirth = ctx._source.dateOfBirth.getYear();",
"lang": "painless"
}
],
"type": "script_exception",
"reason": "runtime error",
"script_stack": [
"ctx._source.yearOfBirth = ctx._source.dateOfBirth.getYear();",
" ^---- HERE"
],
"script": "ctx._source.yearOfBirth = ctx._source.dateOfBirth.getYear();",
"lang": "painless",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "Unable to find dynamic method [getYear] with [0] arguments for class [java.lang.String]."
}
},
"status": 500
}
According to this tutorial Date fields are exposed as ReadableDateTime so they support methods like getYear, and getDayOfWeek. and indeed, the Reference mentions those as supported methods.
Still, the response mentions [java.lang.String] as the type of the dateOfBirth property. I could just parse it to e.g. an OffsetDateTime, but I wonder why it is a string.
Anyone has a suggestion what I'm doing wrong?
Related
Any help would be greatly appreciated. Also, I would like to thank you in advance!
When trying to create the autorest client I am getting the following error, but I cannot figure out how to resolve it. The Web API is a 3rd party API and we cannot change it. I am trying to target net6.0.
Autorest Command:
autorest --csharp --input-file=swagger_doc.json --namespace=MyNameSpace --output-folder=AutoRest --override-client-name=WeatherAPIClient .\custom-config.json --v3
Error Details:
info | AutoRest core version selected from configuration: ^3.2.0.
info | Loading AutoRest core 'C:\Users\dfazekas.autorest#autorestcore#3.9.4\nodemodules#autorest\core\dist' (3.9.4)
info | Loading AutoRest extension '#autorest/csharp' (latest->3.0.0-beta.20230115.1)
info | Loading AutoRest extension '#autorest/modelerfour' (4.25.0->4.25.0)
error | InvalidRef | Ref '#/components/schemas/Microsoft.AspNetCore.JsonPatch.Operations.Operation' is not referencing a valid location. paths,/Item/{Id},patch,requestBody,content,application/json-patch+json,schema,items
- file:///C:/SHI/source/myapp/MyNameSpace.WeatherAPIClient/swagger_doc.json:1211:17
Swagger Doc from error:
"/Item/{Id}": {
"patch": {
"tags": [
"Item"
],
"summary": "Update one or more fields of a resource",
"description": "Update one or more fields of a resource",
"parameters": [
{
"name": "Id",
"in": "path",
"required": true,
"schema": {
"type": "integer",
"format": "int32"
}
}
],
"requestBody": {
"content": {
"application/json-patch+json": {
"schema": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Microsoft.AspNetCore.JsonPatch.Operations.Operation"
}
}
},
"application/json": {
"schema": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Microsoft.AspNetCore.JsonPatch.Operations.Operation"
}
}
}
},
"required": true
},
"responses": {
"200": {
"description": "Success"
},
"400": {
"description": "Bad Request",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Microsoft.AspNetCore.Mvc.ProblemDetails"
}
}
}
},
"500": {
"description": "Server Error"
}
}
}
},
I have a Notion database that I'm trying to write to via the API.
I can write to text fields, but when I try to write to a URL field, I receive the following error:
{
"object": "error",
"status": 400,
"code": "validation_error",
"message": "body failed validation. Fix one:\nbody.properties.O?HT.id should be defined, instead was `undefined`.\nbody.properties.O?HT.name should be defined, instead was `undefined`.\nbody.properties.O?HT.start should be defined, instead was `undefined`."
}
This is the API description of that field:
"LinkedIn": {
"id": "O%3FHT",
"name": "LinkedIn",
"type": "url",
"url": {}
},
This is the body of the API POST request that's failing:
{
"parent": {
"database_id": "f016c02beeff4a34bf298ceb8a8a079f"
},
"properties": {
"title": [
{
"text": {
"content": "Mark Zuckerberg"
}
}
],
"%3DCzE": [
{
"text": {
"content": "CEO"
}
}
],
"HihI": [
{
"text": {
"content": "Facebook"
}
}
],
"JC%3BS": [
{
"text": {
"content": "Menlo Park, CA"
}
}
],
"O%3FHT": {
"url": "https://www.linkedin.com/in/mark-zuckerberg/"
},
"wS%7BN": {
"url": "https://www.linkedin.com/company/facebook/mycompany/verification/"
}
}
}
I'm basing the body contents on the Postman docs provided by Notion: https://www.postman.com/notionhq/workspace/notion-s-api-workspace/documentation/15568543-d990f9b7-98d3-47d3-9131-4866ab9c6df2
I'm following this page in order to create a new PowerBI datasource.
And used this kind of body for the POST request:
{
"connectionDetails": "{\"server\":\"aaa\",\"database\":\"bbb\"}",
"credentialDetails": {
"credentialType": "Basic",
"credentials": "{\"credentialData\":[{\"name\":\"username\", \"value\":\"ccc\"},{\"name\":\"password\", \"value\":\"XoZT6aM1r0puO\"}]}",
"encryptedConnection": "Encrypted",
"encryptionAlgorithm": "RSA-OAEP",
"privacyLevel": "None",
"useEndUserOAuth2Credentials": "False"
},
"datasourceName": "ddd",
"dataSourceType": "Sql"
}
But getting the below error for the POST request:
{
"error": {
"code": "DM_GWPipeline_UnknownError",
"pbi.error": {
"code": "DM_GWPipeline_UnknownError",
"parameters": {},
"details": [
{
"code": "DM_ErrorDetailNameCode_UnderlyingErrorMessage",
"detail": {
"type": 1,
"value": "The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters. "
}
},
{
"code": "DM_ErrorDetailNameCode_UnderlyingHResult",
"detail": {
"type": 1,
"value": "-2146233033"
}
}
]
}
}
}
Any advice would be appreciated!
Thanks in advance.
BTW, as an alternative I would be happy to know if there is a powershell code that was found as working.
I am working on trying to correctly document our rest endpoints. As an example to get this working I created a sample "Healthcheck getStatus()" endpoint which is returning an object called "EndpointStatus" which has 3 fields (class is below). I was able to get this object documenting correctly and using the camel-swagger-java component and the below rest configuration / definition;
restConfiguration()
.apiContextPath(apiContextPath)
.apiProperty("api.title", "Camel Service").apiProperty("api.version", "1.0.0")
// and enable CORS
.apiProperty("cors", "true");
rest()
.path("/healthcheck")
.description("Health Check REST service")
.get("getStatus/{endpointName}")
.param()
.name("endpointName")
.type(RestParamType.path)
.allowableValues(
Stream.of(EndpointName.values())
.map(EndpointName::name)
.collect(Collectors.toList()))
.required(true)
.endParam()
.description("Get Camel Status")
.id("getStatus")
.outType(EndpointStatus.class)
.bindingMode(RestBindingMode.auto)
.responseMessage().code(200).message("Returns an EndpointStatus object representing state of a camel endpoint").endResponseMessage()
.to(CAMEL_STATUS_URI);
Here are the annotations I used on this class:
#ApiModel(description = "Endpoint Status Model")
public class EndpointStatus {
private boolean isAvailable;
private EndpointName name;
private long timestamp;
#ApiModelProperty(value = "Is the endpoint available", required = true)
public boolean isAvailable() {
return isAvailable;
}
public void setAvailable(boolean available) {
isAvailable = available;
}
#ApiModelProperty(value = "The name of the endpoint", required = true)
public EndpointName getName() {
return name;
}
public void setName(EndpointName name) {
this.name = name;
}
#ApiModelProperty(value = "The timestamp the endpoint was checked", required = true)
public long getTimestamp() {
return timestamp;
}
public void setTimestamp(long timestamp) {
this.timestamp = timestamp;
}
}
Along with the generated swagger documentation:
{
"swagger": "2.0",
"info": {
"version": "1.0.0",
"title": "Camel Service"
},
"host": "localhost:9000",
"tags": [
{
"name": "healthcheck",
"description": "Health Check REST service"
}
],
"schemes": [
"http"
],
"paths": {
"/healthcheck/getStatus/{endpointName}": {
"get": {
"tags": [
"healthcheck"
],
"summary": "Get Camel Status",
"operationId": "getStatus",
"parameters": [
{
"name": "endpointName",
"in": "path",
"required": true,
"type": "string",
"enum": [
"ENDPOINTA",
"ENDPOINTB"
]
}
],
"responses": {
"200": {
"description": "Returns an EndpointStatus object representing state of a camel endpoint",
"schema": {
"$ref": "#/definitions/EndpointStatus"
}
}
}
}
}
},
"definitions": {
"EndpointStatus": {
"type": "object",
"required": [
"available",
"name",
"timestamp"
],
"properties": {
"name": {
"type": "string",
"description": "The name of the endpoint",
"enum": [
"ENDPOINTA",
"ENDPOINTB"
]
},
"timestamp": {
"type": "integer",
"format": "int64",
"description": "The timestamp the endpoint was checked"
},
"available": {
"type": "boolean",
"description": "Is the endpoint available"
}
},
"description": "Endpoint Status Model"
}
}
}
However, when trying to move to use camel-openapi-java which supports OpenAPI Specification v3 with the same setup I am getting EndpointStatus without any fields / descriptions in my documentation.
{
"openapi": "3.0.2",
"info": {
"title": "SurePath Camel Service",
"version": "1.0.0"
},
"servers": [
{
"url": ""
}
],
"paths": {
"/healthcheck/getStatus/{endpointName}": {
"get": {
"tags": [
"healthcheck"
],
"parameters": [
{
"name": "endpointName",
"schema": {
"enum": [
"ENDPOINTA",
"ENDPOINTB"
],
"type": "string"
},
"in": "path",
"required": true
}
],
"responses": {
"200": {
"description": "Returns an EndpointStatus object representing state of a camel endpoint"
}
},
"operationId": "getStatus",
"summary": "Get Camel Status",
"x-camelContextId": "camel-1",
"x-routeId": "getStatus"
}
},
"/healthcheck/isAvailable": {
"get": {
"tags": [
"healthcheck"
],
"responses": {
"200": {
"description": "Returns status code 200 when Camel is available"
}
},
"operationId": "verb1",
"summary": "Is Camel Available",
"x-camelContextId": "camel-1",
"x-routeId": "route4"
}
}
},
"components": {
"schemas": {
"EndpointStatus": {
"type": "EndpointStatus",
"x-className": {
"format": "com.sample.bean.EndpointStatus",
"type": "string"
}
}
}
},
"tags": [
{
"name": "healthcheck",
"description": "Health Check REST service"
}
]
}
I have tried adding this into my responseMessage and it is still not documenting correctly;
responseMessage().code(200).responseModel(EndpointStatus.class).message("Returns an EndpointStatus object representing state of a camel endpoint").endResponseMessage()
Do I need different annotations / RestDefinition config to get this EndpointStatus class appearing correctly in the OpenAPI documentation?
This looks to be an issue at the moment with the camel-openapi-java component; waiting for a resolution from this jira https://issues.apache.org/jira/browse/CAMEL-15158
I am trying to change the order of my results but I keep getting an error saying You need an index to execute this query.
In my console, I doesn't say that any indices exist, but I set most of the indexed options to true.
I know in Java, I can create indices that relate to multiple properties either ascending or descending, how do I do this with the REST API?
Following the REST API docs for Google Datastore, my entities are created like this:
{
"mode": "TRANSACTIONAL",
"transaction": "Eb2wksWfYDjkGkkABRmGMQ_vKGijwNwm-tbxAbUPRt8N2RaUCynjSbGT7jFQw3pgaDCT7U0drs3RTPLSIN8TQikdqkdl7pLm2rkMqORmKlO_I_dp",
"mutation": {
"insertAutoId": [
{
"key": {
"path": [
{
"kind": "Attendance"
}
]
},
"properties": {
"section": {
"indexed": true,
"stringValue": "Venturers"
},
"date": {
"dateTimeValue": "2015-01-16T00:00:00+00:00",
"indexed": true
},
"attendee": {
"indexed": true,
"keyValue": {
"path": [
{
"id": "5659313586569216",
"kind": "Attendee"
}
]
}
},
"presence": {
"indexed": false,
"integerValue": 0
}
}
}
]
}
}
And I am trying to query like this:
{
"gqlQuery": {
"allowLiteral": true,
"queryString": "SELECT * FROM Attendance WHERE section = #section ORDER BY date ASC",
"nameArgs": [
{
"name": "section",
"value": {
"stringValue": "Venturers"
}
}
]
}
}
And I get this error:
{
"error": {
"errors": [
{
"domain": "global",
"reason": "FAILED_PRECONDITION",
"message": "no matching index found.",
"locationType": "header",
"location": "If-Match"
}
],
"code": 412,
"message": "no matching index found."
}
}
For future reference:
You can't make a composite index directly through the REST API. You must go through php app engine.
How to build datastore indexes (PHP GAE)