How to get document id (_id) inside an Elasticsearch native script - plugins

I've got a native script that runs as a transform when adding a new document. I need the document id but its not passed in as part of the script params. Only the _source is passed in as the params, but not the _id value. Is there any way to get ElasticSearch to pass in the _id? Or some way of reading it ... somehow?
Below is a contrived ElasticSearch native script example that demonstrates what I'm talking about. setNextVar() is called by ElasticSearch and the "ctx" object is passed in. The value is a Map that only has one object in it, the _source object.
But the _id key/value pair is not passed in by ElasticSearch by default. I'm hoping there is a why to config the native script in the mapping json that tells it to pass in the document id.
public class ExampleNativeScript extends AbstractExecutableScript {
/**
* Factory class that serves native script to ElasticSearch
*/
public static class Factory extends AbstractComponent implements NativeScriptFactory {
#Inject
public Factory(Settings settings, String prefixSettings) {
super(settings, prefixSettings);
}
#Override
public ExecutableScript newScript(#Nullable Map<String, Object> params) {
return new ExampleNativeScript(params);
}
}
private final Map<String, Object> variables;
private Map ctx = null;
private Map source = null;
public ExampleNativeScript(Map<String, Object> params) {
this.variables = params;
}
#Override
public void setNextVar(String name, Object value) {
variables.put(name, value);
if (name.equals("ctx")) {
ctx = (Map<String, LinkedHashMap>) value;
source = (Map<String, LinkedHashMap>) ctx.get("_source");
} else {
//never gets here
System.out.println("Unexpected variable");
}
}
#Override
public Object run() {
//PROBLEM: ctx only has _source. _id does not get passed in so this blows chunks
String documentId = ctx.get("_id").toString();
System.out.println(documentId);
return ctx;
}
}

So just did some digging in the ElasticSearch source and found that its hard coded to just pass in the _source field, and none of the other ctx level fields. So there is no way to get/config ElasticSearch to pass in the _id
Below is the DocumentMapper.transformSourceAsMap() function that calls the native script's setNextVar() and run() functions. It shows that the only thing put in the ctx map is the _source field.
public Map<String, Object> transformSourceAsMap(Map<String, Object> sourceAsMap) {
try {
// We use the ctx variable and the _source name to be consistent with the update api.
ExecutableScript executable = scriptService.executable(language, script, scriptType, ScriptContext.Standard.MAPPING, parameters);
Map<String, Object> ctx = new HashMap<>(1);
ctx.put("_source", sourceAsMap);
executable.setNextVar("ctx", ctx);
executable.run();
ctx = (Map<String, Object>) executable.unwrap(ctx);
return (Map<String, Object>) ctx.get("_source");
} catch (Exception e) {
throw new ElasticsearchIllegalArgumentException("failed to execute script", e);
}
}

Related

How to customize the status code of the Spring Cloud Gateway custom exception based on the exception

I found that I could not get the status code of the exception when customizing the exception in the gateway, so I don’t know how to set the custom status code instead of setting all the exception status codes to the same.
Spring Cloud Gateway custom handlers exception handler. Example for version Spring Cloud 2020.0.3 . Common methods have your own defaultErrorWebexceptionHandler or only ErrorAttributes.
Method 1: ErrorWebexceptionHandler (for schematic only). Customize a globalrrorattributes:
#Component
public class GlobalErrorAttributes extends DefaultErrorAttributes{
#Override
public Map<String, Object> getErrorAttributes(ServerRequest request, ErrorAttributeOptions options) {
Throwable error = super.getError(request);
Map<String, Object> map = super.getErrorAttributes(request, options);
map.put("status", HttpStatus.BAD_REQUEST.value());
map.put("message", error.getMessage());
return map;
}
}
#Component
#Order(-2)
public class GlobalErrorWebExceptionHandler extends AbstractErrorWebExceptionHandler {
public GlobalErrorWebExceptionHandler(GlobalErrorAttributes gea, ApplicationContext applicationContext,
ServerCodecConfigurer serverCodecConfigurer) {
super(gea, new WebProperties.Resources(), applicationContext);
super.setMessageWriters(serverCodecConfigurer.getWriters());
super.setMessageReaders(serverCodecConfigurer.getReaders());
}
// Rendering HTML or JSON.
#Override
protected RouterFunction<ServerResponse> getRoutingFunction(final ErrorAttributes errorAttributes) {
return RouterFunctions.route(RequestPredicates.all(), this::renderErrorResponse);
}
private Mono<ServerResponse> renderErrorResponse(final ServerRequest request) {
final Map<String, Object> errorPropertiesMap = getErrorAttributes(request, ErrorAttributeOptions.defaults());
return ServerResponse.status(HttpStatus.BAD_REQUEST)
.contentType(MediaType.APPLICATION_JSON)
.body(BodyInserters.fromValue(errorPropertiesMap));
}
}
Reference document: https://docs.spring.io/spring-boot/docs/current/api/org/springframework/boot/autoconfigure/web/reactive/error/AbstractErrorWebExceptionHandler.html
Method 2: Only one ErroraTributes to overwrite the default defaultrRorattributes .
#Component
public class GatewayErrorAttributes extends DefaultErrorAttributes {
private static final Logger logger = LoggerFactory.getLogger(GatewayErrorAttributes.class);
#Override
public Map<String, Object> getErrorAttributes(ServerRequest request, ErrorAttributeOptions options) {
Throwable error = super.getError(request);
Map<String, Object> errorAttributes = new HashMap<>(8);
errorAttributes.put("message", error.getMessage());
errorAttributes.put("method", request.methodName());
errorAttributes.put("path", request.path());
MergedAnnotation<ResponseStatus> responseStatusAnnotation = MergedAnnotations
.from(error.getClass(), MergedAnnotations.SearchStrategy.TYPE_HIERARCHY).get(ResponseStatus.class);
HttpStatus errorStatus = determineHttpStatus(error, responseStatusAnnotation);
// Must set, otherwise an error will be reported because the RendereRRRRESPONSE method of DefaultErrorWebexceptionHandler gets this property, re-implementing defaultErrorWebexceptionHandler.
errorAttributes.put("status", errorStatus.value());
errorAttributes.put("code", errorStatus.value());
// html view
errorAttributes.put("timestamp", new Date());
// html view
errorAttributes.put("requestId", request.exchange().getRequest().getId());
errorAttributes.put("error", errorStatus.getReasonPhrase());
errorAttributes.put("exception", error.getClass().getName());
return errorAttributes;
}
// Copy from DefaultErrorWebexceptionHandler
private HttpStatus determineHttpStatus(Throwable error, MergedAnnotation<ResponseStatus> responseStatusAnnotation) {
if (error instanceof ResponseStatusException) {
return ((ResponseStatusException) error).getStatus();
}
return responseStatusAnnotation.getValue("code", HttpStatus.class).orElse(HttpStatus.INTERNAL_SERVER_ERROR);
}
}
Attention please: ErroraTributes.put ("status", errorstatus.value ()), otherwise an error is reported because the rendererrorResponse method of DefaultErrorwebexceptionHandler gets this property. Unless you re-implement the defaultErrorWebexceptionhandler like yourself.
Then visit an unduly service in the gateway to see the effect.
curl 'http://127.0.0.1:8900/fundmain22/abc/gogogo?id=1000' --header 'Accept: application/json'
{"exception":"org.springframework.web.server.ResponseStatusException","path":"/fundmain22/abc/gogogo","code":404,"method":"GET","requestId":"094e53e5-1","message":"404 NOT_FOUND","error":"Not Found","status":404,"timestamp":"2021-08-09T11:07:44.106+0000"}

Writable Classes in mapreduce

How can i use the values from hashset (the docid and offset) to the reduce writable so as to connect map writable with reduce writable?
The mapper (LineIndexMapper) works fine but in the reducer (LineIndexReducer) i get the error that it can't get string as argument when i type this:
context.write(key, new IndexRecordWritable("some string");
although i have the public String toString() in the ReduceWritable too.
I believe the hashset in reducer's writable (IndexRecordWritable.java) maybe isn't taking the values correctly?
I have the below code.
IndexMapRecordWritable.java
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.Writable;
public class IndexMapRecordWritable implements Writable {
private LongWritable offset;
private Text docid;
public LongWritable getOffsetWritable() {
return offset;
}
public Text getDocidWritable() {
return docid;
}
public long getOffset() {
return offset.get();
}
public String getDocid() {
return docid.toString();
}
public IndexMapRecordWritable() {
this.offset = new LongWritable();
this.docid = new Text();
}
public IndexMapRecordWritable(long offset, String docid) {
this.offset = new LongWritable(offset);
this.docid = new Text(docid);
}
public IndexMapRecordWritable(IndexMapRecordWritable indexMapRecordWritable) {
this.offset = indexMapRecordWritable.getOffsetWritable();
this.docid = indexMapRecordWritable.getDocidWritable();
}
#Override
public String toString() {
StringBuilder output = new StringBuilder()
output.append(docid);
output.append(offset);
return output.toString();
}
#Override
public void write(DataOutput out) throws IOException {
}
#Override
public void readFields(DataInput in) throws IOException {
}
}
IndexRecordWritable.java
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.util.HashSet;
import org.apache.hadoop.io.Writable;
public class IndexRecordWritable implements Writable {
// Save each index record from maps
private HashSet<IndexMapRecordWritable> tokens = new HashSet<IndexMapRecordWritable>();
public IndexRecordWritable() {
}
public IndexRecordWritable(
Iterable<IndexMapRecordWritable> indexMapRecordWritables) {
}
#Override
public String toString() {
StringBuilder output = new StringBuilder();
return output.toString();
}
#Override
public void write(DataOutput out) throws IOException {
}
#Override
public void readFields(DataInput in) throws IOException {
}
}
Alright, here is my answer based on a few assumptions. The final output is a text file containing the key and the file names separated by a comma based on the information in the reducer class's comments on the pre-condition and post-condition.
In this case, you really don't need IndexRecordWritable class. You can simply write to your context using
context.write(key, new Text(valueBuilder.substring(0, valueBuilder.length() - 1)));
with the class declaration line as
public class LineIndexReducer extends Reducer<Text, IndexMapRecordWritable, Text, Text>
Don't forget to set the correct output class in the driver.
That must serve the purpose according to the post-condition in your reducer class. But, if you really want to write a Text-IndexRecordWritable pair to your context, there are two ways approach it -
with string as an argument (based on your attempt passing a string when you IndexRecordWritable class constructor is not designed to accept strings) and
with HashSet as an argument (based on the HashSet initialised in IndexRecordWritable class).
Since your constructor of IndexRecordWritable class is not designed to accept String as an input, you cannot pass a string. Hence the error you are getting that you can't use string as an argument. Ps: if you want your constructor to accept Strings, you must have another constructor in your IndexRecordWritable class as below:
// Save each index record from maps
private HashSet<IndexMapRecordWritable> tokens = new HashSet<IndexMapRecordWritable>();
// to save the string
private String value;
public IndexRecordWritable() {
}
public IndexRecordWritable(
HashSet<IndexMapRecordWritable> indexMapRecordWritables) {
/***/
}
// to accpet string
public IndexRecordWritable (String value) {
this.value = value;
}
but that won't be valid if you want to use the HashSet. So, approach #1 can't be used. You can't pass a string.
That leaves us with approach #2. Passing a HashSet as an argument since you want to make use of the HashSet. In this case, you must create a HashSet in your reducer before passing it as an argument to IndexRecordWritable in context.write.
To do this, your reducer must look like this.
#Override
protected void reduce(Text key, Iterable<IndexMapRecordWritable> values, Context context) throws IOException, InterruptedException {
//StringBuilder valueBuilder = new StringBuilder();
HashSet<IndexMapRecordWritable> set = new HashSet<>();
for (IndexMapRecordWritable val : values) {
set.add(val);
//valueBuilder.append(val);
//valueBuilder.append(",");
}
//write the key and the adjusted value (removing the last comma)
//context.write(key, new IndexRecordWritable(valueBuilder.substring(0, valueBuilder.length() - 1)));
context.write(key, new IndexRecordWritable(set));
//valueBuilder.setLength(0);
}
and your IndexRecordWritable.java must have this.
// Save each index record from maps
private HashSet<IndexMapRecordWritable> tokens = new HashSet<IndexMapRecordWritable>();
// to save the string
//private String value;
public IndexRecordWritable() {
}
public IndexRecordWritable(
HashSet<IndexMapRecordWritable> indexMapRecordWritables) {
/***/
tokens.addAll(indexMapRecordWritables);
}
Remember, this is not the requirement according to the description of your reducer where it says.
POST-CONDITION: emit the output a single key-value where all the file names are separated by a comma ",". <"marcello", "a.txt#3345,b.txt#344,c.txt#785">
If you still choose to emit (Text, IndexRecordWritable), remember to process the HashSet in IndexRecordWritable to get it in the desired format.

CacheBuilder using guava cache for query resultant

To reduce the DB hits to read the data from DB using the query, I am planning to keep resultant in the cache. To do this I am using guava caching.
studentController.java
public Map<String, Object> getSomeMethodName(Number departmentId, String departmentType){
ArrayList<Student> studentList = studentManager.getStudentListByDepartmentType(departmentId, departmentType);
----------
----------
}
StudentHibernateDao.java(criteria query )
#Override
public ArrayList<Student> getStudentListByDepartmentType(Number departmentId, String departmentType) {
Criteria criteria =sessionFactory.getCurrentSession().createCriteria(Student.class);
criteria.add(Restrictions.eq("departmentId", departmentId));
criteria.add(Restrictions.eq("departmentType", departmentType));
ArrayList<Student> studentList = (ArrayList)criteria.list();
return studentList;
}
To cache the criteria query resultant i started off with building CacheBuilder, like below.
private static LoadingCache<Number departmentId, String departmentType, ArrayList<Student>> studentListCache = CacheBuilder
.newBuilder().expireAfterAccess(1, TimeUnit.MINUTES)
.maximumSize(1000)
.build(new CacheLoader<Number departmentId, String departmentType, ArrayList<Student>>() {
public ArrayList<Student> load(String key) throws Exception {
return getStudentListByDepartmentType(departmentId, departmentType);
}
});
Here I dont know where to put CacheBuilder function and how to pass multiple key parameters(i.e departmentId and departmentType) to CacheLoader and call it.
Is this the correct way of caching using guava. Am I missing anything?
Guava's cache only accepts two type parameters, a key and a value type. If you want your key to be a compound key then you need to build a new compound type to encapsulate it. Effectively it would need to look like this (I apologize for my syntax, I don't use Java that often):
// Compound Key type
class CompoundDepartmentId {
public CompoundDepartmentId(Long departmentId, String departmentType) {
this.departmentId = departmentId;
this.departmentType = departmentType;
}
}
private static LoadingCache<CompoundDepartmentId, ArrayList<Student>> studentListCache =
CacheBuilder
.newBuilder().expireAfterAccess(1, TimeUnit.MINUTES)
.maximumSize(1000)
.build(new CacheLoader<CompoundDepartmentId, ArrayList<Student>>() {
public ArrayList<Student> load(CompoundDepartmentId key) throws Exception {
return getStudentListByDepartmentType(key.departmentId, key.departmentType);
}
});

How can I ignore a "$" in a DocumentContent to save in MongoDB?

My Problem is, that if I save a Document with a $ inside the content, Mongodb gives me an exception:
java.lang.IllegalArgumentException: Invalid BSON field name $ xxx
I would like that mongodb ignores the $ character in the content.
My Application is written in java. I read the content of the File and put it as a string into an object. After that the object will be saved with a MongoRepository class.
Someone has any ideas??
Example content
Edit: I heard mongodb has the same problem wit dot. Our Springboot found i workaround with dot, but not for dollar.
How to configure mongo converter in spring to encode all dots in the keys of map being saved in mongo db
If you are using Spring Boot you can extend MappingMongoConverter class and add override methods that do the escaping/unescaping.
#Component
public class MappingMongoConverterCustom extends MappingMongoConverter {
protected #Nullable
String mapKeyDollarReplacemant = "characters_to_replace_dollar";
protected #Nullable
String mapKeyDotReplacement = "characters_to_replace_dot";
public MappingMongoConverterCustom(DbRefResolver dbRefResolver, MappingContext<? extends MongoPersistentEntity<?>, MongoPersistentProperty> mappingContext) {
super(dbRefResolver, mappingContext);
}
#Override
protected String potentiallyEscapeMapKey(String source) {
if (!source.contains(".") && !source.contains("$")) {
return source;
}
if (mapKeyDotReplacement == null && mapKeyDollarReplacemant == null) {
throw new MappingException(String.format(
"Map key %s contains dots or dollars but no replacement was configured! Make "
+ "sure map keys don't contain dots or dollars in the first place or configure an appropriate replacement!",
source));
}
String result = source;
if(result.contains(".")) {
result = result.replaceAll("\\.", mapKeyDotReplacement);
}
if(result.contains("$")) {
result = result.replaceAll("\\$", mapKeyDollarReplacemant);
}
//add any other replacements you need
return result;
}
#Override
protected String potentiallyUnescapeMapKey(String source) {
String result = source;
if(mapKeyDotReplacement != null) {
result = result.replaceAll(mapKeyDotReplacement, "\\.");
}
if(mapKeyDollarReplacemant != null) {
result = result.replaceAll(mapKeyDollarReplacemant, "\\$");
}
//add any other replacements you need
return result;
}
}
If you go with this approach make sure you override the default converter from AbstractMongoConfiguration like below:
#Configuration
public class MongoConfig extends AbstractMongoConfiguration{
#Bean
public DbRefResolver getDbRefResolver() {
return new DefaultDbRefResolver(mongoDbFactory());
}
#Bean
#Override
public MappingMongoConverter mappingMongoConverter() throws Exception {
MappingMongoConverterCustom converter = new MappingMongoConverterCustom(getDbRefResolver(), mongoMappingContext());
converter.setCustomConversions(customConversions());
return converter;
}
.... whatever you might need extra ...
}

How can I dump the normal properties on an IEnumerable in Linqpad

If I have an object that among other things is an IEnumerable and I dump this object I get the enumerated values.
Is there a way to get Linqpad to list the other properties:
Se example below:
Can I get Dump to include Hello and digits properties?
void Main()
{
var t = new test();
var d = new Dictionary<string,string> {{"Hello","World"},{"Good by","Sky"}};
t.Dump();
d.Dump();
}
// Define other methods and classes here
public class test : IEnumerable
{
public string Hello { get { return "World"; }}
public List<int> digits { get { return new List<int> {0,1,2,3,4,5,6,7,8,9}; }}
public IEnumerator GetEnumerator() { yield return "Hej"; }
}
You could write a DumpPayload extension method as follows:
void Main()
{
var t = new test();
t.DumpPayload();
}
public static class Extensions
{
public static void DumpPayload (this IEnumerable o)
{
if (o == null)
{
o.Dump();
return;
}
var d = new Dictionary<string,object>();
foreach (var member in o.GetType().GetProperties())
try
{
d[member.Name] = member.GetValue (o, null);
}
catch (Exception ex)
{
d[member.Name] = ex;
}
d["AsEnumerable"] = o;
d.Dump();
}
}
If you put this extension method into "My Extensions", it will be available to all queries.
Another solution is to implement ICustomMemberProvider in the test class:
public class test : IEnumerable, ICustomMemberProvider
{
public string Hello { get { return "World"; }}
public List<int> digits { get { return new List<int> {0,1,2,3,4,5,6,7,8,9}; }}
public IEnumerator GetEnumerator() { yield return "Hej"; }
IEnumerable<string> ICustomMemberProvider.GetNames()
{
return "Hello Enumerator".Split();
}
IEnumerable<Type> ICustomMemberProvider.GetTypes ()
{
return new [] { typeof (string), typeof (IEnumerable) };
}
IEnumerable<object> ICustomMemberProvider.GetValues ()
{
return new object [] { Hello, this.OfType<Object>() };
}
}
Note that if the test class is defined in another assembly, you don't need to reference LINQPad in order to implement ICustomMemberProvider. You can just paste in the following definition into your project and LINQPad will pick it up:
namespace LINQPad
{
public interface ICustomMemberProvider
{
IEnumerable<string> GetNames ();
IEnumerable<Type> GetTypes ();
IEnumerable<object> GetValues ();
}
}
As far as I can tell if the object you're trying to dump implements IEnumerable then LINQPad always wants to dump it as an IEnumerable list. Getting rid of the interface correctly shows the Hello and digits properties in the dumped info.
Going from this link it appears you can write your own dump which accomplishes something like LINQPad by enumerating the collection and all it's properties then outputting the whole thing as an XHTML string. I haven't tried this.
Use a Serializer?
Json.NET will do all of this for you in a json format.
Newtonsoft.Json.JsonConvert.SerializeObject(t, Newtonsoft.Json.Formatting.Indented)
if you don't want json, then pick a serializer you do want, or you'll just have to do what a serializer would do, use reflection to iterate the properties on the object.