How can I ignore a "$" in a DocumentContent to save in MongoDB? - mongodb

My Problem is, that if I save a Document with a $ inside the content, Mongodb gives me an exception:
java.lang.IllegalArgumentException: Invalid BSON field name $ xxx
I would like that mongodb ignores the $ character in the content.
My Application is written in java. I read the content of the File and put it as a string into an object. After that the object will be saved with a MongoRepository class.
Someone has any ideas??
Example content
Edit: I heard mongodb has the same problem wit dot. Our Springboot found i workaround with dot, but not for dollar.
How to configure mongo converter in spring to encode all dots in the keys of map being saved in mongo db

If you are using Spring Boot you can extend MappingMongoConverter class and add override methods that do the escaping/unescaping.
#Component
public class MappingMongoConverterCustom extends MappingMongoConverter {
protected #Nullable
String mapKeyDollarReplacemant = "characters_to_replace_dollar";
protected #Nullable
String mapKeyDotReplacement = "characters_to_replace_dot";
public MappingMongoConverterCustom(DbRefResolver dbRefResolver, MappingContext<? extends MongoPersistentEntity<?>, MongoPersistentProperty> mappingContext) {
super(dbRefResolver, mappingContext);
}
#Override
protected String potentiallyEscapeMapKey(String source) {
if (!source.contains(".") && !source.contains("$")) {
return source;
}
if (mapKeyDotReplacement == null && mapKeyDollarReplacemant == null) {
throw new MappingException(String.format(
"Map key %s contains dots or dollars but no replacement was configured! Make "
+ "sure map keys don't contain dots or dollars in the first place or configure an appropriate replacement!",
source));
}
String result = source;
if(result.contains(".")) {
result = result.replaceAll("\\.", mapKeyDotReplacement);
}
if(result.contains("$")) {
result = result.replaceAll("\\$", mapKeyDollarReplacemant);
}
//add any other replacements you need
return result;
}
#Override
protected String potentiallyUnescapeMapKey(String source) {
String result = source;
if(mapKeyDotReplacement != null) {
result = result.replaceAll(mapKeyDotReplacement, "\\.");
}
if(mapKeyDollarReplacemant != null) {
result = result.replaceAll(mapKeyDollarReplacemant, "\\$");
}
//add any other replacements you need
return result;
}
}
If you go with this approach make sure you override the default converter from AbstractMongoConfiguration like below:
#Configuration
public class MongoConfig extends AbstractMongoConfiguration{
#Bean
public DbRefResolver getDbRefResolver() {
return new DefaultDbRefResolver(mongoDbFactory());
}
#Bean
#Override
public MappingMongoConverter mappingMongoConverter() throws Exception {
MappingMongoConverterCustom converter = new MappingMongoConverterCustom(getDbRefResolver(), mongoMappingContext());
converter.setCustomConversions(customConversions());
return converter;
}
.... whatever you might need extra ...
}

Related

Solr custom query component does not return correct facet counts

I have a simple Solr query component as follows:
public class QueryPreprocessingComponent extends QueryComponent implements PluginInfoInitialized {
private static final Logger LOG = LoggerFactory.getLogger( QueryPreprocessingComponent.class );
private ExactMatchQueryProcessor exactMatchQueryProcessor;
public void init( PluginInfo info ) {
initializeProcessors(info);
}
private void initializeProcessors(PluginInfo info) {
List<PluginInfo> queryPreProcessors = info.getChildren("queryPreProcessors")
.get(0).getChildren("queryPreProcessor");
for (PluginInfo queryProcessor : queryPreProcessors) {
initializeProcessor(queryProcessor);
}
}
private void initializeProcessor(PluginInfo queryProcessor) {
QueryProcessorParam processorName = QueryProcessorParam.valueOf(queryProcessor.name);
switch(processorName) {
case ExactMatchQueryProcessor:
exactMatchQueryProcessor = new ExactMatchQueryProcessor(queryProcessor.initArgs);
LOG.info("ExactMatchQueryProcessor initialized...");
break;
default: throw new AssertionError();
}
}
#Override
public void prepare( ResponseBuilder rb ) throws IOException
{
if (exactMatchQueryProcessor != null) {
exactMatchQueryProcessor.modifyForExactMatch(rb);
}
}
#Override
public void process(ResponseBuilder rb) throws IOException
{
// do nothing - needed so we don't execute the query here.
return;
}
}
This works as expected functionally except when I use this in a distributed request, it has an issue with facets counts returned. It doubles the facet counts.
Note that I am not doing anything related to faceting in plugin. exactMatchQueryProcessor.modifyForExactMatch(rb); does a very minimal processing if the query is quoted otherwise it does nothing. Even if the incoming query is not quoted, facet count issue is there. Even if I comment everything inside prepare function, issue persists.
Note that this component is declared in as first-components in solrconfig.xml.
I resolved this issue by extending the class to SearchComponent instead of QueryComponent. It seems that SearchComponent sits at higher level of abstraction than QueryComponent and is useful when you want to work on a layer above shards.

Writable Classes in mapreduce

How can i use the values from hashset (the docid and offset) to the reduce writable so as to connect map writable with reduce writable?
The mapper (LineIndexMapper) works fine but in the reducer (LineIndexReducer) i get the error that it can't get string as argument when i type this:
context.write(key, new IndexRecordWritable("some string");
although i have the public String toString() in the ReduceWritable too.
I believe the hashset in reducer's writable (IndexRecordWritable.java) maybe isn't taking the values correctly?
I have the below code.
IndexMapRecordWritable.java
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.Writable;
public class IndexMapRecordWritable implements Writable {
private LongWritable offset;
private Text docid;
public LongWritable getOffsetWritable() {
return offset;
}
public Text getDocidWritable() {
return docid;
}
public long getOffset() {
return offset.get();
}
public String getDocid() {
return docid.toString();
}
public IndexMapRecordWritable() {
this.offset = new LongWritable();
this.docid = new Text();
}
public IndexMapRecordWritable(long offset, String docid) {
this.offset = new LongWritable(offset);
this.docid = new Text(docid);
}
public IndexMapRecordWritable(IndexMapRecordWritable indexMapRecordWritable) {
this.offset = indexMapRecordWritable.getOffsetWritable();
this.docid = indexMapRecordWritable.getDocidWritable();
}
#Override
public String toString() {
StringBuilder output = new StringBuilder()
output.append(docid);
output.append(offset);
return output.toString();
}
#Override
public void write(DataOutput out) throws IOException {
}
#Override
public void readFields(DataInput in) throws IOException {
}
}
IndexRecordWritable.java
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.util.HashSet;
import org.apache.hadoop.io.Writable;
public class IndexRecordWritable implements Writable {
// Save each index record from maps
private HashSet<IndexMapRecordWritable> tokens = new HashSet<IndexMapRecordWritable>();
public IndexRecordWritable() {
}
public IndexRecordWritable(
Iterable<IndexMapRecordWritable> indexMapRecordWritables) {
}
#Override
public String toString() {
StringBuilder output = new StringBuilder();
return output.toString();
}
#Override
public void write(DataOutput out) throws IOException {
}
#Override
public void readFields(DataInput in) throws IOException {
}
}
Alright, here is my answer based on a few assumptions. The final output is a text file containing the key and the file names separated by a comma based on the information in the reducer class's comments on the pre-condition and post-condition.
In this case, you really don't need IndexRecordWritable class. You can simply write to your context using
context.write(key, new Text(valueBuilder.substring(0, valueBuilder.length() - 1)));
with the class declaration line as
public class LineIndexReducer extends Reducer<Text, IndexMapRecordWritable, Text, Text>
Don't forget to set the correct output class in the driver.
That must serve the purpose according to the post-condition in your reducer class. But, if you really want to write a Text-IndexRecordWritable pair to your context, there are two ways approach it -
with string as an argument (based on your attempt passing a string when you IndexRecordWritable class constructor is not designed to accept strings) and
with HashSet as an argument (based on the HashSet initialised in IndexRecordWritable class).
Since your constructor of IndexRecordWritable class is not designed to accept String as an input, you cannot pass a string. Hence the error you are getting that you can't use string as an argument. Ps: if you want your constructor to accept Strings, you must have another constructor in your IndexRecordWritable class as below:
// Save each index record from maps
private HashSet<IndexMapRecordWritable> tokens = new HashSet<IndexMapRecordWritable>();
// to save the string
private String value;
public IndexRecordWritable() {
}
public IndexRecordWritable(
HashSet<IndexMapRecordWritable> indexMapRecordWritables) {
/***/
}
// to accpet string
public IndexRecordWritable (String value) {
this.value = value;
}
but that won't be valid if you want to use the HashSet. So, approach #1 can't be used. You can't pass a string.
That leaves us with approach #2. Passing a HashSet as an argument since you want to make use of the HashSet. In this case, you must create a HashSet in your reducer before passing it as an argument to IndexRecordWritable in context.write.
To do this, your reducer must look like this.
#Override
protected void reduce(Text key, Iterable<IndexMapRecordWritable> values, Context context) throws IOException, InterruptedException {
//StringBuilder valueBuilder = new StringBuilder();
HashSet<IndexMapRecordWritable> set = new HashSet<>();
for (IndexMapRecordWritable val : values) {
set.add(val);
//valueBuilder.append(val);
//valueBuilder.append(",");
}
//write the key and the adjusted value (removing the last comma)
//context.write(key, new IndexRecordWritable(valueBuilder.substring(0, valueBuilder.length() - 1)));
context.write(key, new IndexRecordWritable(set));
//valueBuilder.setLength(0);
}
and your IndexRecordWritable.java must have this.
// Save each index record from maps
private HashSet<IndexMapRecordWritable> tokens = new HashSet<IndexMapRecordWritable>();
// to save the string
//private String value;
public IndexRecordWritable() {
}
public IndexRecordWritable(
HashSet<IndexMapRecordWritable> indexMapRecordWritables) {
/***/
tokens.addAll(indexMapRecordWritables);
}
Remember, this is not the requirement according to the description of your reducer where it says.
POST-CONDITION: emit the output a single key-value where all the file names are separated by a comma ",". <"marcello", "a.txt#3345,b.txt#344,c.txt#785">
If you still choose to emit (Text, IndexRecordWritable), remember to process the HashSet in IndexRecordWritable to get it in the desired format.

FlatFileItemWriter not generating the file when using Tasklet approach

I wrote the following code using tasklet approach to generate a file with data.
public class PersonInfoFileWriter implements Tasklet {
#Autowired
PersonInfoFileUtil personInfoFileUtil;
public void write(ExecutionContext executionContext) throws IOException {
List<PersonInfo> personInfoList = null;
FlatFileItemWriter<PersonInfo> flatFileWriter = new FlatFileItemWriter<PersonInfo>();
flatFileWriter.setResource(new FileSystemResource("C:\\test\\"
+ LocalDate.now().format(DateTimeFormatter.BASIC_ISO_DATE) + ".txt"));
try {
flatFileWriter.open(executionContext);
String personName = (String) executionContext.get("personInfo");
//gets the details of the person by name from the database and assign the values to PersonInfo
personInfoList = personInfoFileUtil.setDataForPersonInfoFile(personName);
flatFileWriter.setName("Person-Detail-File");
flatFileWriter.setShouldDeleteIfEmpty(true);
flatFileWriter.setAppendAllowed(true);
flatFileWriter.setLineSeparator("\n");
flatFileWriter.setHeaderCallback(new FlatFileHeaderCallback() {
#Override
public void writeHeader(Writer writer) throws IOException {
writer.write(
"PersonId^Name^Program^ProgramType");
}
});
flatFileWriter.setLineAggregator(new DelimitedLineAggregator<PersonInfo>() {
{
setDelimiter("^");
setFieldExtractor((FieldExtractor<PersonInfo>) new BeanWrapperFieldExtractor<PersonInfo>() {
{
setNames(new String[] { "personId", "name", "program", "programType" });
}
});
}
});
String lines = flatFileWriter.doWrite((List<? extends PersonInfo>) personInfoList);
logger.info(lines); //this prints the information correctly
} finally {
flatFileWriter.close();
}
}
#Override
public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception {
ExecutionContext executionContext = contribution.getStepExecution().getJobExecution().getExecutionContext();
write(executionContext);
return RepeatStatus.FINISHED;
}
}
The above code compiles and runs without errors but it is not generating any file on to the disk.
I tried debugging to check if the fileName and etc values are getting created on to a buffer to write to a disk and everything works as intended except generating and writing the data to a file.
If I write the code using chunk based approach it is working fine.
Please let me know if I am doing any mistake. Thanks for the help in advance.
EDIT: after adding the changes that were suggested the file is getting created on the disk but the file is missing out the header that I have set using setHeaderCallback()
In your write method, you create an instance of FlatFileItemWriter, set some properties on it and then call close on it.
You did not call open() and write() methods, that's why it is not generating an file.

Get and Set attribute values of a class using aspectJ

I am using aspectj to add some field to a existing class and annotate it also.
I am using load time weaving .
Example :- I have a Class customer in which i am adding 3 string attributes. But my issues is that I have to set some values and get it also before my business call.
I am trying the below approach.
In my aj file i have added the below, my problem is in the Around pointcut , how do i get the attribute and set the attribute.
public String net.customers.PersonCustomer.getOfflineRiskCategory() {
return OfflineRiskCategory;
}
public void net.customers.PersonCustomer.setOfflineRiskCategory(String offlineRiskCategory) {
OfflineRiskCategory = offlineRiskCategory;
}
public String net.customers.PersonCustomer.getOnlineRiskCategory() {
return OnlineRiskCategory;
}
public void net.customers.PersonCustomer.setOnlineRiskCategory(String onlineRiskCategory) {
OnlineRiskCategory = onlineRiskCategory;
}
public String net.customers.PersonCustomer.getPersonCommercialStatus() {
return PersonCommercialStatus;
}
public void net.customers.PersonCustomer.setPersonCommercialStatus(String personCommercialStatus) {
PersonCommercialStatus = personCommercialStatus;
}
#Around("execution(* net.xxx.xxx.xxx.DataMigration.populateMap(..))")
public Object invoke(ProceedingJoinPoint joinPoint) throws Throwable {
Object arguments[] = joinPoint.getArgs();
if (arguments != null) {
HashMap<String, String> hMap = (HashMap) arguments[0];
PersonCustomer cus = (PersonCustomer) arguments[1];
return joinPoint.proceed();
}
If anyone has ideas please let me know.
regards,
FT
First suggestion, I would avoid mixing code-style aspectj with annotation-style. Ie- instead of #Around, use around.
Second, instead of getting the arguments from the joinPoint, you should bind them in the pointcut:
Object around(Map map, PersonCustomer cust) :
execution(* net.xxx.xxx.xxx.DataMigration.populateMap(Map, PersonCustomer) && args(map, cust) {
...
return proceed(map, cust);
}
Now, to answer your question: you also need to use intertype declarations to add new fields to your class, so do something like this:
private String net.customers.PersonCustomer.OfflineRiskCategory;
private String net.customers.PersonCustomer.OnlineRiskCategory;
private String net.customers.PersonCustomer.PersonCommercialStatus;
Note that the private keyword here means private to the aspect, not to the class that you declare it on.

How do I add items to GWT ListBox in Uibinder .ui.xml template ?

How to add the listbox items using UiBinder?
It is possible since february 2011 version:
http://code.google.com/p/google-web-toolkit/issues/detail?id=4654
Following this patch you are now able to add items following this syntax:
<g:ListBox>
<g:item value='1'>
first item
</g:item>
<g:item value='2'>
second item
</g:item>
</g:ListBox>
This is a listbox of translations of an enumeration, I suppose this also works for a listbox with string values (version of GWT: 2.1.0)
You only need the renderer for translating the enumeration values.
//UI XML
<g:ValueListBox ui:field="requesterType"/>
//JAVA CODE
#UiField(provided = true)
ValueListBox<RequesterType> requesterType = new ValueListBox<RequesterType>(requesterTypeRenderer);
static EnumRenderer<RequesterType> requesterTypeRenderer = new EnumRenderer<RequesterType>();
public Constructor() {
requesterTypeRenderer.setEmptyValue(Translations.translateEmptyValue(RequesterType.class));
requesterType.setAcceptableValues(Arrays.asList(EnumUtil.getRequesterTypes()));
}
/**
* Translates enum entries. Use setEmptyValue() if you want to have a custom empty value. Default empty value is "".
*
* #param <T>
* an enumeration entry which is to be registered in {#link Translations}
*/
public class EnumRenderer<T extends Enum<?>> extends AbstractRenderer<T> {
private String emptyValue = "";
#Override
public String render(T object) {
if (object == null)
return emptyValue;
return Translations.translate(object);
}
public void setEmptyValue(String emptyValue) {
this.emptyValue = emptyValue;
}
}
GWT ValueListbox otherwise know as a ComboBox or Dropdown component.
Another example that also demonstrates populating the list.
UiBinder...
<g:ValueListBox ui:field="subCategory"/>
Editor...
#UiField(provided = true)
ValueListBox<String> subCategory = new ValueListBox<String>(
new Renderer<String>() {
#Override
public String render(String object) {
String s = "Cats";
if (object != null) {
s = object.toString();
}
return s;
}
#Override
public void render(String object, Appendable appendable)
throws IOException {
render(object);
}
});
Constructor...
List<String> values = new ArrayList<String>();
values.add("Animal Shelters and Rescues");
values.add("Birds");
values.add("Cats");
values.add("Dogs");
values.add("Other Pets");
values.add("Rabbits");
subCategory.setAcceptableValues(values);