When performing a data import from mongodb, Solr throws the following error:
org.apache.solr.common.SolrException: TransactionLog doesn't know how to serialize class org.bson.types.ObjectId; try implementing ObjectResolver?
at org.apache.solr.update.TransactionLog$1.resolve(TransactionLog.java:100)
at org.apache.solr.common.util.JavaBinCodec.writeVal(JavaBinCodec.java:234)
at org.apache.solr.common.util.JavaBinCodec.writeSolrInputDocument(JavaBinCodec.java:589)
at org.apache.solr.update.TransactionLog.write(TransactionLog.java:395)
at org.apache.solr.update.UpdateLog.add(UpdateLog.java:532)
at org.apache.solr.update.UpdateLog.add(UpdateLog.java:516)
at org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:320)
at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239)
at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:194)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:979)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1192)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:748)
at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
at org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:80)
at org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:254)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:526)
at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)
at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)
at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)
at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:415)
at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:474)
at org.apache.solr.handler.dataimport.DataImporter.lambda$runAsync$0(DataImporter.java:457)
at java.lang.Thread.run(Thread.java:748)
My Solr version is 6.6.0. What could be the reason for the error and how can it be resolved?
I came across this issue while trying to import data from multiple collections in mongoDB.
Assuming you are not using mongo-connector, I used the following to import data.
Solr-6.6.0
solr-dataimporthandler-6.6.0
mongo-java-driver-3.5.0
Solr Mongo importer
Since the returned '_id' is of type ObjectId, my work around solution was to convert the '_id' to String before indexing it into solr and while querying with respect to '_id', convert it to type ObjectId before running the query.
Download the solr mongo importer and make the following changes.
MongoMapperTransformer.java
public class MongoMapperTransformer extends Transformer {
#Override
public Object transformRow(Map<String, Object> row, Context context) {
for (Map<String, String> map : context.getAllEntityFields()) {
String mongoFieldName = map.get(MONGO_FIELD);
String mongoId = map.get(MONGO_ID);
if (mongoFieldName == null)
continue;
String columnFieldName = map.get(DataImporter.COLUMN);
//If the field is ObjectId convert it into String
if (mongoId != null && Boolean.parseBoolean(mongoId)) {
Object srcId = row.get(columnFieldName);
row.put(columnFieldName, srcId.toString());
}
else{
row.put(columnFieldName, row.get(mongoFieldName));
}
}
return row;
}
public static final String MONGO_FIELD = "mongoField";
//To identify the _id field
public static final String MONGO_ID = "objectIdToString";
}
Next, Replace the function
public Iterator <Map<String, Object>> getData(String query){...}
in MongoDataSource.java with the following:
#Override
public Iterator<Map<String, Object>> getData(String query) {
DBObject queryObject = new BasicDBObject();
/* If querying by _id, since the id is a string now,
* it has to be converted back to type ObjectId() using the
* constructor
*/
if(query.contains("_id")){
#SuppressWarnings("unchecked")
Map<String, String> queryWithId = (Map<String, String>) JSON.parse(query);
String id = queryWithId.get("_id");
queryObject = new BasicDBObject("_id", new ObjectId(id));
}
else{
queryObject = (DBObject) JSON.parse(query);
}
LOG.debug("Executing MongoQuery: " + query.toString());
long start = System.currentTimeMillis();
mongoCursor = this.mongoCollection.find(queryObject);
LOG.trace("Time taken for mongo :"
+ (System.currentTimeMillis() - start));
ResultSetIterator resultSet = new ResultSetIterator(mongoCursor);
return resultSet.getIterator();
}
After these changes you can build the jar using ant.
Copy the jars (solr mongo importer and the mongo-java-driver) into the lib directory. I copied them into ${solr-install-dir}/contrib/dataimport-handler/lib
Add the lib directives in solr-config.xml for the above jars:
<lib dir="${solr.install.dir:../../../..}/contrib/dataimporthandler/lib" regex=".*\.jar" />
Finally, here's an example of the mongo collections and data-config.xml
User collection
{
"_id" : ObjectId("56e9c892e4b0355017b2fa0f"),
"name" : "User1",
"phone" : "123456789"
}
Address collection
{
"_id" : ObjectId("56e9c892e4b0355017b2fa0f"),
"address" : "#666, Maiden street"
}
data-config.xml
Do not forget to mention objectIdToString="true" for the _id field so that the MongoMapperTransformer can stringify the id.
<dataConfig>
<dataSource name="MyMongo"
type="MongoDataSource"
database="test"
/>
<document name="UserDetails">
<!-- if query="" then it imports everything -->
<entity name="users"
processor="MongoEntityProcessor"
query=""
collection="user"
datasource="MyMongo"
transformer="MongoMapperTransformer">
<field column="_id" name="id" mongoField="_id" objectIdToString="true" />
<field column="phone" name="phone" mongoField="phone"/>
<entity name="address"
processor="MongoEntityProcessor"
query="{_id:'${users._id}'}"
collection="address"
datasource="MyMongo"
transformer="MongoMapperTransformer">
<field column="address" name="adress" mongoField="address"/>
</entity>
</entity>
</document>
</dataConfig>
The managed-schema will have the id field as string.
Also, if you have nested objects in mongodb you will have to use script transformers to index them in solr.
Hope this helps,
Good luck !
According to the error message,
You need to implement JavaBinCodec.ObjectResolver for org.bson.types.ObjectId type, so Solr will know how to serialize instances of this class.
JavaBinCodec.ObjectResolver Documentation
public static interface JavaBinCodec.ObjectResolver Allows extension
of JavaBinCodec to support serialization of arbitrary data types.
Implementors of this interface write a method to serialize a given
object using an existing JavaBinCodec
Once you write your JavaBinCodec.ObjectResolver implementation you should register it using JavaBinCodec
JavaBinCodec Documentation
public class JavaBinCodec extends Object Defines a space-efficient
serialization/deserialization format for transferring data.
JavaBinCodec has built in support many commonly used types. This
includes primitive types (boolean, byte, short, double, int, long,
float), common Java containers/utilities (Date, Map, Collection,
Iterator, String, Object[], byte[]), and frequently used Solr types
(NamedList, SolrDocument, SolrDocumentList). Each of the above types
has a pair of associated methods which read and write that type to a
stream.
Classes that aren't supported natively can still be
serialized/deserialized by providing an JavaBinCodec.ObjectResolver
object that knows how to work with the unsupported class. This allows
JavaBinCodec to be used to marshall/unmarshall arbitrary content.
NOTE -- JavaBinCodec instances cannot be reused for more than one
marshall or unmarshall operation.
Related
I have this camel route:
from("direct:getUser")
.pollEnrich("jpa://User?namedQuery=User.findById&consumeDelete=false");
This is my user Entity:
#Entity
#NamedQueries({
#NamedQuery(name="User.findAll", query="SELECT u FROM User u"),
#NamedQuery(name="User.findById", query="SELECT u FROM User u WHERE u.id = :id")
})
public class User{
#Id
private String id;
}
I have tried this route by setting the header:
from("direct:getUser")
.setHeader("id", simple("myid"))
.pollEnrich("jpa://User?namedQuery=User.findById&consumeDelete=false");
But it is not working
Is there any method to set jpa properties by the headers? The camel documentation quote this in parameters option but i don't found the examples
Options: parameters
This option is Registry based which requires the # notation. This
key/value mapping is used for building the query parameters. It is
expected to be of the generic type java.util.Map where
the keys are the named parameters of a given JPA query and the values
are their corresponding effective values you want to select for. Camel
2.19: it can be used for producer as well. When it's used for producer, Simple expression can be used as a parameter value. It
allows you to retrieve parameter values from the message body header
and etc.
I hope it's not too late to answer. In any case I had a similar issue in my project, the client does a HTTP GET with a parameter id, which is used by the JPA query and the result is finally marshalled back to the HTTP client. I'm running camel in a Spring application.
I finally figured out how to achieve it in a reasonably clean way.
This is the RouteBuilder where the route is defined:
#Override
public void configure() throws Exception {
Class dataClass = SomeClass.class;
JacksonDataFormat format = new JacksonDataFormat();
format.setUnmarshalType(dataClass);
String jpaString = String
.format("jpa://%1$s?resultClass=%1$s&namedQuery=q1" +
"¶meters={\"id\":${headers.id}}", dataClass.getName());
from("jetty://http://localhost:8080/test").toD(jpaString) // note the .toD
.marshal(format)
}
And this is the StringToMapTypeConverter class, otherwise camel cannot convert {"id": X} to a map
public class StringToMapTypeConverter implements TypeConverters {
private static final ObjectMapper mapper = new ObjectMapper();
private static JavaType mapType;
static {
mapType = mapper.getTypeFactory().constructMapType(Map.class,
String.class, Object.class);
}
#Converter
public Map<String, Object> toMap(String map) throws IOException {
return mapper.readValue(map, mapType);
}
}
Remember to add it to the context. In Spring is something like:
<bean id="myStringToMapTypeConverter" class="....StringToMapTypeConverter" />
Refs:
http://camel.apache.org/jpa.html
http://camel.apache.org/message-endpoint.html#MessageEndpoint-DynamicTo
http://camel.apache.org/type-converter.html#TypeConverter-Addtypeconverterclassesatruntime
I've got a Product with a Rating rating attribute. I've got a product update form (updateStart method) which doesn't contain the rating field (since I don't want it to be editable).
The problem is that when I submit the form (with update method), the rating is automatically set to null.
So I tried to add the Rating to the form model in updateStart, retrieving it in the update method, but it keeps being rewritten as well.
I tried to set a #SessionAttributes("rating") annotation in the controller. This time the rating value is kept, but Spring creates a new entry in the database, cloned from the other rating object, and attaches it to the Product.
#Controller
#SessionAttributes("rating")
#RequestMapping("/products")
public class ProductsController {
#RequestMapping("/update_start")
public String updateStart(#RequestParam("id") Long id, Model model) throws BusinessException {
Product product = productService.findProductById(id);
System.out.println("RATING A START "+product.getRating().getAbsoluteRating());
List<Category> categories = productService.findAllCategories();
model.addAttribute("categories", categories);
model.addAttribute("product", product);
model.addAttribute("id", id);
model.addAttribute("rating",product.getRating());
return "products.updateform";
}
#RequestMapping(value="/update", method = RequestMethod.POST)
public String update(#ModelAttribute("rating") Rating rating, #ModelAttribute Product product, BindingResult bindingResult) throws BusinessException {
System.out.println("RATING A UPDATE "+rating.getAbsoluteRating());
validator.validate(product, bindingResult);
List<Image> images = imageService.getProductImages(product.getId());
product.setRating(rating);
productService.updateProduct(product,images,sellerid);
return "redirect:/products/viewsforsellers.do";
}
}
What can I do?
EDIT: I'd prefer to avoid placing a hidden input field with ratingId in my form.
In the form include a hidden input with the name and value specified for the Rating. The value should include
<form>
<input name="product.rating" value="${product.rating.id}"/>
<!-- Other fields -->
</form>
Now when the request comes over the wire it should include a Rating specified by id for the product.
#RequestMapping(value="/update", method = RequestMethod.POST)
public String update(#ModelAttribute Product product, BindingResult bindingResult) throws BusinessException {
//implementation
}
#ModelAttribute should attempt to bind this parameter to the Product however it is not aware of what a Rating is. This is where a Converter comes into play. A Converter is used during databinding to tell Spring MVC how to map a field of type String to a field of type Rating.
public class StringToRatingConverter implements Converter<String, Rating> {
public Rating convert(String source) {
//Use the source String to convert to rating
//Possibly via database call or enum conversion, pending ratings type and definition
//Ultimately the code needs to return the appropriate object of type Rating
return rating; //The above implementation will create the rating object.
}
}
The StringToRatingConverter must then be registered in the dispatcher configuration file.
<!-- Register Converters - Used for data binding-->
<bean id="conversionService" class="org.springframework.context.support.ConversionServiceFactoryBean">
<property name="converters">
<list>
<bean class="fully.qualified.path.to.StringToRatingConverter"/>
</list>
</property>
</bean>
The first time I encountered this scenario, I captured it in a post on my blog, which you may be helpful.
You should add "types" element to your #SessionAttributes("rating") annotation in order properties of attributes to be kept; e.g.
#SessionAttributes(types = Rating.class, names = "rating")
I would like to have an embedded document referred to by a map (as in 'class A' below). The environment is Grails + GORM + MongoDB.
is that possible, and if yes, how?
class A { // fails with IllegalArgumentException occurred when processing request: can't serialize class X in line 234 of org.bson.BasicBSONEncoder
static mapWith = "mongo"
Map<String, X> map = new HashMap<String, X>()
}
class B { // works
static mapWith = "mongo"
List<X> list = new ArrayList<X>()
}
class C { // works with primitive type values
static mapWith = "mongo"
Map<String, String> map = new HashMap<String, String>()
}
class X {
String data
public X(String data) {
this.data = data
}
}
The embedding works perfectly,as Art Hanzel advised.
However your problem comes from the fact that you try and use List genericity as a sort of constraint :
Map<String, X>
The problem is that Grails couldn't cope well with this syntax, first because Groovy doesn't support genericity.
However, the MongoDB plugin offers a very powerful functionality that lets you define custom type as Domain Class Properties : see here.
In your case you could have
class A {
static mapWith = "mongo"
MyClass map = new MyClass()
}
Then in your src/java for example you could for example implement a
class MyClass extends HashMap<String,X> { }
Then, of course, you have to define a special AbstractMappingAwareCustomTypeMarshaller to specify how to read and write the property in the DB.
An additional step could also be to add a custom validator to class A to check the validity of data...
The MongoDB Grails plugin documentation describes how to make embedded documents:
class Foo {
Address address
List otherAddresses
static embedded = ['address', 'otherAddresses']
}
Off the top of my head, you should be able to access these via the object graph. I don't see any reason why you shouldn't.
myFoo.address.myAddressProperty...
I start understanding how jersey works with JAXB. But today i faced a particular case where i want to marshall a Map of (Date,List) entries:
#XmlRootElement
#XmlAccessorType(XmlAccessType.FIELD)
public class MyClass{
#XmlJavaTypeAdapter(MapAdapter.class)
private Map<Date,List<MyObject>> = new TreeMap<Date,List<MyObject>>(new DateCompareDesc());
}
The goal here is to marshall a Map whose entry is a Date with its corresponding list of MyObject. the map is sorted in desc order.
For this i implemented an Adapter for Map (MapAdapter, following #Blaise Doughan's tutorial, http://blog.bdoughan.com/2010/07/xmladapter-jaxbs-secret-weapon.html). The problem is on the Date key. I have an Error : Unable to marshall java.util.Date. So i tried this new Date Adapter :
public class DateAdapter extends XmlAdapter<String, Date> {
#Override
public Date unmarshal(String v) throws Exception {
//not implemented
}
#Override
public String marshal(Date v) throws Exception {
return v.toString();
}
}
Where can i add #XmlJavaTypeAdapter(DateAdapter.class) so that Jersey could marhsall Date as key to my TreeMap?
Thanks.
JAXB supports the marshalling/unmarshalling of java.util.Date to the standard XML schema types: date, time, dateTime. You can control the type used with the #XmlSchemaType annotation.
http://blog.bdoughan.com/2011/01/jaxb-and-datetime-properties.html
If your date information is not represented as one of the standard XML schema types, you can use an XmlAdapter similar to the one I used the following answer to a similar question:
jaxb unmarshal timestamp
If you need to use the XmlAdapter approach, the #XmlJavaTypeAdapter annotation would be placed on the Date field of the adapted object representing the entry in the Map. Below is what this might look like based on my blog: http://blog.bdoughan.com/2010/07/xmladapter-jaxbs-secret-weapon.html.
import javax.xml.bind.annotation.XmlValue;
import javax.xml.bind.annotation.XmlAttribute;
import javax.xml.bind.annotation.adapters.XmlJavaTypeAdapter;
public class MyMapEntryType {
#XmlAttribute
#XmlJavaTypeAdapter(DateAdapter.class)
public Date key;
public List<MyObject> value;
}
I need to convert a json to a native .net object using mongodb. The application is written in javascript/mvc.
One of the field is a datetime object and the toJson function in the mongodb driver formats this as: "Modified":{"$date":1319630804846}
I want to parse this json from the client using the same format, but can't find a function that does this.
In Newtonsoft.Json I used this code, but this fails because of the date field:
var jobject = JObject.parse(jsonAsString)
var myObject = jobject.ToObject<myObject>();
But with the mongoDb driver, all I can do is converting the string to a BsonDocument
var buffer = new JsonBuffer(json);
using (BsonReader reader = new JsonReader(buffer))
{
var doc = BsonDocument.ReadFrom(reader);
....
}
The BSON serialization format for DateTime is an Int64 containing the number of milliseconds since Unix Epoch. So if you were to create a DateTime of kind Utc set to jan 1 1970 and then create a TimeSpan with TotalMilliseconds set to the Int64, and add the two together you'd have the date in Utc. The same algorithm could be used in reverse as needed.
If you're using the official .NET driver, you can work with objects without going through the JSON serialization.
Check the following example of how easy this is:
class Child
{
public ObjectId id;
public string name;
public DateTime birthday;
}
class Program
{
static void Main(string[] args)
{
Child m = new Child();
m.name = "Micaiah";
m.birthday = DateTime.Parse("January 1, 2011");
Children.Insert<Child>(m);
foreach (Child kiddo in Children.FindAllAs<Child>())
{
Console.WriteLine("Kiddo: {0} {1}", kiddo.name, kiddo.birthday);
}
Console.ReadLine();
}
static MongoCollection Children
{
get
{
MongoServer s = MongoServer.Create("mongodb://localhost");
return s["demos"]["children"];
}
}
}
Here's the record as stored in MongoDB:
> db.children.findOne()
{
"_id" : ObjectId("4ea821b2dd316c1e70e34d08"),
"name" : "Micaiah",
"birthday" : ISODate("2011-01-01T06:00:00Z")
}
>
Use JSON.Net to de-serialize your Json into a JObject, and send that to MongoDB... if you have more concrete types in C#, you'll want to serialize/deserialize to/from that to JSON... then persist from your concrete object, or JObject.