Camel mongodb - MongoDbProducer multiple inserts - mongodb

I am trying to do a multiple insert using the camel mongo db component.
My Pojo representation is :
Person {
String firstName;
String lastName;
}
I have a processor which constructs a valid List of Person pojo and is a valid json structure.
When this list of Person is sent to the mongodb producer , on invocation of createDoInsert the type conversion to BasicDBObject fails. This piece of code below looks to be the problem. Should it have more fall backs / checks in place to attempt the list conversion down further below as it fails on the very first cast itself. Debugging the MongoDbProducer the exchange object being received is a DBList which extends DBObject. This causes the singleInsert flag to remain set at true which fails the insertion below as we get a DBList instead of a BasicDBObject :
if(singleInsert) {
BasicDBObject insertObjects = (BasicDBObject)insert;
dbCol.insertOne(insertObjects);
exchange1.getIn().setHeader("CamelMongoOid", insertObjects.get("_id"));
}
The Camel MongoDbProducer code fragment
private Function<Exchange, Object> createDoInsert() {
return (exchange1) -> {
MongoCollection dbCol = this.calculateCollection(exchange1);
boolean singleInsert = true;
Object insert = exchange1.getIn().getBody(DBObject.class);
if(insert == null) {
insert = exchange1.getIn().getBody(List.class);
if(insert == null) {
throw new CamelMongoDbException("MongoDB operation = insert, Body is not conversible to type DBObject nor List<DBObject>");
}
singleInsert = false;
insert = this.attemptConvertToList((List)insert, exchange1);
}
if(singleInsert) {
BasicDBObject insertObjects = (BasicDBObject)insert;
dbCol.insertOne(insertObjects);
exchange1.getIn().setHeader("CamelMongoOid", insertObjects.get("_id"));
} else {
List insertObjects1 = (List)insert;
dbCol.insertMany(insertObjects1);
ArrayList objectIdentification = new ArrayList(insertObjects1.size());
objectIdentification.addAll((Collection)insertObjects1.stream().map((insertObject) -> {
return insertObject.get("_id");
}).collect(Collectors.toList()));
exchange1.getIn().setHeader("CamelMongoOid", objectIdentification);
}
return insert;
};
}
My route is as below :
<route id="uploadFile">
<from uri="jetty://http://0.0.0.0:9886/test"/>
<process ref="fileProcessor"/>
<unmarshal>
<csv>
<header>fname</header>
<header>lname</header>
</csv>
</unmarshal>
<process ref="mongodbProcessor" />
<to uri="mongodb:mongoBean?database=axs175&collection=insurance&operation=insert" />
and the MongoDBProcessor constructing the List of Person Pojo
#Component
public class MongodbProcessor implements Processor {
#Override
public void process(Exchange exchange) throws Exception {
ArrayList<List<String>> personlist = (ArrayList) exchange.getIn().getBody();
ArrayList<Person> persons = new ArrayList<>();
for(List<String> records : personlist){
Person person = new Person();
person.setFname(records.get(0));
person.setLname(records.get(1));
persons.add(person);
}
exchange.getIn().setBody(persons);
}
}
Also requested information here - http://camel.465427.n5.nabble.com/Problems-with-MongoDbProducer-multiple-inserts-tc5792644.html

This issue is now fixed via - https://issues.apache.org/jira/browse/CAMEL-10728

Related

How to ignore exceptions while deserializing xml to an object

I have written a web api which accepts xml and converts to json (a specific object) .
Problem Statement:
If xml contains wrong data type exception is thrown.
Desired situation: xmlserailizer should ignore for the fields where execption is thrown.
following is my sample input xml.
<Invoice>
<ProfileID>bpid:e1212121/ProfileID>
<IssueDate>fault date</IssueDate>
</Invoice>
Following is the code which throws error:
using (var stringreader = new StringReader(requestBody))
{
XmlSerializer xmlSerializer = new XmlSerializer(typeof(Invoice));
response = (Invoice)xmlSerializer.Deserialize(stringreader);//this line throws error
}
fallowing is my invoice object
public class invoice
{
private string profileID;
private DateTime _IssueDate;
public string ProfileID
{
get{
return this.profileID;
}
set {
this.profileID = value;
}
}
public DateTime IssueDate
{
get{
return this._IssueDate;
}
set {
this._IssueDate; = value;
}
}
}
In summary I want that xmlserialzer ignores error thrown for the fields where the data type is mismatch

Update operation adds in a new array instead of adding array elements

My controller:
#RequestMapping(value = "/{id}/giveFacilitiesAccess", method = RequestMethod.POST)
public Object giveFacilityAccessToAnotherUser(#PathVariable("id") String userId)
{
return userService.giveFacilitiesAccessToAnotherUser(userId);
}
My service:
#Override
public Object giveFacilitiesAccessToAnotherUser(String id)
{
String userId = getLoggedInUserId();
User u = userDao.findById(userId);
List<String> facilitiesAccess = u.getFacilitiesAccess();
return userDao.giveFacilitiesAccessToAnotherUser(id,facilitiesAccess);
}
My dao:
#Override
public Object giveFacilitiesAccessToAnotherUser(String userId, List<String>facilitiesAccess)
{
Query query = new Query();
query.addCriteria(Criteria.where("id").is(userId));
Update update = new Update().addToSet("facilitiesAccess.",facilitiesAccess);
mongoTemplate.updateFirst(query, update, User.class);
return null;
}
After updating:
"facilitiesAccess":["5f0996f792691d1b68671da3",["5f0998ba92691d1b68671da4"]]
It's updating like array inside another array, but i need in this format:
"facilitiesAccess":["5f0996f792691d1b68671da3","5f0998ba92691d1b68671da4"]
The update works with the syntax using the addToSet(String key) which returns a Update.AddToSetBuilder - then apply the each(Object... values) method on the builder, to return the Update object.
Update update = new Update().addToSet("facilitiesAccess").each(facilitiesAccess);
You need to use Each like below
#Override
public Object giveFacilitiesAccessToAnotherUser(String userId, List<String>facilitiesAccess)
{
Query query = new Query();
query.addCriteria(Criteria.where("id").is(userId));
Update update = new Update().addToSet("facilitiesAccess.",new Each(facilitiesAccess));
mongoTemplate.updateFirst(query, update, User.class);
return null;
}

SpringBatch - How to track whether Update is happening successfully or not

I have 2 ItemWriters, one for DB Insert and one for DB Update.
With use of ClassifierCustomItemWriter which I am calling the respective ItemWriter for new record and update the existing records.
Here i have concern. How to know the update has been happened or not ? For Example, if the "Application ID" not exists in the DB , the ItemWriter will not throw any error, but i want to know that update has not happened for this record and log it.
How can i track that ?
#Bean
public ClassifierCompositeItemWriter<TRSBatchEntryFormRequest> classifierCompositeItemWriter(ItemWriter<TRSBatchEntryFormRequest> databaseTableItemWriter, ItemWriter<TRSBatchEntryFormRequest> databaseTableUpdateItemWriter) {
ClassifierCompositeItemWriter<TRSBatchEntryFormRequest> classifierCompositeItemWriter = new ClassifierCompositeItemWriter<>();
classifierCompositeItemWriter.setClassifier((Classifier<TRSBatchEntryFormRequest, ItemWriter<? super TRSBatchEntryFormRequest>>) trsBatchEntryFormRequest -> {
if (trsBatchEntryFormRequest.getForm_status().equals("New")) {
return databaseTableItemWriter;
} else {
return databaseTableUpdateItemWriter;
}
});
return classifierCompositeItemWriter;
}
// Writer for DB
#Bean
public ItemWriter<TRSBatchEntryFormRequest> databaseTableItemWriter(DataSource springBatchDatasource) {
JdbcBatchItemWriter<TRSBatchEntryFormRequest> databaseItemWriter = new JdbcBatchItemWriter<TRSBatchEntryFormRequest>();
databaseItemWriter.setDataSource(springBatchDatasource);
logger.info("INSERT QUERY....: " + QUERY_INSERT_TRSEntryForms);
databaseItemWriter.setSql(QUERY_INSERT_TRSEntryForms);
databaseItemWriter.setItemSqlParameterSourceProvider(new TRSDBInputProvider());
return databaseItemWriter;
}
//Update Writer for DB
#Bean
public ItemWriter<TRSBatchEntryFormRequest> databaseTableUpdateItemWriter(DataSource springBatchDatasource) {
JdbcBatchItemWriter<TRSBatchEntryFormRequest> databaseItemWriter = new JdbcBatchItemWriter<TRSBatchEntryFormRequest>();
databaseItemWriter.setDataSource(springBatchDatasource);
logger.info("UPDATE QUERY....: " + QUERY_UPDATE_TRSEntryForms);
databaseItemWriter.setSql(QUERY_UPDATE_TRSEntryForms);
databaseItemWriter.setItemSqlParameterSourceProvider(new TRSDBInputProvider());
return databaseItemWriter;
}
​
Thanks
You can't track that with a CompositeItemWriter. What you can do is use a custom item writer like the following:
import java.util.List;
import org.springframework.batch.item.ItemWriter;
import org.springframework.jdbc.core.JdbcTemplate;
public class TRSBatchEntryFormRequestItemWriter implements ItemWriter<TRSBatchEntryFormRequest> {
private static final String INSERT_ITEM = "insert into item...";
private static final String UPDATE_ITEM = "update item set...";
private JdbcTemplate jdbcTemplate;
public TRSBatchEntryFormRequestItemWriter(JdbcTemplate jdbcTemplate) {
this.jdbcTemplate = jdbcTemplate;
}
#Override
public void write(List<? extends TRSBatchEntryFormRequest> items) throws Exception {
for (TRSBatchEntryFormRequest item : items) {
int updated = jdbcTemplate.update(UPDATE_ITEM);
if (updated == 0) {
jdbcTemplate.update(INSERT_ITEM);
}
}
}
}
The idea is to try to issue an update. If the operation returns 0, this means no rows have been updated and the item does not exist in the database, and you can issue an insert in that case.
Hope this helps.

Why am I getting an InvalidCastException with competing Newtonshoft.Json.Linq.[JArray,JObject] with very similar code/data?

This code works fine - returns the single record that matches the REST query:
Popul8TheGrid("http://localhost:28642/api/subdepartments/1/10");
private void Popul8TheGrid(string URIToPass)
{
try
{
dataGridView1.DataSource = GetRESTData(URIToPass);
}
catch (WebException webex)
{
MessageBox.Show("Eek, a mousey-pooh! ({0})", webex.Message);
}
}
private JArray GetRESTData(string uri)
{
var webRequest = (HttpWebRequest) WebRequest.Create(uri);
var webResponse = (HttpWebResponse) webRequest.GetResponse();
var reader = new StreamReader(webResponse.GetResponseStream());
string s = reader.ReadToEnd();
return JsonConvert.DeserializeObject<JArray>(s);
}
However, this code, which also should return a single record:
private const string BASE_URI = "http://localhost:28642/api/";
. . .
string URIToPass = string.Format("{0}deliveryitems/{1}", BASE_URI, numericUpDownDeliveryItemId.Value);
Popul8TheGrid(URIToPass);
...fails, with "InvalidCastException was unhandled ... Message=Unable to cast object of type 'Newtonsoft.Json.Linq.JObject' to type 'Newtonsoft.Json.Linq.JArray'".
Why might that be? The data returned from the first (working) snippet comes from an MS Access "database"
The data from the second (failing) snippet is from test data:
public DeliveryItemRepository()
{
// Just some bogus/test data for now
Add(new DeliveryItem
{
Id = 1, InvoiceNumber = "123", UPC_PLU = "456", VendorItemId = "789", PackSize = 1, Description = "Something", Quantity = 5, Cost = 1.25M,
Margin = 0.25M, ListPrice = 1.50M, DepartmentNumber = 42, Subdepartment = "5"
});
. . .
This is the Controller method; it works fine when entering the URI in a browser.
// Enter "http://localhost:28642/api/1"
[Route("api/DeliveryItems/{ID:int}")]
public DeliveryItem GetDeliveryItemById(int ID)
{
return _deliveryItemRepository.GetById(ID);
}
...but why that would matter, I know not...
UPDATE
Interestingly enough (perhaps I'm easily amused), this, OTOH, works:
MessageBox.Show(GetRESTScalarVal("http://localhost:28642/api/deliveries/1"));
. . .
private string GetRESTScalarVal(string uri)
{
var client = new WebClient();
return client.DownloadString(uri);
}
By "works," I mean it returns this:
So DownloadString() will even return an entire json "record" and my use of the word "Scalar" was misleading. Maybe I should have said "Single" instead, although that can be confusing, too, what with the data type of the same appellation.
The question still remains as to how I can populate a datagrid with a single json "record"
UPDATE 2
Oddly enough, if I use a different Controller method to get the one record, it works:
private void GetDeliveryItemById()
{
//string uri = string.Format("deliveryitems/{0}", numericUpDownId.Value);
string uri = string.Format("deliveryitems/{0}/1", numericUpDownId.Value);
Popul8TheGrid(uri);
}
The commented out code is what blows up, whereas the other, with a provided const val of 1, works...kludgy, but it works.
UPDATE 3
Perhaps a clue/related to why it won't work when fetching one, but works otherwise, is this Repository code:
public SiteMapping GetById(int ID)
{
return siteMappings.Find(p => p.Id == ID);
}
public IEnumerable<SiteMapping> GetRange(int ID, int CountToFetch)
{
return siteMappings.Where(i => i.Id >= ID).Take(CountToFetch);
}
If GetById() is called with an ID that exists, it works; if one is passed that doesn't exist, though, it fails with, "InvalidOperationException was unhandled by user code . . . Message=Sequence contains no matching element"
Calling GetRange() works robustly - if passed a bogus pair of vals (no records), it simply shrugs its shoulders, rather than getting the old bulgy eye and screaming maniacally.
Changing it to so (see Simon Whitehead's answere here) works:
public SiteMapping GetById(int ID)
{
var entity = siteMappings.Find(p => p.Id == ID);
return entity == null ? null : entity;
}
So trying to find by a particular ID is fragile; trying to find by ID + Count works just fine. Why, I (still) don't know...
This may be somewhat kludgy, but it works:
private JArray GetRESTData(string uri)
{
try
{
var webRequest = (HttpWebRequest)WebRequest.Create(uri);
var webResponse = (HttpWebResponse)webRequest.GetResponse();
var reader = new StreamReader(webResponse.GetResponseStream());
string s = reader.ReadToEnd();
return JsonConvert.DeserializeObject<JArray>(s);
}
catch // This method crashes if only one json "record" is found - try this:
{
try
{
MessageBox.Show(GetScalarVal(uri));
}
catch (Exception ex)
{
MessageBox.Show(ex.Message);
}
}
return null;
}
private string GetScalarVal(string uri)
{
var client = new WebClient();
return client.DownloadString(uri);
}

How to retrieve an embedded list of object of Entity?

I have a simple problem storing and retrieving an embedded collection of entity to mongo. I have checked theses question :
how to serialize class? and Mongodb saves list of object
what I understand is to save a list objects the class of that objects must extends ReflactionDBObject. This worked for saving the object, by retrieving it with the embedded collection does not work.
here a simple test show that retrieving embedded entities does not work !
#Test
public void whatWasStoredAsEmbeddedCollectionIsRetrieved2() {
BasicDBObject country = new BasicDBObject();
country.put("name", "Bulgaria");
List<City> cities = Lists.newArrayList(new City("Tarnovo"));
country.put("listOfCities", cities);
DBCollection collection = db().get().getCollection("test_Collection");
collection.save(country);
DBCursor object = collection.find(new BasicDBObject().append("name", "Bulgaria"));
DBObject returnedCity = object.next();
DBObject embeddedCities = (DBObject) returnedCity.get("listOfCities");
System.out.println(embeddedCities);
}
Here is the City Class
class City extends ReflectionDBObject {
String name;
City() {
}
City(String name) {
this.name = name;
}
public String getName() {
return name;
}
#Override
public boolean equals(Object o) {
if (this == o) return true;
if (!(o instanceof City)) return false;
City city = (City) o;
if (name != null ? !name.equals(city.name) : city.name != null) return false;
return true;
}
#Override
public int hashCode() {
return name != null ? name.hashCode() : 0;
}
#Override
public String toString() {
return "City{" +
"name='" + name + '\'' +
'}';
}
}
The out put of the System.out.println statement is [ { "_id" : null }]
Now how can get back the embedded object and the embedded list in it ?
If you do not have a requirement to define your own class City, you can define subdocuments using the BasicDBObjects. I only added the 'name' field to the citySubDoc1 and citySubDoc2, but of course, you can add more fields to these subdocuments.
// Define subdocuments
BasicDBObject citySubDoc1 = new BasicDBObject();
citySubDoc1.put("name", "Tarnovo");
BasicDBObject citySubDoc2 = new BasicDBObject();
citySubDoc2.put("name", "Sofia");
// add to list
List<DBObject> cities = new ArrayList <DBObject>();
cities.add(citySubDoc1);
cities.add(citySubDoc2);
country.put("listOfCities", cities);
collection.save(country);
// Specify query condition
BasicDBObject criteriaQuery = new BasicDBObject();
criteriaQuery.put("name", "Bulgaria");
// Perform the read
DBCursor cursor = collection.find(criteriaQuery);
// Loop through the results
try {
while (cursor.hasNext()) {
List myReturnedListOfCities = (List) cursor.next().get("listOfCities");
System.out.println(myReturnedListOfCities);
}
} finally {
cursor.close();
}