Our Camel Mongo DB component doesn't support multiple mongo db connections. By default it uses single Mongodb bean in all the mongo endpoint.
Issue:
I have created two different mongo bean instance for two database with respective authentication. And two mongodb beans instance are created in the spring.
When initializing/creating mongo endpoints(multiple endpoints with different DB) with DB its taking first mongo endpoint db connection to all the endpoint. So all the mongo endpoints are pointing to same database which was added first when creating first mongo endpoint.
Code below:
<bean id="DbLocal" name="DbLocal" class="com.mongodb.MongoClient">
<constructor-arg index="0">
<bean class="com.mongodb.MongoClientURI">
<constructor-arg index="0" value="mongodb://<user1>:<pwd1>#<host>:<port>/DB1" />
</bean>
</constructor-arg>
</bean>
<bean id="appDb" name="appDb" class="com.mongodb.MongoClient">
<constructor-arg index="0">
<bean class="com.mongodb.MongoClientURI">
<constructor-arg index="0" value="mongodb://<host>:<port>/DB2" />
</bean>
</constructor-arg>
</bean>
<route id="jobStart" startupOrder="1">
<from uri="direct:jobStart" />
<to uri="bean:jobMonitorInsertQueryBuilder" />
<to uri="mongodb:appDb?database={{requestDb}}&collection={{jobmonitorCollection}}&operation=insert"/>
</route
<route id="jobEnd" startupOrder="2">
<from uri="direct:jobEnd" />
<to uri="bean:jobMonitorUpdateQueryBuilder" />
<to uri="mongodb:DbLocal?database={{requestDb}}&collection={{jobmonitorCollection}}&operation=update" />
</route>
I tried by commenting out the if check in the camel mongo component thus allowing mongo connection look up for each mongo endpoint created in MongoDbComponent.java . This way it is working for multiple DB connections.
Any idea why mongo component is allowing only a single mongoDB connection in camel , is it because any existing issue?
protected Endpoint createEndpoint(String uri, String remaining, Map parameters) throws Exception {
// TODO: this only supports one mongodb
//if (db == null) {
db = CamelContextHelper.mandatoryLookup(getCamelContext(), remaining, Mongo.class);
LOG.debug("Resolved the connection with the name {} as {}", remaining, db);
//}
https://github.com/apache/camel/blob/master/components/camel-mongodb/src/main/java/org/apache/camel/component/mongodb/MongoDbComponent.java
Appreciate any help in this regard.
Camel forum link : http://camel.465427.n5.nabble.com/Camel-mongo-db-does-not-support-multiple-mongo-DB-tp5765468.html
-Sree
This feature has been implemented in camel 2.18.2, 2.19.0. Kindly use these version to implement multiple DB's in your Project .
FYI :
https://issues.jboss.org/browse/ENTESB-5884
Jira Ticket Link - https://issues.apache.org/jira/browse/CAMEL-10644
Thanks,
Panneer
Related
I am working on Spring Boot Spring Data Mongo Pageable example. I want to set pagination value default page=1 and size=5, if no one provided any values.
Now when user provided any values, I want Pageable object to use those values and not default one.
Crated Bug: https://jira.spring.io/browse/DATAMONGO-2313
Its not working in my case. I'm not using Spring HATEOAS in my application.
#GetMapping()
public ResponseEntity<Page<Student>> getAllstudents(Pageable pageable){
return studentService.findAllstudents(pageable);
}
in application.yml file
# Mongo DB details
data:
mongodb:
database: TEST
host: localhost
port: 27017
web:
pageable:
default-page-size: 2
size-parameter: 0
one-indexed-parameters: true
page-parameter: page
Or I want exactly like below using Spring boot
<mvc:argument-resolvers>
<bean class="org.springframework.data.web.PageableHandlerMethodArgumentResolver" >
<property name="oneIndexedParameters" value="true"></property>
<property name="pageParameterName" value="page"></property>
<property name="sizeParameterName" value="size"></property>
<property name="fallbackPageable">
<bean class="org.springframework.data.domain.PageRequest">
<constructor-arg name="page" value="1" />
<constructor-arg name="size" value="${paging.default.pageSize}" />
</bean>
</property>
</bean>
</mvc:argument-resolvers>
Swagger UI:
I'm seeing reports that those pageable settings in application.yml only work with Spring HATEOAS. I would try using the #PageableDefault annotation instead.
I'm working on an existing Spring application that uses JDBC (DAO's extend NamedParameterJdbcDaoSupport). There were four datasources configured, each with it's own DataSourceTransactionManager. (though only one was registered with tx:annotation-driven for some reason)
I've recently added JPA (Spring-data-JPA) into the application and configured two entityManagerFactories (for now I don't need the other two datasources). I also configured two JpaTransactionManagers and removed the corresponding DataSourceTransactionManagers for these dataSources, since the JpaTransactionManagers can also be used for JDBC transactions. (correct me if I'm wrong)
It appears I need to be able to have distributed transactions, since the two datasources (to two different databases) need to be accessed (through JPA) in one service method. Since I did not have all I need to set up JTA (missing XA-driver for one of the databases) I've decided to give the Spring ChainedTransactionManager a try. Sadly this didn't work out as expected. All works fine if I just call a service method that only uses JPA.
Though when I call an existing service method that uses a JDBC find that has a class level #transactional annotation with it's propagation set to NOT_SUPPORTED and call another service method after that with a JPA call and a #transactional, I get an exception:
Caused by: java.lang.IllegalStateException: Already value [org.springframework.jdbc.datasource.ConnectionHolder#462cf9d9] for key [org.jboss.jca.adapters.jdbc.WrapperDataSource#3fbb4c32] bound to thread [http-/127.0.0.1:8080-5]
at org.springframework.transaction.support.TransactionSynchronizationManager.bindResource(TransactionSynchronizationManager.java:189) [spring-tx-3.2.5.RELEASE.jar:3.2.5.RELEASE]
at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:403) [spring-orm-3.2.5.RELEASE.jar:3.2.5.RELEASE]
After some debugging, I found out that the transactions in Spring get added to a map on a ThreadLocal in the "TransactionSynchronizationManager.bindResource" method. The problem is that when using a JDBC call with #transactional and propogation NOT_SUPPORTED, a transaction is made anyway and registered through that method. When the JpaTransactionManager tries to bind it's resource, it is already on the map (and not marked as void) which causes the error to occur.
Changing the propagation to the default "REQUIRED" for the service call that encapsulates the JDBC call fixes the problem.
I have no idea why Spring is still creating that transaction when the transactional annotation is NOT_SUPPORTED. And if it creates that transaction, it should not bypass the JpaTransactionManager.
So what I'd like to know is if there is some way to tell Spring to use the JpaTransactionManager also when it creates a transaction itself inside the NamedParameterJdbcDaoSupport. (Well actually the JdbcDaoSupport... Well actually the DataSourceUtils)
We're using Spring 3.2.5, spring-data-jpa 1.6.0 and I've used Hibernate 4.2.0 as JpaVendor.
This problem doesn't occur without the ChainedTransactionManager.
Datasources:
<bean id="dataSourceCompta" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="java:jboss/datasources/comptaDS"/>
</bean>
<bean id="dataSourceUnisys" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="java:jboss/datasources/insoverDS"/>
</bean>
<bean id="dataSourceInsoverwebMysql" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="java:jboss/datasources/insoverWebDS"/>
</bean>
<bean id="dataSourceBatch" class="org.springframework.jndi.JndiObjectFactoryBean">
<property name="jndiName" value="java:jboss/datasources/batchDS"/>
</bean>
Single remaining JDBC transaction manager (no JPA counterpart):
<bean id="transactionManagerBatch" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSourceBatch"/>
</bean>
JPA Transaction Managers:
<bean id="jpaUnisysTransactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="entityManagerFactoryUnisys"/>
<qualifier value="unisys" />
</bean>
<bean id="jpaMysqlTransactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
<property name="entityManagerFactory" ref="entityManagerFactoryMysql"/>
<qualifier value="mysql" />
</bean>
My ChainedTransactionManager:
<bean id="chainedTransactionManager" class="org.springframework.data.transaction.ChainedTransactionManager">
<constructor-arg>
<list>
<ref bean="jpaUnisysTransactionManager" />
<ref bean="jpaMysqlTransactionManager" />
</list>
</constructor-arg>
</bean>
JPA Entity manager factories:
<bean name="jpaVendorAdapter" class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter"/>
<bean id="entityManagerFactoryUnisys" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="persistenceXmlLocation" value="classpath:META-INF/some-persistence.xml"/>
<property name="dataSource" ref="dataSourceUnisys"/>
<property name="persistenceUnitName" value="unisysPU"/>
<property name="jpaVendorAdapter" ref="jpaVendorAdapter"/>
<property name="jpaProperties">
<!-- properties -->
</property>
</bean>
<bean id="entityManagerFactoryMysql" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
<property name="persistenceXmlLocation" value="classpath:META-INF/some-persistence.xml"/>
<property name="dataSource" ref="dataSourceCompta"/>
<property name="persistenceUnitName" value="mysqlPU"/>
<property name="jpaVendorAdapter" ref="jpaVendorAdapter"/>
<property name="jpaProperties">
<!-- properties -->
</property>
</bean>
For now I've "fixed" this, by changing all the class-level transactional annotations to have propagation.REQUIRED (default) instead of NOT_SUPPORTED. Though I do not really like this solutions, since it might be somebody set those propagations to NOT_SUPPORTED with a good reason. I've also tried SUPPORTED, but using that had the same issue as NOT_SUPPORTED: a transaction was being made anyway by the Spring DataSourceUtils when the query was being executed by the NamedParameterJdbcDaoSupport DAO.
When no transactional annotation is set on the service, all works well too.
We are in the process of changing some of the applications that use a relation database, to OrientDB (2.1.9).
This transaction will be made in small steps, we will first start with the JDBC driver approach, and maybe we will switch to a native implementation later.
We have an application that is based on Spring Integration, where we changed from an Oracle Database to OrientDB, using the JDBC driver.
The database connections are managed by Tomcat 8 Connection Pool, and the transactional behavior is handled by Spring's Transaction Manager:
<bean id="dataSource" class="org.apache.tomcat.jdbc.pool.DataSource" destroy-method="close">
<property name="driverClassName" value="com.orientechnologies.orient.jdbc.OrientJdbcDriver" />
...
</bean>
<bean id="txManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
<property name="dataSource" ref="dataSource" />
</bean>
After changing the database to OrientDB, we encountered the following error:
com.orientechnologies.orient.core.exception.ODatabaseException: Database instance is not set in current thread. Assure to set it with: ODatabaseRecordThreadLocal.INSTANCE.set(db);
The OrientDB documentation, on Multi threading, states:
OrientDB supports multi-threads access to the database. ODatabase* and OrientGraph* instances are not thread-safe, so you've to get an instance per thread and each database instance can be used only in one thread per time. For more information about how concurrency is managed by OrientDB look at Concurrency.
Since v2.1 OrientDB doesn't allow implicit usage of multiple database instances from the same thread. Any attempt to manage multiple instances in the same thread must explicitly call the method db.activateOnCurrentThread() against the database instance BFORE you use it.
This posed a problem, as all the database connection management is handled by the connection pool (as it should be).
So, in order to implement the solution proposed by OrientDB, I had to have access to the underlying ODatabaseDocumentTx database, in order to apply the activateOnCurrentThread() method.
To be able to do it, I had to create a wrapper from both the JDBC driver class, and the OrientJdbcConnection.
In the OrientJdbcConnection, I obtained the underlying database by the use of Java Reflection:
public class OrientJdbcConnectionWrapper extends OrientJdbcConnection {
#Override
public void commit() throws SQLException {
final ODatabaseDocumentTx database = getPrivateDatabaseFieldByReflection();
database.activateOnCurrentThread();
super.commit();
}
And the JDBC Driver class wrapper, now returns a wrapped connection:
public class OrientJdbcDriverWrapper extends OrientJdbcDriver {
public Connection connect(final String url, final Properties info) throws SQLException {
if (!acceptsURL(url)) {
throw new SQLException(DO_NOT_ACCEPT_URL_ERROR + url);
}
return new OrientJdbcConnectionWrapper(url, info);
}
}
Finally, the configuration of the Connection pool now uses the wrapped driver:
<bean id="dataSource" class="org.apache.tomcat.jdbc.pool.DataSource" destroy-method="close">
<property name="driverClassName" value="x.y.z.driver.orientdb.OrientJdbcDriverWrapper" />
</bean>
As expected, this works perfectly, but i'm not entirely sure that this is the correct way to do it as i haven't see anyone with this issue.
On the other hand, the OrientDB documentation does suggest the use of activateOnCurrentThread() on a multi-threading environment.
OrientDB has its own database pool, that can be easily configured:
<bean id="dataSource" class="com.orientechnologies.orient.jdbc.OrientDataSource">
<constructor-arg name="url" value="jdbc:orient:plocal:xx\\databases\\test" />
<constructor-arg name="username" value="xx"/>
<constructor-arg name="password" value="xx"/>
<constructor-arg name="info" ref="databaseProperties" />
</bean>
<util:map id="databaseProperties" value-type="java.lang.String">
<entry key="db.usePool" value="true" />
<entry key="db.pool.min" value="1" />
<entry key="db.pool.max" value="10" />
Note: I'm using OrientDB through its orient-jdbc driver.
I use restlet in camel route in from("restlet:http/myLink") clause. When user's requests more then ten per second, I begin recieve errors processing request like a "org.restlet.engine.connector.Controller run
INFO: Connector overload detected. Stop accepting new work"
I think, that error is caused by number of threads,request query's size or number,or something like that. I try set to maxThreads param different values in spring config
<bean id="restlet" class="org.apache.camel.component.restlet.RestletComponent">
<property name="maxThreads" value="15"/>
</bean>
but I am not succeed. In documentation http://camel.apache.org/restlet.html I ddin't find ant param for setting size\number of request queue. I need help :(
P.S. camel-restlet version is 2.12.2
Update
I try to set big numbers to maxThreads,maxConnectionsPerHost,maxTotalConnections, but it's useless. If inject org.restlet.Component to camel's config like that:
<bean id="restletComponent" class="org.restlet.Component" />
<bean id="restlet" class="org.apache.camel.component.restlet.RestletComponent">
<constructor-arg index="0">
<ref bean="restletComponent" />
</constructor-arg>
<property name="maxThreads" value="255"/>
<property name="maxConnectionsPerHost" value="1000"/>
<property name="maxTotalConnections" value="1000" />
</bean>
How I can override properties, that use BaseHelper params?
After go through the options of lowThread as well.
But I found current released camel doesn't support it.
I'm running MongoDB 2.2. I can add credentials to access my single instance MongoDB like this:
<bean id="mongoTemplate" class="org.springframework.data.mongodb.core.MongoTemplate">
<property name="writeResultChecking" value="EXCEPTION"/>
<property name="writeConcern" value="FSYNC_SAFE"/>
<constructor-arg ref="myRs"/>
<constructor-arg name="databaseName" value="mydb"/>
<constructor-arg name="userCredentials" ref="userCredentials"/>
</bean>
<bean id="userCredentials" class="org.springframework.data.authentication.UserCredentials">
<constructor-arg name="username" value="username" />
<constructor-arg name="password" value="password" />
</bean>
but the docs say that only keyfile authentication is available for replica sets:
http://docs.mongodb.org/v2.2/administration/replica-sets/#replica-set-security
How can I use DB credentials preferrably (or even a keyfile) in the app context XML file so I can authenticate to the MongoDB replica set when using Spring Data MongoDB?
Thanks.
--keyfile option enables key based authentication between replicaset members. It has no effect the communication between your application and the replicaset.
You need to add a user to the database. Make sure you are connected to Primary.
$ mongo
PRIMARY> use the_database_i_want_to_be_authenticated
switched to db the_database_i_want_to_be_authenticated
PRIMARY> db.addUser("myusername","mypassword")
{
"user" : "myusername",
"readOnly" : false,
"pwd" : "a6de521abefc2fed4f5876855a3484f5",
"_id" : ObjectId("5194932444ef978a730c22d4")
}
After you created a user in the database, you will be able to connect it with the following
<bean id="userCredentials" class="org.springframework.data.authentication.UserCredentials">
<constructor-arg name="username" value="username" />
<constructor-arg name="password" value="password" />
</bean>