In my spring batch task I have two datasources configured, one for oracle and another for h2.
H2 I'm using for batch and task execution tables and oracle is for real data for batch processing. I'm able to successfully run the task from ide but when I run it from SCDF server I get following error
Caused by: java.sql.SQLException: Unable to start the Universal Connection Pool: oracle.ucp.UniversalConnectionPoolException: Cannot get Connection from Datasource: org.h2.jdbc.JdbcSQLInvalidAuthorizationSpecException: Wrong user name or password [28000-200]
Question is, why is it connecting to H2 for oracle db connection as well?
Following is my DB configuration:
#Bean(name = "OracleUniversalConnectionPool")
public DataSource secondaryDataSource() {
PoolDataSource pds = null;
try {
pds = PoolDataSourceFactory.getPoolDataSource();
pds.setConnectionFactoryClassName(driverClassName);
pds.setURL(url);
pds.setUser(username);
pds.setPassword(password);
pds.setMinPoolSize(Integer.valueOf(minPoolSize));
pds.setInitialPoolSize(10);
pds.setMaxPoolSize(Integer.valueOf(maxPoolSize));
} catch (SQLException ea) {
log.error("Error connecting to the database: ", ea.getMessage());
}
return pds;
}
#Bean(name = "batchDataSource")
#Primary
public DataSource dataSource() throws SQLException {
final SimpleDriverDataSource dataSource = new SimpleDriverDataSource();
dataSource.setDriver(new org.h2.Driver());
dataSource.setUrl("jdbc:h2:tcp://localhost:19092/mem:dataflow");
dataSource.setUsername("sa");
dataSource.setPassword("");
return dataSource;
}
I got it resolved, problem was I was using this for setting values to oracle db.
spring.datasource.url: ******
spring.datasource.username: ******
It worked fine from IDE, but when I ran it on SCDF server it was overwritten by default properties being used for database connection of SCDF server. So I just updated the connection properties to :
spring.datasource.oracle.url: ******
spring.datasource.oracle.username: ******
And now it's working as expected.
Related
The following ItemReader gets a list of thousands accounts (acc).
The database that the ItemReader will connected to in order to retrieve the data is HIVE. I don’t have permission to create any table, only read option.
#Bean
#StepScope
public ItemReader<OmsDto> omsItemReader(#Value("#{stepExecutionContext[acc]}") List<String> accountList) {
String inParams = String.join(",", accountList.stream().map(id ->
"'"+id+"'").collect(Collectors.toList()));
String query = String.format("SELECT ..... account IN (%s)", inParams);
BeanPropertyRowMapper<OmsDto> rowMapper = new BeanPropertyRowMapper<>(OmsDto.class);
rowMapper.setPrimitivesDefaultedForNullValue(true);
JdbcCursorItemReader<OmsDto> reader = new JdbcCursorItemReader<OmsDto>();
reader.setVerifyCursorPosition(false);
reader.setDataSource(hiveDataSource());
reader.setRowMapper(rowMapper);
reader.setSql(query);
reader.open(new ExecutionContext());
return reader;
}
This is the error message that I get when using ItemReader:
Caused by: org.springframework.batch.item.ItemStreamException: Failed to initialize the reader
at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader.open(AbstractItemCountingItemStreamItemReader.java:153) ~[spring-batch-infrastructure-4.2.4.RELEASE.jar:4.2.4.RELEASE]
Caused by: java.sql.SQLException: Error executing query
at com.facebook.presto.jdbc.PrestoStatement.internalExecute(PrestoStatement.java:279) ~[presto-jdbc-0.243.2.jar:0.243.2-128118e]
at com.facebook.presto.jdbc.PrestoStatement.execute(PrestoStatement.java:228) ~[presto-jdbc-0.243.2.jar:0.243.2-128118e]
at com.facebook.presto.jdbc.PrestoPreparedStatement.<init>(PrestoPreparedStatement.java:84) ~[presto-jdbc-0.243.2.jar:0.243.2-128118e]
at com.facebook.presto.jdbc.PrestoConnection.prepareStatement(PrestoConnection.java:130) ~[presto-jdbc-0.243.2.jar:0.243.2-128118e]
at com.facebook.presto.jdbc.PrestoConnection.prepareStatement(PrestoConnection.java:300) ~[presto-jdbc-0.243.2.jar:0.243.2-128118e]
at org.springframework.batch.item.database.JdbcCursorItemReader.openCursor(JdbcCursorItemReader.java:121) ~[spring-batch-infrastructure-4.2.4.RELEASE.jar:4.2.4.RELEASE]
... 63 common frames omitted
Caused by: java.lang.RuntimeException: Error fetching next at https://prestoanalytics-ch2-p.sys.comcast.net:6443/v1/statement/executing/20201118_131314_11079_v3w47/yf55745951e0beccc234c98f36005723457073854/0 returned an invalid response: JsonResponse{statusCode=502, statusMessage=Bad Gateway, headers={cache-control=[no-cache], content-length=[107], content-type=[text/html]}, hasValue=false} [Error: <html><body><h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
</body></html>
]
I was sure that the root cause is because of the driver but I have tested the driver with the same SQL this time using DriverManager and its run perfectly.
#Component
public class OmsItemReader implements ItemReader<OmsDto>, StepExecutionListener {
private ItemReader<OmsDto> delegate;
public SikOmsItemReader() {
Properties properties = new Properties();
properties.setProperty("user", "....");
properties.setProperty("password", "...");
properties.setProperty("SSL", "true");
Connection connection = null;
try {
connection = DriverManager.getConnection("jdbc:presto://.....", properties);
Statement statement = connection.createStatement();
ResultSet resultSet = statement.executeQuery(
I am not sure what is the different ? Is it the driver or sparing batch ?
I am looking for a workaround. How can I retrieve thousands of accounts via IN clauses with spring batch ?
Thank you
I have a dockerized .net core container and am trying to query a MongoDB database. I have a REST API that is called to query the database from the .net core container. It seems as if I am able to establish a connection to the container when the instance of the service is created:
private readonly IMongoCollection<DbObject> _dbObjects;
public TaxService(IDbConfig dbConfig)
{
Console.Out.WriteLine("got here: " + dbConfig.ConnectionString);
MongoClient client = new MongoClient(dbConfig.ConnectionString);
IMongoDatabase database = client.GetDatabase(dbConfig.DatabaseName);
_dbObjects = database.GetCollection<DbObject>(dbConfig.SalesTaxCollectionName);
Console.Out.WriteLine("db initialized");
}
Here is the dbConfig object:
"DbConfig": {
"SalesTaxCollectionName": "ExampleCollection",
"ConnectionString": "mongodb://mongo:27017",
"DatabaseName": "ExampleDb"
}
It gets through this code, but when the database is actually queried, a PlatformNotSupportedException is thrown:
public DbObject GetByValue(string value)
{
Console.Out.WriteLine("querying db");
List<DbObject> matches = _dbObjects.Find(dbObject => dbObject.Value.Equals(value)).ToList();
Console.Out.WriteLine("successfully queried"); // Does not reach this point
if (matches.Any())
{
return matches.First();
}
else
{
throw new ArgumentException($"No values matched");
}
}
I have put the rest of the files in gists for convenience:
docker-compose.yml: https://gist.github.com/MinhazMurks/1fbb47afd360bbac48df45b1f0609e33
Dockerfile: https://gist.github.com/MinhazMurks/bb2b7f76d28894a81136d940b5997165
InitExampleDb.js: https://gist.github.com/MinhazMurks/9aae03daceee1e689c4e821419966f41
Full-Log: https://gist.github.com/MinhazMurks/0de9cd822fcf2930065527191127c83b
Any insight would be appreciated!
You can try to set the following line in the host file:
(It is located here in case of Windows: C:\Windows\System32\drivers\etc\hosts)
127.0.0.1 mongo
After this try connect using RoboMongo. In the Address textbox insert mongo and try to connect to it.
I am trying to insert my application logs into MongoDB. I have created a custom appender and have overridden the append method as follows:
public void append(LoggingEvent loggingEvent) {
Document doc = convertToMongoDocument(loggingEvent);
pushDocToDB(doc);
}
public Document convertToMongoDocument(LoggingEvent event) {
Document doc = new Document();
// will read from the actual logging event later
doc.append("logger", "logger");
doc.append("user", "user");
doc.append("message", "message");
doc.append("timestamp", "timestamp");
return doc;
}
public void pushDocToDB(Document docList) {
getCollection().insertOne(docList);
}
I keep running into the below error when trying to insert a document from a java client. I have not created any replica sets and am using a standalone instance of MongoDB in my local.
No server chosen by WritableServerSelector from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, all=[ServerDescription{address=localhost:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 1000000 ms before timing out
I am using mongo-java-driver - version 3.4.0 and jdk 1.7
I am new to orient-db and have run into a major block even trying to open a simple in memory database.
Here is my two lines of code (in java)
OrientGraphFactory factory = new
OrientGraphFactory("memory:test").setupPool(1,10);
// EVERY TIME YOU NEED A GRAPH INSTANCE
OrientGraph g = factory.getTx();
try {
} finally {
g.shutdown();
}
I get the following error:
Exception in thread "main" com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage 'test' with mode=rw
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:210)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:223)
at com.orientechnologies.orient.core.db.OPartitionedDatabasePool.acquire(OPartitionedDatabasePool.java:287)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.<init>(OrientBaseGraph.java:163)
at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.<init>(OrientTransactionalGraph.java:78)
at com.tinkerpop.blueprints.impls.orient.OrientGraph.<init>(OrientGraph.java:128)
at com.tinkerpop.blueprints.impls.orient.OrientGraphFactory.getTx(OrientGraphFactory.java:74)
Caused by: com.orientechnologies.orient.core.exception.OStorageException:
Cannot open the storage 'test' because it does not exist in path: test
at
com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage .open(OAbstractPaginatedStorage.java:154)
... 7 more
What 'path' is it talking about? How is a path even relevant when trying to open a simple in memory database? Furthermore I have also tried this with plocal:/..... ,,, and I always get the above error.
Regards,
Bhargav.
Try to create the database first :
OrientGraphNoTx graph = new OrientGraphNoTx ("memory:test");
Then use the pool :
OrientGraphFactory factory = new OrientGraphFactory ("memory:test").setupPool (1, 10);
By the way which db version are you using ?
Databases created as in-memory only needs to be created first and the pool didn't allow it (fixed in last snapshot). Try acquiring an instance from the factory without pool, like:
OrientGraphFactory factory = newOrientGraphFactory("memory:test");
factory.getTx().shutdown(); // AUTO-CREATE THE GRAPH IF NOT EXISTS
factory.setupPool(1,10);
// EVERY TIME YOU NEED A GRAPH INSTANCE
OrientGraph g = factory.getTx();
try {
} finally {
g.shutdown();
}
I am using playframework 2.0 and after every recompile I get a socket timeout the first time my app tries to go to the database. I am using the Mongo Driver directly. here is a typical stack trace:
play.core.ActionInvoker$$anonfun$receive$1$$anon$1: Execution exception [[Network: can't call something : ds031907.mongolab.com/107.21.99.26:31907/heroku_app4620908]]
at play.core.ActionInvoker$$anonfun$receive$1.apply(Invoker.scala:82) [play_2.9.1.jar:2.0]
at play.core.ActionInvoker$$anonfun$receive$1.apply(Invoker.scala:63) [play_2.9.1.jar:2.0]
at akka.actor.Actor$class.apply(Actor.scala:290) [akka-actor.jar:2.0]
at play.core.ActionInvoker.apply(Invoker.scala:61) [play_2.9.1.jar:2.0]
at akka.actor.ActorCell.invoke(ActorCell.scala:617) [akka-actor.jar:2.0]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:179) [akka-actor.jar:2.0]
Caused by: com.mongodb.MongoException$Network: can't call something : ds031907.mongolab.com/107.21.99.26:31907/heroku_app4620908
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:227) ~[mongo-java-driver-2.7.3.jar:na]
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:305) ~[mongo-java-driver-2.7.3.jar:na]
at com.mongodb.DBCollection.findOne(DBCollection.java:647) ~[mongo-java-driver-2.7.3.jar:na]
at com.mongodb.DBCollection.findOne(DBCollection.java:626) ~[mongo-java-driver-2.7.3.jar:na]
at models.daos.ModuleDAO.findPublishedModuleById(ModuleDAO.java:445) ~[classes/:na]
at controllers.LearnController.viewModule(LearnController.java:31) ~[classes/:2.0]
Caused by: java.net.SocketException: Operation timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.6.0_31]
at java.net.SocketInputStream.read(SocketInputStream.java:129) ~[na:1.6.0_31]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) ~[na:1.6.0_31]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258) ~[na:1.6.0_31]
at java.io.BufferedInputStream.read(BufferedInputStream.java:317) ~[na:1.6.0_31]
at org.bson.io.Bits.readFully(Bits.java:35) ~[mongo-java-driver-2.7.3.jar:na]
And here is my initialization code:
public static DB getDB(){
ensureMongo();
DB db = mongo.getDB(MOJULO_DB);
if(!db.isAuthenticated()){
db.authenticate(MONGO_USERNAME, MONGO_PASSWORD);
if(db.isAuthenticated())
System.out.println("authentication success on db:" + db.getName());
else
System.out.println("db authentication failure");
}
return db;
}
private static synchronized void ensureMongo(){
if(mongo == null){
try{
MongoURI mongoURI = new MongoURI(MONGO_URI);
mongo = new Mongo(mongoURI);
DB db = mongo.getDB(MOJULO_DB);
db.command("ping");
}catch(UnknownHostException ex){
mongo = null;
System.out.println("failed to connect to mongo");
ex.printStackTrace();
}
}
}
public static void disconnect(){
System.out.println("disconnecting from mongo");
if(mongo != null){
mongo.close();
mongo = null;
}
}
I use the getDB method from outside the class to get the db. The method is meant to create the mongo singleton if it does not exists. I always get the authentication success println, but then on the first hit to the database, I get the socket timeout exception.
In my Global class, I close the connection to the database when the application is closed.
#Override
public void onStop(Application app) {
System.out.println("stop");
Logger.info("Application shutdown...");
DBManager.disconnect();
}
Any Ideas?
I am not an expert on MongoDB but can see similarity with other DB connectivity.
How is your connection configured?
It looks (to me) like it may be attempting to load all mappings/DB and table definitions/everything else when you attempt to use DB connection (find method) for the first time.
It may be better to run a simple DB query in your ensureMongo() method to allow the system to re-initialise everything it needs (you may have to set longer time-out on this method).