The following ItemReader gets a list of thousands accounts (acc).
The database that the ItemReader will connected to in order to retrieve the data is HIVE. I don’t have permission to create any table, only read option.
#Bean
#StepScope
public ItemReader<OmsDto> omsItemReader(#Value("#{stepExecutionContext[acc]}") List<String> accountList) {
String inParams = String.join(",", accountList.stream().map(id ->
"'"+id+"'").collect(Collectors.toList()));
String query = String.format("SELECT ..... account IN (%s)", inParams);
BeanPropertyRowMapper<OmsDto> rowMapper = new BeanPropertyRowMapper<>(OmsDto.class);
rowMapper.setPrimitivesDefaultedForNullValue(true);
JdbcCursorItemReader<OmsDto> reader = new JdbcCursorItemReader<OmsDto>();
reader.setVerifyCursorPosition(false);
reader.setDataSource(hiveDataSource());
reader.setRowMapper(rowMapper);
reader.setSql(query);
reader.open(new ExecutionContext());
return reader;
}
This is the error message that I get when using ItemReader:
Caused by: org.springframework.batch.item.ItemStreamException: Failed to initialize the reader
at org.springframework.batch.item.support.AbstractItemCountingItemStreamItemReader.open(AbstractItemCountingItemStreamItemReader.java:153) ~[spring-batch-infrastructure-4.2.4.RELEASE.jar:4.2.4.RELEASE]
Caused by: java.sql.SQLException: Error executing query
at com.facebook.presto.jdbc.PrestoStatement.internalExecute(PrestoStatement.java:279) ~[presto-jdbc-0.243.2.jar:0.243.2-128118e]
at com.facebook.presto.jdbc.PrestoStatement.execute(PrestoStatement.java:228) ~[presto-jdbc-0.243.2.jar:0.243.2-128118e]
at com.facebook.presto.jdbc.PrestoPreparedStatement.<init>(PrestoPreparedStatement.java:84) ~[presto-jdbc-0.243.2.jar:0.243.2-128118e]
at com.facebook.presto.jdbc.PrestoConnection.prepareStatement(PrestoConnection.java:130) ~[presto-jdbc-0.243.2.jar:0.243.2-128118e]
at com.facebook.presto.jdbc.PrestoConnection.prepareStatement(PrestoConnection.java:300) ~[presto-jdbc-0.243.2.jar:0.243.2-128118e]
at org.springframework.batch.item.database.JdbcCursorItemReader.openCursor(JdbcCursorItemReader.java:121) ~[spring-batch-infrastructure-4.2.4.RELEASE.jar:4.2.4.RELEASE]
... 63 common frames omitted
Caused by: java.lang.RuntimeException: Error fetching next at https://prestoanalytics-ch2-p.sys.comcast.net:6443/v1/statement/executing/20201118_131314_11079_v3w47/yf55745951e0beccc234c98f36005723457073854/0 returned an invalid response: JsonResponse{statusCode=502, statusMessage=Bad Gateway, headers={cache-control=[no-cache], content-length=[107], content-type=[text/html]}, hasValue=false} [Error: <html><body><h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
</body></html>
]
I was sure that the root cause is because of the driver but I have tested the driver with the same SQL this time using DriverManager and its run perfectly.
#Component
public class OmsItemReader implements ItemReader<OmsDto>, StepExecutionListener {
private ItemReader<OmsDto> delegate;
public SikOmsItemReader() {
Properties properties = new Properties();
properties.setProperty("user", "....");
properties.setProperty("password", "...");
properties.setProperty("SSL", "true");
Connection connection = null;
try {
connection = DriverManager.getConnection("jdbc:presto://.....", properties);
Statement statement = connection.createStatement();
ResultSet resultSet = statement.executeQuery(
I am not sure what is the different ? Is it the driver or sparing batch ?
I am looking for a workaround. How can I retrieve thousands of accounts via IN clauses with spring batch ?
Thank you
I'm able to create a new schema using the MeshRestClient, and get a successful response back.
Although, just after that, I try to create a node using the schema and I get an exception about missing the referenced schema.
Log output from creating a schema -
12:30:13.177 [] INFO [vert.x-worker-thread-9] [JULLogDelegate.java:167] - 127.0.0.1 - POST /api/v1/schemas/f0ee56b03d514a5fae56b03d519a5f04 HTTP/1.1 201 835 - 20 ms
12:30:13.179 [] INFO [main] [MeshService.java:81] - created schema - uuid: f0ee56b03d514a5fae56b03d519a5f04, name: form_definition
Then when creating a new node using that schema reference -
Caused by: com.gentics.mesh.rest.client.MeshRestClientMessageException: Error:404 in POST /api/v1/demo/nodes : Not Found Info: Object with uuid "f0ee56b03d514a5fae56b03d519a5f04" could not be found.
I tried setting the schema name and the schema reference in the NodeCreateRequest, but both complain.
public MeshRequest<NodeResponse> saveFormDefinition(Map<String, Object> form) {
NodeCreateRequest nodeCreateRequest = new NodeCreateRequest()
.setSchema(formDefinitionSchema.toReference())
.setLanguage("en")
.setParentNodeUuid(formsFolderNode);
String formName = (String)form.get("name");
nodeCreateRequest.getFields().putString("name", formName);
return this.client.createNode(this.meshProjectName, nodeCreateRequest);
}
Is there a time period I need to wait before it's available?
Or any other thoughts?
Thanks!
The problem was that I never subscribed to the assignSchemaToProject request -
client.assignSchemaToProject(meshProjectName, response.getUuid())
So after I subscribed, the request was executed and now available for the CreateNodeRequest.
client.assignSchemaToProject(meshProjectName, response.getUuid()).blockingGet();
I have a sql query which returns DateTime as one of the objects. I am getting an error when it is being to added to JsonArray.
Stack Trace:
SEVERE: An exception occurred
java.lang.IllegalStateException: Illegal type in JsonObject: class org.joda.time.DateTime
at io.vertx.core.json.Json.checkAndCopy(Json.java:120)
at io.vertx.core.json.JsonArray.add(JsonArray.java:437)
at io.vertx.ext.asyncsql.impl.AsyncSQLConnectionImpl$2.apply(AsyncSQLConnectionImpl.java:286)
at io.vertx.ext.asyncsql.impl.AsyncSQLConnectionImpl$2.apply(AsyncSQLConnectionImpl.java:274)
at scala.collection.Iterator$class.foreach(Iterator.scala:743)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1195)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at com.github.mauricio.async.db.general.ArrayRowData.foreach(ArrayRowData.scala:22)
at io.vertx.ext.asyncsql.impl.AsyncSQLConnectionImpl.rowToJsonArray(AsyncSQLConnectionImpl.java:274)
at io.vertx.ext.asyncsql.impl.AsyncSQLConnectionImpl.access$000(AsyncSQLConnectionImpl.java:46)
at io.vertx.ext.asyncsql.impl.AsyncSQLConnectionImpl$1.apply(AsyncSQLConnectionImpl.java:265)
at io.vertx.ext.asyncsql.impl.AsyncSQLConnectionImpl$1.apply(AsyncSQLConnectionImpl.java:262)
at scala.collection.Iterator$class.foreach(Iterator.scala:743)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1195)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at com.github.mauricio.async.db.general.MutableResultSet.foreach(MutableResultSet.scala:27)
at io.vertx.ext.asyncsql.impl.AsyncSQLConnectionImpl.rowDataSeqToJsonArray(AsyncSQLConnectionImpl.java:262)
at io.vertx.ext.asyncsql.impl.AsyncSQLConnectionImpl.queryResultToResultSet(AsyncSQLConnectionImpl.java:250)
at io.vertx.ext.asyncsql.impl.AsyncSQLConnectionImpl.lambda$null$10(AsyncSQLConnectionImpl.java:130)
at io.vertx.ext.asyncsql.impl.ScalaUtils$3.apply(ScalaUtils.java:81)
at io.vertx.ext.asyncsql.impl.ScalaUtils$3.apply(ScalaUtils.java:77)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at io.vertx.ext.asyncsql.impl.VertxEventLoopExecutionContext.lambda$execute$5(VertxEventLoopExecutionContext.java:70)
at io.vertx.core.impl.ContextImpl.lambda$wrapTask$18(ContextImpl.java:335)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:358)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
at java.lang.Thread.run(Thread.java:745)
I believe the error is occuring here:
private List<JsonArray> rowDataSeqToJsonArray(com.github.mauricio.async.db.ResultSet set) {
List<JsonArray> list = new ArrayList<>();
set.foreach(new AbstractFunction1<RowData, Void>() {
#Override
public Void apply(RowData row) {
list.add(rowToJsonArray(row));
return null;
}
});
return list;
}
My Rowdata looks like this:
Some(MutableResultSet(ArrayRowData(, , hphan, 2016-04-26T00:00:00.000-07:00, 1), ArrayRowData(, , hphan, 2016-04-28T00:00:00.000-07:00, 2), ArrayRowData(BXBSVA, BLUE CROSS BLUE SHIELD VIRGINIA, null, 2016-04-26T00:00:00.000-07:00, 1)))
Does anyone know how to fix this ?
This should work with the upcoming release 3.3. However note that the asynchronous client is marked as technology preview so there are rough edges.
Having that said if you need more features that I'd suggest to use the jdbc client instead. While not so high performance it is feature complete.
I am new to orient-db and have run into a major block even trying to open a simple in memory database.
Here is my two lines of code (in java)
OrientGraphFactory factory = new
OrientGraphFactory("memory:test").setupPool(1,10);
// EVERY TIME YOU NEED A GRAPH INSTANCE
OrientGraph g = factory.getTx();
try {
} finally {
g.shutdown();
}
I get the following error:
Exception in thread "main" com.orientechnologies.orient.core.exception.OStorageException: Cannot open local storage 'test' with mode=rw
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.open(OAbstractPaginatedStorage.java:210)
at com.orientechnologies.orient.core.db.document.ODatabaseDocumentTx.open(ODatabaseDocumentTx.java:223)
at com.orientechnologies.orient.core.db.OPartitionedDatabasePool.acquire(OPartitionedDatabasePool.java:287)
at com.tinkerpop.blueprints.impls.orient.OrientBaseGraph.<init>(OrientBaseGraph.java:163)
at com.tinkerpop.blueprints.impls.orient.OrientTransactionalGraph.<init>(OrientTransactionalGraph.java:78)
at com.tinkerpop.blueprints.impls.orient.OrientGraph.<init>(OrientGraph.java:128)
at com.tinkerpop.blueprints.impls.orient.OrientGraphFactory.getTx(OrientGraphFactory.java:74)
Caused by: com.orientechnologies.orient.core.exception.OStorageException:
Cannot open the storage 'test' because it does not exist in path: test
at
com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage .open(OAbstractPaginatedStorage.java:154)
... 7 more
What 'path' is it talking about? How is a path even relevant when trying to open a simple in memory database? Furthermore I have also tried this with plocal:/..... ,,, and I always get the above error.
Regards,
Bhargav.
Try to create the database first :
OrientGraphNoTx graph = new OrientGraphNoTx ("memory:test");
Then use the pool :
OrientGraphFactory factory = new OrientGraphFactory ("memory:test").setupPool (1, 10);
By the way which db version are you using ?
Databases created as in-memory only needs to be created first and the pool didn't allow it (fixed in last snapshot). Try acquiring an instance from the factory without pool, like:
OrientGraphFactory factory = newOrientGraphFactory("memory:test");
factory.getTx().shutdown(); // AUTO-CREATE THE GRAPH IF NOT EXISTS
factory.setupPool(1,10);
// EVERY TIME YOU NEED A GRAPH INSTANCE
OrientGraph g = factory.getTx();
try {
} finally {
g.shutdown();
}
I'm trying to create an ejabberd module which uses mongodb-erlang as the driver.
I'm doing this as follows:
ejabberd starts my module, which starts a child process with gen_server
I start mongodb application and setup a replset pool
Replset = {<<"rs1">>, [{localhost, 27017}, {localhost, 27018}]},
PoolSize = gen_mod:get_opt(poolsize, Opts, 2),
Pool = resource_pool:new(mongo:rs_connect_factory(Replset), PoolSize),
to get access to the pool, I insert the pool into an ets table
try ets:new(mymodule_pool, [named_table, public]) of
mymodule_pool -> ok
catch
_:_ -> alredy_existing
end,
ets:insert(mymodule_pool, {pool, Pool}),
when my registered hook gets called by ejabberd, I fetch the pool and a connection from it
Pool = element(2, lists:nth(1, ets:lookup(mod_log_chat_mongodb_pool, pool))),
{ok, Conn} = resource_pool:get(Pool),
then I try to insert the new doc. This is where it fails
mongo:do(unsafe, master, Conn, Db, fun() ->
mongo:insert(Coll, Doc)
end),
This is the error I get:
{badarg,
[{ets,
update_counter,
[mongodb_app,
requestid_counter,
1]},
{mongodb_app,
next_requestid,0},
{mongo_connect,
'-messages_binary/2-fun-0-',
3},
{lists,foldl,3},
{mongo_connect,
call,3},
{mongo_query,
find_one,2},
{mongo_query,
command,3},
{mongo_replset,
connect_member,2}]}
And from erlang.log:
=CRASH REPORT==== 25-Jan-2012::16:01:23 ===
crasher:
initial call: mymodule:init/1
pid: <0.289.0>
registered_name: mymodule_localhost
exception exit: {badarg,[{ets,update_counter,
[mongodb_app,requestid_counter,1]},
{mongodb_app,next_requestid,0},
{mongo_connect,'-messages_binary/2-fun-0-',3},
{lists,foldl,3},
{mongo_connect,call,3},
{mongo_query,find_one,2},
{mongo_query,command,3},
{mongo_replset,connect_member,2}]}
in function gen_server:terminate/6
ancestors: [ejabberd_sup,<0.36.0>]
messages: []
links: [<0.210.0>]
dictionary: []
trap_exit: false
status: running
heap_size: 1597
stack_size: 24
reductions: 1182
neighbours:
I tried a lot of different things but I don't have a clue why this fails.
In ets:update_counter manual is the reason for badarg errors described as follows:
the table is not of type set or ordered_set,
no object with the right key exists,
the object has the wrong arity,
the element to update is not an integer,
the element to update is also the key, or,
any of Pos, Incr, Threshold or SetValue is not an integer
The relevant code of mongodb_app is as follows:
%#doc Create global vars which will be owned by this supervisor (and die with it)
init ([]) ->
ets:new (?MODULE, [named_table, public]),
ets:insert (?MODULE, [
{oid_counter, 0},
{oid_machineprocid, oid_machineprocid()},
{requestid_counter, 0} ]),
{ok, {{one_for_one,3,10}, []}}.
%% API functions
-spec next_requestid () -> mongo_protocol:requestid(). % IO
%#doc Fresh request id
next_requestid() -> ets:update_counter (?MODULE, requestid_counter, 1).
I hope someone can help me for this, I'm helpless :(
Thank you very much,
Michael
I took a quick look at mongodb_app and I suspect one of two things.
The first thing I suspect is that mongodb_app isn't actually running. Run this in debug or live mode and make sure mongodb_app is in the list.
application:which_applications().
If it isn't then make sure that application:start(mongodb_app) is in
your module startup. If that is in your module start function then it
is probably time to check the return value of the application:start(mongodb_app).
The second thing I suspect is that there is a problem with the ets table. Investigate the status of the table with
ets:info(mongdb_app). and ets:tab2list(mongdb_app).
mongdb_app:init() declares the table as a public table so
there shouldn't be any owning process issues. I'm not sure what else
to suggest other than debugging the state of the mongodb_app ets table and
working your way back into your code from any clues you find there.