How to execute JPA Entity manager operations inside Quarkus-Kafka consumer method - apache-kafka

Dears, I want to update my domain entities according to message being received by Kafka topic. I am using Quarkus latest and Smallrye reactive messaging with Kafka. Pub-sub model is working fine with me, but in consumer method I am unable to update my entities using entityManager or HibernatePanache.
Whenever I try to use entityManager code inside consumer message, an exception is being thrown and handled silently. Here is my consumer code :
#Transactional
#Incoming("new-payment")
public CompletionStage<Void> updateTotalBuyers(String paymentEvent) {
return CompletableFuture.runAsync(() -> {
PaymentEvent event = jsonb.fromJson(paymentEvent, PaymentEvent.class);
TypedQuery<Book> query = em.createQuery("SELECT b FROM Book b where b.isbn=:isbn", Book.class);
query.setParameter("isbn", event.getIsbn());
Book book = query.getSingleResult();
book.setTotalBuyers(book.getTotalBuyers() + 1);
em.merge(book);
});
}
If anyone has a working code snippet for my problem it would be great. Also, how can I print that silent exception for further debugging ?
Update :
I surrounded the code with try/catch block and the below exception is being thrown :
javax.enterprise.context.ContextNotActiveException: interface javax.enterprise.context.RequestScoped
at io.quarkus.hibernate.orm.runtime.RequestScopedEntityManagerHolder_ClientProxy.arc$delegate(RequestScopedEntityManagerHolder_ClientProxy.zig:83)
at io.quarkus.hibernate.orm.runtime.RequestScopedEntityManagerHolder_ClientProxy.getOrCreateEntityManager(RequestScopedEntityManagerHolder_ClientProxy.zig:191)
at io.quarkus.hibernate.orm.runtime.entitymanager.TransactionScopedEntityManager.getEntityManager(TransactionScopedEntityManager.java:78)
at io.quarkus.hibernate.orm.runtime.entitymanager.TransactionScopedEntityManager.createQuery(TransactionScopedEntityManager.java:317)
at io.quarkus.hibernate.orm.runtime.entitymanager.ForwardingEntityManager.createQuery(ForwardingEntityManager.java:142)
at io.quarkus.hibernate.orm.panache.runtime.JpaOperations.find(JpaOperations.java:208)
at io.quarkus.hibernate.orm.panache.runtime.JpaOperations.find(JpaOperations.java:200)
at org.ibra.ebs.book.model.Book.find(Book.java)
at org.ibra.ebs.book.service.BookService.lambda$updateTotalBuyers$0(BookService.java:106)
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
at java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1618)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
I added annotation #ActivateRequestContext on both class and method with no luck.
Update : I tried to elevate context-propagation using
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-smallrye-reactive-streams-operators</artifactId>
</dependency>
Also same exception is being thrown with some context-propagation classes (which means it is being activated).
Regards.

Finally it worked :), Yes the answer was around context-propagation and here is what I did :
#Transactional
#Incoming("new-payment")
public CompletionStage<?> updateTotalBuyers(Message<String> paymentEvent) {
//TODO: move to CDI producer
ManagedExecutor executor = ManagedExecutor.builder()
.maxAsync(5)
.propagated(ThreadContext.CDI,
ThreadContext.TRANSACTION)
.build();
//TODO: move to CDI producer
ThreadContext threadContext = ThreadContext.builder()
.propagated(ThreadContext.CDI,
ThreadContext.TRANSACTION)
.build();
return executor.runAsync(threadContext.contextualRunnable(() -> {
try {
log.info("Into update total buyers");
PaymentEvent event = jsonb.fromJson(paymentEvent.getPayload(), PaymentEvent.class);
Book book = Book.find("isbn", event.getIsbn()).singleResult();
book.totalBuyers++;
book.persist();
log.info("Total books {}", book.totalBuyers);
} catch(Exception e) {
log.error("Something wrong happened !!!", e);
} finally {
paymentEvent.ack();
}
}));
}
I will try to make it more standard now.

Add #ApplicationScoped and #ActivateRequestContext both these annotations on the class and #Transactional annotation on the updateTotalBuyers method.
Cheers!

I had simillar issue using Quarkus + a single thread executor to collect AWS SQS messages.
Initial - wrong approach:
private final ExecutorService scheduler = Executors.newSingleThreadExecutor();
scheduler.submit(/*MyRunnable*/)
Solution: Injecting ManagedExecutor which is provided by quarkus:
#Inject
ManagedExecutor managedExecutor;
managedExecutor.submit(/*MyRunnable*/);
In your particular case you can replace following code with injecting ManagedExecutor.
ManagedExecutor executor = ManagedExecutor.builder()
.maxAsync(5)
.propagated(ThreadContext.CDI,
ThreadContext.TRANSACTION)
.build();
//TODO: move to CDI producer
ThreadContext threadContext = ThreadContext.builder()
.propagated(ThreadContext.CDI,
ThreadContext.TRANSACTION)
.build();

Related

Spring kafka setErrorHandler deprecated replacement (boot 2.6.4)

On spring boot 2.6.4, this method is deprecated.
public ConcurrentKafkaListenerContainerFactory<Object, Object> kafkaListenerContainerFactory(
ConcurrentKafkaListenerContainerFactoryConfigurer configurer) {
var factory = new ConcurrentKafkaListenerContainerFactory<Object, Object>();
configurer.configure(factory, consumerFactory());
// deprecated
factory.setErrorHandler(new GlobalErrorHandler());
return factory;
}
The global error handler class
public class GlobalErrorHandler implements ConsumerAwareErrorHandler {
private static final Logger log = LoggerFactory.getLogger(GlobalErrorHandler.class);
#Override
public void handle(Exception thrownException, ConsumerRecord<?, ?> data, Consumer<?, ?> consumer) {
// my custom global logic (e.g. notify ops team via slack)
}
}
What is the replacement sample for this? The doc says I should use setCommonErrorHandler, but how to implements the CommonErrorHandler interface, as no method to be overriden there.
Point is, I have to send slack notification to ops team, based on certain condition (the message tpye, which is available on kafka message header)
This is not blocking, just an annoying deprecated message though.
Thanks
See the Spring for Apache Kafka documentation; legacy error handlers are replaced with CommonErrorHandler implementations.
What's New?
https://docs.spring.io/spring-kafka/docs/current/reference/html/#x28-eh
The legacy GenericErrorHandler and its sub-interface hierarchies for record an batch listeners have been replaced by a new single interface CommonErrorHandler with implementations corresponding to most legacy implementations of GenericErrorHandler. See Container Error Handlers for more information.
Container Error Handlers
https://docs.spring.io/spring-kafka/docs/current/reference/html/#error-handlers
Starting with version 2.8, the legacy ErrorHandler and BatchErrorHandler interfaces have been superseded by a new CommonErrorHandler. These error handlers can handle errors for both record and batch listeners, allowing a single listener container factory to create containers for both types of listener. CommonErrorHandler implementations to replace most legacy framework error handler implementations are provided and the legacy error handlers deprecated. The legacy interfaces are still supported by listener containers and listener container factories; they will be deprecated in a future release.
I was facing exactly the same problem, so I changed the method implementation ConsumerAwareErrorHandler by
CommonErrorHandler
and implemented
handleRecord
like described in the docs and it works!
public class GlobalErrorHandler implements CommonErrorHandler {
private static final Logger log = LoggerFactory.getLogger(GlobalErrorHandler.class);
#Override
public void handleRecord(
Exception thrownException,
ConsumerRecord<?, ?> record,
Consumer<?, ?> consumer,
MessageListenerContainer container) {
log.warn("Global error handler for message: {}", record.value().toString());
}
}
In KafkaConfig.class
#Bean(value = "kafkaListenerContainerFactory")
public ConcurrentKafkaListenerContainerFactory<Object, Object> kafkaListenerContainerFactory(
ConcurrentKafkaListenerContainerFactoryConfigurer configurer) {
var factory = new ConcurrentKafkaListenerContainerFactory<>();
configurer.configure(factory, consumerFactory());
factory.setCommonErrorHandler(new GlobalErrorHandler());
return factory;
}

How to call multiple DB calls from different threads, under same transaction?

I have a requirement to perform clean insert (delete + insert), a huge number of records (close to 100K) per requests. For sake testing purpose, I'm testing my code with 10K. With 10K also, the operation is running for 30 secs, which is not acceptable. I'm doing some level of batch inserts provided by spring-data-JPA. However, the results are not satisfactory.
My code looks like below
#Transactional
public void saveAll(HttpServletRequest httpRequest){
List<Person> persons = new ArrayList<>();
try(ServletInputStream sis = httpRequest.getInputStream()){
deletePersons(); //deletes all persons based on some criteria
while((Person p = nextPerson(sis)) != null){
persons.add(p);
if(persons.size() % 2000 == 0){
savePersons(persons); //uses Spring repository to perform saveAll() and flush()
persons.clear();
}
}
savePersons(persons); //uses Spring repository to perform saveAll() and flush()
persons.clear();
}
}
#Transactional
public void savePersons(List<Persons> persons){
System.out.println(new Date()+" Before save");
repository.saveAll(persons);
repository.flush();
System.out.println(new Date()+" After save");
}
I have also set below properties
spring.jpa.properties.hibernate.jdbc.batch_size=40
spring.jpa.properties.hibernate.order_inserts=true
spring.jpa.properties.hibernate.order_updates=true
spring.jpa.properties.hibernate.jdbc.batch_versioned_data=true
spring.jpa.properties.hibernate.id.new_generator_mappings=false
Looking at logs, I noticed that the insert operation is taking around 3 - 4 secs to save 2000 records, but not much on iteration. So I believe the time taken to read through the stream is not a bottleneck. But the inserts are. I also checked the logs and confirm that Spring is doing a batch of 40 inserts as per the property set.
I'm trying to see, if there is a way, I can improve the performance, by using multiple threads (say 2 threads) that would read from a blocking queue, and once accumulated say 2000 records, will call save. I hope, in theory, this may provide better results. But the problem is as I read, Spring manages Transactions at the thread level, and Transaction can not propagate across threads. But I need the whole operation (delete + insert) as atomic. I looked into few posts about Spring transaction management and could not get into the correct direction.
Is there a way I can achieve this kind of parallelism using Spring transactions? If Spring transactions is not the answer, are there any other techniques that can be used?
Thanks
Unsure if this will be helpful to you - it is working well in a test app. Also, do not know if it will be in the "good graces" of senior Spring personnel but my hope is to learn so I am posting this suggestion.
In a Spring Boot test app, the following injects a JPA repository into the ApplicationRunner which then injects the same into Runnables managed by an ExecutorService. Each Runnable gets a BlockingQueue that is being continually filled by a separate KafkaConsumer (which is acting like a producer for the queue). The Runnables use queue.takes() to pop from the queue and this is followed by a repo.save(). (Can readily add batch insert to thread but haven't done so since application has not yet required it...)
The test app currently implements JPA for Postgres (or Timescale) DB and is running 10 threads with 10 queues being fed by 10 Consumers.
JPA repository is provide by
public interface DataRepository extends JpaRepository<DataRecord, Long> {
}
Spring Boot Main program is
#SpringBootApplication
#EntityScan(basePackages = "com.xyz.model")
public class DataApplication {
private final String[] topics = { "x0", "x1", "x2", "x3", "x4", "x5","x6", "x7", "x8","x9" };
ExecutorService executor = Executors.newFixedThreadPool(topics.length);
public static void main(String[] args) {
SpringApplication.run(DataApplication.class, args);
}
#Bean
ApplicationRunner init(DataRepository dataRepository) {
return args -> {
for (String topic : topics) {
BlockingQueue<DataRecord> queue = new ArrayBlockingQueue<>(1024);
JKafkaConsumer consumer = new JKafkaConsumer(topic, queue);
consumer.start();
JMessageConsumer messageConsumer = new JMessageConsumer(dataRepository, queue);
executor.submit(messageConsumer);
}
executor.shutdown();
};
}
}
And the Consumer Runnable has a constructor and run() method as follows:
public JMessageConsumer(DataRepository dataRepository, BlockingQueue<DataRecord> queue) {
this.queue = queue;
this.dataRepository = dataRepository;
}
#Override
public void run() {
running.set(true);
while (running.get()) {
// remove record from FIFO blocking queue
DataRecord dataRecord;
try {
dataRecord = queue.take();
} catch (InterruptedException e) {
logger.error("queue exception: " + e.getMessage());
continue;
}
// write to database
dataRepository.save(dataRecord);
}
}
Into learning so any thoughts/concerns/feedback is appreciated...

Cannot remove a JPA entity using Spring Integration

When I try to remove an entity using Outbound Channel Adapter I always get removing a detached instance exception.
I know that an entity should be retrieved and deleted in the same transaction to avoid this exception, but how can I achieve it with Spring Integration?
To demonstrate the problem I modified the JPA sample:
PersonService.java
public interface PersonService {
...
void deletePerson(Person person);
}
Main.java
private static void deletePerson(final PersonService service) {
final List<Person> people = service.findPeople();
Person p1 = people.get(0);
service.deletePerson(p1);
}
spring-integration-context.xml
<int:gateway id="personService"
service-interface="org.springframework.integration.samples.jpa.service.PersonService"
default-request-timeout="5000" default-reply-timeout="5000">
<int:method name="createPerson" request-channel="createPersonRequestChannel"/>
<int:method name="findPeople" request-channel="listPeopleRequestChannel"/>
<int:method name="deletePerson" request-channel="deletePersonChannel"/>
</int:gateway>
<int:channel id="deletePersonChannel"/>
<int-jpa:outbound-channel-adapter entity-manager-factory="entityManagerFactory"
channel="deletePersonChannel" persist-mode="DELETE" >
<int-jpa:transactional transaction-manager="transactionManager" />
</int-jpa:outbound-channel-adapter>
When I call deletePerson I get the exception:
Exception in thread "main" java.lang.IllegalArgumentException:
Removing a detached instance
org.springframework.integration.samples.jpa.Person#1001
UPDATE:
Apparently I should've chosen a sample closer to my actual project, because here you can just create a new transaction programmatically and wrap both retrieve and delete function calls in it, as Artem did.
In my project I have a transformer connected to an outbound-channel-adapter. The transformer retrieves an entity and the outbound-channel-adapter removes it. How can I get the transformer and the outbound-channel-adapter to use the same transaction in this case?
To get it worked you should wrap all operations in the deletePerson to transaction, e.g.
private static void deletePerson(final PersonService service) {
new new TransactionTemplate(transactionManager)
.execute(new TransactionCallbackWithoutResult() {
protected void doInTransactionWithoutResult(TransactionStatus status) {
final List<Person> people = service.findPeople();
Person p1 = people.get(0);
service.deletePerson(p1);
}
});
}
In this case you should somehow provide to your method transactionManager bean too.
UPDATE:
I shown you a sample for use-case in the original question.
Now re. <transformer> -> <jpa:outbound-channel-adapter>.
In this you should understand where your message flow is started:
If it is <inbound-channel-adapter> with poller, so just make the <poller> <transactional>
If it <gateway>, who call <transformer>, so it's just enough to mark gateway's method with #Transactional
Here is one more transactional advice trick: Keep transaction within Spring Integration flow
In all cases you should get rid of <transactional> from your <jpa:outbound-channel-adapter>

Autofac, OrchardProject and AsyncControllers

I'm working on trying to get an AsyncController to work in OrchardProject. The current version I'm using is 2.2.4.9.0.
I've had 2 people eyeball my code: http://www.pastie.org/2117952 (AsyncController) which works fine in a regular MVC3 vanilla application.
Basically, I can route to IndexCompleted, but I can't route to Index. I am going to assume i'm missing something in the Autofac configuration of the overall project.
I think the configuration is in the global.asax: http://pastie.org/2118008
What I'm looking for is some guidance on if this is the correct way to implement autofac for AsyncControllers, or if there is something/someplace else I need to implement/initialize/etc.
~Dan
Orchard appears to register its own IActionInvoker, called Orchard.Mvc.Filters.FilterResolvingActionInvoker.
This class derives from ControllerActionInvoker. At a guess, in order to support async actions, it should instead derive from AsyncControllerActionInvoker.
Hope this helps!
Nick
The Autofac setup looks ok, and as long as you can navigate to something I cannot say that your assumption makes sense. Also, the pattern you are using for initialization in global.asax is used by others too.
The AsyncController requires that async methods come in pairs, in your case IndexAsync & IndexCompleted. These together represent the Index action. When you say you can navigate to IndexCompleted, do you mean that you open a url "..../IndexCompleted"?
Also, and this I cannot confirm from any documentation, but I would guess that AsyncController requires that all actions are async. Thus, your NewMessage action causes trouble and should be converted to an async NewMessageAsync & NewMessageCompleted pair.
I did too needed to have AsyncController which I easily changed FilterResolvingActionInvoker to be based on AsyncControllerActionInvoker instead of ControllerActionInvoker.
But there was other problems because of automatic transaction disposal after completion of request. In AsyncController starting thread and the thread that completes the request can be different which throws following exception in Dispose method of TransactionManager class:
A TransactionScope must be disposed on the same thread that it was created.
This exception is suppressed without any logging and really was hard to find out. In this case session remains not-disposed and subsequent sessions will timeout.
So I made dispose method public on ITransactionManager and now in my AsyncController, whenever I need a query to database I wrap it in:
using (_services.TransactionManager) {
.....
}
new TransactionManager :
public interface ITransactionManager : IDependency, IDisposable {
void Demand();
void Cancel();
}
public class TransactionManager : ITransactionManager {
private TransactionScope _scope;
private bool _cancelled;
public TransactionManager() {
Logger = NullLogger.Instance;
}
public ILogger Logger { get; set; }
public void Demand() {
if (_scope == null) {
Logger.Debug("Creating transaction on Demand");
_scope = new TransactionScope(
TransactionScopeOption.Required,
new TransactionOptions {
IsolationLevel = IsolationLevel.ReadCommitted
});
_cancelled = false;
}
}
void ITransactionManager.Cancel() {
Logger.Debug("Transaction cancelled flag set");
_cancelled = true;
}
void IDisposable.Dispose() {
if (_scope != null) {
if (!_cancelled) {
Logger.Debug("Marking transaction as complete");
_scope.Complete();
}
Logger.Debug("Final work for transaction being performed");
try {
_scope.Dispose();
}
catch {
// swallowing the exception
}
Logger.Debug("Transaction disposed");
}
_scope = null;
}
}
Please notice that I have made other small changes to TransactionManager.
I tried the AsyncControllerActionInvoker route as well to no avail. I would get intermittent errors from Orchard itself with the following errors:
Orchard.Exceptions.DefaultExceptionPolicy - An unexpected exception was caught
System.TimeoutException: The operation has timed out.
at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult`1.End()
at System.Web.Mvc.Async.ReflectedAsyncActionDescriptor.EndExecute(IAsyncResult asyncResult)
at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass3f.<BeginInvokeAsynchronousActionMethod>b__3e(IAsyncResult asyncResult)
at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult`1.End()
at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult)
at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass37.<>c__DisplayClass39.<BeginInvokeActionMethodWithFilters>b__33()
at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass4f.<InvokeActionMethodFilterAsynchronously>b__49()
at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass4f.<InvokeActionMethodFilterAsynchronously>b__49()
at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass4f.<InvokeActionMethodFilterAsynchronously>b__49()
NHibernate.Util.ADOExceptionReporter - While preparing SELECT this_.Id as Id236_2_, this_.Number as Number236_2_,...<blah blah blah>
NHibernate.Util.ADOExceptionReporter - The connection object can not be enlisted in transaction scope.
So I don't think just wrapping your own database calls with a transaction object will help. The innards of Orchard would have to modified as well.
Go vote for this issue if you want AsyncControllers supported in Orchard:
https://orchard.codeplex.com/workitem/18012

gwt-rpc + appengine + persistence using restlet throws exception

I was trying to rebuild the Restlet sample Application for GWT + GAE ( http://wiki.restlet.org/docs_2.1/13-restlet/21-restlet/318-restlet/303-restlet.html ) .
I changed it a bit, since I am planning something diffrent but I thought it would be a good start.
It was going okish until now. The "Put" was coming through to app engine but when i tried to persist the Objects using JPA i get the following Exception:
Caused by: org.datanucleus.exceptions.ClassNotResolvedException: Class "de.fr1zle.shoplist.web.gae.client.ShoppingListRessourceProxy" was not found in the CLASSPATH. Please check your specification and your CLASSPATH.
at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:250)
at org.datanucleus.JDOClassLoaderResolver.classForName(JDOClassLoaderResolver.java:415)
at org.datanucleus.metadata.MetaDataManager.loadPersistenceUnit(MetaDataManager.java:767)
... 79 more
As you can see, datanucleus somehow tries to access the GWT Proxy class when loading the info from the persistence.xml.
I use the following in my ServerRessource:
#Put
public void putShoppingList(ShoppingList shoppingList) {
ShoppingListDOA shoppingListDOA = new ShoppingListDOA(shoppingList);
EntityManagerFactory emf = Persistence
.createEntityManagerFactory("transactions-optional");
try {
EntityManager entityManager = emf.createEntityManager();
EntityTransaction transaction = entityManager.getTransaction();
transaction.begin();
entityManager.persist(shoppingListDOA);
entityManager.flush();
transaction.commit();
entityManager.close();
} catch (Exception e) {
e.printStackTrace();
} finally {
if (emf != null)
emf.close();
}
}
I somehow have the feeling that DataNucleus enhances the Proxy Class, too although I changed the properites for it to not do so.
Using: GAE 1.4.2 (tried 1.4.3, too) , GWT 2.2 and Restlet 2.1m3
Am I missing a point here? Your help is appricated :-)
Thanks in advance!
fr1zle