BeanInitializationException with NPE during initialization when branching with Kafka Streams - apache-kafka

I am currently working on an application using Spring Cloud Stream framework and I'm having troubles upgrading kafka deprecated method KStream.branch() to KStream.split(). I used to seperate my KStream input in several KStreams using branch() with predicates and then apply several business logic to these streams but now I encounter a BeanInitializationException during the initialization of my tests such as :
15:31:48.425 [main] WARN o.s.b.w.s.c.AnnotationConfigServletWebServerApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'kafkaStreamsFunctionProcessorInvoker': Invocation of init method failed; nested exception is org.springframework.beans.factory.BeanInitializationException: Cannot setup function invoker for this Kafka Streams function.; nested exception is java.lang.NullPointerException: Cannot invoke "org.apache.kafka.streams.kstream.KStream.toTable()" because the return value of "java.util.Map.get(Object)" is null
15:31:48.465 [main] ERROR o.s.b.SpringApplication - Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'kafkaStreamsFunctionProcessorInvoker': Invocation of init method failed; nested exception is org.springframework.beans.factory.BeanInitializationException: Cannot setup function invoker for this Kafka Streams function.; nested exception is java.lang.NullPointerException: Cannot invoke "org.apache.kafka.streams.kstream.KStream.toTable()" because the return value of "java.util.Map.get(Object)" is null
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:160)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:440)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1796)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:620)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:953)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:730)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:412)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:302)
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:164)
Here is a simplified version of my code :
#Bean
public BiFunction<
KStream<Integer, Integer>,
KTable<String, String>,
KStream<Integer, Integer>>
process() {
return (inputA, inputB) -> {
Map<String, KStream<Integer, Integer>> mapInputA = inputA
.split()
.branch((key, value) -> value > 50, // Some business predicates
Branched.as("branchOver50"))
.defaultBranch(Branched.as("branchUnder50"));
mapInputA
.get("branchOver50")
.toTable( // <- BeanCreationException (NPE : map.get() is null)
Named.as("KTable_Over_50"),
Materialized.with(Serdes.Integer(), Serdes.Integer())
).join(
inputB,
...
From my understanding, I encounter a NullPointerException during initialization because the map is empty since there is no data yet, but since the deprecated method KStream.branch() didn't require any action to handle empty streams, I was wondering if this was an intended behavior of the new method KStream.split() or if I was doing something wrong.
The error happens during a JUnit test using an EmbeddedKafkaBroker to simulate the Kafka environment (I am using kafka-streams version 3.0.0).

Related

java.lang.NullPointerException: Cannot invoke "org.apache.kafka.streams.processor.api.ProcessorSupplier.get()" because this.processorSupplier is null

I use Kafka Streams SDK 3.3.1 and pretty simple topology (that calculates overall number of messages) but got rather weird exception when creating topology:
java.lang.NullPointerException: Cannot invoke "org.apache.kafka.streams.processor.api.ProcessorSupplier.get()" because "this.processorSupplier" is null
at org.apache.kafka.streams.kstream.internals.graph.ProcessorParameters.toString(ProcessorParameters.java:133)
at java.base/java.lang.String.valueOf(String.java:4225)
at java.base/java.lang.StringBuilder.append(StringBuilder.java:173)
at org.apache.kafka.streams.kstream.internals.graph.BaseRepartitionNode.toString(BaseRepartitionNode.java:77)
at org.apache.kafka.streams.kstream.internals.graph.OptimizableRepartitionNode.toString(OptimizableRepartitionNode.java:63)
My code to create topology:
private static Topology createTopology() {
StreamsBuilder builder = new StreamsBuilder();
KTable<Integer, Long> table = builder.stream("messages")
.selectKey((key, value) -> 1)
.groupByKey()
.count();
table.toStream().to("stats");
return builder.build();
}
This is a bug that has already been fixed:
https://issues.apache.org/jira/browse/KAFKA-14325
It should only affect you when calling toString on the topology. Please upgrade to 3.3.2 or, as a workaround, catch the exception around the code that calls toString (could be Kafka itself if DEBUG log level is enabled).

Spring Boot #Scheduled method can not access database relation

Jpa repositoy methods called from inside of a #Scheduled method is throwing PSQLException and saying the relation does not exist, where the same repository method is being executed fine without any problems from other parts of the code.
This is my #Configuration class for Scheduler,
#Configuration
#EnableScheduling
public class SchedulingConfig {
#Bean
public TaskScheduler taskScheduler() {
ScheduledThreadPoolExecutor executor = new ScheduledThreadPoolExecutor(1);
executor.setThreadFactory(new ThreadFactoryBuilder().setNameFormat("Scheduled-%d").build());
ConcurrentTaskScheduler taskScheduler = new ConcurrentTaskScheduler(executor);
taskScheduler.setTaskDecorator(new MasterTaskDecorator());
return taskScheduler;
}
}
And this is the #Scheduled annotated method,
#Scheduled(fixedRate = 60000, initialDelay = 60000)
public void scheduledRevalidationOfOpenJourney() {
List<JourneyEntity> openJourneys = journeyRepository.findAllByState(JourneyState.OPEN);
openJourneys.forEach(openJourney -> openJourney.getDatasets().forEach(validationService::removeRelatedViolation));
}
From there on, this is the exception being thrown,
2022-12-02 15:22:58.894 ERROR 32704 --- [ scheduling-1] o.h.engine.jdbc.spi.SqlExceptionHelper : ERROR: relation "journey" does not exist
org.springframework.dao.InvalidDataAccessResourceUsageException: could not extract ResultSet; SQL [n/a]; nested exception is org.hibernate.exception.SQLGrammarException: could not extract ResultSet
2022-12-02 15:23:58.896 ERROR 32704 --- [ scheduling-1] o.s.s.s.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task
Caused by: org.hibernate.exception.SQLGrammarException: could not extract ResultSet
Caused by: org.postgresql.util.PSQLException: ERROR: relation "journey" does not exist
As I mentioned before, the table 'journey' does already exist and its repository methods are already being executed without any problems elsewhere in the code. Can anybody please point out what is causing this scenario and what could be the solution to it?

I get exception with axon-mongo 4.5, spring boot 2.4.3 and mongodb 4.2 in AxonConfig class

When running the project I get this exception:
org.springframework.beans.factory.BeanCreationException: Error creating bean with name
'axonMongoTemplate' defined in com.springbank.user.core.configuration.AxonConfig: Bean
instantiation via factory method failed; nested exception is
org.springframework.beans.BeanInstantiationException: Failed to instantiate
[org.axonframework.extensions.mongo.MongoTemplate]: Factory method 'axonMongoTemplate'
threw exception; nested exception is java.lang.IllegalStateException: #Bean method
AxonConfig.mongo called as bean reference for type [com.mongodb.MongoClient] but overridden
by non-compatible bean instance of type [com.mongodb.client.internal.MongoClientImpl].
Overriding bean of same name declared in: class path resource
[org/springframework/boot/autoconfigure/mongo/MongoAutoConfiguration.class]
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate
[org.axonframework.extensions.mongo.MongoTemplate]: Factory method 'axonMongoTemplate'
threw exception; nested exception is java.lang.IllegalStateException: #Bean method
AxonConfig.mongo called as bean reference for type [com.mongodb.MongoClient] but overridden
by non-compatible bean instance of type [com.mongodb.client.internal.MongoClientImpl].
Overriding bean of same name declared in: class path resource
[org/springframework/boot/autoconfigure/mongo/MongoAutoConfiguration.class]
Caused by: java.lang.IllegalStateException: #Bean method AxonConfig.mongo called as bean
reference for type [com.mongodb.MongoClient] but overridden by non-compatible bean instance
of type [com.mongodb.client.internal.MongoClientImpl]. Overriding bean of same name declared
in: class path resource
[org/springframework/boot/autoconfigure/mongo/MongoAutoConfiguration.class]
this is my axon configuration class:
#Configuration
public class AxonConfig {
#Value("${spring.data.mongodb.host:127.0.0.1}")
private String mongoHost;
#Value("${spring.data.mongodb.port:27017}")
private int mongoPort;
#Value("${spring.data.mongodb.database:user}")
private String mongoDatabase;
#Bean
public MongoClient mongo() {
var mongoFactory = new MongoFactory();
mongoFactory.setMongoAddresses(Collections.singletonList(new ServerAddress(mongoHost, mongoPort)));
return mongoFactory.createMongo();
}
#Bean
public MongoTemplate axonMongoTemplate() {
return DefaultMongoTemplate.builder()
.mongoDatabase(mongo(), mongoDatabase)
.build();
}
...
}
The spring boot version and dependencies I use are:
org.springframework.boot:spring-boot-starter-parent:2.4.3
org.axonframework:axon-spring-boot-starter:4.5.9
org.axonframework.extensions.mongo:axon-mongo:4.3
this is my application.properties:
#spring
server.port=8082
#mongodb
spring.data.mongodb.host=localhost
spring.data.mongodb.port=27017
spring.data.mongodb.database=user
spring.main.allow-bean-definition-overriding=true
spring.main.allow-circular-references=true
I think you can just remove the MongoClient Bean, as Spring Boot is already doing that for you.
After several tests of dependency versions, I found the dependencies that did not conflict with each other.
Here's the list of versions used:
Java JDK 14
spring-boot-starter-parent:2.2.6.RELEASE
axon-spring-boot-starter:4.3.1
lombok:1.18.20
Using other versions of spring boot exceptions occurred.

Error creating unit test with Spring cloud stream using kafka

i dunno how make one sample test using kafka, i tried to follow the spring guide but dont work.
Can someone help me?
zzzzz zz z z z z z z z z z z z
#RunWith(SpringRunner.class)
#SpringBootTest
#DirtiesContext
public class EnrollSenderTest {
#Autowired
public EnrollSender producer;
#Autowired
private BinderFactory<MessageChannel> binderFactory;
#Autowired
private MessageCollector messageCollector;
#SuppressWarnings("unchecked")
#Test
public void test() {
Message<String> message = new GenericMessage<>("hello");
producer.sendEnroll(message);
Message<String> received = (Message<String>) messageCollector.forChannel(producer.getOutput()).poll();
assertThat(received.getPayload(), equalTo("hello"));
}
}
And my class Producer is:
#Service
#EnableBinding(Source.class)
public class EnrollSender {
private final MessageChannel output;
public EnrollSender(Source output) {
this.output = output.output();
}
public void sendEnroll(Object enroll) {
output.send(MessageBuilder.withPayload(enroll).build());
}
public MessageChannel getOutput() {
return output;
}
}
But gives the following error:
java.lang.IllegalStateException: Failed to load ApplicationContext
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'messageCollector' defined in class path resource [org/springframework/cloud/stream/test/binder/TestSupportBinderAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.cloud.stream.test.binder.MessageCollector]: Factory method 'messageCollector' threw exception; nested exception is java.lang.NoSuchMethodError: org.springframework.cloud.stream.binder.BinderFactory.getBinder(Ljava/lang/String;Ljava/lang/Class;)Lorg/springframework/cloud/stream/binder/Binder;
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.cloud.stream.test.binder.MessageCollector]: Factory method 'messageCollector' threw exception; nested exception is java.lang.NoSuchMethodError: org.springframework.cloud.stream.binder.BinderFactory.getBinder(Ljava/lang/String;Ljava/lang/Class;)Lorg/springframework/cloud/stream/binder/Binder;
Caused by: java.lang.NoSuchMethodError: org.springframework.cloud.stream.binder.BinderFactory.getBinder(Ljava/lang/String;Ljava/lang/Class;)Lorg/springframework/cloud/stream/binder/Binder;
Marius Bogoevici, my dependencys
dependencyManagement {
imports {
mavenBom "org.springframework.cloud:spring-cloud-dependencies:Camden.SR4"
}
}
compile 'org.springframework.cloud:spring-cloud-starter-stream-kafka'
compile group: 'org.springframework.cloud', name: 'spring-cloud-stream-test-support', version: '1.1.1.RELEASE'
Looks like you have a mismatched dependency set on the classpath (i.e. an older version of Spring Cloud Stream core).
You can solve this by removing the version for spring-cloud-stream-test-support because the Camden.SR4 BOM will provide the correct one.
Moreover, if you want to test with an embedded Kafka instance, you can find an example here: https://github.com/spring-cloud/spring-cloud-stream-samples/blob/master/multibinder/src/test/java/multibinder/RabbitAndKafkaBinderApplicationTests.java#L57
(The example shows you how to configure the Kafka binder with an embedded broker for testing - it also shows how to use two different binders within the same app, but probably you don't care about that).
This is because of the incompatible versions as pointed out by Marius above.
You would either need Camden.SR5 that has compatible versions of Spring Cloud Stream and Spring Cloud Stream test support or Camden.SR4 with Spring Cloud Stream test support version 1.1.0.RELEASE.
This is change that went in between 1.1.0.RELEASE and 1.1.1.RELEASE of Spring Cloud Steram:

How to see nested exceptions trace, hidden by Spring?

When I am debugging Spring application in Eclipse, I am getting long exception chains. For example, I have
Error creating bean with name '...' defined in file [...Tester.xml]: Error setting property values; nested exception is org.springframework.beans.PropertyBatchUpdateException; nested PropertyAccessExceptions (1) are:
PropertyAccessException 1: org.springframework.beans.MethodInvocationException: Property '...' threw exception; nested exception is java.lang.IllegalArgumentException: ...
And so on. Multiple stacks with exceptions inside Spring, which is uninteresting. It should be my exceptions somewhere below, but Spring does not show them.
And I can't click into exception and navigate to problem place as usual.
How to say Spring to output all exceptions?
UPDATE
Below is the full output. One can see that the place where IllegalArgumentException occurred was probably truncated.
Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mybean' defined in file [D:\mypath\myconfig.xml]: Error setting property values; nested exception is org.springframework.beans.PropertyBatchUpdateException; nested PropertyAccessExceptions (1) are:
PropertyAccessException 1: org.springframework.beans.MethodInvocationException: Property 'myproperty' threw exception; nested exception is java.lang.IllegalArgumentException: my exception message
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1361)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1086)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:580)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
at org.springframework.context.support.FileSystemXmlApplicationContext.(FileSystemXmlApplicationContext.java:140)
at org.springframework.context.support.FileSystemXmlApplicationContext.(FileSystemXmlApplicationContext.java:84)
at springtests.SpringRunner.main(SpringRunner.java:8)
Caused by: org.springframework.beans.PropertyBatchUpdateException; nested PropertyAccessExceptions (1) are:
PropertyAccessException 1: org.springframework.beans.MethodInvocationException: Property 'target.partner' threw exception; nested exception is java.lang.IllegalArgumentException: Illegal frame length 1 in explicit constructor
at org.springframework.beans.AbstractPropertyAccessor.setPropertyValues(AbstractPropertyAccessor.java:102)
at org.springframework.beans.AbstractPropertyAccessor.setPropertyValues(AbstractPropertyAccessor.java:58)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1358)
... 13 more
Because there are potentially multiple exceptions, you need to catch the PropertyBatchUpdateException and call getPropertyAccessExceptions() to examine the stack trace of the particular exception.
Edit
Actually I'm not quite sure what's going on here
Here is PropertyBatchUpdateException's printStackTrace method:
public void printStackTrace(PrintWriter pw) {
synchronized (pw) {
pw.println(getClass().getName() + "; nested PropertyAccessException details (" +
getExceptionCount() + ") are:");
for (int i = 0; i < this.propertyAccessExceptions.length; i++) {
pw.println("PropertyAccessException " + (i + 1) + ":");
this.propertyAccessExceptions[i].printStackTrace(pw);
}
}
}
It should be including the nested stack traces. Are you using the latest version of Spring?
Edit:
The best I can suggest is to run in debug mode, then put a breakpoint at AbstractAutowireCapableBeanFactory.java:1361 and see what's going on.