I am trying to create a Spring Boot application with Spring Cloud Stream and Kafka integration. I created a sample Topic in Kafka with 1 partition and have published to the topic from the Spring Boot application created based on the directions given here
http://docs.spring.io/spring-cloud-stream/docs/1.0.2.RELEASE/reference/htmlsingle/index.html
and
https://blog.codecentric.de/en/2016/04/event-driven-microservices-spring-cloud-stream/
Spring Boot App -
#SpringBootApplication
public class MyApplication {
private static final Log logger = LogFactory.getLog(MyApplication.class);
public static void main(String[] args) {
SpringApplication.run(MyApplication.class, args);
}
}
Kafka Producer Class
#Service
#EnableBinding(Source.class)
public class MyProducer {
private static final Log logger = LogFactory.getLog(MyProducer.class);
#Bean
#InboundChannelAdapter(value = Source.OUTPUT, poller = #Poller(fixedDelay = "10000", maxMessagesPerPoll = "1"))
public MessageSource<TimeInfo> timerMessageSource() {
TimeInfo t = new TimeInfo(new Timestamp(new Date().getTime())+"","Label");
MessageBuilder<TimeInfo> m = MessageBuilder.withPayload(t);
return () -> m.build();
}
public static class TimeInfo{
private String time;
private String label;
public TimeInfo(String time, String label) {
super();
this.time = time;
this.label = label;
}
public String getTime() {
return time;
}
public String getLabel() {
return label;
}
}
}
All is working well except for when I want to handle exceptions.
If the Kafka Topic went down, I can see the ConnectionRefused exception being thrown in the log files for the app, but the retry logic built in seems to be going at retrying continuously without stopping!
There is no exception thrown at all for me to handle and do further exception processing. I have read through the Producer options and the Binder options for Kafka in the Spring Cloud Stream documentation above and I cannot see any customization options possible to get this exception thrown above all the way for me to capture.
I am new to Spring Boot / Spring Cloud Stream / Spring Integration (which seems to be the underlying implementation to the cloud stream project).
Is there anything else you guys know to get this exception cascaded to my Spring Cloud Stream app?
Related
I'm using Spring Cloud Stream 3.1.3.
I'm migrate from a pre 3.1 version, so I wrote my producer using a java.util.Function (I know I can use Supplier but this is what I need)
application.yaml file is configured with function definition, input and output bindings, and this is what I have:
#EnableAutoConfiguration
#Service
public class Producer {
public void produce(int messageId, Object data) {
Message<Object> message = MessageBuilder
.withPayload(data)
.setHeader(PARTITION_KEY, messageId)
.build();
streamBridge.send("produceMessage-in-0", message);
}
#Bean
public Function<Message<Object>, Message<Object>> produceMessage() {
return (input) -> {
int messageId = input.getHeaders().get(PARTITION_KEY, Integer.class);
Object message = input.getPayload();
return MessageBuilder
.withPayload(message)
.setHeader(MessageHeaders.CONTENT_TYPE, MimeTypeUtils.APPLICATION_JSON)
.setHeader(PARTITION_KEY, messageId)
.setHeader("type", "MyMessage")
.build();
};
}
}
Now, I would like to test this implmentation, so I wrote this test class
#SpringBootTest
class ProducerTest {
#Autowired
private Producer producer;
#Autowired
private ObjectMapper objectMapper;
#Test
void produceOk() {
try (ConfigurableApplicationContext context = new SpringApplicationBuilder(TestChannelBinderConfiguration.getCompleteConfiguration(Producer.class)).run()) {
producer.produce(1, new MyMessage(1, "Hello"));
OutputDestination output = context.getBean(OutputDestination.class);
Message<byte[]> received = output.receive();
Assertions.assertNotNull(received);
}
}
}
Test fails because output.receive() returns null.
Is this the right way to test my code?
Thanks
It is difficult to see what your issue may be since we don't see the entire setup of your project, but here are few pointers that may help. . .
Please look at any of the tests we use in the framework as well as checkout the Testing section of the reference manual.
There is also s dedicated StreamBridgeTests.java which is what I believe you are looking for.
We have a spring boot rest api (spring boot 2.3.0.RELEASE) that uses spring cloud sleuth (version 2.2.3.RELEASE).
At some point, we use the trace id from spring sleuth as data. The trace id is fetched by autowiring the Tracing bean and then accessing the current span. Lets say we defined a bean SimpleCorrelationBean with:
#Autowired
private Tracer tracer;
public String getCorrelationId() {
return tracer.currentSpan().context().traceIdString();
}
This seem to work perfectly when running the spring boot application, but when we try to access the tracer.currentSpan() in the unit tests, this is null. It looks like spring cloud sleuth is not creating any span while running tests..
I think it has something to do with the application context that is set up during the unit test, but I don't know how to enable spring cloud sleuth for the test application context.
Below is a simple test class where the error occurs in simpleTest1. In simpleTest2, no error occurs.
simpleTest1 errors because tracer.currentSpan() is null
#ExtendWith({ RestDocumentationExtension.class, SpringExtension.class })
#SpringBootTest(classes = MusicService.class)
#WebAppConfiguration
#ActiveProfiles("unit-test")
#ComponentScan(basePackageClasses = datacast2.data.JpaConfig.class)
public class SimpleTest {
private static final Logger logger = LoggerFactory.getLogger(SimpleTest.class);
#Autowired
private WebApplicationContext context;
private MockMvc mockMvc;
#Autowired
private FilterChainProxy springSecurityFilterChain;
#Autowired
private SimpleCorrelationBean simpleCorrelationBean;
#Autowired
private Tracer tracer;
#BeforeEach
public void setup(RestDocumentationContextProvider restDocumentation) throws Exception {
this.mockMvc = MockMvcBuilders.webAppContextSetup(this.context)
.apply(documentationConfiguration(restDocumentation))
.addFilter(springSecurityFilterChain).build();
}
#Test
public void simpleTest1() throws Exception {
try {
String correlationId = simpleCorrelationBean.getCorrelationId();
}catch(Exception e) {
logger.error("This seem to fail.", e);
}
}
#Test
public void simpleTest2() throws Exception {
//It looks like spring cloud sleuth is not creating a span, so we create one ourselfs
Span newSpan = this.tracer.nextSpan().name("simpleTest2");
try (Tracer.SpanInScope ws = this.tracer.withSpanInScope(newSpan.start())) {
String correlationId = simpleCorrelationBean.getCorrelationId();
}
finally {
newSpan.finish();
}
}
}
The question is: how to enable spring cloud sleuth for a mockMvc during unit tests?
The issue here is that MockMvc is created manually instead of relying on autoconfiguration. In this particular case custom configuration of MockMvc could be necessary. However, at least for my version of Spring Boot (2.7.6), there is no need to manually configure MockMvc, even though I use Spring Security and Spring Security Test. I couldn't figure out how to enable tracing when manually configuring MockMvc though.
I have created Webservice using Spring boot and in this there is a rest controller which hits the database via vendor based JDBC driver and fetch the records. In this process number of records retrieved are more than 80K records. Due to this when ever we are hitting the the rest endpoint as client , we are getting time out errors.
I have tried setting up the asynchronous calls using the below tutorial.But unfortunately , rest calls are still timing out.
https://howtodoinjava.com/spring-boot2/enableasync-async-controller/
Controller
#RequestMapping(value = "/v1/lr/fullpositionasync", produces = {APPLICATION_JSON_UTF8_VALUE}, method = RequestMethod.GET)
#ResponseBody
public CompletableFuture<List<Position>> retrieveTradePositionsFullAsync(HttpServletRequest request, HttpServletResponse response) throws ExecutionException, InterruptedException {
CompletableFuture<List<Position>> positionList =null;
try {
positionList = positionService.getFullPosition();
}
catch(Exception e){
log.info("Error Occurred in Controller is:"+e.getMessage());
}
CompletableFuture.allOf(positionList).join();
log.info(String.valueOf(positionList.get()));
return positionList;
}
Service
#Service
#Slf4j
public class PositionServiceImpl implements PositionService {
#Autowired
private PositionDao positionDao;
#Async("asyncExecutor")
#Override
public CompletableFuture<List<Position>> getFullPosition() {
List<Position> fullpositionList = null;
log.info("Getting the full Position process started");
fullpositionList = positionDao.retrieveData();
log.info("Total Positions retrieved:"+fullpositionList.size());
try {
log.info("Thread is about to sleep 1000 milliseconds");
Thread.sleep(1000);
}catch(InterruptedException e){
log.info(e.getMessage());
}
log.info("Full Positions retrieval completed");
return CompletableFuture.completedFuture(fullpositionList);
}
}
Configuration
#Configuration
#EnableAsync
#Slf4j
public class AsyncConfiguration
{
#Bean(name = "asyncExecutor")
public Executor asyncExecutor()
{
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(20);
executor.setMaxPoolSize(1000);
executor.setWaitForTasksToCompleteOnShutdown(true);
executor.setThreadNamePrefix("AsynchThreadForEndPoint-");
executor.initialize();
log.info("Executor is :"+executor.toString());
return executor;
}
}
DAO
#Repository
public class PositionDaoImpl implements PositionDao {
#Autowired
private JdbcTemplate jdbcTemplate;
private static final String ALL_POSITION_QUERY = "call AllPositionProcedure()";
public List<Position> retrieveData() {
return jdbcTemplate.query(ALL_POSITION_QUERY, new BeanPropertyRowMapper(Position.class));
// List<Map<String, Object>> mapList = jdbcTemplate.queryForList(sql);
}
You can't perform async operations over the Database by using JDBC. JDBC is blocking, so it will block your thread until the operations will be executed. If you want to execute operations in an async manner, use R2DBC instead of JDBC.
For your use case, the best way to do is convert your application into Reactive Streams (Flux).
A Flux is a Reactive Streams Publisher. it is a fully non-blocking reactive programming foundation for the JVM, with efficient demand management (in the form of managing "backpressure"). It integrates directly with the Java 8 functional APIs, notably CompletableFuture, Stream, and Duration. It offers composable asynchronous sequence APIs Flux (for [N] elements) and Mono (for [0|1] elements), extensively implementing the Reactive Streams specification.
it is very simple to implement in existing app. just change your repository return type Flux instead of List or Future.
For more info, you can take reference Here
As of now, I'm able to connect to Cassandra via the following code:
import com.datastax.driver.core.Cluster;
import com.datastax.driver.core.Session;
public static Session connection() {
Cluster cluster = Cluster.builder()
.addContactPoints("IP1", "IP2")
.withCredentials("user", "password")
.withSSL()
.build();
Session session = null;
try {
session = cluster.connect("database_name");
session.execute("CQL Statement");
} finally {
IOUtils.closeQuietly(session);
IOUtils.closeQuietly(cluster);
}
return session;
}
The problem is that I need to write to Cassandra in a Spring Batch project. Most of the starter kits seem to use a JdbcBatchItemWriter to write to a mySQL database from a chunk. Is this possible? It seems that a JdbcBatchItemWriter cannot connect to a Cassandra database.
The current itemwriter code is below:
#Bean
public JdbcBatchItemWriter<Person> writer() {
JdbcBatchItemWriter<Person> writer = new JdbcBatchItemWriter<Person>();
writer.setItemSqlParameterSourceProvider(new
BeanPropertyItemSqlParameterSourceProvider<Person>());
writer.setSql("INSERT INTO people (first_name, last_name) VALUES
(:firstName, :lastName)");
writer.setDataSource(dataSource);
return writer;
}
Spring Data Cassandra provides repository abstractions for Cassandra that you should be able to use in conjunction with the RepositoryItemWriter to write to Cassandra from Spring Batch.
It is possible to extend Spring Batch to support Cassandra by customising ItemReader and ItemWriter.
ItemWriter example:
public class CassandraBatchItemWriter<Company> implements ItemWriter<Company>, InitializingBean {
protected static final Log logger = LogFactory.getLog(CassandraBatchItemWriter.class);
private final Class<Company> aClass;
#Autowired
private CassandraTemplate cassandraTemplate;
#Override
public void afterPropertiesSet() throws Exception { }
public CassandraBatchItemWriter(final Class<Company> aClass) {
this.aClass = aClass;
}
#Override
public void write(final List<? extends Company> items) throws Exception {
logger.debug("Write operations is performing, the size is {}" + items.size());
if (!items.isEmpty()) {
logger.info("Deleting in a batch performing...");
cassandraTemplate.deleteAll(aClass);
logger.info("Inserting in a batch performing...");
cassandraTemplate.insert(items);
}
logger.debug("Items is null...");
}
}
Then you can inject it as a #Bean through #Configuration
#Bean
public ItemWriter<Company> writer(final DataSource dataSource) {
final CassandraBatchItemWriter<Company> writer = new CassandraBatchItemWriter<Company>(Company.class);
return writer;
}
Full source code can be found in Github repo: Spring-Batch-with-Cassandra
I have a web service that via a GET Http method, the user requests for a person object. This person is sent to a JMS Queue and then with the help of Spring Integration, I send it to a fake email address (https://papercut.codeplex.com/). I have written the code with Spring Integration Java DSL. I would like to ask:
Is there a more flexible way to send the email message?
If an exception is thrown, how can the mail be redelivered with the help of Spring Integration? (e.g. for 5 times and if it is not sent then the exception gets handled and the program stops)
Here is my code:
Web Service
public Person findById(Integer id) {
Person person = jpaPersonRepository.findOne(id);
jmsTemplate.convertAndSend("testQueue", person);
return jpaPersonRepository.findOne(id);
}
Java Confiuration
#Configuration
#EnableIntegration
#ComponentScan
public class JavaConfig {
private static final String DEFAULT_BROKER_URL = "tcp://localhost:61616";
private static final String DEFAULT_QUEUE = "testQueue";
#Bean
public ActiveMQConnectionFactory connectionFactory() {
ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory();
connectionFactory.setBrokerURL(DEFAULT_BROKER_URL);
return connectionFactory;
}
#Bean
public JmsTemplate jmsTemplate() {
JmsTemplate template = new JmsTemplate();
template.setConnectionFactory(this.connectionFactory());
template.setDefaultDestinationName(DEFAULT_QUEUE);
return template;
}
#Bean
public DefaultMessageListenerContainer defaultMessageListenerContainer() {
DefaultMessageListenerContainer defaultMessageListenerContainer = new DefaultMessageListenerContainer();
defaultMessageListenerContainer.setDestinationName(DEFAULT_QUEUE);
defaultMessageListenerContainer.setConnectionFactory(this.connectionFactory());
return defaultMessageListenerContainer;
}
#Bean(name="inputChannel")
public DirectChannel directChannel() {
return new DirectChannel();
}
#Bean
public IntegrationFlow orders() {
return IntegrationFlows
.from(Jms.messageDrivenChannelAdapter(defaultMessageListenerContainer()))
.transform(new ObjectToStringTransformer())
.enrichHeaders(p -> p.header(MailHeaders.TO, "Papercut0#test.com"))
.handle(Mail.outboundAdapter("127.0.0.1")
.credentials("test","test").port(25)
.javaMailProperties(p -> p.put("mail.debug", "true")),
e -> e.id("sendMailEndpoint"))
.get();
}
}
Is there a more flexible way to send the email message?
Sorry, the question isn't clear. You have enough short code to do that. Mail.outboundAdapter() and all its fluent API. What should be more flexible?
If an exception is thrown, how can the mail be redelivered with the help of Spring Integration?
For this purpose Spring Integration suggests RequestHandlerRetryAdvice. And Mail.outboundAdapter() can be configured with that as:
#Bean
public Advice retryAdvice() {
RequestHandlerRetryAdvice advice = new RequestHandlerRetryAdvice();
RetryTemplate retryTemplate = new RetryTemplate();
SimpleRetryPolicy retryPolicy = new SimpleRetryPolicy();
retryPolicy.setMaxAttempts(5);
retryTemplate.setRetryPolicy(retryPolicy);
advice.setRetryTemplate(retryTemplate);
advice.setRecoveryCallback(new ErrorMessageSendingRecoverer(emailErrorChannel()));
return advice;
}
...
.handle(Mail.outboundAdapter("127.0.0.1")
.credentials("test","test").port(25)
.javaMailProperties(p -> p.put("mail.debug", "true")),
e -> e.id("sendMailEndpoint")
.advice(retryAdvice())) // HERE IS THE TRICK!
See its JavaDocs and Reference Manual on the matter.