I am using Spring kafka transaction for my producer and consumer applications.
The requirement is on producer side there are multiple steps: send message to kafka and then save to db. If save to db failed want to rollback the message send to kafka as well.
So on the consumer side, i set the isolation.leve to read_committed, then if the message is rollback from kafka, the consumer shouldn't read it.
Code for Producer application is:
#Configuration
#EnableKafka
public class KafkaConfiguration {
#Bean
public ProducerFactory<String, Customer> producerFactory() {
DefaultKafkaProducerFactory<String, Customer> pf = new DefaultKafkaProducerFactory<>(producerConfigs());
pf.setTransactionIdPrefix("customer.txn.tx-");
return pf;
}
#Bean
public Map<String, Object> producerConfigs() {
Map<String, Object> props = new HashMap<>();
// create a minimum Producer configs
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "http://127.0.0.1:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class);
props.put("schema.registry.url", "http://127.0.0.1:8081");
// create safe Producer
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.RETRIES_CONFIG, Integer.toString(Integer.MAX_VALUE));
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "5"); // kafka 2.0 >= 1.1 so we can keep this as 5. Use 1 otherwise.
// high throughput producer (at the expense of a bit of latency and CPU usage)
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "snappy");
props.put(ProducerConfig.LINGER_MS_CONFIG, "20");
props.put(ProducerConfig.BATCH_SIZE_CONFIG, Integer.toString(32 * 1024)); // 32 KB batch size
return props;
}
#Bean
public KafkaTemplate<String, Customer> kafkaTemplate() {
return new KafkaTemplate<>(producerFactory());
}
#Bean
public KafkaTransactionManager kafkaTransactionManager(ProducerFactory<String, Customer> producerFactory) {
KafkaTransactionManager<String, Customer> ktm = new KafkaTransactionManager<>(producerFactory);
ktm.setTransactionSynchronization(AbstractPlatformTransactionManager.SYNCHRONIZATION_ON_ACTUAL_TRANSACTION);
return ktm;
}
#Bean
#Primary
public JpaTransactionManager jpaTransactionManager(EntityManagerFactory entityManagerFactory) {
return new JpaTransactionManager(entityManagerFactory);
}
#Bean(name = "chainedTransactionManager")
public ChainedTransactionManager chainedTransactionManager(JpaTransactionManager jpaTransactionManager,
KafkaTransactionManager kafkaTransactionManager) {
return new ChainedTransactionManager(kafkaTransactionManager, jpaTransactionManager);
}
}
#Component
#Slf4j
public class KafkaProducerService {
private KafkaTemplate<String, Customer> kafkaTemplate;
private CustomerConverter customerConverter;
private CustomerRepository customerRepository;
public KafkaProducerService(KafkaTemplate<String, Customer> kafkaTemplate, CustomerConverter customerConverter, CustomerRepository customerRepository) {
this.kafkaTemplate = kafkaTemplate;
this.customerConverter = customerConverter;
this.customerRepository = customerRepository;
}
#Transactional(transactionManager = "chainedTransactionManager", rollbackFor = Exception.class)
public void sendEvents(String topic, CustomerModel customer) {
LOGGER.info("Sending to Kafka: topic: {}, key: {}, customer: {}", topic, customer.getKey(), customer);
// kafkaTemplate.send(topic, customer.getKey(), customerConverter.convertToAvro(customer));
kafkaTemplate.executeInTransaction(kt -> kt.send(topic, customer.getKey(), customerConverter.convertToAvro(customer)));
customerRepository.saveToDb();
}
}
So i explicitly throw an exception in the saveToDb method and I can see exception throw out. But the consumer application can still see the message.
Code for consumer:
#Slf4j
#Configuration
#EnableKafka
public class KafkaConfiguration {
#Bean
ConcurrentKafkaListenerContainerFactory<String, Customer> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, Customer> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setAfterRollbackProcessor(new DefaultAfterRollbackProcessor<String, Customer>(-1));
// SeekToCurrentErrorHandler errorHandler =
// new SeekToCurrentErrorHandler((record, exception) -> {
// // recover after 3 failures - e.g. send to a dead-letter topic
//// LOGGER.info("***in error handler data, {}", record);
//// LOGGER.info("***in error handler headers, {}", record.headers());
//// LOGGER.info("value: {}", new String(record.headers().headers("springDeserializerExceptionValue").iterator().next().value()));
// }, 3);
//
// factory.setErrorHandler(errorHandler);
return factory;
}
#Bean
public ConsumerFactory<String, Customer> consumerFactory() {
return new DefaultKafkaConsumerFactory<>(consumerConfigs());
}
#Bean
public Map<String, Object> consumerConfigs() {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
// props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
// props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, KafkaAvroDeserializer.class);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer2.class);
props.put(ErrorHandlingDeserializer2.VALUE_DESERIALIZER_CLASS, KafkaAvroDeserializer.class);
props.put("schema.registry.url", "http://127.0.0.1:8081");
props.put("specific.avro.reader", "true");
props.put("isolation.level", "read_committed");
// props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false"); // disable auto commit of offsets
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "100"); // disable auto commit of offsets
return props;
}
}
#Component
#Slf4j
public class KafkaConsumerService {
#KafkaListener(id = "demo-consumer-stream-group", topics = "customer.txn")
#Transactional
public void process(ConsumerRecord<String, Customer> record) {
LOGGER.info("Customer key: {} and value: {}", record.key(), record.value());
LOGGER.info("topic: {}, partition: {}, offset: {}", record.topic(), record.partition(), record.offset());
}
}
Did I miss something here?
executeInTransaction will run in a separate transaction. See the javadocs:
/**
* Execute some arbitrary operation(s) on the operations and return the result.
* The operations are invoked within a local transaction and do not participate
* in a global transaction (if present).
* #param callback the callback.
* #param <T> the result type.
* #return the result.
* #since 1.1
*/
<T> T executeInTransaction(OperationsCallback<K, V, T> callback);
Just use send() to participate in the existing transaction.
Related
I'm building a simple Kafka application with a producer and a consumer. I'm sending a string through postman and pushing through the topic. The topic is receiving the message but the consumer isn't consuming it.
ConsumerConfig.Java
#EnableKafka
#Configuration
#ConditionalOnProperty(name = "kafka.enabled", havingValue = "true")
public class KafkaConsumerConfig {
#Bean
public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, String>> kafkaListenerContainerFactory(){
ConcurrentKafkaListenerContainerFactory<String,String> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
return factory;
}
#Bean
public Map<String,Object> config(){
Map<String,Object> config = new HashMap<>();
config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "127.0.0.1:9092");
config.put(ConsumerConfig.GROUP_ID_CONFIG, "group_Id");
config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
config.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
config.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
config.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "100");
config.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "30000");
return config;
}
#Bean
public ConsumerFactory<String,String> consumerFactory(){
return new DefaultKafkaConsumerFactory<>(config());
}
}
CosumerService.Java
#Service
#ConditionalOnProperty(name = "kafka.enabled", havingValue = "true")
#Component
public class KafkaConsumerService {
private static final Logger log = LoggerFactory.getLogger(KafkaConsumerService.class);
private static final String TOPIC = "Kafka_Test";
#KafkaListener(topics = TOPIC, groupId= "group_Id")
public void consumeOTP(String otp) {
log.debug("The OTP Sent to Kafka is:" + otp);
}
}
Based on your question I am assuming you're using spring-kafka with Spring Boot. For a simple example, with this setup you can avoid all the Bean configuration and use the DefaultBean from Spring Kafka so you can basically do the setup using the application.yml file, there's better explanation in this post but basically:
Producer:
#Service
public class SimpleProducer {
private KafkaTemplate<String, String> simpleProducer;
public SimpleProducer(KafkaTemplate<String, String> simpleProducer) {
this.simpleProducer = simpleProducer;
}
public void send(String message) {
simpleProducer.send("simple-message", message);
}
}
Consumer:
#Slf4j
#Service
public class SimpleConsumer {
#KafkaListener(id = "simple-consumer", topics = "simple-message")
public void consumeMessage(String message) {
log.info("Consumer got message: {}", message);
}
}
Api so you can produce sending a message:
#RestController
#RequestMapping("/api")
public class MessageApi {
private final SimpleProducer simpleProducer;
public MessageApi(SimpleProducer simpleProducer) {
this.simpleProducer = simpleProducer;
}
#PostMapping("/message")
public ResponseEntity<String> message(#RequestBody String message) {
simpleProducer.send(message);
return ResponseEntity.ok("Message received: " + message);
}
}
Because you're using the defaults with String as key and String as value you don't even have to add any specific configuration to the spring-boot props or yaml files.
Spring Kafka cannot read environment variables like SPRING_KAFKA_CONSUME_PROPERTIES_SASL_JAAS_CONFIG, but using spring.kafka.consume.properties.sasl.jaas.config works, kafka factory config is
public class KafkaConfig {
#Autowired
private KafkaProperties kafkaProperties;
#Bean
public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, String>> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
return factory;
}
private ConsumerFactory<String, String> consumerFactory() {
kafkaProperties.getProperties().put(ConsumerConfig.ALLOW_AUTO_CREATE_TOPICS_CONFIG, "true");
kafkaProperties.getProperties().put(GROUP_ID_CONFIG, UUID.randomUUID().toString());
return new DefaultKafkaConsumerFactory<>(kafkaProperties.buildConsumerProperties());
}
I'm reading a ton of questions and answers about this topic, but I can't solve my problem.
I initialized a Springboot project with Kafka and spring-data-jdbc.
What I'm trying to do is
Configure a Kafka JDBC Connector in order to push record changes from a PostgreSQL DB into a Kafka topic
Setup a Kafka Consumer in order to consume records pushed into the topic by inserting them into another PostgresSQL DB.
For point 1 is everything ok.
For point 2 I'm having some problem.
This is how is organized the project
com.migration
- MigrationApplication.java
com.migration.config
- KafkaConsumerConfig.java
com.migration.db
- JDBCConfig.java
- RecordRepository.java
com.migration.listener
- MessageListener.java
com.migration.model
- Record.java
- AbstractRecord.java
- PostgresRecord.java
This is the MessageListener class
#EnableJdbcRepositories("com.migration.db")
#Transactional
#Configuration
public class MessageListener {
#Autowired
private RecordRepository repository;
#KafkaListener(topics={"author"}, groupId = "migrator", containerFactory = "migratorKafkaListenerContainerFactory")
public void listenGroupMigrator(Record record) {
repository.insert(message);
throw new RuntimeException();
}
I think is pretty clear, it setup a Kafka Consumer in order to listen on "author" topic and consume the record by inserting it into DB.
As you can see, inside listenGroupMigrator() method is performed the insert into DB of the record and then is thrown RuntimeException because I'm checking if #Transactional works and if rollback is performed.
But not, rollback is not performed, even if the class is annotated with #Transactional.
For completeness these are other classes
RecordRepository class
#Repository
public class RecordRepository {
public RecordRepository() {}
public void insert(Record record) {
JDBCConfig jdbcConfig = new JDBCConfig();
SimpleJdbcInsert messageInsert = new SimpleJdbcInsert(jdbcConfig.postgresDataSource());
messageInsert.withTableName(record.tableName()).execute(record.content());
}
}
JDBCConfig class
#Configuration
public class JDBCConfig {
#Bean
public DataSource postgresDataSource() {
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName("org.postgresql.Driver");
dataSource.setUrl("jdbc:postgresql://localhost:5432/db");
dataSource.setUsername("postgres");
dataSource.setPassword("root");
return dataSource;
}
}
KafkaConsumerConfig class:
#EnableKafka
#Configuration
public class KafkaConsumerConfig {
#Value(value = "${kafka.bootstrap-server}")
private String bootstrapServer;
private <T extends Record> ConsumerFactory<String, T> consumerFactory(String groupId, Class<T> clazz) {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServer);
props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
props.put(JsonSerializer.ADD_TYPE_INFO_HEADERS, false);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);
return new DefaultKafkaConsumerFactory<>(props, new StringDeserializer(), new JsonDeserializer<>(clazz));
}
private <T extends Record> ConcurrentKafkaListenerContainerFactory<String, T> kafkaListenerContainerFactory(String groupId, Class<T> clazz) {
ConcurrentKafkaListenerContainerFactory<String, T> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory(groupId, clazz));
return factory;
}
#Bean
public ConcurrentKafkaListenerContainerFactory<String, PostgresRecord> migratorKafkaListenerContainerFactory() {
return kafkaListenerContainerFactory("migrator", PostgresRecord.class);
}
}
MigrationApplication class
#SpringBootApplication
public class MigrationApplication {
public static void main(String[] args) {
ConfigurableApplicationContext context = SpringApplication.run(MigrationApplication.class, args);
MessageListener listener = context.getBean(MessageListener.class);
}
}
How can I make the listenGroupMigrator method transactional?
I need to read message from topic1 completely and then read message from topic2. I will be receiving messages in these topic everyday once. I managed to stop reading messages from topic2 before reading all the messages in topic1, but this is happening for me only once when the server is started. Can someone help me with this scenario.
ListenerConfig code
#EnableKafka
#Configuration
public class ListenerConfig {
#Value("${spring.kafka.bootstrap-servers}")
private String bootstrapServers;
#Bean
public Map<String, Object> consumerConfigs() {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.GROUP_ID_CONFIG, "batch");
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "5");
return props;
}
#Bean
public ConsumerFactory<String, String> consumerFactory() {
return new DefaultKafkaConsumerFactory<>(consumerConfigs());
}
#Bean
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setBatchListener(true);
return factory;
}
#Bean("kafkaListenerContainerTopic1Factory")
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerTopic1Factory() {
ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.getContainerProperties().setIdleEventInterval(60000L);
factory.setBatchListener(true);
return factory;
}
#Bean("kafkaListenerContainerTopic2Factory")
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerTopic2Factory() {
ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setBatchListener(true);
return factory;
}
}
Listner code
#Service
public class Listener {
private static final Logger LOG = LoggerFactory.getLogger(Listener.class);
#Autowired
private KafkaListenerEndpointRegistry registry;
#KafkaListener(id = "first-listener", topics = "topic1", containerFactory = "kafkaListenerContainerTopic1Factory")
public void receive(#Payload List<String> messages,
#Header(KafkaHeaders.RECEIVED_PARTITION_ID) List<Integer> partitions,
#Header(KafkaHeaders.OFFSET) List<Long> offsets) {
for (int i = 0; i < messages.size(); i++) {
LOG.info("received first='{}' with partition-offset='{}'",
messages.get(i), partitions.get(i) + "-" + offsets.get(i));
}
}
#KafkaListener(id = "second-listener", topics = "topic2", containerFactory = "kafkaListenerContaierTopic2Factory" , autoStartup="false" )
public void receiveRel(#Payload List<String> messages,
#Header(KafkaHeaders.RECEIVED_PARTITION_ID) List<Integer> partitions,
#Header(KafkaHeaders.OFFSET) List<Long> offsets) {
for (int i = 0; i < messages.size(); i++) {
LOG.info("received second='{}' with partition-offset='{}'",
messages.get(i), partitions.get(i) + "-" + offsets.get(i));
}
}
#EventListener()
public void eventHandler(ListenerContainerIdleEvent event) {
LOG.info("Inside event");
this.registry.getListenerContainer("second-listener").start();
}
Kindly help me in resolving , as this cycle should happen everyday. Reading topic1 message completely and then reading message from topic2.
You are already using an idle event listener to start the second listener - it should also stop the first listener.
When the second listener goes idle; stop it.
You should be checking which container the event is for to decide which container to stop and/or start.
Then, using a TaskScheduler, schedule a start() of the first listener at the next time you want it to start.
Topic in Kafka is an abstraction where stream of records are published. Streams are naturally unbounded, so they have a start but they do not have a defined end. For your case, first you need to clearly define what is the end of your topic1 and your topic2 so that you can stop/presume your consumers when needed. Maybe you know how many messages you will process for each topic, so you can use: position or commmited to stop one consumer and presume the other one in that moment. Or if you are using a streaming framework they usually have a session window where the framework detects a groups elements by sessions of activity. You can also prefer to put that logic into the application side so that you don't need to stop/start any consumer threads.
I am new to Kakfa and learning on to produce and consume messages to and from a Kafka Topic.
I am using the Kafka configuration using #EnableKafka
#EnableKafka
#Configuration
public class ConsumerConfig implements ApplicationContextAware {
#Value("${kafka.servers}")
private String kafkaServerAddress;
#Value("${kafka.ca.groupid}")
private String groupId;
private ApplicationContext context;
public DefaultKafkaConsumerFactory<String, Object> consumerFactory() {
Map<String, Object> props = new HashMap<>();
return new DefaultKafkaConsumerFactory<>(props);
}
#Bean
public ConcurrentKafkaListenerContainerFactory<String, Object> binlogListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
DefaultKafkaConsumerFactory<String, Object> defaultFactory = consumerFactory();
defaultFactory.setKeyDeserializer(new StringDeserializer());
defaultFactory.setValueDeserializer(new JsonDeserializer(BinlogMessage.class));
factory.setConsumerFactory(defaultFactory);
return factory;
}
#Override
public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
context = applicationContext;
}
}
Got the answer, it can be done by setting the property AUTO_OFFSET_RESET_CONFIG to latest as follows:
public DefaultKafkaConsumerFactory<String, Object> consumerFactory() {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");
return new DefaultKafkaConsumerFactory<>(props);
}