Spring Kafka cannot read environment variables like SPRING_KAFKA_CONSUME_PROPERTIES_SASL_JAAS_CONFIG, but using spring.kafka.consume.properties.sasl.jaas.config works, kafka factory config is
public class KafkaConfig {
#Autowired
private KafkaProperties kafkaProperties;
#Bean
public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, String>> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
return factory;
}
private ConsumerFactory<String, String> consumerFactory() {
kafkaProperties.getProperties().put(ConsumerConfig.ALLOW_AUTO_CREATE_TOPICS_CONFIG, "true");
kafkaProperties.getProperties().put(GROUP_ID_CONFIG, UUID.randomUUID().toString());
return new DefaultKafkaConsumerFactory<>(kafkaProperties.buildConsumerProperties());
}
Related
I'm building a simple Kafka application with a producer and a consumer. I'm sending a string through postman and pushing through the topic. The topic is receiving the message but the consumer isn't consuming it.
ConsumerConfig.Java
#EnableKafka
#Configuration
#ConditionalOnProperty(name = "kafka.enabled", havingValue = "true")
public class KafkaConsumerConfig {
#Bean
public KafkaListenerContainerFactory<ConcurrentMessageListenerContainer<String, String>> kafkaListenerContainerFactory(){
ConcurrentKafkaListenerContainerFactory<String,String> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
return factory;
}
#Bean
public Map<String,Object> config(){
Map<String,Object> config = new HashMap<>();
config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "127.0.0.1:9092");
config.put(ConsumerConfig.GROUP_ID_CONFIG, "group_Id");
config.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
config.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
config.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
config.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
config.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "100");
config.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "30000");
return config;
}
#Bean
public ConsumerFactory<String,String> consumerFactory(){
return new DefaultKafkaConsumerFactory<>(config());
}
}
CosumerService.Java
#Service
#ConditionalOnProperty(name = "kafka.enabled", havingValue = "true")
#Component
public class KafkaConsumerService {
private static final Logger log = LoggerFactory.getLogger(KafkaConsumerService.class);
private static final String TOPIC = "Kafka_Test";
#KafkaListener(topics = TOPIC, groupId= "group_Id")
public void consumeOTP(String otp) {
log.debug("The OTP Sent to Kafka is:" + otp);
}
}
Based on your question I am assuming you're using spring-kafka with Spring Boot. For a simple example, with this setup you can avoid all the Bean configuration and use the DefaultBean from Spring Kafka so you can basically do the setup using the application.yml file, there's better explanation in this post but basically:
Producer:
#Service
public class SimpleProducer {
private KafkaTemplate<String, String> simpleProducer;
public SimpleProducer(KafkaTemplate<String, String> simpleProducer) {
this.simpleProducer = simpleProducer;
}
public void send(String message) {
simpleProducer.send("simple-message", message);
}
}
Consumer:
#Slf4j
#Service
public class SimpleConsumer {
#KafkaListener(id = "simple-consumer", topics = "simple-message")
public void consumeMessage(String message) {
log.info("Consumer got message: {}", message);
}
}
Api so you can produce sending a message:
#RestController
#RequestMapping("/api")
public class MessageApi {
private final SimpleProducer simpleProducer;
public MessageApi(SimpleProducer simpleProducer) {
this.simpleProducer = simpleProducer;
}
#PostMapping("/message")
public ResponseEntity<String> message(#RequestBody String message) {
simpleProducer.send(message);
return ResponseEntity.ok("Message received: " + message);
}
}
Because you're using the defaults with String as key and String as value you don't even have to add any specific configuration to the spring-boot props or yaml files.
I am writing a Kafka Streams application, and I would like to include two application id in this application, but I keep getting error saying that "Topology with no input topics will create no stream threads and no global thread, must subscribe to at least one source topic or global table." Could you please let me know where I made a mistake? Thank you so much!
public class KafkaStreamsConfigurations {
...
#Bean(name = KafkaStreamsDefaultConfiguration.DEFAULT_STREAMS_CONFIG_BEAN_NAME)
#Primary
public KafkaStreamsConfiguration kStreamsConfigs() {
Map<String, Object> props = new HashMap<>();
setDefaults(props);
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "default");
return new KafkaStreamsConfiguration(props);
}
public void setDefaults(Map<String, Object> props) {...}
#Bean("snowplowStreamBuilder")
public StreamsBuilderFactoryBean streamsBuilderFactoryBean() {
Map<String, Object> props = new HashMap<>();
setDefaults(props);
...
props.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 0);
props.put(StreamsConfig.REPLICATION_FACTOR_CONFIG, 1);
Properties properties = new Properties();
props.forEach(properties::put);
StreamsBuilderFactoryBean streamsBuilderFactoryBean = new StreamsBuilderFactoryBean();
streamsBuilderFactoryBean.setStreamsConfiguration(properties);
return streamsBuilderFactoryBean;
}
}
Here is my application class.
public class SnowplowStreamsApp {
#Bean("snowplowStreamsApp")
public KStream<String, String> [] startProcessing(
#Qualifier("snowplowStreamBuilder") StreamsBuilder builder) {
KStream<String, String>[] branches = builder.stream(inputTopicPubsubSnowplow, Consumed
.with(Serdes.String(), Serdes.String()))
.mapValues(snowplowEnrichedGoodDataFormatter::formatEnrichedData)
.branch(...);
return branches;
}
}
Name your factory bean DEFAULT_STREAMS_BUILDER_BEAN_NAME instead of snowplowStreamBuilder - otherwise, the default factory bean will be started with no defined streams.
I'm reading a ton of questions and answers about this topic, but I can't solve my problem.
I initialized a Springboot project with Kafka and spring-data-jdbc.
What I'm trying to do is
Configure a Kafka JDBC Connector in order to push record changes from a PostgreSQL DB into a Kafka topic
Setup a Kafka Consumer in order to consume records pushed into the topic by inserting them into another PostgresSQL DB.
For point 1 is everything ok.
For point 2 I'm having some problem.
This is how is organized the project
com.migration
- MigrationApplication.java
com.migration.config
- KafkaConsumerConfig.java
com.migration.db
- JDBCConfig.java
- RecordRepository.java
com.migration.listener
- MessageListener.java
com.migration.model
- Record.java
- AbstractRecord.java
- PostgresRecord.java
This is the MessageListener class
#EnableJdbcRepositories("com.migration.db")
#Transactional
#Configuration
public class MessageListener {
#Autowired
private RecordRepository repository;
#KafkaListener(topics={"author"}, groupId = "migrator", containerFactory = "migratorKafkaListenerContainerFactory")
public void listenGroupMigrator(Record record) {
repository.insert(message);
throw new RuntimeException();
}
I think is pretty clear, it setup a Kafka Consumer in order to listen on "author" topic and consume the record by inserting it into DB.
As you can see, inside listenGroupMigrator() method is performed the insert into DB of the record and then is thrown RuntimeException because I'm checking if #Transactional works and if rollback is performed.
But not, rollback is not performed, even if the class is annotated with #Transactional.
For completeness these are other classes
RecordRepository class
#Repository
public class RecordRepository {
public RecordRepository() {}
public void insert(Record record) {
JDBCConfig jdbcConfig = new JDBCConfig();
SimpleJdbcInsert messageInsert = new SimpleJdbcInsert(jdbcConfig.postgresDataSource());
messageInsert.withTableName(record.tableName()).execute(record.content());
}
}
JDBCConfig class
#Configuration
public class JDBCConfig {
#Bean
public DataSource postgresDataSource() {
DriverManagerDataSource dataSource = new DriverManagerDataSource();
dataSource.setDriverClassName("org.postgresql.Driver");
dataSource.setUrl("jdbc:postgresql://localhost:5432/db");
dataSource.setUsername("postgres");
dataSource.setPassword("root");
return dataSource;
}
}
KafkaConsumerConfig class:
#EnableKafka
#Configuration
public class KafkaConsumerConfig {
#Value(value = "${kafka.bootstrap-server}")
private String bootstrapServer;
private <T extends Record> ConsumerFactory<String, T> consumerFactory(String groupId, Class<T> clazz) {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServer);
props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
props.put(JsonSerializer.ADD_TYPE_INFO_HEADERS, false);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonDeserializer.class);
return new DefaultKafkaConsumerFactory<>(props, new StringDeserializer(), new JsonDeserializer<>(clazz));
}
private <T extends Record> ConcurrentKafkaListenerContainerFactory<String, T> kafkaListenerContainerFactory(String groupId, Class<T> clazz) {
ConcurrentKafkaListenerContainerFactory<String, T> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory(groupId, clazz));
return factory;
}
#Bean
public ConcurrentKafkaListenerContainerFactory<String, PostgresRecord> migratorKafkaListenerContainerFactory() {
return kafkaListenerContainerFactory("migrator", PostgresRecord.class);
}
}
MigrationApplication class
#SpringBootApplication
public class MigrationApplication {
public static void main(String[] args) {
ConfigurableApplicationContext context = SpringApplication.run(MigrationApplication.class, args);
MessageListener listener = context.getBean(MessageListener.class);
}
}
How can I make the listenGroupMigrator method transactional?
I am using Spring kafka transaction for my producer and consumer applications.
The requirement is on producer side there are multiple steps: send message to kafka and then save to db. If save to db failed want to rollback the message send to kafka as well.
So on the consumer side, i set the isolation.leve to read_committed, then if the message is rollback from kafka, the consumer shouldn't read it.
Code for Producer application is:
#Configuration
#EnableKafka
public class KafkaConfiguration {
#Bean
public ProducerFactory<String, Customer> producerFactory() {
DefaultKafkaProducerFactory<String, Customer> pf = new DefaultKafkaProducerFactory<>(producerConfigs());
pf.setTransactionIdPrefix("customer.txn.tx-");
return pf;
}
#Bean
public Map<String, Object> producerConfigs() {
Map<String, Object> props = new HashMap<>();
// create a minimum Producer configs
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "http://127.0.0.1:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class);
props.put("schema.registry.url", "http://127.0.0.1:8081");
// create safe Producer
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.RETRIES_CONFIG, Integer.toString(Integer.MAX_VALUE));
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "5"); // kafka 2.0 >= 1.1 so we can keep this as 5. Use 1 otherwise.
// high throughput producer (at the expense of a bit of latency and CPU usage)
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "snappy");
props.put(ProducerConfig.LINGER_MS_CONFIG, "20");
props.put(ProducerConfig.BATCH_SIZE_CONFIG, Integer.toString(32 * 1024)); // 32 KB batch size
return props;
}
#Bean
public KafkaTemplate<String, Customer> kafkaTemplate() {
return new KafkaTemplate<>(producerFactory());
}
#Bean
public KafkaTransactionManager kafkaTransactionManager(ProducerFactory<String, Customer> producerFactory) {
KafkaTransactionManager<String, Customer> ktm = new KafkaTransactionManager<>(producerFactory);
ktm.setTransactionSynchronization(AbstractPlatformTransactionManager.SYNCHRONIZATION_ON_ACTUAL_TRANSACTION);
return ktm;
}
#Bean
#Primary
public JpaTransactionManager jpaTransactionManager(EntityManagerFactory entityManagerFactory) {
return new JpaTransactionManager(entityManagerFactory);
}
#Bean(name = "chainedTransactionManager")
public ChainedTransactionManager chainedTransactionManager(JpaTransactionManager jpaTransactionManager,
KafkaTransactionManager kafkaTransactionManager) {
return new ChainedTransactionManager(kafkaTransactionManager, jpaTransactionManager);
}
}
#Component
#Slf4j
public class KafkaProducerService {
private KafkaTemplate<String, Customer> kafkaTemplate;
private CustomerConverter customerConverter;
private CustomerRepository customerRepository;
public KafkaProducerService(KafkaTemplate<String, Customer> kafkaTemplate, CustomerConverter customerConverter, CustomerRepository customerRepository) {
this.kafkaTemplate = kafkaTemplate;
this.customerConverter = customerConverter;
this.customerRepository = customerRepository;
}
#Transactional(transactionManager = "chainedTransactionManager", rollbackFor = Exception.class)
public void sendEvents(String topic, CustomerModel customer) {
LOGGER.info("Sending to Kafka: topic: {}, key: {}, customer: {}", topic, customer.getKey(), customer);
// kafkaTemplate.send(topic, customer.getKey(), customerConverter.convertToAvro(customer));
kafkaTemplate.executeInTransaction(kt -> kt.send(topic, customer.getKey(), customerConverter.convertToAvro(customer)));
customerRepository.saveToDb();
}
}
So i explicitly throw an exception in the saveToDb method and I can see exception throw out. But the consumer application can still see the message.
Code for consumer:
#Slf4j
#Configuration
#EnableKafka
public class KafkaConfiguration {
#Bean
ConcurrentKafkaListenerContainerFactory<String, Customer> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, Customer> factory = new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setAfterRollbackProcessor(new DefaultAfterRollbackProcessor<String, Customer>(-1));
// SeekToCurrentErrorHandler errorHandler =
// new SeekToCurrentErrorHandler((record, exception) -> {
// // recover after 3 failures - e.g. send to a dead-letter topic
//// LOGGER.info("***in error handler data, {}", record);
//// LOGGER.info("***in error handler headers, {}", record.headers());
//// LOGGER.info("value: {}", new String(record.headers().headers("springDeserializerExceptionValue").iterator().next().value()));
// }, 3);
//
// factory.setErrorHandler(errorHandler);
return factory;
}
#Bean
public ConsumerFactory<String, Customer> consumerFactory() {
return new DefaultKafkaConsumerFactory<>(consumerConfigs());
}
#Bean
public Map<String, Object> consumerConfigs() {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
// props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
// props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, KafkaAvroDeserializer.class);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer2.class);
props.put(ErrorHandlingDeserializer2.VALUE_DESERIALIZER_CLASS, KafkaAvroDeserializer.class);
props.put("schema.registry.url", "http://127.0.0.1:8081");
props.put("specific.avro.reader", "true");
props.put("isolation.level", "read_committed");
// props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false"); // disable auto commit of offsets
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "100"); // disable auto commit of offsets
return props;
}
}
#Component
#Slf4j
public class KafkaConsumerService {
#KafkaListener(id = "demo-consumer-stream-group", topics = "customer.txn")
#Transactional
public void process(ConsumerRecord<String, Customer> record) {
LOGGER.info("Customer key: {} and value: {}", record.key(), record.value());
LOGGER.info("topic: {}, partition: {}, offset: {}", record.topic(), record.partition(), record.offset());
}
}
Did I miss something here?
executeInTransaction will run in a separate transaction. See the javadocs:
/**
* Execute some arbitrary operation(s) on the operations and return the result.
* The operations are invoked within a local transaction and do not participate
* in a global transaction (if present).
* #param callback the callback.
* #param <T> the result type.
* #return the result.
* #since 1.1
*/
<T> T executeInTransaction(OperationsCallback<K, V, T> callback);
Just use send() to participate in the existing transaction.
I am new to Kakfa and learning on to produce and consume messages to and from a Kafka Topic.
I am using the Kafka configuration using #EnableKafka
#EnableKafka
#Configuration
public class ConsumerConfig implements ApplicationContextAware {
#Value("${kafka.servers}")
private String kafkaServerAddress;
#Value("${kafka.ca.groupid}")
private String groupId;
private ApplicationContext context;
public DefaultKafkaConsumerFactory<String, Object> consumerFactory() {
Map<String, Object> props = new HashMap<>();
return new DefaultKafkaConsumerFactory<>(props);
}
#Bean
public ConcurrentKafkaListenerContainerFactory<String, Object> binlogListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
DefaultKafkaConsumerFactory<String, Object> defaultFactory = consumerFactory();
defaultFactory.setKeyDeserializer(new StringDeserializer());
defaultFactory.setValueDeserializer(new JsonDeserializer(BinlogMessage.class));
factory.setConsumerFactory(defaultFactory);
return factory;
}
#Override
public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
context = applicationContext;
}
}
Got the answer, it can be done by setting the property AUTO_OFFSET_RESET_CONFIG to latest as follows:
public DefaultKafkaConsumerFactory<String, Object> consumerFactory() {
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");
return new DefaultKafkaConsumerFactory<>(props);
}