Kafka configuration properties:
Can i have the same property "key"(and maybe different "value") in
(1) application.properties,
(2) bean(ProducerFactory/ProducerConfig) and
If yes, who is the "last-win"?
P.S Yes, i know, test it! But it will also be handy to have this question/answer on SO.
EDIT:
Example:
(1) spring.kafka.producer.properties.enable.idempotence=true
(2) props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "false");
With props defined as:
#Configuration
public class KafkaProducerConfiguration {
#Bean
public ProducerFactory<Object, Object> producerFactory() {
Map<String, Object> props = new HashMap<>();
The boot property (1) is only used when Boot auto-configures the producer factory for you. Since you are defining your own producer factory #Bean (2), Boot's is disabled and the properties are ignored.
If you want to use the Boot application.properties, simply remove your producerFactory #Bean and let Boot configure the producer factory for you.
I have no idea what config/producer.properties (3) is.
Related
Tracing information do not propagate over kafka messages due to the method SleuthKafkaAspect.wrapProducerFactory() is not triggered.
On the producer side, the message is correctly sent and the tracing information is correctly logged. On consumer side, instead a new traceId and spanId is created.
The following two logging lines show different values for traceId,spanId (and parentId):
2021-03-23 11:42:30.158 [http-nio-9185-exec-2] INFO my.company.Producer - /4afe07273872918b/4afe07273872918b// - Sending event='MyEvent'
2021-03-23 11:42:54.374 [org.springframework.kafka.KafkaListenerEndpointContainer#1-0-C-1] INFO my.company.Consumer /1fec3bf6a3c91773/ff4bd26b2e509ed8/1fec3bf6a3c91773/ - Received new event='MyEvent'
In first instance, using Krafdrop and also debugging, I verified that the message header doesn't contains any tracing information.
After that, I figured out that the method SleuthKafkaAspect.wrapProducerFactory() is never triggered, instead on consumer side the method SleuthKafkaAspect.anyConsumerFactory() is.
The libraries versions used are the following:
spring boot: 2.3.7.RELEASE
spring cloud bom: Hoxton.SR10
spring cloud: 2.2.7.RELEASE (and 2.2.5.RELEASE)
spring kafka: 2.5.10.RELEASE
kakfa client: 2.4.1
spring-cloud-starter-sleuth: 2.2.7.RELEASE
spring-cloud-sleuth-zipkin:2.2.7.RELEASE
The kakfa client library version is 2.4.1 is due to a version downgrade related to production bug on 2.5.1 version of kafka client that increase the cpu usage.
I also tried to use the following libraries versions combination with no success:
spring boot: 2.3.7.RELEASE
spring cloud bom: Hoxton.SR10 (and Hoxton.SR8)
spring cloud: 2.2.7.RELEASE (and 2.2.5.RELEASE)
spring kafka: 2.5.10.RELEASE
kakfa client: 2.5.1
spring-cloud-starter-sleuth: 2.2.7.RELEASE (and 2.2.5.RELEASE)
spring-cloud-sleuth-zipkin:2.2.7.RELEASE (and 2.2.5.RELEASE)
spring boot: 2.3.7.RELEASE
spring cloud bom: Hoxton.SR10 (and Hoxton.SR8)
spring cloud: 2.2.7.RELEASE (and 2.2.5.RELEASE)
spring kafka: 2.5.10.RELEASE
kakfa client: 2.6.0
spring-cloud-starter-sleuth: 2.2.7.RELEASE (and 2.2.5.RELEASE)
spring-cloud-sleuth-zipkin:2.2.7.RELEASE (and 2.2.5.RELEASE)
spring boot: 2.3.7.RELEASE
spring cloud bom: Hoxton.SR10 (and Hoxton.SR8)
spring cloud: 2.2.7.RELEASE (and 2.2.5.RELEASE)
spring kafka: 2.6.x
kakfa client: 2.6.0
spring-cloud-starter-sleuth: 2.2.7.RELEASE (and 2.2.5.RELEASE)
spring-cloud-sleuth-zipkin:2.2.7.RELEASE (and 2.2.5.RELEASE)
We migrated our project to a different spring boot version, from 2.3.0.RELEASE to 2.3.7.RELEASE. Before everthing was working correctly.
Below the old libraries versions:
spring-boot: 2.3.0.RELEASE
spring-kafka: 2.5.0.RELEASE
kafka-clients: 2.4.1
spring-cloud: 2.2.5.RELEASE
spring-cloud-starter-sleuth: 2.2.5.RELEASE
spring-cloud-sleuth-zipkin:2.2.5.RELEASE
We also introduced a log42/log4j (before it was slf4j with logback).
Below the related libraries:
- org.springframework.boot:spring-boot-starter-log4j2:jar:2.3.7.RELEASE:compile
- org.slf4j:jul-to-slf4j:jar:1.7.30:compile
- io.projectreactor:reactor-test:jar:3.3.12.RELEASE:test
- io.projectreactor:reactor-core:jar:3.3.12.RELEASE:test
- org.reactivestreams:reactive-streams:jar:1.0.3:test
The properties configured are the following:
spring.sleuth.messaging.enabled=true
spring.kafka.consumer.auto-offset-reset=latest
spring.kafka.consumer.enable-auto-commit=false
spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.client-id=myClientIdentifier
spring.kafka.consumer.group-id=MyConsumerGroup
spring.kafka.producer.value-serializer=org.springframework.kafka.support.serializer.JsonSerializer
The configuration class for the ProducerFactory creation is the following:
#Configuration
#EnableTransactionManagement
public class KafkaProducerConfig {
KafkaProperties kafkaProperties;
#Autowired
public KafkaProducerConfig(
KafkaProperties kafkaProperties) {
this.kafkaProperties = kafkaProperties;
}
#Bean
public KafkaTemplate<String, Object> kafkaTemplate() {
KafkaTemplate<String, Object> kafkaTemplate = new KafkaTemplate<>(producerFactory());
return kafkaTemplate;
}
private ProducerFactory<String, Object> producerFactory() {
DefaultKafkaProducerFactory<String, Object> defaultKafkaProducerFactory =
new DefaultKafkaProducerFactory<>(producerConfigs());
//defaultKafkaProducerFactory.transactionCapable();
//defaultKafkaProducerFactory.setTransactionIdPrefix("tx-");
return defaultKafkaProducerFactory;
}
private Map<String, Object> producerConfigs() {
Map<String, Object> configs = kafkaProperties.buildProducerProperties();
configs.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
configs.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, JsonSerializer.class);
return configs;
}
}
My spring boot application class:
#Profile("DEV")
#SpringBootApplication(
scanBasePackages = {"my.company"},
exclude = {
DataSourceAutoConfiguration.class,
DataSourceTransactionManagerAutoConfiguration.class,
HibernateJpaAutoConfiguration.class
}
)
#EnableSwagger2
#EnableFeignClients(basePackages = {"my.company.common", "my.company.integration"})
#EnableTransactionManagement
#EnableMongoRepositories(basePackages = {
"my.company.repository"})
#EnableMBeanExport(registration = RegistrationPolicy.IGNORE_EXISTING)
#ServletComponentScan
public class DevAppStartup extends SpringBootServletInitializer {
public static void main(String[] args) {
SpringApplication.run(DevAppStartup.class, args);
}
}
Here you can find the output of command "mvn dependency:tree"
mvn_dependency_tree.txt
As the documentation suggests, you need to create a ProducerFactory bean if you want to use your own KafkaTemplate:
#Configuration
public class KafkaProducerConfig {
#Bean
public ProducerFactory<String, Object>producerFactory(KafkaProperties kafkaProperties) {
return new DefaultKafkaProducerFactory<>(kafkaProperties.buildProducerProperties());
}
#Bean
public KafkaTemplate<String, Object> kafkaTemplate(ProducerFactory<String, Object>producerFactory) {
return new KafkaTemplate<>(producerFactory);
}
}
Based on the document from Spring Sleuth: https://docs.spring.io/spring-cloud-sleuth/docs/current-SNAPSHOT/reference/html/integrations.html#sleuth-kafka-integration
We decorate the Kafka clients (KafkaProducer and KafkaConsumer) to create a span for each event that is produced or consumed. You can disable this feature by setting the value of spring.sleuth.kafka.enabled to false.
You have to register the Producer or Consumer as beans in order for Sleuth’s auto-configuration to decorate them. When you then inject the beans, the expected type must be Producer or Consumer (and NOT e.g. KafkaProducer).
Problem Statement :
Need to handle exceptions occur while consuming messages in kafka
Commit failed offset
Seek to the next unprocessed offset, so that next polling start from this offset.
Seems all these are handled as part of SeekToCurrentErrorHandler.java in Spring-Kafka.
How to leverage this functionality in Spring-Integration-Kafka ?
Please help with this.
Versions used :
Spring-Integration-Kafka - 3.3.1
Spring for apache kafka - 2.5.x
#Bean(name ="kafkaConsumerFactory")
public ConsumerFactory consumerFactory0(
HashMap<String, String> properties = new HashMap<>();
properties.put("bootstrap.servers", "kafkaServerl");
properties.put("key.deserializer", StringDeserializer.class);
properties.put("value.deserializer", StringDeserializer.class);
properties.put("auto.offset.reset", "earliest");
} return new DefaultKafkaConsumerFactoryo(properties); I
#Bean("customKafkalistenerContainer")
public ConcurrentMessagelistenerContainerCtring, AddAccountReqRes> customKafkaListenerContainer() (
ContainerProperties containerProps = new ContainerProperties("Topici");
containerProps.setGroupld("Groupldl");
return (ConcurrentMessagelistenerContainerCtring, CustomReqRes>) new ConcurrentMessageListenerContainer<>(
} kafkaConsumerFactory, containerProps);
IntegrationFlows.from(Kafka.messageDrivenChannelAdapter(customKafkalistenerContainer, KafkaMessageDrivenChannelAdapter.ListenerMode.record)
.errorChannel(errorChannel()))
.handle(transformationProcessor, "process")
.channel("someChannel")
.get();
spring-integration-kafka uses spring-kafka underneath, so you just need to configure the adapter's container with the error handler.
spring-integration-kafka was moved to spring-integration starting with 5.4 (it was an extension previously). So, the current versions of both jars is 5.4.2.
I am developing a consumer application using spring-kafka. I am planning to keep it running by 24*7 using pod. But, recently I got to know there are some batch processes which would be running in between. And, when those batches are running, our processing shouldn't occur. So, probably, somehow I have to stop polling for records and when the batches are finished then I can resume my processing. But, I have no clue how to achieve this..
Whether the batches are running or not, I can query and get the details from table, by looking into some flag. But, how can I stop polling for records ? and will it not cause re balancing if I just keep consumer application running without processing anything ?
Config class :
#Bean
public ConsumerFactory<String, GenericRecord> consumerFactory(){
Map<String, Object> props = new HashMap<>();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,KAFKA_BROKERS);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, OFFSET_RESET);
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, KafkaAvroDeserializer.class.getName());
props.put(ConsumerConfig.GROUP_ID_CONFIG, GROUP_ID_CONFIG);
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, MAX_POLL_RECORDS);
props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, MAX_POLL_INTERVAL);
props.put(KafkaAvroDeserializerConfig.SCHEMA_REGISTRY_URL_CONFIG, SCHEMA_REGISTRY_URL);
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, SSL_PROTOCOL);
props.put(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG,SSL_TRUSTSTORE_LOCATION_FILE_NAME);
props.put(SslConfigs.SSL_TRUSTSTORE_PASSWORD_CONFIG, SSL_TRUSTSTORE_SECURE);
props.put(SslConfigs.SSL_KEYSTORE_LOCATION_CONFIG,SSL_KEYSTORE_LOCATION_FILE_NAME);
props.put(SslConfigs.SSL_KEYSTORE_PASSWORD_CONFIG, SSL_KEYSTORE_SECURE);
props.put(SslConfigs.SSL_KEY_PASSWORD_CONFIG, SSL_KEY_SECURE);
return new DefaultKafkaConsumerFactory<>(props);
}
#Bean
ConcurrentKafkaListenerContainerFactory<String, GenericRecord>
kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, GenericRecord> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setConcurrency(KAFKA_CONCURRENCY);
factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE); // manual async comm
return factory;
}
Code :
#KafkaListener(topics = "${app.topic}", groupId = "${app.group_id_config}")
public void run(ConsumerRecord<String, GenericRecord> record, Acknowledgment acknowledgement) throws Exception {
try {
int flag = getBatchIndquery ();
// How to stop and resume based on the flag value---?
// business logic process once the consumer resumes
processRecords();
InsertDb();
acknowledgement.acknowledge();
}catch (Exception ex) {
System.out.println(record);
System.out.println(ex.getMessage());
}
}
Use the endpoint registry to stop and start the container...
#KafkaListener(id = "myListener" ...)
#Autowired
KafkaListenerEndpointRegistry registry;
...
registry.getListenerContainer("myListener").stop();
See #KafkaListener Lifecycle Management.
The listener containers created for #KafkaListener annotations are not beans in the application context. Instead, they are registered with an infrastructure bean of type KafkaListenerEndpointRegistry. This bean is automatically declared by the framework and manages the containers' lifecycles; it will auto-start any containers that have autoStartup set to true. All containers created by all container factories must be in the same phase. See Listener Container Auto Startup for more information. You can manage the lifecycle programmatically by using the registry. Starting or stopping the registry will start or stop all the registered containers. Alternatively, you can get a reference to an individual container by using its id attribute. You can set autoStartup on the annotation, which overrides the default setting configured into the container factory. You can get a reference to the bean from the application context, such as auto-wiring, to manage its registered containers.
For the native Java Kafka client, there is a Kafka configuration called, enable.idempotence and we can set it to be true to enable idempotence producer.
However, for Spring Kafka, I can't find similar idempotence property in KafkaProperties class.
So I am wondering, if I manually set in my Spring Kafka configuration file, whether this property will take effect or Spring will totally ignore this config for Spring Kafka?
There are two ways to specify this property
application.properties You can use this property to specify any additional properties on producer
spring.kafka.producer.properties.*= # Additional producer-specific properties used to configure the client.
If you have any additional common config between Producer and Consumer
spring.kafka.properties.*= # Additional properties, common to producers and consumers, used to configure the client.
Through Code You can also override and customize the configs
#Bean
public ProducerFactory<String, String> producerFactory() {
Map<String, Object> configProps = new HashMap<>();
configProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,bootstrapAddress);
configProps.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
configProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
StringSerializer.class);
configProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
StringSerializer.class);
return new DefaultKafkaProducerFactory<>(configProps);
}
#Bean
public KafkaTemplate<String, String> kafkaTemplate() {
return new KafkaTemplate<>(producerFactory());
}
}
You're trying to add features that are not handled by Spring KafkaProperties, if you look at the documentation, you can do as the following:
Only a subset of the properties supported by Kafka are available directly through the KafkaProperties class.
If you wish to configure the producer or consumer with additional properties that are not directly supported, use the following properties:
spring.kafka.properties.prop.one=first
spring.kafka.admin.properties.prop.two=second
spring.kafka.consumer.properties.prop.three=third
spring.kafka.producer.properties.prop.four=fourth
spring.kafka.streams.properties.prop.five=fifth
https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-messaging.html#boot-features-kafka-extra-props
Yannick
You can find it with ProducerConfig as it is producer configuration. In order to enable this, you need to add below line in producerConfigs:
Properties producerProperties = new Properties();
producerProperties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
producerProperties.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
Using spring integration Kafka dsl, I wonder why listener not receive messages? But the same application If I replace spring integration DSL with a method annotated with KafkaListener is able to consume messages fine.
What am I missing with DSL?
DSL code that does not consume:
#Configuration
#EnableKafka
class KafkaConfig {
//consumer factory provided by Spring boot
#Bean
IntegrationFlow inboundKafkaEventFlow(ConsumerFactory consumerFactory) {
IntegrationFlows
.from(Kafka
.messageDrivenChannelAdapter(consumerFactory, "kafkaTopic")
.configureListenerContainer({ c -> c.groupId('kafka-consumer-staging') })
.id("kafkaTopicListener").autoStartup(true)
)
.channel("logChannel")
.get()
}
}
logChannel (or any other channel I use), does not reflect inbound messages.
Instead of the above code, If I use plain listener, it works fine to consume messages.
#Component
class KafkaConsumer {
#KafkaListener(topics = ['kafkaTopic'], groupId = 'kafka-consumer-staging')
void inboundKafkaEvent(String message) {
log.debug("message is {}", message)
}
}
Both approaches uses same application.properties for Kafka consumer.
You are missing the fact that you use Spring Integration, but you haven't enable it in your application. You don't need to do that for Kafka though, since you are not going to consume it with the #KafkaListener. So, to enable Spring Integration infrastructure, you need to add #EnableIntegration on your #Configuration class: https://docs.spring.io/spring-integration/docs/5.1.6.RELEASE/reference/html/#configuration-enable-integration