I m newbie on the apache kafka and I have problem about the connection to apache kafka.When I run configuration with java based apache kafka producer its working but when I added packaged jar file and run on apache metron kafka broker,error occure as seen below
2017-03-17 07:32:32 WARN NetworkClient:568 - Bootstrap broker node1:6667 disconnected
configuration
bootstrap.servers=node1:6667
acks=all
retries=0
batch.size=16384
linger.ms=1
buffer.memory=33554432
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
Code :
sender.send(new ProducerRecord<S, String>(getSenderTopic(), (S) getSenderKey(),
data), new Callback() {
public void onCompletion(RecordMetadata metadata, Exception e) {
if (e != null) {
LOGGER.error("exception ", e);
sender.close();
}
}
});
Apache kafka broker run on the apache metron and configuration host is setted node1:6667
I think its network side problem but what is your suggesition on it?.
Thanks
so after research , I have solve the problem , problem on the kafka versions apache metron use 10.0.1 and we use 10.1.0 , changed on the pom xml problem solved.
Related
I am using spring kafka with the embedded kafka for JUnit test, it gives an error for every test on windows:
Error deleting C:\Users:LXX691\AppData\Local\Temp\kafka-1103610162480947200/.lock: The process cannot access the file because it is being used by another process.
I just did the basic configuration like below
#SpringBootTest(webEnvironment = RANDOM_PORT)
#RunWith(SpringRunner.class)
public class KafkaTest {
#Autowired
EmbeddedKafkaBroker broker;
#Before
void setUp() throws Exception() {
// setup producer and consumers
}
#Test
void test() {
producer.send(new ProducerRecord<>("topic", "content"));
}
}
Any suggestion to resolve or any workaround is appreciated.
This is a known issue in Apache Kafka: https://issues.apache.org/jira/browse/KAFKA-8145.
Unfortunately there is nothing in Spring Kafka we can do on the matter.
See more info here: Kafka: unable to start Kafka - process can not access file 00000000000000000000.timeindex and here https://github.com/spring-projects/spring-kafka/issues/194
We are using Spring Kafka to consume records in batches. We are sometimes facing an issue where the application starts and it doesn't consume any records even though there are enough unread messages. Instead we continuously see info logs saying.
[INFO]-[FetchSessionHandler:handleError:440] - [Consumer clientId=consumer-2, groupId=groupId] Error sending fetch request (sessionId=INVALID, epoch=INITIAL) to node 1027: org.apache.kafka.common.errors.DisconnectException.
People are facing this issue and everyone says to ignore it, since it is just a info log. Even, we see after sometime the application starts picking up the records without doing anything. But, it is very unpredictable on how long it might take to start consuming records :(
We didn't see this error when we were using Spring cloud stream. Not sure if we have missed any configuration in spring-kafka.
Anyone faced this issue in past, please let us know if we are missing something. We have huge load in our topics and if there is a lot of lag, could this happen?
We are using Spring Kafka of 2.2.2.RELEASE
Spring boot 2.1.2.RELEASE
Kafka 0.10.0.1 (We understand it's very old, because of unavoidable reasons we are having to use this :()
Here is our code:
application.yml
li.topics: CUSTOM.TOPIC.JSON
spring:
application:
name: DataPublisher
kafka:
listener:
type: batch
ack-mode: manual_immediate
consumer:
enable-auto-commit: false
max-poll-records: 500
fetch-min-size: 1
fetch-max-wait: 1000
group-id: group-dev-02
key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
value-deserializer:CustomResourceDeserialiser
auto-offset-reset: earliest
Consumer:
public class CustomKafkaBatchConsumer {
#KafkaListener(topics = "#{'${li.topics}'.split(',')}", id = "${spring.kafka.consumer.group-id}")
public void receiveData(#Payload List<CustomResource> customResources,
Acknowledgment acknowledgment,
#Header(KafkaHeaders.RECEIVED_PARTITION_ID) List<Integer> partitions,
#Header(KafkaHeaders.OFFSET) List<Long> offsets) {
}
}
Deserialiser:
public class CustomResourceDeserialiser implements Deserializer<CustomResource> {
#Override
public void configure(Map<String, ?> configs, boolean isKey) {
}
#Override
public CustomResource deserialize(String topic, byte[] data) {
if (data != null) {
try {
ObjectMapper objectMapper = ObjectMapperFactory.getInstance();
return objectMapper.readValue(data, CustomResource.class);
} catch (IOException e) {
log.error("Failed to deserialise with {}",e.getMessage());
}
}
return null;
}
#Override
public void close() {
}
}
This could be because of this Kafka-8052 - Intermittent INVALID_FETCH_SESSION_EPOCH error on FETCH request issue. This is fixed in Kafka 2.3.0
Unfortunately, as of Aug 21, 2019 Spring cloud streams haven't upgraded it's dependencies yet with 2.3.0 release of kafka-clients yet.
You can try adding these as explicit dependencies in your gradle
compile ('org.apache.kafka:kafka-streams:2.3.0')
compile ('org.apache.kafka:kafka-clients:2.3.0')
compile ('org.apache.kafka:connect-json:2.3.0')
compile ('org.apache.kafka:connect-api:2.3.0')
Update
This could also be caused by kafka Broker - client incompatibility. If your cluster is behind the client version you might see all kinds of odd problems such as this. Example, let's say, your kafka broker is on 1.x.x and your kafka-consumer is on 2.x.x, this could happen
I have faced the same problem before, solution was either the decrease current partition count or increase the number of consumers. In my case, we have ~100M data on 60 partition and I came across the same error when single pod is running. I scaled 30 pods (30 consumers) and the problem was solved.
i was using kafka producer which version is 0.8.2.1 to write to kafka broker which version is 1.0.1 async.
my code is like bellow:
KafkaProducer producer = new KafkaProducer(configs);
ProducerRecord producerRecord = new ProducerRecord("topic", "key", "value");
producer.send(producerRecord, new CallBack(){
#override
public void onCompletion(RecordMetadata metadata,
java.lang.Exception exception){
if(metadata != null){
System.out.println(metadata.partition() + "|" + metadata.offset());
}
});
i found that partition offset printed in my producer app's log at "onCompletion" method was bigger than kafka broker's offset which was query by shell command "./kafka-run-class.sh kafka.tools.GetOffsetShell ".
my producer was set with the config "acks=all"
for example, partition 0's offset is 30000 in log, but is 10000 queryed by shell command.
is it caused by version compatible problem?
The producer API was rewriten around Kafka 0.9 such that offsets are stored in Kafka, not Zookeeper. It's not clear if you've used GetOffsetShell with Zookeeper option or not.
Newer brokers are mostly backwards compatible down to version 0.10.2, but you shouldn't expect older clients less than those versions to work correctly with newer broker versions
https://cwiki.apache.org/confluence/display/KAFKA/Compatibility+Matrix
I am using Apache Flink, and trying to connect to Azure eventhub by using Apache Kafka protocol to receive messages from it. I manage to connect to Azure eventhub and receive messages, but I can't use flink feature "setStartFromTimestamp(...)" as described here (https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#kafka-consumers-start-position-configuration).
When I am trying to get some messages from timestamp, Kafka said that the message format on the broker side is before 0.10.0.
Is anybody faced with this?
Apache Kafka client version is 2.0.1
Apache Flink version is 1.7.2
UPDATED: tried to use Azure-Event-Hub quickstart examples (https://github.com/Azure/azure-event-hubs-for-kafka/tree/master/quickstart/java) in consumer package added code to get offset with timestamp, it returns null as expected if message version under 0.10.0 kafka version.
List<PartitionInfo> partitionInfos = consumer.partitionsFor(TOPIC);
List<TopicPartition> topicPartitions = partitionInfos.stream().map(pi -> new TopicPartition(pi.topic(), pi.partition())).collect(Collectors.toList());
Map<TopicPartition, Long> topicPartitionToTimestampMap = topicPartitions.stream().collect(Collectors.toMap(tp -> tp, tp -> 0L));
Map<TopicPartition, OffsetAndTimestamp> offsetAndTimestamp = consumer.offsetsForTimes(topicPartitionToTimestampMap);
System.out.println(offsetAndTimestamp);
Sorry we missed this. Kafka offsetsForTimes() is now supported in EH (previously unsupported).
Feel free to open an issue against our Github in the future. https://github.com/Azure/azure-event-hubs-for-kafka
I try to use logstash with kafka broker. It can not integrated together.
Each version are,
logstash 2.4
kafka 0.8.2.2 with scala 2.10
The kafka input config file is:
input {
stdin{
}
}
output{
stdout {
codec => rubydebug
}
kafka {
bootstrap_servers => 10.120.16.202:6667,10.120.16.203:6667,10.120.16.204:6667'
topic_id => "cephosd1"
}
}
I can list topic cephosd1 from kafka.
The stdout could print out content, also.
But I can not read anything from kafka-console-consumer.sh .
I think you have a compatibility issue. If you check the version compatibility matrix between Logstash, Kafka and the kafka output plugin, you'll see that the kafka output plugin in Logstash 2.4 uses the Kafka 0.9 client version.
If you have a Kafka broker 0.8.2.2, it is not compatible with the client version 0.9 (the other way around would be ok). You can either downgrade to Logstash 2.0 or upgrade your Kafka broker to 0.9.