I am new to Apache Kafka and I'm trying to build a Python app which is able to handle Kafka messages. I've set Kafka up to produce and consume messages locally. Now I also want this to work non-locally, so that I can send messages from everywhere to my Python app.
My idea was to just expose the specific port that Kafka is using by using Localtunnel. I thought this would just mirror the local messages, so that I can consume them via the generated URL. But surprise, it doesn't work.
I just don't receive any messages at all. Do you have an idea why this is? Do I maybe have to configure the listeners in the Kafka server.properties first?
Thanks!
We are using a kafka broker setup with a kafka streams application that runs using Spring cloud stream kafka. Although it seems to run fine, we do get the following error statements in our log:
2019-02-21 22:37:20,253 INFO kafka-coordinator-heartbeat-thread | anomaly-timeline org.apache.kafka.clients.FetchSessionHandler [Consumer clientId=anomaly-timeline-56dc4481-3086-4359-a8e8-d2dae12272a2-StreamThread-1-consumer, groupId=anomaly-timeline] Node 2 was unable to process the fetch request with (sessionId=1290440723, epoch=2089): INVALID_FETCH_SESSION_EPOCH.
I searched the internet but there is not much information on this error. I guessed that it could have something to do with a difference in time settings between the broker and the consumer, but both machines have the same timeserver settings.
Any idea how this can be resolved?
There is a concept of fetch session, introduced within KIP-227 since 1.1.0 release: https://cwiki.apache.org/confluence/display/KAFKA/KIP-227%3A+Introduce+Incremental+FetchRequests+to+Increase+Partition+Scalability
Kafka brokers, which are replica followers, fetch messages from the leader. In order to avoid sending full metadata each time for all partitions, only those partitions which changed are sent within the same fetch session.
When we look into Kafka's code, we can see an example, when this is returned:
if (session.epoch != expectedEpoch) {
info(s"Incremental fetch session ${session.id} expected epoch $expectedEpoch, but " +
s"got ${session.epoch}. Possible duplicate request.")
new FetchResponse(Errors.INVALID_FETCH_SESSION_EPOCH, new FetchSession.RESP_MAP, 0, session.id)
} else {
src: https://github.com/axbaretto/kafka/blob/ab2212c45daa841c2f16e9b1697187eb0e3aec8c/core/src/main/scala/kafka/server/FetchSession.scala#L493
In general, if you don't have thousands of partitions and, at the same time, this doesn't happen very often, then it shouldn't worry you.
It seems as this might be caused by Kafka-8052 issue, which was fixed for Kafka 2.3.0
Indeed, you can have this message when rolling or retention-based deletion occurs, as zen pointed out in comments. It's not a problem if it doesn't happen all the time. If it does, check your log.roll and log.retention configurations.
Updating the client version to 2.3 (same version from broker) fix it for me.
In our case, The root cause was kafka Broker - client incompatibility. If your cluster is behind the client version you might see all kinds of odd problems such as this.
Our kafka broker is on 1.x.x and our kafka-consumer was on 2.x.x. As soon as we downgraded our spring-cloud-dependencies to Finchley.RELEASE our problem was solved.
dependencyManagement {
imports {
mavenBom "org.springframework.cloud:spring-cloud-dependencies:Finchley.RELEASE"
}
}
We noticed on our kafka broker logs that our stream applications are trying to describe topics that are not part of the topology. It looks like it is periodically happening every 10 minutes.
Why is this occurring? Why does streams need to know about topics that are not part of the topology? Anyway to stop this? Since the stream app does not have access to describe those topics we are getting a bunch of error logs.
We are using version 1.1.1
It's a know issue, that will get fixed in the 2.5 release: https://issues.apache.org/jira/browse/KAFKA-7317
I'm having some troubles with kafka topics.
We have on our development environment kafka version 2.0,
and I've checked for topic deletion to be enabled.
Infact I've been able to delete some topics, but there are few that won't be deleted no matter what I do. Brokers are running fine by the way.
Not only that, but it seems that those topics don't receive any message from producers, even though those are correctly configured. We tried change the topic where to send messages and it was fine.
I didn't find anything regard these 2 problems, any of you know anything?
I'm new with kafka and I'm trying to publish data from external application via http but I cannot find the way to do this.
I already created a topic in kafka and test it producing and consuming the message but I don't know how to insert/publish message via http, I tried to invoke the following url to retrieve the topics but it does not retrieve any data http://servername:2181/topics/
I'm using cloudera 5.12.1.
You can access to your topics, if it was already created, using APIs. The easy way...(see client list)
Or see Connects Config to manage connectors by REST (rest.host.name, rest.port parameters). But only connectors...
To consume or produce message in a topic, use a middleware. IT is more feaseble.
Check out the open source Kafka REST Proxy from Confluent. It does exactly what you want.
You can get it standalone, or as part of Confluent Platform.