I want to use AWS lambda to consumer kafka messages, but the events I received don't contain kafka headers, that are very critical. Is there a way to get messages with headers info?
The Lambda Kafka integration did not support headers before, but they have (very recently, in the last few days) silently added support for headers. You can see this in the latest event example in the docs (or try it yourself): https://docs.aws.amazon.com/lambda/latest/dg/with-msk.html
Related
Currently, I am using Confluent Rest Proxy 5.5.1 for collecting data in production. Duplicated events can come. I found the solution for de-duplication using Kafka Stream API. Is it possible to get an idempotent producer guarantee in Rest Proxy?
You can modify the REST Proxy's producer client with the producer. prefix
producer.enable.idempotence=true
Idempotent producers are not supported in Kafka REST. Ignoring 'enable.idempotence'. (io.confluent.kafkarest.KafkaRestConfig:880)
I am writing HTTP Sink plugin for Kafka connect. Its purpose is to send HTTP requests for each message in the configured Kafka topic. I want to send the message to dead letter queue in case HTTP request fails. Can I make use of dead letter queue configuration provided in sink plugin ?
The reason for this question is that, it has been mentioned in kafka connect documentation and several blogs that only errors in transformer and converter will be send to dead letter queue and not the ones during PUT. Since the task of sending the http request is done in PUT. So I am think, is there a way to send failed http messages to DLQ ? If not, is it possible to send the message to some other kafka topic for further processing ?
According to #Neil this might be informative,
Kip 610 (implemented in Kafka 2.6) added DLQ support for issues when interacting with end system. Kip 298 added DLQ but only on issues prior to sink system interaction.
Check the versions of your connect cluster and sink version and see if it supports it.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-610%3A+Error+Reporting+in+Sink+Connectors
I'm developing a Kafka Sink connector on my own. My deserializer is JSONConverter. However, when someone send a wrong JSON data into my connector's topic, I want to omit this record and send this record to a specific topic of my company.
My confuse is: I can't find any API for me to get my Connect's bootstrap.servers.(I know it's in the confluent's etc directory but it's not a good idea to write hard code of the directory of "connect-distributed.properties" to get the bootstrap.servers)
So question, is there another way for me to get the value of bootstrap.servers conveniently in my connector program?
Instead of trying to send the "bad" records from a SinkTask to Kafka, you should instead try to use the dead letter queue feature that was added in Kafka Connect 2.0.
You can configure the Connect runtime to automatically dump records that failed to be processed to a configured topic acting as a DLQ.
For more details, see the KIP that added this feature.
I'm new with kafka and I'm trying to publish data from external application via http but I cannot find the way to do this.
I already created a topic in kafka and test it producing and consuming the message but I don't know how to insert/publish message via http, I tried to invoke the following url to retrieve the topics but it does not retrieve any data http://servername:2181/topics/
I'm using cloudera 5.12.1.
You can access to your topics, if it was already created, using APIs. The easy way...(see client list)
Or see Connects Config to manage connectors by REST (rest.host.name, rest.port parameters). But only connectors...
To consume or produce message in a topic, use a middleware. IT is more feaseble.
Check out the open source Kafka REST Proxy from Confluent. It does exactly what you want.
You can get it standalone, or as part of Confluent Platform.
How to delete Kafka topic using Kafka REST Proxy? I tried the following command, but it returns the error message:
curl -X DELETE XXX.XX.XXX.XX:9092/topics/test_topic
If it's impossible, then how to update delete the messages and update the scheme of a topic?
According to the documentation API Reference, you cannot delete topics via REST Proxy, and I agree with them because such a destructive operation should not be available via interface that is exposed to outside.
The topic deletion operation can be performed on the server where broker runs using command line utility. See How to Delete a topic in apache kafka
You can update the schema for a message when you publish it using the POST /topics/(string: topic_name) REST endpoint. If the schema for the new messages is not backward compatible with the older messages in the same topic you will have to configure your Schema Registry to allow publishing of incompatible messages, otherwise you will get an error.
See the "Example Avro request" here:
http://docs.confluent.io/3.1.1/kafka-rest/docs/api.html#post--topics-(string-topic_name)
See how to configure Schema Registry for forward, backward, or no compatibility see the documentation here:
http://docs.confluent.io/3.1.1/schema-registry/docs/api.html#compatibility
I confirmed that it is supported from version 5.5.0 or higher, and the test result worked normally. (REST Proxy API v3)
https://docs.confluent.io/current/kafka-rest/api.html#topic