How to delete Kafka topic using Kafka REST Proxy? I tried the following command, but it returns the error message:
curl -X DELETE XXX.XX.XXX.XX:9092/topics/test_topic
If it's impossible, then how to update delete the messages and update the scheme of a topic?
According to the documentation API Reference, you cannot delete topics via REST Proxy, and I agree with them because such a destructive operation should not be available via interface that is exposed to outside.
The topic deletion operation can be performed on the server where broker runs using command line utility. See How to Delete a topic in apache kafka
You can update the schema for a message when you publish it using the POST /topics/(string: topic_name) REST endpoint. If the schema for the new messages is not backward compatible with the older messages in the same topic you will have to configure your Schema Registry to allow publishing of incompatible messages, otherwise you will get an error.
See the "Example Avro request" here:
http://docs.confluent.io/3.1.1/kafka-rest/docs/api.html#post--topics-(string-topic_name)
See how to configure Schema Registry for forward, backward, or no compatibility see the documentation here:
http://docs.confluent.io/3.1.1/schema-registry/docs/api.html#compatibility
I confirmed that it is supported from version 5.5.0 or higher, and the test result worked normally. (REST Proxy API v3)
https://docs.confluent.io/current/kafka-rest/api.html#topic
Related
I want to use AWS lambda to consumer kafka messages, but the events I received don't contain kafka headers, that are very critical. Is there a way to get messages with headers info?
The Lambda Kafka integration did not support headers before, but they have (very recently, in the last few days) silently added support for headers. You can see this in the latest event example in the docs (or try it yourself): https://docs.aws.amazon.com/lambda/latest/dg/with-msk.html
Currently, I am using Confluent Rest Proxy 5.5.1 for collecting data in production. Duplicated events can come. I found the solution for de-duplication using Kafka Stream API. Is it possible to get an idempotent producer guarantee in Rest Proxy?
You can modify the REST Proxy's producer client with the producer. prefix
producer.enable.idempotence=true
Idempotent producers are not supported in Kafka REST. Ignoring 'enable.idempotence'. (io.confluent.kafkarest.KafkaRestConfig:880)
I have a requirement in my project where I have to get the data after hitting a REST service. Is there any Kafka connector that does this work or I have to write the custom code using streams or producer ?
I tried finding REST connector on ( https://www.confluent.io/hub/ ) but could not find anything. Can you please suggest?
Seems you're asking for the HTTP Sink Connector, which does exist on that page.
There is also a kafka-connect-rest project on Github
I'm developing a Kafka Sink connector on my own. My deserializer is JSONConverter. However, when someone send a wrong JSON data into my connector's topic, I want to omit this record and send this record to a specific topic of my company.
My confuse is: I can't find any API for me to get my Connect's bootstrap.servers.(I know it's in the confluent's etc directory but it's not a good idea to write hard code of the directory of "connect-distributed.properties" to get the bootstrap.servers)
So question, is there another way for me to get the value of bootstrap.servers conveniently in my connector program?
Instead of trying to send the "bad" records from a SinkTask to Kafka, you should instead try to use the dead letter queue feature that was added in Kafka Connect 2.0.
You can configure the Connect runtime to automatically dump records that failed to be processed to a configured topic acting as a DLQ.
For more details, see the KIP that added this feature.
I'm new with kafka and I'm trying to publish data from external application via http but I cannot find the way to do this.
I already created a topic in kafka and test it producing and consuming the message but I don't know how to insert/publish message via http, I tried to invoke the following url to retrieve the topics but it does not retrieve any data http://servername:2181/topics/
I'm using cloudera 5.12.1.
You can access to your topics, if it was already created, using APIs. The easy way...(see client list)
Or see Connects Config to manage connectors by REST (rest.host.name, rest.port parameters). But only connectors...
To consume or produce message in a topic, use a middleware. IT is more feaseble.
Check out the open source Kafka REST Proxy from Confluent. It does exactly what you want.
You can get it standalone, or as part of Confluent Platform.