I want to do something like this :
kafka-avro-console-producer --broker-list [kafka-url] --topic test2 --property value.schema=[some-value-format] --property schema.registry.url=[schema-url] --property Authorization: Bearer=$TOKEN
Related
I tried to create Kafka topic but I am getting the error.
$ docker exec -ti kafka-server1 \
> /opt/bitnami/kafka/bin/kafka-topics.sh --create \
> --zookeeper zookeeper-server:2181 \
> --replication-factor 1 \
> --partitions 1 \
> --topic topic_model_local
Exception in thread "main" joptsimple.UnrecognizedOptionException: zookeeper is not a recognized option
at joptsimple.OptionException.unrecognizedOption(OptionException.java:108)
at joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:510)
at joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56)
at joptsimple.OptionParser.parse(OptionParser.java:396)
at kafka.admin.TopicCommand$TopicCommandOptions.<init>(TopicCommand.scala:517)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:47)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
From the quickstart docs:
Option zookeeper is deprecated, use --bootstrap-server instead.
I would like to use the kafka-avro-console-producer with the schema registry. I have big schemas (over 10k chars) and I can't really past them as a command line argument. Besides that I'd like to use the schema registry directly so I can use a specific schema id.
I'm thinking about something like this, but it doesn't work:
kafka-avro-console-producer \
--broker-list <broker-list> \
--topic <topic> \
--property schema.registry.url=http://localhost:8081 \
--property value.schema=`curl http://localhost:8081/schemas/ids/419`
For the current version of the CLI tool
kafka-avro-console-producer \
--broker-list <broker-list> \
--topic <topic> \
--property schema.registry.url=http://localhost:8081 \
--property value.schema.id=419
For older version
You'll need to extract the schema from the API request using jq, for example
value.schema="$(curl http://localhost:8081/schemas/ids/419 | jq -r .schema)"
You can use the property value.schema.id:
kafka-avro-console-producer \
--broker-list <broker-list> \
--topic <topic> \
--property schema.registry.url=http://localhost:8081 \
--property value.schema.id=419
I'd like to use the kafka-console-producer.sh to fire a few JSON messages with Kafka headers.
Is this possible?
docker exec -it kafka_1 /opt/kafka_2.12-2.3.0/bin/kafka-console-producer.sh --broker-list localhost:9093 --topic my-topic --producer.config /opt/kafka_2.12-2.3.0/config/my-custom.properties
No, but you can with kafkacat's -H argument:
Produce:
echo '{"col_foo":1}'|kafkacat -b localhost:9092 -t test -P -H foo=bar
Consume:
kafkacat -b localhost:9092 -t test -C -f '-----\nTopic %t[%p]\nOffset: %o\nHeaders: %h\nKey: %k\nPayload (%S bytes): %s\n'
-----
Topic test[0]
Offset: 0
Headers: foo=bar
Key:
Payload (9 bytes): col_foo:1
% Reached end of topic test [0] at offset 1
Starting from kafka 3.1.0 there is an option to turn on headers parsing parse.headers=true and then you just place them before your record value, info form docs:
| parse.headers=true:
| "h1:v1,h2:v2...\tvalue"
So your command will look like
kafka-console-producer.sh --bootstrap-server localhost:9092 --topic topic_name --property parse.headers=true
and then you pass
header_name:header_value\nrecord_value
when I typed this command line:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
But,the final result returns like this:
it returns nothing,it is unusual.Somebody could help?Thanks in advance.
According to the docs, you just modify a topic to increase the replication factor.
> bin/kafka-topics.sh --zookeeper zk_host:port/chroot --create --topic my_topic_name
--partitions 20 --replication-factor 3 --config x=y
Unfortunately, it doesn't specify what happens then after you modify the topic. Do existing log segments get replicated to the new replicas, or only new messages?
For kafka 10 you need to invoke the kafka-reassign-partitions.sh script to change the replication factor of the topic.
Here is a demo of a script that will display the topic before and after the change:
updateTopicReplication() {
TOPIC_NAME=$1
REPLICAS=$2
echo "****************************************"
echo "describe $TOPIC_NAME"
bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic $TOPIC_NAME
JSON_FILE=./configChanges/${TOPIC_NAME}-update.json
echo $JSON_FILE
[ -e $JSON_FILE ] && rm $JSON_FILE
touch $JSON_FILE
echo -e "{\"version\":1, \"partitions\":[{\"topic\":\"${TOPIC_NAME}\",\"partition\":0,\"replicas\":[${REPLICAS}]}]}" >> $JSON_FILE
echo "****************************************"
echo "updateing $TOPIC_NAME"
bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file $JSON_FILE --execute
echo "****************************************"
echo "describe $TOPIC_NAME"
bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic $TOPIC_NAME
}
It seems that kafka actually doesn't support increasing (or decreasing) the replication factor for a topic, according to the same docs I mentioned in the question.