I need to produce batch messages to Kafka so I have a file that I feed kafkacat:
kafkacat -b localhost:9092 -t <my_topic> -T -P -l /tmp/msgs
The content of /tmp/msgs is as follows
-H "id=1"
{"key" : "value0"}
-H "id=2"
{"key" : "value1"}
When I run the kafkacat command above, it inserts four messages to kafka - one message per line in /tmp/msgs.
I need to instruct kafkacat to parse the file correctly - that is -H "id=1" is the header for the message {"key" = "value0"}.
How do I achieve this?
Thanks
You need to pass the headers as follows.
kcat -b localhost:9092 -t topic-name -P -H key1=value1 -H key2=value2 /temp/payload.json
Related
I am using rest api to get the topic if from the kafka instance on confluent cloud
I am using below curl command
curl "http://(myhost):9092/topics/(topicname)" --output op.txt
but I am getting junk value in op.txt like
"U^C^C^#^B^BP"
Is there any solution to this?
You can't use REST to consume from Confluent Cloud (yet). With your data in Confluent Cloud you can use tools like ccloud or kafkacat to access your data from the commandline.
Confluent Cloud CLI (ccloud)
$ ccloud kafka topic consume --from-beginning rmoff_test_topic_01
Starting Kafka Consumer. ^C or ^D to exit
Hello world!
This is a message on a topic in Confluent Cloud
kafkacat
You can run kafkacat locally, or use it from Docker.
docker run --rm --interactive edenhill/kafkacat:1.6.0 \
-X security.protocol=SASL_SSL -X sasl.mechanisms=PLAIN \
-X ssl.ca.location=./etc/ssl/cert.pem -X api.version.request=true \
-b $CCLOUD_BROKER_HOST \
-X sasl.username="$CCLOUD_API_KEY" \
-X sasl.password="$CCLOUD_API_SECRET" \
-t rmoff_test_topic_01 -C -u -e
Hello world!
This is a message on a topic in Confluent Cloud
You can also output the message in JSON which can be useful:
docker run --rm --interactive edenhill/kafkacat:1.6.0 \
-X security.protocol=SASL_SSL -X sasl.mechanisms=PLAIN \
-X ssl.ca.location=./etc/ssl/cert.pem -X api.version.request=true \
-b $CCLOUD_BROKER_HOST \
-X sasl.username="$CCLOUD_API_KEY" \
-X sasl.password="$CCLOUD_API_SECRET" \
-t rmoff_test_topic_01 -C -u -J -e
{
"topic": "rmoff_test_topic_01",
"partition": 0,
"offset": 0,
"tstype": "create",
"ts": 1604571163960,
"broker": 7,
"key": null,
"payload": "Hello world!"
}
{
"topic": "rmoff_test_topic_01",
"partition": 3,
"offset": 0,
"tstype": "create",
"ts": 1604571168723,
"broker": 1,
"key": null,
"payload": "This is a message on a topic in Confluent Cloud"
}
I'm very new to kafka and confluent. I wrote a Producer nearly identical to the tutorial on https://www.confluent.fr/blog/schema-registry-avro-in-spring-boot-application-tutorial/ with my own dummy model. The application.yaml is the same as well. When I send the message to ccloud - the messages being received is gibberish
Any idea as to how to fix this? When I do a System.out.println of the avro POJO before sending to kafka, the object looks good with all the proper values.
{
"locationId": 1,
"time": 1575950400,
"temperature": 9.45,
"summary": "Overcast",
"icon": "cloudy",
"precipitationProbability": 0.24,
...
Whereas when I download the message from ccloud, the value looks like this
[
{
"topic":"Weather",
"partition":0,
"offset":14,
"timestamp":1576008230509,
"timestampType":"CREATE_TIME",
"headers":[],
"key":"dummyKey",
"value":"\u0000\u0000\u0001��\u0002\u0002����\u000b\u0002fffff�\"#\
...
}
You're actually doing everything right :) What you're hitting is just a current limitation in the Confluent Cloud GUI in rendering Avro messages.
If you consume the message as Avro you'll see that everything is fine. Here's an example of consuming the message from Confluent Cloud using kafkacat:
$ source .env
$ docker run --rm edenhill/kafkacat:1.5.0 \
-X security.protocol=SASL_SSL -X sasl.mechanisms=PLAIN \
-X ssl.ca.location=./etc/ssl/cert.pem -X api.version.request=true \
-b ${CCLOUD_BROKER_HOST}:9092 \
-X sasl.username="${CCLOUD_API_KEY}" \
-X sasl.password="${CCLOUD_API_SECRET}" \
-r https://"${CCLOUD_SCHEMA_REGISTRY_API_KEY}":"${CCLOUD_SCHEMA_REGISTRY_API_SECRET}"#${CCLOUD_SCHEMA_REGISTRY_HOST} \
-s avro \
-t mssql-04-mssql.dbo.ORDERS \
-f '"'"'Topic %t[%p], offset: %o (Time: %T)\nHeaders: %h\nKey: %k\nPayload (%S bytes): %s\n'"'"' \
-C -o beginning -c1
Topic mssql-04-mssql.dbo.ORDERS[2], offset: 110 (Time: 1576056196725)
Headers:
Key:
Payload (53 bytes): {"order_id": {"int": 1345}, "customer_id": {"int": 11}, "order_ts": {"int": 18244}, "order_total_usd": {"double": 2.4399999999999999}, "item": {"string": "Bread - Corn Muffaleta Onion"}}
This is the same topic shown here, with the binary Avro value field:
I'd like to use the kafka-console-producer.sh to fire a few JSON messages with Kafka headers.
Is this possible?
docker exec -it kafka_1 /opt/kafka_2.12-2.3.0/bin/kafka-console-producer.sh --broker-list localhost:9093 --topic my-topic --producer.config /opt/kafka_2.12-2.3.0/config/my-custom.properties
No, but you can with kafkacat's -H argument:
Produce:
echo '{"col_foo":1}'|kafkacat -b localhost:9092 -t test -P -H foo=bar
Consume:
kafkacat -b localhost:9092 -t test -C -f '-----\nTopic %t[%p]\nOffset: %o\nHeaders: %h\nKey: %k\nPayload (%S bytes): %s\n'
-----
Topic test[0]
Offset: 0
Headers: foo=bar
Key:
Payload (9 bytes): col_foo:1
% Reached end of topic test [0] at offset 1
Starting from kafka 3.1.0 there is an option to turn on headers parsing parse.headers=true and then you just place them before your record value, info form docs:
| parse.headers=true:
| "h1:v1,h2:v2...\tvalue"
So your command will look like
kafka-console-producer.sh --bootstrap-server localhost:9092 --topic topic_name --property parse.headers=true
and then you pass
header_name:header_value\nrecord_value
I have a use case where I need to copy the data from one topic to another topic in a different cluster but I need to copy only from a given offset. What can I use for the above use case?
I have looked into mirror maker as it copies data from one cluster to another but how to mention the offset part, I am not getting that.
Is there any utility I can use?
If as you say "This will be a one time operation" you can use kafkacat this -o option.
For example (the easiest case):
kafkacat -C -b mybrocker_cluster_1:9092 -t mytopic1 -o <offset> | \
kafkacat -P -b mybrocker_cluster_2:9092 -t mytopic1
You probably still need to add a few parameters to the consumer:
-X message.max.bytes=<value> -X fetch.message.max.bytes=<value> -X receive.message.max.bytes=<value>
Is there a way to POST to an ActiveMQ topic without requiring body= ?
e.g. instead of this:
curl -u test:test -d "body=message" http://server:8161/api/message/testtopic?type=topic
use this:
curl -u test:test -d "message" http://server:8161/api/message/testtopic?type=topic
It turns out that specifying the Content-Type allows you to omit body=
curl -u test:test -d "message" -H "Content-Type: text/plain" http://server:8161/api/message/testtopic?type=topic