I'm trying to use Apache Kafka with Clickhouse. I have a kafka username and password which I added to config.xml file in clickhouse files liked this:
<kafka>
<sasl_mechanisms>SCRAM-SHA-256</sasl_mechanisms>
<sasl_username>some_user</sasl_username>
<sasl_password>some_pass</sasl_password>
</kafka>
This way I can use Kafka topics which is available for that one user. How can I use multiple user and use different topics available for different users while using a Kafka Engine in Clickhouse?
Is there a way to configure Kafka user settings while writing Kafka table with SQL in Clickhouse?
https://clickhouse.com/docs/en/engines/table-engines/integrations/kafka/#configuration
each topic can have own settings
<kafka_mytopic>
<sasl_mechanisms>SCRAM-SHA-256</sasl_mechanisms>
<sasl_username>yyyy</sasl_username>
<sasl_password>xxxx</sasl_password>
</kafka_mytopic>
<kafka_mytopic2>
<sasl_mechanisms>SCRAM-SHA-256</sasl_mechanisms>
<sasl_username>ddd</sasl_username>
<sasl_password>zzz</sasl_password>
</kafka_mytopic2>
Related
Debezium Kafka Debezium Connect was configured on PostgreSQL payments(schema) outbox(table) to Kafka topic(payments-transactions).
Means at Kafka Connect perform below actions
payments.outbox ==streams=> payments-transactions Topic
As part of the new CDC flow, we plan to configure webhooks(schema) outbox(table) to different Kafka topic(webhooks)
Means at Kafka Connect perform below actions
payments.outbox ==streams=> payments-transactions topic
webhooks.outbox ==streams=> webhooks topic
How does the Debezium Kafka connect configuration looks? Is it possible to configure two table with separate topics?
If you configure table.whitelist to multiple tables of the same schema, the each table will generate a corresponding output topic.
https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-topic-names
I am using Confluent MongoDB Atlas Source Connector to pull data from MongoDB collection to Kafka. I have noticed that the connector is creating multiple topics in the Kafka Cluster. I need the data to be available on one topic so that the consumer application can consume the data from the topic. How can I do this?
Besides, why the Kafka connector is creating so many topics? isn't is difficult for consumer applications to retrieve the data with that approach?
Kafka Connect creates 3 internal topics for the whole cluster for managing its own workload. You should never need/want external consumers to use these
In addition to that, connectors can create their own topics. Debezium for example creates a "database history topic", and again, this shouldn't be read outside of the Connect framework.
Most connectors only need to create one for the source to pull data into, which is what consumers actually should care about
I am using MongoDB Source Connector to get the data from a MongoDB collection into Kafka. What this connector does is that it automatically creates a topic using the following naming convention:
[prefix_provided_in_the_connector_properties].[db_name].[collection_name]
In the MongoDB Source Connector's documentation, there is no mention of overriding the topic configuration such as number of partitions or replication factor. I have the following questions:
Is it possible to override the topic configs in the connector.properties file?
If not, is it then done on Kafka's end? If so, can we individually configure each topics' settings or it will globally affect all the topics?
Thank you!
Sounds like you have auto.create.topics.enable=true on your brokers. It is recommended to disable this and enforce manual topic creation.
Connect only creates internal topics for itself. Source connectors should ideally have their topics created ahead of time, otherwise, you get the defaults set in the broker server.properties. Changing the values will not change existing topics
Does Kafka Connect creates the topic on the fly if it doesn't exist (but provided as a destination) or fails to copy messages to it?
I need to create such topics on the fly or programmatically (Java API) at least, not manually using scripts.
I searched this info, but it seems topics have to be already created before migration
Kafka Connect doesn't really control this.
There's a setting in Kafka that enables/disables automatic topic creation.
If this is turned on - Kafka Connect will create its' own topics, if not - you have to create them yourselves.
By default, Kafka will not create a new topic when a consumer subscribes to a non-existing topic. you should enable the auto.create.topics.enable=truein your Kafka server configuration file which enables auto-creation of topics on the server.
Once you turn on this feature Kafka will automatically create topics on the fly. When an application tries to connect to a non-existing topic, Kafka will create that topic automatically.
I'm using the Kafka connect JDBC source connector to read from a view in a database and post it on kafka, it is working fine.
My use case is that a user can create multiple objects and the order of the objects is important in my application. I would like to use the user id as the message key of all the messages I'm posting into the topic to maintain their order.
My question is how can I define a message key in the Kafka connect source connector?
You can use Kafka Connect's SMT (Single Message Transforms) feature by adding following code to connect-file-source configuration file.
transforms=createKey
transforms.createKey.type=org.apache.kafka.connect.transforms.ValueToKey
transforms.createKey.fields=UserId <name of user id column>
More information about SMT here