Dezebium is not recreating topics for a table after they are deleted. In this question I was able to create a Kafka topic to stream updates from the table specified in table.include.list. I deleted the topic and reran bin/connect-standalone.sh with the same config, reproduced below, but the same topic is not recreated. I removed table.include.list, reran bin/connect-standalone.sh, and all the topics other than for table myschema.ipaddrs were recreated. I deleted those topics, reran bin/connect-standalone.sh, and no topics are recreated.
name=testdb
connector.class=io.debezium.connector.postgresql.PostgresConnector
topic.prefix=test
database.hostname=localhost
database.port=5432
database.user=postgres
database.password=root
database.dbname=testdb
database.server.name=testdb
table.include.list=myschema.ipaddrs
plugin.name=pgoutput
Before running bin/connect-standalone.sh, I ran curl -i -X DELETE localhost:8083/connectors/testdb/ to delete the connector. I don't have any other connectors.
I see these logs related to the above table. I don't see any errors.
[2022-10-20 11:52:42,655] INFO [testdb|task-0] table.include.list = myschema.ipaddrs (io.debezium.connector.common.BaseSourceTask:131)
[2022-10-20 11:52:42,894] INFO [testdb|task-0] REPLICA IDENTITY for 'myschema.ipaddrs' is 'FULL'; UPDATE AND DELETE events will contain the previous values of all the columns (io.debezium.connector.postgresql.PostgresSchema:100)
[2022-10-20 11:52:42,965] INFO [testdb|task-0] REPLICA IDENTITY for 'myschema.ipaddrs' is 'FULL'; UPDATE AND DELETE events will contain the previous values of all the columns (io.debezium.connector.postgresql.PostgresSchema:100)
The topics are not recreated if I change either the name of the connector or topic.prefix
I observe this same behavior if I run bin/connect-distributed.sh and pass the above config as JSON to the API.
The topics are not created if the underlying table has no data. The topic was created once I added a row into the table.
Related
Using debezium mongo connector, I am pushing data out of mongodb to kafka. I have 4 collections: transactions, wallets, cards and users.
Before launching kafka-connector, I created a topic "mongo_conn.digi.users". and when i re-launch it Kafka-connector created "mongo_conn.digi.cards" and "mongo_conn.digi.transactions" topic on it's own and got data pushed to them. However, it did not create a topic for wallets and did not push data to "mongo_conn.digi.users".
So, I deleted "mongo_conn.digi.users" and re-launched connector. As expected, "mongo_conn.digi.users" was recreated by kafka-connector but nothing has been pushed too. also no topic was created for wallets collection same issue.
I am trying to find a reason on why connector did not create any topic for wallets topic & why kafka-connector is not pushing any data for users topic
this is my connector output
I'm trying to delete Kafka topic __Consumer_offset as it is causing a lot of confusion for my brokers.
When i do so, it says this topic can't be marked for deletion.
i'm using the zookeeper cli to delete it such as rmr /brokers/topic __consumer_offset, but it is not working!
__consumer_offsets is a kafka internal topic and it is not allowed to be deleted through delete topic command. It contains information about committed offsets for each topic:partition for each group of consumers (groupID). If you want to wipe it out entirely you have to delete the zookeeper dataDir location. That implies, you lose all the metadata.
Also if you just want to get rid of the existing consumer groups, you can as well reset the offsets or consider deleting them.
AFAIK you cannot delete that topic. It is a internal topic and should not be deleted manually.
If it is must, then you will have to manually clean/remove your data directory. When you deploy Kafka brokers and Zookeepers it creates data directory.
Note: By removing data directory you will loose all topics and related data. So this is not feasible option in Production.
I know how to delete Kafka connector as mentioned here Kafka Connect - How to delete a connector
But I am not sure if it also delete/erase specific connector related configs, offsets and status from *.sorage.topic for that worker?
For e.g:
Lets say I delete a connector having connector-name as"connector-abc-1.0.0" and Kafka connect worker was started with following config.
offset.storage.topic=<topic.name>.internal.offsets
config.storage.topic=<topic.name>.internal.configs
status.storage.topic=<topic.name>.internal.status
Now after DELETE call for that connector, will it erased all records from above internal topics for that specific connector?
So that I can create new connector with "same name" on same worker but different config(different offset.start or connector.class)?
When you delete a connector, the offsets are retained in the offsets topic.
If you recreate the connector with the same name, it will re-use the offsets from the previous execution (even if the connector was deleted in between).
Since Kafka is append only, then only way the messages in those Connect topics would be removed is if it were published with the connector name as the message key, and null as the value.
You could inspect those topics using console consumer to see what data is in them including --property print.key=true, and keep the consumer running when you delete a connector.
You can PUT a new config at /connectors/{name}/config, but any specific offsets that are used are dependent upon the actual connector type (sink / source); for example, there is the internal Kafka __consumer_offsets topic, used by Sink connectors, as well as the offset.storage.topic, optionally used by source connectors.
"same name" on same worker but different config(different offset.start or connector.class)?
I'm not sure changing connector.class would be a good idea with the above in mind since it'd change the connector behavior completely. offset.start isn't a property I'm aware of, so you'll need to see the documentation of that specific connector class to know what it does.
Using KSQL (Confluent: Version: 5.0.1) I'm able to drop a table / stream normally (using DROP [TABLE|STREAM]) <NAME> when the linked topic exist and when it is registered (Registered=true).
However, if the topic is dropped first (Registered=false) then the associated stream or table can't be dropped with KSQL pointing out that "No topic with name Foo was registered".
The problem is that the stream / table still shows in the list and no new stream / table can be added using the same name.
Is there any way to delete them after the topic was dropped?
From the official documentation, I would do it like this:
SHOW QUERIES;
Write down the QueryID;
TERMINATE <QueryID>;
DROP TABLE <tablename>;
[See documentation...]
https://docs.ksqldb.io/en/latest/developer-guide/create-a-table/#delete-a-ksqldb-table
https://docs.ksqldb.io/en/latest/developer-guide/ksqldb-reference/terminate/
I think I found an answer, or at least in the right direction.
You need to re-register the topic with the stream. After a bit of hacking around, I managed to reattach my topics to streams, the drop the streams.
REGISTER TOPIC <ksql_topic_name> WITH (KAFKA_TOPIC='<kafka_topic_name>', VALUE_FORMAT='<format>');
Then,
DROP STREAM <stream_name>;
I don't know if this works in all cases, but it got me back to where I want to be. If anyone else can iterate on this and find a more robust solution I would like to hear about it.
I've been wrestling with these type issues for a while. Here's where I'm at, from my notes:
Having trouble deleting streams/tables/topics? Delete them in zookeeper:
Shut down any producers or consumers associated with topics you'll be deleting
Attempt to drop your tables and streams from KSQL CLI
Shut down KSQL server
Delete topics with 'kafka-topic' CLI
Delete in zookeeper like this:
zookeeper-shell localhost:2181
ls /brokers/topics
_confluent-controlcenter-5-0-0-1-TriggerEventsStore-changelog, _confluent-controlcenter-5-0-0-1-error-topic, ...
rmr /brokers/topics/yourTopicName
Now do a " kafka-topics --list --zookeeper localhost:2181' and they should be gone for good.
OR, for the ABSOLUTE NUCLEAR OPTION:
Attempt to drop your tables and streams from KSQL CLI
Shut down KSQL server
Shut down your brokers and zookeepers
Delete eveything in the kafka logs dir on each broker node (location of dir is defined in your properties file).
Start up your zookeepers again and delete the /brokers/topics from zookeeper CLI
Start everything back up and it should all be gone. But be careful, you'll lose
everything, all your data.
You would have to terminate any running queries first.
SHOW QUERIES; --> that will show you the active queries (COPY QUERY ID)
Then:
TERMINATE [QUERY ID];
Then:
DROP [TABLE OR STREAM];
I was using a kafka topic, and it's metadata as well in my application. I hard deleted the topic from the zookeeper shell, by deleting the directories corresponding to that topic. After creating the topic again, I described the topic and found that no leaders have been assigned to this newly created topic. In the consumer, I can see repeated logs printing LEADER_NOT_AVAILABLE. Any reason as to what am I doing wrong? Or maybe is there a way to delete the metadata related to the kafka topic as well that I'm unaware of? Thanks in advance!
Deleting topics in Kafka hasn't been straightforward until recently. In general, you shouldn't attempt to delete Kafka topics by deleting metadata in Zookeeper. You should always use the included command line utilities.
First you need to make sure that deleting topics is enabled in the server.properties file on all brokers, and do a rolling restart if needed:
delete.topic.enable=true
After you restart the brokers to enable topic deletion, you should issue the delete command using the command line utilities:
./kafka-topics.sh —zookeeper <zookeeper_host>:2181 —delete —topic <topic_name>
If at this point, it's still stuck, try to run these two commands from the zookeeper shell to make sure and remove all metadata for that particular topic:
rmr /brokers/topics/<topic_name>
rmr /admin/delete_topics/<topic_name>
A few more details here:
https://medium.com/#contactsunny/manually-delete-apache-kafka-topics-424c7e016ff3