Exporting data from volt to kafka - apache-kafka

Question

We are trying to do a POC where we try to export data from a volt db table to kafka below is the steps I followed:-
Step1:- prepared the deployment.xml to enable the export to kafka
<?xml version="1.0"?>
<deployment>
<cluster hostcount="1" kfactor="0" schema="ddl" />
<httpd enabled="true">
<jsonapi enabled="true" />
</httpd>
<export enabled="true" target="kafka">
<configuration>
<property name="metadata.broker.list">localhost:9092</property>
<property name="batch.mode">false</property>
</configuration>
</export>
</deployment>
Step2:- Then Strted the voltdb server
./voltdb create -d deployment-noschema.xml --zookeeper=2289
Step3:- Create a export only table and insert some data into it
create table test(x int);
export table test;
insert into test values(1);
insert into test values(2);
After this I tried to verify if any topic has been created in kafka but there was none.
./kafka-topics.sh --list --zookeeper=localhost:2289
Also I can see logging of all the data in exportoverflow directory. Could anyone please let me know what's the missing part here.

Answer 1

Prabhat,
In your specific case, a possible explanation of the behavior you observe is you started Kafka with out the auto create topics options set to true. The export process requires Kafka to have this enabled to be able to create topics on the fly. If not you will have to manually create the topic and then export from VoltDB.
As a side note, while you can use the zookeeper that starts with VoltDB to start your Kafka, it is not the recommended approach since when you bring down VoltDB server, then your Kafka is left with no zookeeper. It is best approach to use Kafka's own zookeeper to manager your Kafka instance.
Let me know if this helped - Thx.

Answer 2

Some Questions and Possible answers.
Are you using enterprise version?
Can you call #Quiesce from sqlcmd and see if your data pushes to kafka.
Which version you are using?
VoltDB embeds a zookeeper are you using standalone zookeeper or VoltDB's ? we dont test with embedded one as its not exactly same as kafka supported.
Let us know or email support At voltdb.com
Looking forward.