Can i have kafka connect service as a standalone service - apache-kafka

Question

I am using apache kafka in 1st server and apache zookeeper in 2nd server.
I want to have kafka connect service in other server.Is it possible to use a standalone service. I need to have apache kafka connect or confluent kafka connect

Answer 1

There is no such thing as "Confluent Kafka (Connect)"
Kafka Connect is Apache 2.0 Licensed and released as part of Apache Kafka. Confluent and other vendors write plugins (free, or enterprise-licensed) for Kafka Connect.
Yes, it it recommended to run Connect as a separate set of servers than either then brokers or zookeepers. In order to run it, you will need to download all of Kafka, then use bin/connect-distributed, or you can run it via Docker containers.

Answer 2

You can easily run a Kafka Connect Standalone (Single Process) service from any server, provided you have configured both connector and workers properties correctly.
A gist of it here, if you are interested.
In standalone mode all work is performed in a single process. It is easy to setup and get started but it does not benefit from some of the features of Kafka Connect such as fault tolerance.
You can start a standalone process with the following command:
bin/connect-standalone.sh config/connect-standalone.properties connector1.properties connector2.properties
The first parameter is the configuration for the worker, it includes connection parameters, serialization format, and how frequently to commit offsets.
All workers (both standalone and distributed) require a few configs:
bootstrap.servers - List of Kafka servers used to bootstrap connections to Kafka
Key/Value.converter - Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka.
offset.storage.file.filename - File to store offset data
A Simple standalone connecter (Import Data from a file into KAFKA)
kafka-topics --create --zookeeper localhost:2181 --replication-factor 2 --partitions 10 --topic connect-test to create a topic called connect-test with 10 partitions (Up to us) and replication factor of 2
To start a standalone Kafka Connector, we need following three configuration files located under C:\kafka_2.11-1.1.0\config. Update these configuration files as following
connect-standalone.properties
offset.storage.file.filename=C:/kafka_2.11-1.1.0/connectConfig/connect.offsets
connect-file-source.properties
file=C:/kafka/Workspace/kafka.txt
connect-file-sink.properties
file=test.sink.txt
Execute the following command
Connect-standalone.bat C:\kafka_2.11-1.1.0\config\connect-standalone.properties C:\kafka_2.11-1.1.0\config\connect-file-source.properties C:\kafka_2.11-1.1.0\config\connect-file-sink.properties
Once the Connector is started, initially the data in kafka.txt would be synced to test.sync.txt and the data is published to the Kafka Topic named, connect-test. Then any changes to the kafka.txt file would be synced to test.sync.txt and published to connect-test topic.
Create a Consumer
kafka-console-consumer.bat --bootstrap-server localhost:9096 --topic connect-test --from-beginning
CLI Output
kafka-console-consumer --bootstrap-server localhost:9096 --topic connect-test --from-beginning
{"schema":{"type":"string","optional":false},"payload":"This is the stream data for the KAFKA Connector"}
Add a new line into “Consuming the events now” into the kafka.txt
CLI Output
kafka-console-consumer --bootstrap-server localhost:9096 --topic connect-test --from-beginning
{"schema":{"type":"string","optional":false},"payload":"This is the stream data for the KAFKA Connector"}
{"schema":{"type":"string","optional":false},"payload":"Consuming the events now"}