I have a 3 node Confluent Kafka Cluster Running. I have installed the Confluent 5.0.0, Started the connect service with nohup <Start connect-distributed> >/dev/null 2>&1 command.
I have an existing MSSQL connector task which was running earlier. When I tried starting the connector, connector was not started, and the task status was unassigned. Tried restarting the task as well as the connector, but no luck.
Do we have any option to recover the connector and the task?
Related
I'm new to PySpark and I want to connect remote Hadoop Cluster (CDP) through Linux server by using spark-submit command.
Any help would be appreciated.
I need spark-submit command to connect remote CDP.
You can use Apache Livy to submit remote jobs to a CDP cluster. Here is detailed info on how to install and use Livy to submit jobs :
After downloading and unzipping Livy you should add following lines in livy.conf file. Then start livy service.
livy.spark.master = yarn
livy.spark.deploy-mode = cluster
You can find examples of how to create a spark submit script on following links:
https://community.cloudera.com/t5/Community-Articles/Submit-a-Spark-Job-to-CDP-Data-Hub-using-the-Livy-REST-API/ta-p/322481
https://livy.apache.org/examples/
I'm aware, that the kafka server can be shutdown using shell script kafka-server-stop.sh and zookeeper can be shutdown using zookeeper-server-stop.sh
But, how do we stop connect-distributed.sh gracefully, I didn't find any stop shell script for connect-distributed.
Unfortunately, there isn't a stop script.
Best options other than kill command would be to use systemctl to manage the service, or use pre-built Docker images to run the server that can be stopped.
I am trying to install Kafka following a tutorial from DigitalOcean.com here.
I am doing this on Windows WSL2 with Ubuntu. So, after creating the zookeeper.service and kafka.service as per the tutorial, I do this command (the tutorial uses sudo systemctl start kafka instead), following advice from this thread:
sudo service kafka start
I received :
kafka: unrecognized service
When I do service --status-all to see if kafka is in the list, it is not there.
What am I missing?
There is lack of support in WSL for systemd
why systemd is disabled in WSL?
I have a Kafka Connect source and sink connector for putting data into Kafka and taking it back out.
I am running Kafka and Kafka Connect using docker-compose which runs connect in distributed mode. see that it finds my plugin when connect starts up, but it doesn't actually do anything unless I do a POST to the /connectors API, including the configuration in JSON.
I have a properties file with the configuration in it and I've tried putting it under /etc where I find similar properties files for the other plugins that are installed.
Am I missing a step when installing my plugin, or is it required to register the connector via the REST API before it will be assigned to workers?
Yes, you have to configure Kafka Connect using the REST API when using distributed mode.
It's possible to script the creation of connectors though, using a Docker Compose like this:
command:
- bash
- -c
- |
/etc/confluent/docker/run &
echo "Waiting for Kafka Connect to start listening on kafka-connect ⏳"
while [ $$(curl -s -o /dev/null -w %{http_code} http://kafka-connect:8083/connectors) -eq 000 ] ; do
echo -e $$(date) " Kafka Connect listener HTTP state: " $$(curl -s -o /dev/null -w %{http_code} http://kafka-connect:8083/connectors) " (waiting for 200)"
sleep 5
done
nc -vz kafka-connect 8083
echo -e "\n--\n+> Creating Kafka Connect Elasticsearch sink"
/scripts/create-es-sink.sh
sleep infinity
where /scripts/create-es-sink.sh is the REST call from curl in a file mounted locally to the container.
(source)
You can install a Kafka connector before you start the distributed Connect worker using "confluent-hub install" as shown here: Install Kafka connector manually). However, I'm not sure what the magic is if you aren't using confluent-hub though.
Kafka server failed to start on confluent start command.
command lines:
~]# sudo confluent start
zookeeper is already running. Try restarting if needed
Starting kafka
-Kafka failed to start
kafka is [DOWN]
Cannot start Schema Registry, Kafka Server is not running. Check your deployment
Run
confluent log kafka
to see the log from Kafka trying to start, and see what the error is.