connect self managed ksqlDB to Confluent Cloud | managed ksql server - confluent-platform

This is a question on how to connect self managed ksqlDB / ksql server to confluent cloud.
I have a confluent basic cluster running in https://confluent.cloud/ in the GCP asia south.
In this cluster i want to connect self managed ksqlDB to Confluent Cloud Control center.
Here is my configurations which i copied from confluent cloud and put in the managed ksqldb.
This self managed ksqldb is a single machine GCP compute unit.
The same configuration is present in following properties.
/home/confluent/confluent-5.5.1/etc/ksqldb/ksql-server.properties
and the ksql server was started using following commands.
nohup /home/confluent/confluent/confluent-5.5.1/bin/ksql-server-start /home/confluent/confluent/confluent-5.5.1/etc/ksqldb/ksql-server.properties &
Command line :
/home/confluent/confluent-5.5.1/bin/ksql
Couple of things were noted in ksql terminal :
STREAM was created successfully in the terminal but not available in the cloud.
On the command "show streams;" It is able to show the specific STREAM.
print {STREAM}; It doesnt show up data even while data is pushed to STREAM.
I have not set any host entries.
On show connectors following exception is generated in ksql terminal.
ksql> show connectors;
io.confluent.ksql.util.KsqlServerException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8083 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to
localhost:8083 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed:
Connection refused (Connection refused)
Caused by: Could not connect to the server.
Caused by: Could not connect to the server.
I am expecting my ksqlDB shows up in the confluent cloud but unable to see though.
I dont know what more configurations are required, so that my self managed ksql server works and show up in confluent cloud.

seems that you are confusing some terminology here, self-managed != managed.
Managed KSQLDB is the service available on your Confluent Cloud console (last image). In there you have to add applications which spin up a KSQLDB cluster for your queries.
For self-managed KSQLDB instance running in GCP, you can connect it to Confluent Cloud, but it is not going to appear on the list of KSQLDB Applications, as you'll have to operate it yourself.
Docs:
Self-managed KSQLDB with Confluent Cloud: https://docs.confluent.io/current/cloud/cp-component/ksql-cloud-config.html
Confluent Cloud (managed) KSQLDB: https://docs.confluent.io/current/quickstart/cloud-quickstart/ksql.html#cloud-ksql-create-application

Related

Unable to start debezium connector in distributed mode

Trying to deploy debezium using kafka connect distributed mode is causing issues .
The connect worker shut down with not a clear exception.
The group-id decleration along with the topic.regex is not forcing the connect to read only the regex topics it is trying to consume from all the topics in the cluster
Has anyone able to run debezium on connect-distributed?
Followed the instructions to run debezium using connect-distributed.

How to connect already setup kafka cluster to mongodb?

How can I connect kafka events to a mongodb sink?
The resources I found on the net using confluent they make a cluster for you and didn't find how to connect my already existing cluster
You need to install the Mongo Connector into the plugin.path of your connect properties file, then start Kafka Connect using one of the bin/connect- scripts in your Kafka installation

error conctting to the cluster.Unable to connecting to xxx.com kafka tool

I tried to connect openShift cluster via kafka tool 2.0.8
I defined all the fields and I got:
![1]

Kafka Connect with Amazon MSK

How do I use Kafka Connect adapters with Amazon MSK?
As per the AWS documentation, it supports Kafka connect but not documented about how to setup adapters and use it.
Edit Oct 2021: MSK Connect has been launched, see https://aws.amazon.com/blogs/aws/introducing-amazon-msk-connect-stream-data-to-and-from-your-apache-kafka-clusters-using-managed-connectors/
AFAIK Amazon MSK does not provide managed connectors, so you have to run them yourself. This is done by running the Kafka Connect worker process (a JVM) and then providing it one or more connector configurations to run.
From the point of view of a Kafka Connect worker it just needs a Kafka cluster to connect to; it shouldn't matter whether it's MSK or on-premises, since it's ultimately 'just' a consumer/producer underneath.
You can see more, including a live demo, here: https://rmoff.dev/bbuzz19-kafka-connect
For an example of configuring Kafka Connect to use a cloud-hosted Kafka platform (in this case, Confluent Cloud), see this article.
If you are interested in managed connectors in the Cloud, check out the connectors that are provided in Confluent Cloud.
Disclaimer: I work for Confluent :)
AWS now supports MSK Connect, a new feature of MSK service based on Kafka Connect allowing you to deploy managed Kafka connectors built for Kafka connect
Check the announcement here: https://aws.amazon.com/blogs/aws/introducing-amazon-msk-connect-stream-data-to-and-from-your-apache-kafka-clusters-using-managed-connectors/
There are two aspects to this
Kafka Connect is a framework which should be deployed separately from kafka brokers. MSK only provides kafka brokers. If you want to use Kafka Connect with MSK you would need to use EC2 instances and deploy the kafka binaries.Kafka Connect framework is bundled along with kafka
Coming to connectors if you donot have a confluent subscription or similar - I am afraid your choices get very limited. But having said you can always write your own connectors. Writing new connectors is not that difficult rather you can apply your business specific logic and be on your way quite quickly.

Connecting REST Proxy to Confluent Cloud trying to connect to zookeeper

I'm trying to connect kafka rest proxy to confluent cloud.
kafka-rest-start ccloud-kafka-rest.properties
Here is my file with properties ccloud-kafka-rest.properties
client.ssl.endpoint.identification.algorithm=https
client.sasl.mechanism=PLAIN
consumer.request.timeout.ms=20000
bootstrap.servers=***-****.us-east-1.aws.confluent.cloud:9092
consumer.retry.backoff.ms=500
client.security.protocol=SASL_SSL
id=kafka-rest-with-ccloud
producer.acks=1
admin.request.timeout.ms=50000
client.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="***" password="***";
After I run kafka-rest-start it tries connect to zookeeper (zookeeper.connect = localhost:2181).
ERROR Server died unexpectedly: (io.confluent.kafkarest.KafkaRestMain:63)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server 'localhost:2181' with timeout of 30000 ms
Also I tried to set zookeeper host empty, It didn't help.
How to turn off connecting to zookeeper?
The default value for the zookeeper.connect property is localhost:2181. See here
You need to populate it with the address of the zookeeper of the confluent cloud.
See the instructions for connecting a rest-proxy to kafka cloud here