I installed confluent using CFK (Confluent for Kubernetes) way of deployment, setup went fine, using the vanilla yaml file for the entire components (zookeeper, kafka, connect, ksql, control-center, schema-registry).
I tried to use kind: Connector to configure my sqlserver source connector, connector created succesfully.
The problem came when I tried to list the connector via the below curl request (after port-forwarding to the pod)
curl localhost:8083/connectors | jq .
I got nothing registered, however when I ran the below command:
kubectl confluent connector list
it shows that I have a registered connectors as below, I assume both are 2 faces for the same coin.
NAME STATUS TASKS-READY TASKS-FAILED AGE
bq-sink-conn 0 9h
mssql-source-conn 0 9h
My question is, why is this discrepancy?, or I am missing something
also after a week of searching and looking on the internet, I can't find enough resources with example on how to use CFK and specifically Connectors CR.
Thanks,
Related
To run cp S3-connect to consume kafka topic in my local mac, I did something like below
1. Installed Confluent Kafka connector and ran the kafka connect-standalone.sh
ML-C02Z605SLVDQ:kafka_2.12-2.5.0 e192270$ confluent-hub install confluentinc/kafka-connect-s3:latest --component-dir /usr/local/share/java --worker-configs config/connect-distributed.properties
ML-C02Z605SLVDQ:kafka_2.12-2.5.0 e192270$ cd kafka_2.12-2.5.0
ML-C02Z605SLVDQ:kafka_2.12-2.5.0 e192270$ bin/connect-standalone.sh config/connect-standalone.properties s3-sink.properties. // s3-sink.properties connector.class=io.confluent.connect.s3.S3SinkConnector
Now, to run Kafka S3 connect in minikube I have installed Kafka-connect(kafka-connect-s3) in minikube using cp-helm-charts with help this tutorial Using a connector with Helm-installed Kafka/Confluent.
How to copy kafka config and script files inside kafka-connect pod ?
Should I need to login kafka-connect pod to run
connect-standalone.sh command?
There is a from scratch procedure here. The only requirement is Minikube.
The steps you need are the following:
Start Minikube
Deploy a Kafka cluster using the Strimzi Operator
Build your own custom image including required plugins and dependencies
Deploy Kafka Connect cluster in distributed mode using that image
Create a KafkaConnector instance passing a configuration YAML
How to copy kafka config and script files inside kafka-connect pod
You shouldn't copy anything. Everything is configured by env-vars. The Helm charts should be mostly documenting how those variables are working.
The Docker image uses Connect Distributed, which is started via a REST API, not a property file. And confluentinc/cp-kafka-connect already contains S3 Connect
You can also take a look at https://strimzi.io/.
The project is aimed at making the installation and management of a Kafka and Kafka Connect cluster on Kubernetes very easy.
I have started confluent platform on windows with the help of docker. I am able to start Kafka Broker, Zookeeper & control center.
I have setup the confluent-cli as well in docker. Now when I try to run the commands on confluent-cli, it's expecting --url param. Confluent docs says that Contact your IT admin to get the HTTP address for the <service url> (for example, http://127.0.0.1:8080/) for each Confluent Platform component. Since I am running docker on my local now, what would be my service url ?
Now, I have started schema-registry as well and I can use the schema-registry port to check the cluster details.
confluent cluster describe --url http://<my_ip>:<schema_registry_port>
I got the below output when I run the describe command
Scope:
Type | ID
+-------------------------+------------------------+
kafka-cluster | d60cQ7BWQTSz5v9fNuvQRw
schema-registry-cluster | schema-registry
Reference : https://docs.confluent.io/current/cli/command-reference/confluent_cluster_describe.html#example
I am deploying Kafka-connect on Google Kubernetes Engine (GKE) using cp-kafka-connect Helm chart in distributed mode.
A working Kafka cluster with broker and zookeeper is already running on the same GKE cluster. I understand I can create connectors by sending post requests to http://localhost:8083/connectors endpoint once it is available.
However, Kafka-connect container goes into RUNNING state and then starts loading the jar files and till all the jar files are loaded the endpoint mentioned above is unreachable.
I am looking for a way to automate the step of manually exec the pod, check if the endpoint is ready and then send the post requests. I have a shell script that has a bunch of curl -X POST requests to this endpoint to create the connectors and also have config files for these connectors which work fine with standalone mode (using Confluent platform show in this confluent blog).
Now there are only two ways to create the connector:
Somehow identify when the container is actually ready (when the endpoint has started listening) and then run the shell script containing the curl requests
OR use the configuration files as we do in standalone mode (Example: $ <path/to/CLI>/confluent local load connector_name -- -d /connector-config.json)
Which of the above approach is better?
Is the second approach (config files) even doable with distributed mode?
If YES: How to do that?
If NO: How to successfully do what is explained in the first approach?
EDIT:
With reference to his github issue(thanks to #cricket_007's answer below) I added the following as the container command and connectors got created after the endpoint gets ready:
...
command:
- /bin/bash
- -c
- |
/etc/confluent/docker/run &
echo "Waiting for Kafka Connect to start listening on kafka-connect "
while : ; do
curl_status=`curl -s -o /dev/null -w %{http_code} http://localhost:8083/connectors`
echo -e `date` " Kafka Connect listener HTTP state: " $curl_status " (waiting for 200)"
if [ $curl_status -eq 200 ] ; then
break
fi
sleep 5
done
echo -e "\n--\n+> Creating Kafka Connector(s)"
/tmp/scripts/create-connectors.sh
sleep infinity
...
/tmp/scripts/create-connectors.sh is a script mounted externally containing a bunch of POST requests using CURL to the Kafka-connect API.
confluent local doesn't interact with a remote Connect cluster, such as one in Kubernetes.
Please refer to the Kafka Connect REST API
You'd connect to it like any other RESTful api running in the cluster (via a Nodeport, or an Ingress/API Gateway for example)
the endpoint mentioned above is unreachable.
Localhost is the physical machine you're typing the commands into, not the remote GKE cluster
Somehow identify when the container is actually ready
Kubernetes health checks are responsible for that
kubectl get services
there are only two ways to create the connector
That's not true. You could additional run Landoop's Kafka Connect UI or Confluent Control Center in your cluster to point and click.
But if you have local config files, you could also write code to interact with the API
Or try and see if you can make a PR for this issue
https://github.com/confluentinc/cp-docker-images/issues/467
Do I need to install an instance of Spring Cloud Data Flow on the master server myself, or is this getting installed "automatically" as part of the deployment?
This isn't quite clear from the description at
http://docs.spring.io/spring-cloud-dataflow-server-kubernetes/docs/current-SNAPSHOT/reference/htmlsingle/#_deploying_streams_on_kubernetes
I've followed the guide, though removed every config for MySQL. Maybe this is required. Though I'm somewhat stuck since it's just not assigning an external IP and I do not see why, how to debug, and whether I missed to install some required component.
Edit:
To clarify, I see a scdf service entry when I run
kubectl get svc
But this service never gets an external IP.
Do I need to install an instance of Spring Cloud Data Flow on the master server myself, or is this getting installed "automatically" as part of the deployment?
Spring Cloud Data Flow server needs to be setup either outside (that knows how to connect to the kubernetes environment) or you can use the Spring Cloud Data Flow server docker image to run inside the kubernetes while the latter approach is better.
Step 6 in the link you posted above runs the SCDF docker image inside the kubernetes cluster:
```
Deploy the Spring Cloud Data Flow Server for Kubernetes using the Docker image and the configuration settings you just modified.
$ kubectl create -f src/etc/kubernetes/scdf-config-kafka.yml
$ kubectl create -f src/etc/kubernetes/scdf-secrets.yml
$ kubectl create -f src/etc/kubernetes/scdf-service.yml
$ kubectl create -f src/etc/kubernetes/scdf-controller.yml
```
MySql is required, that's why it's in the steps.
Spring Cloud Data Flow uses an RDBMS instead of Redis for stream/task
definitions, application registration, and for job repositories.
You can also use any of the other supported RDMBSes.
You can install it using Helm Charts.
https://dataflow.spring.io/docs/installation/kubernetes/helm/
At first install Helm
Then install Spring Cloud Data Flow
helm install --name my-release stable/spring-cloud-data-flow
It will install and config relevant pods such as spring-cloud-dataflow-server, mysql, skipper, rabbitmq, etc.
Also you can customize versions and configurations.
enter image description hereI tried to used the instructions from this link https://github.com/kubernetes/heapster/blob/master/docs/influxdb.md but I was not able to install it. specifically I dont know what this instruction means "Ensure that kubecfg.sh is exported." I dont even know where I can find this I did this sudo find / -name "kubecfg.sh" and I found no results.
moving on to the next step "kubectl create -f deploy/kube-config/influxdb/" when I did this it says kube-system not found I am using latest version of kubernetes version 1.0.1
These instructions are broken can any one provide some instructions on how to install this? I have kubernetes cluster up and running I was able to create and delete pods and so on and default is the only namespace I have when i do kubectl get pods,svc,rc --all-namespaces
Changing kube-system to default in the yaml files is just getting me one step further but I am unable to access the UI and so on. so installing kube-system makes more sense however I dont know how to do it and any instructions on installing influxdb and grafana to get it up and running will be very helpful
I am using latest version of kubernetes version 1.0.1
FYI, the latest version is v1.2.3.
... it says kube-system not found
You can create the kube-system namespace by running
kubectl create namespace kube-system.
Hopefully once you've created the kube-system namespace the rest of the instructions will work.
We had the same issue deploying grafana/influxdb. So we dug into it:
Per https://github.com/kubernetes/heapster/blob/master/docs/influxdb.md since we don’t have an external load balancer, we changed the port type on the grafana service to NodePort which made it accessible at port 30397.
Then looked at the controller configuration here: https://github.com/kubernetes/heapster/blob/master/deploy/kube-config/influxdb/influxdb-grafana-controller.yaml and noticed the comment about using the api-server proxy which we wouldn’t be doing by exposing the NodePort, so we deleted the GF_SERVER_ROOT_URL environment variable from the config. At that point Grafana at least seemed to be running, but it looked like it was having trouble reaching influxdb.
We then changed the datasource to use localhost instead of monitoring-influxd and was able to connect. We're getting data on the cluster usage now, though individual pod data doesn’t seem to be working.