Is it possible to capture IBM MQ data with Kafka-Cloudera?
The confluent company offers connectors to capture IBM MQ data, but I'm not sure if I can do the same with Kafka-Cloudera.
Yes.
Kafka Connect is not a framework specific to Confluent or Cloudera. It is built into all Apache Kafka offerings.
If Confluent Platform includes a specific connector as part of the OSS offering, for which you can individually download and use the connector, then that's a separate issue.
Related
I have usecase to start using Kafka and was looking for opensource free (production) kafka.
When check Confluent 7.1 platform looks suitable as it has zookeeper / kafka / schema registry / kafka UI bundled together.
Before deciding to go ahead with it just want to check if the Confluent Platform 7.1 is free and open source? Am I required to purchase licensing or paid support?
The Confluent Community License covers several components of Confluent Platform, including KSQLDB, the Schema Registry, REST Proxy, and various Kafka Connect plugins. Confluent Control Center (what you call Kafka UI) is only available on a trial basis, outside of which requires Enterprise license payment.
Majority of Confluent Platform individual components are "source-available", and free with limitations. Many of the plugin features like RBAC, Tiered Storage, Cluster Linking, and server-side Kafka record Schema Validation require payment. This is an Enterprise license and also includes Control Center, on-call Support, and several other connectors.
Apache Kafka, it's clients, and Zookeeper are Apache 2.0 licensed.
If you want a completely Apache 2.0 stack, you can replace Confluent Schema Registry with Apicurio and replace Control Center with various Kafka GUI projects that exist on Github, such as AKHQ or CMAK
I'm trying to use Debezium platform to make a Kafka-cdc.But I was confused.
What is really difference between Confluent platform and Debezium?
Confluent (https://www.confluent.io/) is a platform which mainly integrate Apache-Kafka (https://kafka.apache.org/) and its ecosystems. So let say the basic Confluent platform has Zookeeper, apache kafka, KSql and thier Control Center.
Debezium is another platform to focus Database Streaming.
So you think Confluent is the general Streaming, and Debezium actually has a connector https://debezium.io/documentation/reference/stable/connectors/index.html that can be integreted to Confluent like in https://www.confluent.io/hub/debezium/debezium-connector-postgresql
At the time of writing, Confluent Platform does not have any CDC connectors, and you don't really need it. Apache Kafka Connect that is bundled as part of the Confluent Platform is all that's needed, and can be downloaded directly from Apache Kafka site instead.
Debezium is built on Kafka Connect API, and provided as a plug in.
I have a Kafka cluster that I work with which is managed by my team and runs on Kubernetes. We want to install the Kafka connect via helm into our cluster to work with our Kafka. This Kafka we are running is NOT the confluent platform Kafka. Is there a good way to do this? I was wondering if this would work cp-helm-charts. Will using the confluentinc Kafka connect container be compatible with my Kafka cluster that is on non-confluent platform?
Kafka Connect has never been labelled as a Confluent Platform exclusive product.
The Framework is entirely Apache 2.0 Licensed and Open Source.
Similarly, "Confluent Platform Kafka" is just Apache Kafka
Any Opensource tool to monitor confluent Kafka? Most of the opensource tools available are specific to Apache Kafka but not for Confluent Kafka.
we want to monitor atleast the connectors, streams and cluster health
The Kafka that is distributed in the Confluent Platform is Apache Kafka. There really is no such thing as "Confluent Kafka". Any tools that work with the latest version of Apache Kafka (including Kafka Connect and Kafka Streams) will work with the same versions of Kafka included with Confluent Open Source.
Confluent 3.3 includes Apache Kafka 0.11
Confluent 3.2 includes Apache Kafka 0.10.2
Confluent 3.1 includes Apache Kafka 0.10.1
Confluent 3.0 includes Apache Kafka 0.10.0
Confluent 2.0 includes Apache Kafka 0.9
Confluent 1.0 includes Apache Kafka 0.8.2
Note: Confluent Enterprise includes its own monitoring and management GUI called Control Center. Control Center is a separate process so the Apache Kafka is still the same as the open source version.
You can use updated version of KafkaOffsetMonitor. It supports SSL/TLS and Kerbros. Also uses Kafka 1.1.0 library.
You should be able to use kafka-monitor for monitoring your cluster's health as well as Burrow and KafkaOffsetMonitor for monitoring your consumer application lag. Also, you should definitely use something like jmxtrans for collecting your Kafka broker metrics.
I'm trying to install Confluent over HDP for Kafka Streams which think may not be possible could you people suggest me what to do
It seems like you are trying to install Confluent Platform using Ambari. If that's the case then you want to use a custom service install or you will need to wait for HDP to support Kafka 0.10 which includes Kafka Streams. The alternative route is to install Confluent Platform in parallel with HDP and just not activate the Kafka version that ships with HDP. This will require that you monitor and manage the Confluent Platform independently.
for Kafka Streams
Kafka Streams is a client library that can use with any Kafka cluster above version 0.10. You do not specifically require "Confluent Platform"
That being said, if the version that HDP/Cloudera provide of Kafka is not useful for you, then you should provision external infrastructure for it.