Every search returns the Confluent pages for Kafka REST proxy.
Is there any other way to setup with the vanilla application or any open source ones?
I think the community licensed ones are free to you use. Please check the following page https://docs.confluent.io/platform/current/installation/docker/image-reference.html
Also you can consider this : https://strimzi.io/docs/bridge/latest/#api_reference-bridge if you need only a bridge for integration.
Related
I would like to use a HTTP proxy on top of Kafka. I see two projects with same purpose :
https://github.com/confluentinc/kafka-rest
https://github.com/strimzi/strimzi-kafka-bridge
I use strimzi operator to spin up Kafka on Kubernetes.
Both are open source
Both can be used in commercial - self hosted cloud applications.
Both provide REST proxy on top of Kafka
How do they differ ? When to use which one ?
The licensing and nature of the projects are a bit different. Strimzi is an independent project under the Cloud Native Computing Foundation. And the Strimzi Bridge (and all other Strimzi components) is licensed under Apache License 2.0 as a recognized open-source license. The Confluent REST Proxy on the other hand uses a proprietary Confluent Community license - whether that works for you or not depends on how you use it.
On the other hand, if I remember correctly, the Confluent REST Proxy has more features around things such as topic and cluster management etc. For many use cases, both will do the job. But it is good to check what exact features you need as they might not support the same things.
The Strimzi Bridge is also directly supported by the Strimzi Operator. So if you already use it, it might be easier for you to stick with it. Similarly, if you would be already using the Confluent Platform, it might be easier to go with the Confluent REST proxy.
Disclaimer: I'm one of the Strimzi maintainers.
I have usecase to start using Kafka and was looking for opensource free (production) kafka.
When check Confluent 7.1 platform looks suitable as it has zookeeper / kafka / schema registry / kafka UI bundled together.
Before deciding to go ahead with it just want to check if the Confluent Platform 7.1 is free and open source? Am I required to purchase licensing or paid support?
The Confluent Community License covers several components of Confluent Platform, including KSQLDB, the Schema Registry, REST Proxy, and various Kafka Connect plugins. Confluent Control Center (what you call Kafka UI) is only available on a trial basis, outside of which requires Enterprise license payment.
Majority of Confluent Platform individual components are "source-available", and free with limitations. Many of the plugin features like RBAC, Tiered Storage, Cluster Linking, and server-side Kafka record Schema Validation require payment. This is an Enterprise license and also includes Control Center, on-call Support, and several other connectors.
Apache Kafka, it's clients, and Zookeeper are Apache 2.0 licensed.
If you want a completely Apache 2.0 stack, you can replace Confluent Schema Registry with Apicurio and replace Control Center with various Kafka GUI projects that exist on Github, such as AKHQ or CMAK
I'm about to start deploying to production a couple of Kafka cluster in 2 different DCs. My main use is for replication using MirrorMaker. To continuously stream/replicate ElasticSearch and Postgres between DCs in order to have a (near) real-time backup and failover.
What I can't get my head around is this simplest question: should I use Confluent or apache Kafka?
I can see that Confluent adds many niceties but what I don't get it: why would someone pick plain Apache Kafka then? I've seen this answer and it seems clear: "pick Confluent, has way more stuff".
As answered in linked post, you can add whatever external processes you want to Apache Kafka.
Note: You are not picking either or, you are always picking Apache Kafka. Confluent Platform adds on top of, similar to Cloudera's Data Platform, as an alternative consideration.
If you want to connect Elasticsearch and Postgres (via JDBC), both of those connectors are Open-Source under the Confluent Community License, so that would be one potential reason for not using Confluent products.
Other reason: Do you need the "more stuff"? Are you able to get support from elsewhere? For example, AWS support on MSK or IBM Streams or Azure EventHub are not using Confluent Platform (because it's against the above license)
MirrorMaker and MirrorMaker2 are both under the Apache License, so they have no such usage / redistribution restrictions.
should I use Confluent or apache Kafka?
When deciding on deploying a vanilla Apache or a commercially supported product you should think about the O&M (operation and maintenance) timeline and what you gain and lose. Whatever you choose will be very difficult to replace once in production.
I'll also agree with #OneCricketeer's answer.
Do you need the "more stuff"?
I work as a professional services consultant with some Apache products. My advice is keep your stack (whatever it is) as simple as possible. So if you don't need the additional tools and functionality of Confluent, don't use them. It's how they make the product "sticky" (re: vendor lock-in).
Vanilla Apache Kafka
Pro No vendor lock-in or dependencies
Pro Faster updates and feature development
Con No nice dashboards
Con Harder to secure
Confluent
Pro Commercial support and professional services available
Pro More stable with fast and easy security patches
Pro Nice dashboard and management tools
Pro Easier to properly secure
Con Expensive
Con Expect vendor lock-in and frequent up-sells
My Opinion
If you have money to spare and this will be a critical piece of infrastructure I'd recommend buying through Confluent. If you try to avoid paying for them, you'll have to hire someone (expensive) who knows it anyway and you'll have to deal with the patching nightmare of open source projects.
If this is something you want to kick the tires on, can allow for downtime, or think you'll replace in 2 years, I'd just use the Apache Kafka with one of the open source dashboards.
I want to activate the Kafka Spark pipeline for the Thingsboard platform (community edition).
As per the mentioned Stack question "Couldn't able to find plugins in ThingsBoard 2.0.3 Home screen"
It's been said that we can do it via Rule chains itself since the plugin section has been removed, but I am not able to understand how to configure it using rule chains. I am not able to get the complete documentation to configure Kafka via rule chains. So need help on that.
I figured it out. By using this link it can be done easily "https://thingsboard.io/docs/samples/analytics/kafka-streams/"
The thing is that using the Thingsboard CE we can get data into Kafka-topic. However, to fetch data from Kafka you will need to have TB Professional Edition integration.
The alternate option to Thingsboard PE is to write your own REST API script to push the insights back to ThingsBoard.
Can I create and break cross-cluster replication links in some way other than the web console?
Web console is just UI for Couchbase REST API, therefore you can try to create CLI utility that manipulates HTTP requests like "controller/createReplication".
Unfortunately neither official tools nor numerous sdk support this feature.
Docs about the Rest API can be found here:
Managing Cross Data Center Replication (XDCR)
http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-admin-restapi-xdcr.html
Creating replications:
http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-admin-restapi-xdcr-create-repl.html