We have a microservice architecture and we want to publish events into Kafka topics to then be consumed by Salesforce. What is the best approach to do this ? Is there any example Apex code that do that ?
Best Regards.
Please note that most Salesforce Kafka connectors are using the Salesforce Streaming API for which following limits apply and might be very limiting:
https://developer.salesforce.com/docs/atlas.en-us.188.0.api_streaming.meta/api_streaming/limits.htm
Kafka Salesforce connector paid version is available on Confluent Connector Hub.As of now ,Kafka Salesforce connector is not available on any Open Source Platforms but some source code is available on GitHub. You can use and customized the code according to your requirement.
https://github.com/Entanet/kafka-salesforce-connect
https://github.com/nodefluent/salesforce-kafka-connect
https://javalibs.com/artifact/com.github.jcustenborder.kafka.connect/kafka-connect-salesforce
Related
I'm currently using Gatling & a home grown solution, but I was wondering if Kafka offer anything?
Yes, Kafka has both kafka-{producer,consumer}-perf-test.sh included
I am trying to find options for Open Source Kafka writing directly to Azure Data Lake storage Gen2 . It seems I have few options and mostly circling around Confluent like below :
Use Confluent Cloud with Apache Kafka - Need to Subscribe with Confluent and pay charges (Confluent Cloud with ADLS
Use Azure VM with Confluent Hub and Install Confluent Platform
At present I am not wiling to pay Confluent licensing and not want to test with confluent package (more and more wrappers and hoops around)
Any option to use Open source Kafka directly to write data to ADLS Gen2 ? If yes how can we achieve this any useful information to share ?
Firstly, Kafka Connect is Apache2 licensed product and an open-platform consisting of plugins; Confluent Platform/Cloud is not a requirement to use it. You can download the Azure connector as a ZIP file and install it like any other
However, it is at Confluent's (or any developer) discretion to provide a paid license agreement for their software and any support, and there might otherwise be a limited trial period where you can use the plugin for some time.
That being said, you do not "need" Confluent Platform, and there are no "hoops" to using it if you did because it only adds extras to Apache Kafka+Zookeeper, it is not its own thing (you can use your existing Kafka installation with the other Confluent products)
Regarding other open-source things. StackOverflow is not the place for software recommendations or seeking tools/libraries. You can use Spark/Flink/Nifi, though, I'm sure to reimplement a similar pipeline as Kafka Connect, or you can write your own Kafka Connector based on the open-source kafka-connect-storage-cloud project that is used as a base for S3, GCS, and Azure, AFAIK.
There is Apache Camel Connectors, which has an Azure Datalake connector for sending and receiving data. (sink and source) Check this out: https://camel.apache.org/camel-kafka-connector/latest/connectors/camel-azure-storage-datalake-kafka-sink-connector.html
This is a free solution that doesnt require Confluent licenses or technologies to be used.
I wonder if I can use Kafka Streams in production.
Is it really open source? Or Should we buy some license?
I am looking up documentation Kafka Streams and Licence FAQs but it isn't clear for me.
Kafka Streams is Apache 2.0 Licensed, not sure why you're looking at Confluent License pages
Its source code exists with the Kafka Broker, JVM clients, and Kafka Connect sources
Is it possible to capture IBM MQ data with Kafka-Cloudera?
The confluent company offers connectors to capture IBM MQ data, but I'm not sure if I can do the same with Kafka-Cloudera.
Yes.
Kafka Connect is not a framework specific to Confluent or Cloudera. It is built into all Apache Kafka offerings.
If Confluent Platform includes a specific connector as part of the OSS offering, for which you can individually download and use the connector, then that's a separate issue.
i have two question
1) I want to use Kafka with Google cloud Dataflow Pipeline program. in my pipeline program I want to read data from kafka is it possible?
2) I created Instance with BigQuery enabled now i want to enable Pubsub how can i do ?
(1) Ad mentioned by Raghu, support for writing to/reading from Kafka was added to Apache Beam in mid-2016 with the KafkaIO package. You can check the package's documentation[1] to see how to use it.
(2) I'm not quite sure what you mean. Can you provide more details?
[1] https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/kafka/KafkaIO.html
Kafka support was added to Dataflow (and Apache Beam) in mid 2016. You can read and write to Kafka streaming pipelines. See JavaDoc for KafkaIO in Apache Beam.
(2) As of April 27, 2015, you can enable Cloud Pub/Sub API as follows:
Go to your project page on the Developer Console
Click APIs & auth -> APIs
Click More within Google Cloud APIs
Click Cloud Pub/Sub API
Click Enable API