I need to put data from OPC-UA into a topic in kafka.
I tried searching for connector or technology to accomplish this, but I didn't find anything.
You can try to use Visual Logger for OPC: https://onewayautomation.com/visual-logger
Its community edition is free. Can be deployed in Docker container, as well available to run in Windows and Linux.
Related
i have setup Aerospike with all the configuration required for Kakfa and m using confluent local cluster for Kafka and have installed https://www.confluent.io/hub/aerospike/kafka-connect-aerospike-source and have started confluent cluster but still connector is not starting
also i found that there is no jar in share folder of confluent is it still under development ?
It works and is fully generally available but requires extra licensing on the Aerospike side. I would not expect it to work with Community Edition.
Which is the better way to install kafka, zookeeper and schema registry? I already have installed kafka and zookeeper from their official site, but I noticed that on the confluent schema registry package it comes with kafka and zookeeper too, so will I need download the package and only use schema registry? or there is another link where I can download schema registry without the kafka and zookeper part?, thanks in advance
I recommend to install it all from the same place. Confluent offers the Confluent Platform community edition that includes:
Zookeeper
Kafka
Schema Registry
Kafka Connect
REST Proxy
KSQL Server
That way you will avoid any version incompatibilities. I followed the instructions from Confluent to get the entire suite installed and running.
Together with the command line tool you will be able to start and stop all necessary services with only one command:
confluent local start
They also offer Docker installation which might be of interest to you as well.
Can kafka server be installed with dotnet?
I'm finding it goes with JVM. Without JVM can it work in dotnet tech stack?
Kafka Server requires the JVM, yes. It cannot run without the JVM.
Kafka Clients support multiple languages, including .NET
Currently microsoft kafka client is not actively developed/supported. Instead the author recommends to use rdkafka-dotnet It has better documentation and is easy to use
https://github.com/Microsoft/CSharpClient-for-Kafka
https://github.com/ah-/rdkafka-dotnet
But confluent has a great .net kafka library. It's easy to implement consumer groups and highly configurable.
https://www.nuget.org/packages/Confluent.Kafka/
I want to activate the Kafka Spark pipeline for the Thingsboard platform (community edition).
As per the mentioned Stack question "Couldn't able to find plugins in ThingsBoard 2.0.3 Home screen"
It's been said that we can do it via Rule chains itself since the plugin section has been removed, but I am not able to understand how to configure it using rule chains. I am not able to get the complete documentation to configure Kafka via rule chains. So need help on that.
I figured it out. By using this link it can be done easily "https://thingsboard.io/docs/samples/analytics/kafka-streams/"
The thing is that using the Thingsboard CE we can get data into Kafka-topic. However, to fetch data from Kafka you will need to have TB Professional Edition integration.
The alternate option to Thingsboard PE is to write your own REST API script to push the insights back to ThingsBoard.
I'm new in Confluent world, and I know how to start kafka, zookeepers from confluent, but it's not that what I need.
I have already 3 kafka nodes and 2 zookeepers installed by Ambari. Afterwards I downloaded 3.0.0 version of Confluent and now I want to connect Confluent with already running Kafka and zookeeper. I don't want to instance new kafka server or zookeeper server which confluent is giving.
Does anyone has an idea how to accomplish that, what to actually run from Confluent and what to change.
By now I was only chaning files in ./etc/kafka or ./etc/zookeeper which are in Confluent dir. Thank you!
clarify some basics about Confluent and how manage communication between Confluent and Kafka
First things first, there is no single application called "Confluent" that can be started all on its own.
There's is nothing to configure for Kafka or Zookeeper. The Confluent Platform doesn't add anything on top of the existing Apache Kafka you have (presumably, via Hortonworks or Cloudera).
In fact, those companies do add patches to Kafka that would be slightly different than the base Apache versions you would get from Confluent.
That being said, if you read through each of extra services that Confluent provides, you'll notice either a Zookeeper or a Bootstrap server configuration option. Fill out those fields, start the respective services, and you're good to go.
what to actually run from Confluent
Look in the bin directory, you can find all the start scripts. From the comments, looks like you're trying to use Connect Distributed (which should already be installed by any recent Kafka installation, it's not Confluent specific), and Schema Registry. You'll have to be more specific about the errors that you get, but the config files are all in the etc path.
Unless you're using KSQL, REST Proxy or Control Center, there's not much to run because, as mentioned, Kafka Connect is included with the base Apache Kafka project and Hortonworks is maintaining their own Schema Registry project
2 zookeepers installed by Ambari
This is a highly non-recommended setup. Please install an odd number of Zookeepers. 3 or 5, preferably