Setting up a Hyperledger Fabric Kafka Implementation - apache-kafka

I have been recently working on a custom implementation of hyperledger fabric with kafka implementation to no avail.
I was wondering if anyone has done this successfully and if so could you share your configtx.yaml, docker-compose.yaml(and any base files), and crypto-config.yaml.
Also looking for any extra articles anyone might have found on this topic that may help me progress.

Yes, I have done that. Configured multiple orderer(kafka-based) for one network. Here is the code: https://github.com/lissdy/fabric-composer/tree/master/fabric-network/kafka-orderers

Related

Spring Cloud Integration test, embedded kafka vs testcontainers

I have a Spring cloud stream application which I need to make an integration test for (to be specific using cucumber). The application communicate with other services using Kafka message broker. From what I know I could make this work using either a kafka testcontainers or using spring provided embedded kafka. But what I don't know is which one would be the best solution so are there anything that the testcontainer could do but embedded can't or the other way around? (use cases or example would be appreciate!)
p.s This integration should be able to run on ci/cd pipeline.
It is called embedded for a reason. It really can be only accessed from the process which spawned it. With Testcontainers you really can reuse existing container and have access to it from the other process. But that's probably to exotic.
I guess with properly configured Testcontainers you can reach as much as possible similarities with the prod you'd deploy your solution. The embedded Kafka might be limited in some areas, e.g. SSL configuration or so.

Respository code deployment to multiple cluster with different configuration

Issue/References Needed
I am looking for a setup where a repository code could be deployed to multiple clusters and all cluster are working on different configurations, I want to understand the possibility of going this using GitHub.
I tried looking for some solution on the internet but was unable to find any referenced article which can support this. Any references could be quite helpful.
Thanks in advance!
I wanted to create multiple deployments from the same container image in the same namespace with different configurations.
These Documentations helped me:
Doc 1
Doc 2

UI tools to connect to strimzi kafka cluster

Looking for UI tools to connect to Strimzi kafka cluster to get visibility to kafka topics, read messages within topics, broker and partition details and ability to connect with or without SSL/SASL connections.I have already tried using kafka tool and facing issues with it hence looking for an alternative). Kindly suggest some UI tools for the same ? ( similar to confluent center/kafka tool) which are either free or with minimal cost.
There are no UI tools for now but a new issue was started about it. I would follow it https://github.com/strimzi/strimzi-kafka-operator/issues/3287
There is a project in early stage of developement: https://github.com/strimzi/strimzi-ui
Strimzi UI provides a way for managing Strimzi and Kafka clusters (+
other components) deployed by it using a graphical user interface.
But unforunately in the moment of writing:
This UI is currently not in a state where it can be used. Is it still
early on in it's development but we hope to have something usable very
soon!
So, keep an eye on it.

Kafka - Confluent Hub - Exploit only part of it

I already saw a similar question in SO, but not clearly answer my doubts.
We have different Kafka clusters and lot of exploitation operational habits around it. We have our way to start/stop the cluster, lots of exploit scripts that help maintain the cluster etc..
Now we would like to use Kafka connect connectors for new needs, but from what I saw, Kafka connect is extremely coupled to confluent-hub.
It's like I can't even use the connectors without having to install a full operational confluent-hub.
This makes it very difficult for us to use Kafka connect connectors, I understand that confluent-hub might be a framework that help running those connectors, but it's like we can't even use a dissociated Kafka cluster ( a one not exploited by confluent-hub..).
But maybe I miss something..
Do you know if there is any way to use properly Kafka connectors on a already existing Kafka cluster ( completely independent from confluent-hub) ?
EDITED :
It's more a question regarding the high coupled behaviour between confluent-hub and Kafka-connect. All the features that comes with Kafka connect ( distributed workers to handle different fail over scenarios, etc..) are not usable without confluent-hub, thus a "need" to have Kafka cluster running exclusively via confluent-hub, which is not an easy task when you already have an existing big Kafka cluster with lots of OPS habits on it.
Kafka Connect is part of Apache Kafka. It's a pluggable framework for streaming integration between systems in and out of Kafka.
To use Kafka Connect you need connectors for the specific technology with which you want to integrate. For example, S3 sink, Elasticsearch sink, JDBC source or sink, and so on.
The connector API is part of Apache Kafka, and available for anyone who wants to develop a connector.
Connectors are written by various people and organisations, and available in various different ways. How you obtain a connector depends on which connector you want, how its licensed, and how the author has made it available for distribution. It could be you go to github, clone the repo and build the JAR. It could be you can download the JAR directly.
All that Confluent Hub does is make lots of these connectors available for you in one place, easily searchable, and with an optional CLI tool that will install them for you.
Do you have to use Confluent Hub? No, not at all. Might it make your life easier in locating connectors that you want to use, and make it easier to install them? Hopefully :)
Disclaimer: I work for Confluent.

Problem with embedding activemq to jboss application server

I'm totally a newbie in Java Enterprise and I have a lot stuff yet to learn. Right now I'm working which involves JMS using JBoss and ActiveMQ. An application sends messages to queues in JBoss and my goal is to access those messages using any message broker (in this case I tried to use ActiveMQ). So I think it's better for me to embed ActiveMQ to JBoss. I used the link below as the guidance :
http://activemq.apache.org/integrating-apache-activemq-with-jboss.html
I followed every single steps in the guide except that I used JBoss 4.2.3 and ActiveMQ 5.1.1. If this problem results from the different version of the tools, I think I can't help it out because other versions seem does not work on my machine. When I run JBoss to test whether the embedding is working or not, I can't see anything running on port localhost:61616 which is the default port for ActiveMQ, although JBoss seems run well. My question is :
Is any one know how to fix this? Or has anyone ever experienced such problem?
Is there any way to access queue in Jboss?
If this question is pretty ambiguous or need more details, let me know. Thanks in advance for any help.
I'm quite sure the embedded broker is started (transport vm://localhost) which results in a working JMS broker for that VM. But this broker isn't reachable from outside (via tcp or anything else)
Do you see any error in the log when launching jboss? I managed to make it work with Jboss 4.2 and ActiveMQ 5.4, and once integrated it only gives you a small hint that is working correctly with some info message saying "broker started in port ..." .
Also you can try to use the sample producers and receivers in ActiveMQ to test if the queue is working correctly.
If you still have problems I can try to help