Would Kafka need to be installed on the consumer cluster?
Presently the same cluster YAML configuration is:
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
name: sample-topic
spec:
type: bindings.kafka
version: v1
metadata:
# Kafka broker connection setting
- name: brokers
value: dapr-kafka.kafka:9092
# consumer configuration: topic and consumer group
- name: topics
value: sample
- name: consumerGroup
value: group1
# publisher configuration: topic
- name: publishTopic
value: sample
- name: authRequired
value: "false"
On different clusters, does each cluster require only either "name: publishTopic" or "name: consumerGroup" and not the other?
I'm not familiar with Dapr, but Kafka does not need installed in k8s, or any specific location. Your only requirement should be client connectivity to that bootstrap-servers list.
According to the Kafka Binding spec, consumerGroup is for incoming events, and publishTopic is for outgoing, so two different use cases, although one app should be able to have both event types. If the app only uses incoming or outgoing events, then use the appropriate binding for that case.
Related
TL;DR - Argo Events Kafka eventsource triggers do not currently parse headers of consumed Kafka message, which is needed to enable distributed tracing. I submitted a feature request (here) - if you face the same problem please upvote, and curious if anyone figured out a workaround.
====================================
Context
Common pattern of Argo Workflows we deploy are Kafka event-driven, asynchronous distributed workloads, e.g.:
Service "A" Kafka producer that emits message to topic
Argo Events eventsource Kafka trigger listening to that topic
Argo Workflow gets triggered, and post-processing...
... service "B" Kafka producer at end of workflow emits that work is done.
To monitor the entire system for user-centric metrics "how long did it take & where are the bottle necks", I'm looking to instrument distributed tracing from service "A" to service "B". We use Datadog as aggregator, with dd-trace.
Pattern I've seen is manual propagation of trace ctx via Kafka headers - by injecting headers to Kafka messages before emitting (similar to HTTP headers, with parent trace metadata), and receiving Consumer once done processing the message will then add child_span to that parent_span received from upstream.
ex) of above: https://newrelic.com/blog/how-to-relic/distributed-tracing-with-kafka
Issue
Argo-Events Kafka event source trigger does not parse any headers, only passing the body json for downstream Workflow to use at eventData.Body.
[source code]
Simplified views of my Argo Eventsource -> Trigger -> Workflow:
# eventsource/my-kafka-eventsource.yaml
apiVersion: argoproj.io/v1alpha1
kind: EventSource
spec:
kafka:
my-kafka-eventsource:
topic: <my-topic>
version: "2.5.0"
# sensors/trigger-my-workflow.yaml
apiVersion: argoproj.io/v1alpha1
kind: Sensor
spec:
dependencies:
- name: my-kafka-eventsource-dep
eventSourceName: my-kafka-eventsource
eventName: my-kafka-eventsource
triggers:
- template:
name: start-my-workflow
k8s:
operation: create
source:
resource:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
spec:
entrypoint: my-sick-workflow
arguments:
parameters:
- name: proto_message
value: needs to be overriden
# I would like to be able to add this
- name: msg_headers
value: needs to be overriden
templates:
- name: my-sick-workflow
dag:
tasks:
- name: my-sick-workflow
templateRef:
name: my-sick-workflow
template: my-sick-workflow
parameters:
# content/body of consumed message
- src:
dependencyName: my-kafka-eventsource-dep
dataKey: body
dest: spec.arguments.parameters.0.value
# I would like to do this - get msg.headers() if exists.
- src:
dependencyName: my-kafka-eventsource-dep
dataKey: headers
dest: spec.arguments.parameters.1.value
# templates/my-sick-workflow.yaml
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
spec:
templates:
- name: my-sick-workflow
container:
image: <image>
command: [ "python", "/main.py" ]
# I want to add the 2nd arg - msg_headers - here
args: [ "{{workflow.parameters.proto_message}}", "{{workflow.parameters.msg_headers}}" ]
# so that in my Workflow Dag step source code,
# I can access headers of Kafka msg from upstream by....
# body=sys.argv[1], headers=sys.argv[2]
Confluent-Kafka API docs on accessing message headers: [doc]
Q's
Has anyone found a workaround on passing tracing context from upstream to downstream service that travels between Kafka Producer<>Argo Events?
I considered changing my Argo-Workflows sensor trigger to HTTP trigger accepting payloads, by a new Kafka consumer listening for the message that is currently triggering my Argo Workflow --> then forward HTTP payload with parent trace metadata in headers.
it's anti-pattern to rest of my workflows, so I would like to avoid if there's a simpler solution.
As you pointed out, the only real workaround without forking some part of Argo Events, or implementing your own Source/Sensor yourself would be to use a Kafka Consumer (or Kafka Connect), and call a WebHook EventSource (or another, which can extract the information you need).
A graph is always better than the last sentences, so here is what I would like to do :
To sum up:
I want to have a Redis master instance outside (or inside, this is not relevant here) my K8S cluster
I want to have a Redis slave instance per node replicating the master instance
I want that when removing a node, the Redis slave pod gets unregistered from master
I want that when adding a node, a Redis slave pod is added to the node and registered to the master
I want all pods in one node to consume only the data of the local Redis slave (easy part I think)
Why do I want such an architecture?
I want to take advantage of Redis master/slave replication to avoid dealing with cache invalidation myself
I want to have ultra-low latency calls to Redis cache, so having one slave per node is the best I can get (calling on local host network)
Is it possible to automate such deployments, using Helm for instance? Are there domcumentation resources to make such an architecture with clean dynamic master/slave binding/unbinding?
And most of all, is this architecture a good idea for what I want to do? Is there any alternative that could be as fast?
i remember we had a discussion on this topic previously here, no worries adding more here.
Read more about the Redis helm chart : https://github.com/bitnami/charts/tree/master/bitnami/redis#choose-between-redis-helm-chart-and-redis-cluster-helm-chart
You should also be asking the question of how my application will be
connecting to POD on same Node without using the service of Redis.
For that, you can use the `environment variables and expose them to application POD.
Something like :
env:
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
It will give you the value of Node IP on which the POD is running, then you can use that IP to connect to DeamonSet (Redis slave if you are running).
You can read more at : https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/
Is it possible to automate such deployments, using Helm for instance?
Yes, you can write down your own Helm chart and deploy the generated YAML manifest.
And most of all, is this architecture a good idea for what I want to
do? Is there any alternative that could be as fast?
If you think then it is a good idea, as per my consideration this could create the $$$ issue & higher cluster resources usage.
What if you are running the 200 nodes on each you will be running the slave of Redis ? Which might consume resources on each node and add cost to your infra.
OR
if you are planning for specific deployment
Your above suggestion is also good, but still, if you are planning to use the Redis with only Specific deployment you can use the sidecar pattern also and connect multiple Redis together using configuration.
apiVersion: v1
kind: Service
metadata:
name: web
labels:
app: web
spec:
ports:
- port: 80
name: redis
targetPort: 5000
selector:
app: web
type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
selector:
matchLabels:
app: web
replicas: 3
template:
metadata:
labels:
app: web
spec:
containers:
- name: redis
image: redis
ports:
- containerPort: 6379
name: redis
protocol: TCP
- name: web-app
image: web-app
env:
- name: "REDIS_HOST"
value: "localhost"
I've got sequences working in my Knative environment. We're trying to configure and confirm the DLQ/Dead Letter Sink works so we can write tests and things against sequences. I can't for the life of my get Knative to send anything to the Dead Letter Sink. I've approached this two ways, the first was setting up a Broker, Trigger, Services and a Sequence. I've defined in the Broker a service to use for the DLQ. I then setup a service in the sequence to intentionally returned a non-200 status. When I view the logs for the channel dispatcher in the knative-eventing namespace, I believe what I read is that it thinks there was a failure.
I read some things about the default MT Broker maybe not handling DLQ correctly so then I installed Kafka. Got that all working and essentially, it appears to do the same thing.
I started to wonder, ok, maybe within a sequence you can't do DLQ. After all the documentation only talks about DLQ with subscriptions and brokers, and maybe Knative believes that the message was successfully delivered from the broker to the sequence, even if it dies within the sequence. So I manually setup and channel and a subscription and sent the data straight to the channel and again, what I got was essentially the same thing, which is:
The sequence will stop on whatever step doesn't return a 2XX status code, but nothing gets sent to the DLQ. I even made the subscription go straight to the service (instead of a sequence) and that service returned a 500 and still nothing to the DLQ.
The log item below is from the channel dispatcher pod running in the knative-eventing namespace. It basically looks the same with In memory channel or Kafka, i.e. expected 2xx got 500.
{"level":"info","ts":"2021-11-30T16:01:05.313Z","logger":"kafkachannel-dispatcher","caller":"consumer/consumer_handler.go:162","msg":"Failure while handling a message","knative.dev/pod":"kafka-ch-dispatcher-5bb8f84976-rpd87","knative.dev/controller":"knative.dev.eventing-kafka.pkg.channel.consolidated.reconciler.dispatcher.Reconciler","knative.dev/kind":"messaging.knative.dev.KafkaChannel","knative.dev/traceid":"957c394a-1636-44ad-b024-fb0dde9c8440","knative.dev/key":"kafka/test-sequence-kn-sequence-0","topic":"knative-messaging-kafka.kafka.test-sequence-kn-sequence-0","partition":0,"offset":4,"error":"unable to complete request to http://cetf.kafka.svc.cluster.local: unexpected HTTP response, expected 2xx, got 500"}
{"level":"warn","ts":"2021-11-30T16:01:05.313Z","logger":"kafkachannel-dispatcher","caller":"dispatcher/dispatcher.go:314","msg":"Error in consumer group","knative.dev/pod":"kafka-ch-dispatcher-5bb8f84976-rpd87","error":"unable to complete request to http://cetf.kafka.svc.cluster.local: unexpected HTTP response, expected 2xx, got 500"}
Notes on setup. I deployed literally everything to the same namespace for testing. I followed the guide here essentially to setup my broker when doing the broker/trigger and for deploying Kafka. My broker looked like this:
apiVersion: eventing.knative.dev/v1
kind: Broker
metadata:
annotations:
# case-sensitive
eventing.knative.dev/broker.class: Kafka
name: default
namespace: kafka
spec:
# Configuration specific to this broker.
config:
apiVersion: v1
kind: ConfigMap
name: kafka-broker-config
namespace: knative-eventing
# Optional dead letter sink, you can specify either:
# - deadLetterSink.ref, which is a reference to a Callable
# - deadLetterSink.uri, which is an absolute URI to a Callable (It can potentially be
out of the Kubernetes cluster)
delivery:
deadLetterSink:
ref:
apiVersion: serving.knative.dev/v1
kind: Service
name: dlq
namespace: kafka
When I manually created the subscription and channel my subscription looked like this:
apiVersion: messaging.knative.dev/v1
kind: Subscription
metadata:
name: test-sub # Name of the Subscription.
namespace: kafka
spec:
channel:
apiVersion: messaging.knative.dev/v1beta1
kind: KafkaChannel
name: test-channel
delivery:
deadLetterSink:
backoffDelay: PT1S
backoffPolicy: linear
retry: 1
ref:
apiVersion: serving.knative.dev/v1
kind: Service
name: dlq
namespace: kafka
subscriber:
ref:
apiVersion: serving.knative.dev/v1
kind: Service
name: cetf
No matter what I do I NEVER see the dlq pod spin up. I've adjusted retry stuff, waited and waited, used the default channel/broker, Kafka, etc. I simply cannot see the pod ever run. Is there something I'm missing, what on earth could be wrong? I can set the subscriber to be a junk URI and then the DLQ pod spins up, but shouldn't it also spin up if the service it sends events to returns error codes?
Can anyone provide a couple of very basic YAML files to deploy the simplest version of a working DLQ to test with?
there was some issue with Dead Letter Sinks not being propagated at pre-GA releases. Can you make sure you are using Knative 1.0?
This is working for me as expected using the inmemory channel:
https://gist.github.com/odacremolbap/f6ce029caf4fa6fbb3cc1e829f188788
curl producing cloud events to a broker
broker with DLS configured to an event-display
event display service as DLS receiver
trigger from broker to a replier service
replier service (can ack and nack depending on the incoming event)
I never found an example of this in the docs, but the API docs for the SequenceStep does show a delivery property. Which, when assigned, uses the DLQ.
steps:
- ref:
apiVersion: serving.knative.dev/v1
kind: Service
name: service-step
delivery:
# DLS to an event-display service
deadLetterSink:
ref:
apiVersion: serving.knative.dev/v1
kind: Service
name: dlq-service
namespace: ns-name
It seems odd to have to specify a delivery for EVERY step and not just the sequence as a whole.
I am using Kafka with strimzi operator, I created a Kafka cluster and and also deployed Kafka connect using yml file. But after this I am totally blank what to do next . I read that Kafka connect is used to copy data from a source to Kafka cluster or from Kafka cluster to another destination.
I want to use Kafka connect to copy the data from a file to Kafka cluster's any topic.
Can any one please help me how can I do that I am sharing the yml file using which I created my Kafka connect cluster.
apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaConnect
metadata:
name: my-connect-cluster
# annotations:
# # use-connector-resources configures this KafkaConnect
# # to use KafkaConnector resources to avoid
# # needing to call the Connect REST API directly
# strimzi.io/use-connector-resources: "true"
spec:
version: 2.6.0
replicas: 1
bootstrapServers: my-cluster-kafka-bootstrap:9093
tls:
trustedCertificates:
- secretName: my-cluster-cluster-ca-cert
certificate: ca.crt
config:
group.id: connect-cluster
offset.storage.topic: connect-cluster-offsets
config.storage.topic: connect-cluster-configs
status.storage.topic: connect-cluster-status
#kubeclt create -f kafka-connect.yml -n strimzi
After that pod for Kafka connect is in running status ,I don't know what to do next. Please help me.
Kafka Connect exposes a REST API, so you need to expose that HTTP endpoint from the Connect pods
I read that Kafka connect is used to copy data from a source to Kafka cluster or from Kafka cluster to another destination.
That is one application, but sounds like you want MirrorMaker2 instead for that
If you don't want to use the REST API, then uncomment this line
# strimzi.io/use-connector-resources: "true"
and use another YAML file to configure the Connect resources , as shown here for Debezium. See kind: "KafkaConnector"
Look at this simple example from scratch. Not really what you want to do, but pretty close. We are sending messages to a topic using the kafka-console-producer.sh and consuming them using a file sink connector.
The example also shows how to include additional connectors by creating your own custom Connect image, based on the Strimzi one. This step would be needed for more complex examples involving external systems.
Can anyone share me the yaml file for creating kafka cluster with two kafka broker and zookeeper cluster with 3 servers.I'm new to kubernetes.
Take look at https://github.com/Yolean/kubernetes-kafka, Make sure the broker memory limit is 2 GB or above.
Maintaining a reliable kafka cluster in kubernetes is still a challenge, good luck.
I recommend you to try Strimzi Kafka Operator. Using it you can define a Kafka cluster just like other Kubernetes object - writing a yaml file. Moreover, also users, topics and Kafka Connect cluster are just a k8s objects. Some (by not all!) features of Strimzi Kafka Operator:
Secure communication between brokers and between brokers and zookeeper with TLS
Ability to expose the cluster outside k8s cluster
Deployable as a helm chart (it simplifies things a lot)
Rolling updates when changing cluster configuration
Smooth scaling out
Ready to monitor the cluster using Prometheus and Grafana.
It's worth to mention a great documentation.
Creating a Kafka cluster is as simple as applying a Kubernetes manifest like this:
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
name: my-cluster
spec:
kafka:
version: 2.2.0
replicas: 3
listeners:
plain: {}
tls: {}
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
log.message.format.version: "2.2"
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: 100Gi
deleteClaim: false
zookeeper:
replicas: 3
storage:
type: persistent-claim
size: 100Gi
deleteClaim: false
entityOperator:
topicOperator: {}
userOperator: {}
I think that you could take a look at the Strimzi project here https://strimzi.io/.
It's based on the Kubernetes operator pattern and provide a simple way to deploy and manage a Kafka cluster on Kubernetes using custom resources.
The Kafka cluster is described through a new "Kafka" resource YAML file for setting all you need.
The operator takes care of that and deploys the Zookeeper ensemble + the Kafka cluster for you.
It also deploys more two operators for handling topics and users (but they are optional).
Another simple configuration of Kafka/Zookeeper on Kubernetes in DigitalOcean with external access:
https://github.com/StanislavKo/k8s_digitalocean_kafka
You can connect to Kafka from outside of AWS/DO/GCE by regular binary protocol. Connection is PLAINTEXT or SASL_PLAINTEXT (username/password).
Kafka cluster is StatefulSet, so you can scale cluster easily.