kafka is not ingesting logs - apache-kafka

Question

I created a Kafka service on a kube cluster based on bitnami chart. The deployment goes well.
Next, I installed a filebeat to send logs to that service. It seems to me that the filebeat communicates with the cluster but does not ingest the logs. Indeed, after starting the filebeat service, I find a topic called "logs-topic" which is created by filebeat. However, this topic remains empty. My configuration is given below :
filebeat.inputs:
- type: filestream
enabled: true
paths:
- /var/log/test.log
fields:
level: debug
review: 1
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
setup.template.settings:
index.number_of_shards: 1
setup.kibana:
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
output.kafka:
hosts: ["ip-172-31-26-181:30092"]
topic: "logs-topic"
codec.json:
pretty: false
Kafka topic is present
I have no name!#kafka-release-client:/$ kafka-topics.sh --list --bootstrap-server kafka-release.default.svc.cluster.local:9092
.......
logs-topic
syslog output
Dec 30 08:28:45 ip-172-31-23-248 filebeat[29968]: 2021-12-30T08:28:45.928Z#011INFO#011[file_watcher]#011filestream/fswatch.go:137#011Start next scan
Dec 30 08:28:53 ip-172-31-23-248 filebeat[29968]: 2021-12-30T08:28:53.186Z#011INFO#011[publisher]#011pipeline/retry.go:219#011retryer: send unwait signal to consumer
Dec 30 08:28:53 ip-172-31-23-248 filebeat[29968]: 2021-12-30T08:28:53.186Z#011INFO#011[publisher]#011pipeline/retry.go:223#011 done
Dec 30 08:28:55 ip-172-31-23-248 filebeat[29968]: 2021-12-30T08:28:55.927Z#011INFO#011[file_watcher]#011filestream/fswatch.go:137#011Start next scan

Answer 1

I've found a workaround for this issue and maybee it will help others.
My problem was the same described in the post above and i struggled a long time, because i didn't find any logs from filebeat, which indicates the problem. Also i couldn't find a way to increase logging level instead of the debug option, like:
filebeat -e -c filebeat.yml -v -d "publisher"
The Kafka Broker Logs pointed to some SSL Handshake Failures at INFO Level:
journalctl -f -u kafka
Jan 22 18:18:22 kafka sh[15893]: [2022-01-22 18:18:22,604] INFO [SocketServer listenerType=ZK_BROKER, nodeId=1] Failed authentication with /172.20.3.10 (SSL handshake failed) (org.apache.kafka.common.network.Selector)
Also with DEBUG level configured, i couldn't see the actual problem.
I decided to produce some data with a python script and was able to reproduce the problem:
from kafka import KafkaConsumer, KafkaProducer
import logging
logging.basicConfig(level=logging.DEBUG)
try:
topic = "test"
sasl_mechanism = "PLAIN"
username = "test"
password = "fake"
security_protocol = "SASL_SSL"
producer = KafkaProducer(bootstrap_servers='kafka.domain.com:9092',
security_protocol=security_protocol,
ssl_check_hostname=True,
ssl_cafile='ca.crt',
sasl_mechanism=sasl_mechanism,
sasl_plain_username=username,
sasl_plain_password=password)
producer.send(topic, "test".encode('utf-8'))
producer.flush()
print("Succeed")
except Exception as e:
print("Exception:\n\n")
print(e)
Output:
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'kafka'. (_ssl.c:1129)
DEBUG:kafka.producer.sender:Node 1 not ready; delaying produce of accumulated batch
WARNING:kafka.conn:SSL connection closed by server during handshake.
INFO:kafka.conn:<BrokerConnection node_id=1 host=kafka:9092 <handshake> [IPv4 ('198.168.1.11', 9092)]>: Closing connection. KafkaConnectionError: SSL connection closed by server during handshake
=> The Hostname can not be successfully verified in the certificate. I changed and added a lot of parameter (listener / advertised.listeners / host.name) in the servers.properties of Kafka and was not able to configure another/full domain name which will be returned to the client, in the metadata. It always returns "kafka:9092", and this is not the domainname which is indicated in the certificat.
Workaround:
Disable Hostname Verification on both side (server/client).
kafka-server: ssl.endpoint.identification.algorithm=
python-client: ssl_check_hostname=True
Filebeat can do this too, but it's not realy clear:
output.kafka:
ssl.verification_mode: certificate
certificate
Verifies that the provided certificate is signed by a trusted authority (CA), but does not perform any hostname verification.
Source: https://www.elastic.co/guide/en/beats/filebeat/current/configuration-ssl.html