I am running Kafka via Confluent platform. I have followed the steps as per mentioned, java.net.BindException: Address already in use
As per documentation here, https://docs.confluent.io/2.0.0/quickstart.html#quickstart
start zookeeper,
$ ./bin/zookeeper-server-start ./etc/kafka/zookeeper.properties
start kafka,
$ ./bin/kafka-server-start ./etc/kafka/server.properties
next when I run schema-registry command,
$ ./bin/schema-registry-start ./etc/schema-registry/schema-registry.properties
I have observed error as, java.net.BindException: Address already in use
I am running all this locally in a macbook. Could somoene please help me to solve this address already in use error?
Console log:
EFGHS-MER648W:confluent-4.0.0 user$ sudo ./bin/schema-registry-start ./etc/schema-registry/schema-registry.properties
Password:
[2018-01-09 13:09:03,510] INFO SchemaRegistryConfig values:
metric.reporters = []
kafkastore.sasl.kerberos.kinit.cmd = /usr/bin/kinit
response.mediatype.default = application/vnd.schemaregistry.v1+json
kafkastore.ssl.trustmanager.algorithm = PKIX
authentication.realm =
ssl.keystore.type = JKS
kafkastore.topic = _schemas
metrics.jmx.prefix = kafka.schema.registry
kafkastore.ssl.enabled.protocols = TLSv1.2,TLSv1.1,TLSv1
kafkastore.topic.replication.factor = 3
ssl.truststore.password = [hidden]
kafkastore.timeout.ms = 500
host.name = 192.168.0.13
kafkastore.bootstrap.servers = []
schema.registry.zk.namespace = schema_registry
kafkastore.sasl.kerberos.ticket.renew.window.factor = 0.8
kafkastore.sasl.kerberos.service.name =
schema.registry.resource.extension.class =
ssl.endpoint.identification.algorithm =
compression.enable = false
kafkastore.ssl.truststore.type = JKS
avro.compatibility.level = backward
kafkastore.ssl.protocol = TLS
kafkastore.ssl.provider =
kafkastore.ssl.truststore.location =
response.mediatype.preferred = [application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json]
kafkastore.ssl.keystore.type = JKS
ssl.truststore.type = JKS
kafkastore.ssl.truststore.password = [hidden]
access.control.allow.origin =
ssl.truststore.location =
ssl.keystore.password = [hidden]
port = 8081
kafkastore.ssl.keystore.location =
master.eligibility = true
ssl.client.auth = false
kafkastore.ssl.keystore.password = [hidden]
kafkastore.security.protocol = PLAINTEXT
ssl.trustmanager.algorithm =
authentication.method = NONE
request.logger.name = io.confluent.rest-utils.requests
ssl.key.password = [hidden]
kafkastore.zk.session.timeout.ms = 30000
kafkastore.sasl.mechanism = GSSAPI
kafkastore.sasl.kerberos.ticket.renew.jitter = 0.05
kafkastore.ssl.key.password = [hidden]
zookeeper.set.acl = false
schema.registry.inter.instance.protocol = http
authentication.roles = [*]
metrics.num.samples = 2
ssl.protocol = TLS
schema.registry.group.id = schema-registry
kafkastore.ssl.keymanager.algorithm = SunX509
kafkastore.connection.url = localhost:2181
debug = false
listeners = [http://0.0.0.0:8081]
kafkastore.group.id =
ssl.provider =
ssl.enabled.protocols = []
shutdown.graceful.ms = 1000
ssl.keystore.location =
ssl.cipher.suites = []
kafkastore.ssl.endpoint.identification.algorithm =
kafkastore.ssl.cipher.suites =
access.control.allow.methods =
kafkastore.sasl.kerberos.min.time.before.relogin = 60000
ssl.keymanager.algorithm =
metrics.sample.window.ms = 30000
kafkastore.init.timeout.ms = 60000
(io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig:175)
[2018-01-09 13:09:03,749] INFO Logging initialized #629ms (org.eclipse.jetty.util.log:186)
[2018-01-09 13:09:04,202] INFO Initializing KafkaStore with broker endpoints: PLAINTEXT://192.168.0.13:9092 (io.confluent.kafka.schemaregistry.storage.KafkaStore:103)
[2018-01-09 13:09:04,475] INFO Validating schemas topic _schemas (io.confluent.kafka.schemaregistry.storage.KafkaStore:228)
[2018-01-09 13:09:04,482] WARN The replication factor of the schema topic _schemas is less than the desired one of 3. If this is a production environment, it's crucial to add more brokers and increase the replication factor of the topic. (io.confluent.kafka.schemaregistry.storage.KafkaStore:242)
[2018-01-09 13:09:04,567] INFO Initialized last consumed offset to -1 (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread:138)
[2018-01-09 13:09:04,567] INFO [kafka-store-reader-thread-_schemas]: Starting (io.confluent.kafka.schemaregistry.storage.KafkaStoreReaderThread:72)
[2018-01-09 13:09:04,651] INFO Wait to catch up until the offset of the last message at 7 (io.confluent.kafka.schemaregistry.storage.KafkaStore:277)
[2018-01-09 13:09:04,675] INFO Joining schema registry with Zookeeper-based coordination (io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry:210)
[2018-01-09 13:09:04,682] INFO Created schema registry namespace localhost:2181/schema_registry (io.confluent.kafka.schemaregistry.masterelector.zookeeper.ZookeeperMasterElector:161)
[2018-01-09 13:09:04,705] INFO Successfully elected the new master: {"host":"192.168.0.13","port":8081,"master_eligibility":true,"scheme":"http","version":1} (io.confluent.kafka.schemaregistry.masterelector.zookeeper.ZookeeperMasterElector:102)
[2018-01-09 13:09:04,715] INFO Wait to catch up until the offset of the last message at 8 (io.confluent.kafka.schemaregistry.storage.KafkaStore:277)
[2018-01-09 13:09:04,778] INFO Adding listener: http://0.0.0.0:8081 (io.confluent.rest.Application:182)
[2018-01-09 13:09:04,844] INFO jetty-9.2.22.v20170606 (org.eclipse.jetty.server.Server:327)
[2018-01-09 13:09:05,411] INFO HV000001: Hibernate Validator 5.1.3.Final (org.hibernate.validator.internal.util.Version:27)
[2018-01-09 13:09:05,547] INFO Started o.e.j.s.ServletContextHandler#54c62d71{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:744)
[2018-01-09 13:09:05,555] WARN FAILED NetworkTrafficServerConnector#4879dfad{HTTP/1.1}{0.0.0.0:8081}: java.net.BindException: Address already in use (org.eclipse.jetty.util.component.AbstractLifeCycle:212)
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.server.Server.doStart(Server.java:366)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:44)
[2018-01-09 13:09:05,557] WARN FAILED io.confluent.rest.Application$1#388526fb: java.net.BindException: Address already in use (org.eclipse.jetty.util.component.AbstractLifeCycle:212)
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.server.Server.doStart(Server.java:366)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:44)
[2018-01-09 13:09:05,558] ERROR Server died unexpectedly: (io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain:51)
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.server.Server.doStart(Server.java:366)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:44)
O2771-C02K648W:confluent-4.0.0 user$
Please help to solve this error.
Thanks,
When I run command, ps aux | grep schema-registry
O2771-C02K648W:~ user$ ps aux | grep schema-registry
root 20888 0.1 1.5 4584980 255588 ?? S 6:14PM 1:05.15 /usr/bin/java -Xmx512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dschema-registry.log.dir=/Users/user/Downloads/confluent-4.0.0/bin/../logs -Dlog4j.configuration=file:/Users/user/Downloads/confluent-4.0.0/bin/../etc/schema-registry/log4j.properties -cp :/Users/user/Downloads/confluent-4.0.0/bin/../package-schema-registry/target/kafka-schema-registry-package-*-development/share/java/schema-registry/*:/Users/user/Downloads/confluent-4.0.0/bin/../share/java/confluent-common/*:/Users/user/Downloads/confluent-4.0.0/bin/../share/java/rest-utils/*:/Users/user/Downloads/confluent-4.0.0/bin/../share/java/schema-registry/* io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain ./etc/schema-registry/schema-registry.properties
root 20887 0.0 0.0 2456112 3452 ?? S 6:14PM 0:00.02 sudo ./bin/schema-registry-start ./etc/schema-registry/schema-registry.properties
user 25256 0.0 0.0 2432804 1984 s001 S+ 1:41PM 0:00.00 grep schema-registry
O2771-C02K648W:~ user$
You can try to find out which process is using port:8081 by using below commands
netstat -vanp tcp | grep 8081
For OSX El Capitan and newer (or if your netstat doesn't support -p), use lsof
sudo lsof -i tcp:8081
Or any other command for windows and force kill that process using
kill -9 {PID}
You can also try changing schema registry default port to another port which is not used.
Check the Kafka schema registry port using below command
[root#in-ibmibm3718 /]# netstat -tnpl |grep 8093
tcp 0 0 0.0.0.0:8093 0.0.0.0:* LISTEN 16862/java
[root#in-ibmibm3718 /]# ps -eaf |grep -i 16862
kafka 16862 1 35 14:35 ? 01:15:19 /usr/jdk64/java-1.8.0-openjdk/bin/java -Xmx512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dschema-registry.log.dir=/var/log/kafka -Dlog4j.configuration=file:/opt/IBM/basecamp/basecamp-schema-registry/bin/../etc/schema-registry/log4j.properties -cp :/opt/IBM/basecamp/basecamp-schema-registry/bin/../package-schema-registry/target/kafka-schema-registry-package-*-development/share/java/schema-registry/*:/opt/IBM/basecamp/basecamp-schema-registry/bin/../share/java/confluent-common/*:/opt/IBM/basecamp/basecamp-schema-registry/bin/../share/java/rest-utils/*:/opt/IBM/basecamp/basecamp-schema-registry/bin/../share/java/schema-registry/* io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain /opt/IBM/basecamp/basecamp-schema-registry/etc/schema-registry/schema-registry.properties
Kill the process and start the schema registry again
I had same issue,
just force kill currently working zookeeper process
$ps -ef|grep zookeeper
$kill -9 <process number>
then start again zookeeper and kafka.
Related
I've Strimzi Kafka setup on GKE, and it is working fine.
I've a requirement to setup MirrorMaker2 to push data from source kafka topic to target Kafka topic.
From what i understand, MirrorMaker2 requires KafkaConnect.
I'm trying to install KafkaConnect on the GKE cluster in namespace kafka-connect, the yaml used is the following.
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: my-connect-cluster
# annotations:
# strimzi.io/use-connector-resources: "true"
spec:
version: 3.0.0
replicas: 1
bootstrapServers: versa-kafka-gke-kafka-bootstrap:9093
tls:
trustedCertificates:
- secretName: versa-kafka-gke-cluster-ca-cert
certificate: ca.crt
config:
group.id: connect-cluster
offset.storage.topic: connect-cluster-offsets
config.storage.topic: connect-cluster-configs
status.storage.topic: connect-cluster-status
# -1 means it will use the default replication factor configured in the broker
config.storage.replication.factor: -1
offset.storage.replication.factor: -1
status.storage.replication.factor: -1
Command run to install:
kubectl apply -f kafka-connect.yaml -n kafka-connect
Note : Strimzi kafka is installed in namespace 'kafka', version - 3.0.0
I was expecting the KafkaConnect to get installed & the pods to be installed as well, however KafkaConnect resource is created, but not showing as Ready.
Also, the pods are not getting created.
(base) Karans-MacBook-Pro:kafkaConnect karanalang$ kc get kafkaconnect my-connect-cluster -n kafka-connect
NAME DESIRED REPLICAS READY
my-connect-cluster 1
On describing the my-connect-cluster, here is the output (not showing any errors)
(base) Karans-MacBook-Pro:kafkaConnect karanalang$ kc describe kafkaconnect my-connect-cluster -n kafka-connect
Name: my-connect-cluster
Namespace: kafka-connect
Labels: <none>
Annotations: <none>
API Version: kafka.strimzi.io/v1beta2
Kind: KafkaConnect
Metadata:
Creation Timestamp: 2023-02-16T06:38:41Z
Generation: 1
Managed Fields:
API Version: kafka.strimzi.io/v1beta2
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:bootstrapServers:
f:config:
.:
f:config.storage.replication.factor:
f:config.storage.topic:
f:group.id:
f:offset.storage.replication.factor:
f:offset.storage.topic:
f:status.storage.replication.factor:
f:status.storage.topic:
f:replicas:
f:tls:
.:
f:trustedCertificates:
f:version:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2023-02-16T06:38:41Z
Resource Version: 266457732
UID: bdd697f8-e38a-466b-8ddf-81ed1ae54efe
Spec:
Bootstrap Servers: versa-kafka-gke-kafka-bootstrap:9093
Config:
config.storage.replication.factor: -1
config.storage.topic: connect-cluster-configs
group.id: connect-cluster
offset.storage.replication.factor: -1
offset.storage.topic: connect-cluster-offsets
status.storage.replication.factor: -1
status.storage.topic: connect-cluster-status
Replicas: 1
Tls:
Trusted Certificates:
Certificate: ca.crt
Secret Name: versa-kafka-gke-cluster-ca-cert
Version: 3.0.0
Events: <none>
How do i debug/fix this ?
tia!
Update :
Based on note from Jakub, i re-installed kafkaconnect in the same namespace as Strimzi kafka (i.e. namespace - kafka), and pods are coming up now. However the logs show error as shown below :
(base) Karans-MacBook-Pro:kafkaConnect karanalang$ kc logs -f pod/my-connect-cluster-connect-67f76f5d89-nv9sj -n kafka
Preparing truststore
Certificate was added to keystore
Preparing truststore is complete
Starting Kafka Connect with configuration:
# Bootstrap servers
bootstrap.servers=versa-kafka-gke-w-kafka-bootstrap:9093
# REST Listeners
rest.port=8083
rest.advertised.host.name=10.6.0.199
rest.advertised.port=8083
# Plugins
plugin.path=/opt/kafka/plugins
# Provided configuration
offset.storage.topic=connect-cluster-offsets
value.converter=org.apache.kafka.connect.json.JsonConverter
config.storage.topic=connect-cluster-configs
key.converter=org.apache.kafka.connect.json.JsonConverter
group.id=connect-cluster
status.storage.topic=connect-cluster-status
config.storage.replication.factor=-1
offset.storage.replication.factor=-1
status.storage.replication.factor=-1
security.protocol=SSL
producer.security.protocol=SSL
consumer.security.protocol=SSL
admin.security.protocol=SSL
# TLS / SSL
ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
ssl.truststore.password=[hidden]
ssl.truststore.type=PKCS12
producer.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
producer.ssl.truststore.password=[hidden]
consumer.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
consumer.ssl.truststore.password=[hidden]
admin.ssl.truststore.location=/tmp/kafka/cluster.truststore.p12
admin.ssl.truststore.password=[hidden]
# Additional configuration
client.rack=
2023-02-17 06:43:08,952 INFO WorkerInfo values:
jvm.args = -Xms128M, -XX:+UseG1GC, -XX:MaxGCPauseMillis=20, -XX:InitiatingHeapOccupancyPercent=35, -XX:+ExplicitGCInvokesConcurrent, -XX:MaxInlineLevel=15, -Djava.awt.headless=true, -Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -Dkafka.logs.dir=/opt/kafka, -Dlog4j.configuration=file:/opt/kafka/custom-config/log4j.properties
jvm.spec = Red Hat, Inc., OpenJDK 64-Bit Server VM, 11.0.12, 11.0.12+7-LTS
jvm.classpath = /opt/kafka/bin/../libs/accessors-smart-2.4.7.jar:/opt/kafka/bin/../libs/activation-1.1.1.jar:/opt/kafka/bin/../libs/annotations-13.0.jar:/opt/kafka/bin/../libs/aopalliance-repackaged-2.6.1.jar:/opt/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/bin/../libs/automaton-1.11-8.jar:/opt/kafka/bin/../libs/checker-qual-3.5.0.jar:/opt/kafka/bin/../libs/commons-cli-1.4.jar:/opt/kafka/bin/../libs/commons-lang-2.6.jar:/opt/kafka/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/bin/../libs/connect-api-3.0.0.jar:/opt/kafka/bin/../libs/connect-basic-auth-extension-3.0.0.jar:/opt/kafka/bin/../libs/connect-file-3.0.0.jar:/opt/kafka/bin/../libs/connect-json-3.0.0.jar:/opt/kafka/bin/../libs/connect-mirror-3.0.0.jar:/opt/kafka/bin/../libs/connect-mirror-client-3.0.0.jar:/opt/kafka/bin/../libs/connect-runtime-3.0.0.jar:/opt/kafka/bin/../libs/connect-transforms-
......
2023-02-17 06:43:24,944 INFO Added alias 'InsertHeader' to plugin 'org.apache.kafka.connect.transforms.InsertHeader' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-02-17 06:43:24,945 INFO Added alias 'RegexRouter' to plugin 'org.apache.kafka.connect.transforms.RegexRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-02-17 06:43:24,945 INFO Added alias 'TimestampRouter' to plugin 'org.apache.kafka.connect.transforms.TimestampRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-02-17 06:43:24,945 INFO Added alias 'ValueToKey' to plugin 'org.apache.kafka.connect.transforms.ValueToKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-02-17 06:43:24,945 INFO Added alias 'HasHeaderKey' to plugin 'org.apache.kafka.connect.transforms.predicates.HasHeaderKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-02-17 06:43:24,945 INFO Added alias 'RecordIsTombstone' to plugin 'org.apache.kafka.connect.transforms.predicates.RecordIsTombstone' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-02-17 06:43:24,945 INFO Added alias 'TopicNameMatches' to plugin 'org.apache.kafka.connect.transforms.predicates.TopicNameMatches' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-02-17 06:43:24,945 INFO Added alias 'BasicAuthSecurityRestExtension' to plugin 'org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-02-17 06:43:24,945 INFO Added aliases 'AllConnectorClientConfigOverridePolicy' and 'All' to plugin 'org.apache.kafka.connect.connector.policy.AllConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-02-17 06:43:24,945 INFO Added aliases 'NoneConnectorClientConfigOverridePolicy' and 'None' to plugin 'org.apache.kafka.connect.connector.policy.NoneConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-02-17 06:43:24,945 INFO Added aliases 'PrincipalConnectorClientConfigOverridePolicy' and 'Principal' to plugin 'org.apache.kafka.connect.connector.policy.PrincipalConnectorClientConfigOverridePolicy' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader) [main]
2023-02-17 06:43:25,347 INFO DistributedConfig values:
access.control.allow.methods =
access.control.allow.origin =
admin.listeners = null
bootstrap.servers = [versa-kafka-gke-w-kafka-bootstrap:9093]
client.dns.lookup = use_all_dns_ips
client.id =
config.providers = []
config.storage.replication.factor = -1
config.storage.topic = connect-cluster-configs
connect.protocol = sessioned
connections.max.idle.ms = 540000
connector.client.config.override.policy = All
group.id = connect-cluster
header.converter = class org.apache.kafka.connect.storage.SimpleHeaderConverter
heartbeat.interval.ms = 3000
inter.worker.key.generation.algorithm = HmacSHA256
inter.worker.key.size = null
inter.worker.key.ttl.ms = 3600000
inter.worker.signature.algorithm = HmacSHA256
inter.worker.verification.algorithms = [HmacSHA256]
key.converter = class org.apache.kafka.connect.json.JsonConverter
listeners = [http://:8083]
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
offset.flush.interval.ms = 60000
offset.flush.timeout.ms = 5000
offset.storage.partitions = 25
offset.storage.replication.factor = -1
offset.storage.topic = connect-cluster-offsets
plugin.path = [/opt/kafka/plugins]
rebalance.timeout.ms = 60000
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 40000
response.http.headers.config =
rest.advertised.host.name = 10.6.0.199
rest.advertised.listener = null
rest.advertised.port = 8083
rest.extension.classes = []
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
scheduled.rebalance.max.delay.ms = 300000
security.protocol = SSL
send.buffer.bytes = 131072
session.timeout.ms = 10000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.client.auth = none
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = /tmp/kafka/cluster.truststore.p12
ssl.truststore.password = [hidden]
ssl.truststore.type = PKCS12
status.storage.partitions = 5
status.storage.replication.factor = -1
status.storage.topic = connect-cluster-status
task.shutdown.graceful.timeout.ms = 5000
topic.creation.enable = true
topic.tracking.allow.reset = true
topic.tracking.enable = true
value.converter = class org.apache.kafka.connect.json.JsonConverter
worker.sync.timeout.ms = 3000
worker.unsync.backoff.ms = 300000
(org.apache.kafka.connect.runtime.distributed.DistributedConfig) [main]
2023-02-17 06:43:25,355 INFO Creating Kafka admin client (org.apache.kafka.connect.util.ConnectUtils) [main]
2023-02-17 06:43:25,359 INFO AdminClientConfig values:
bootstrap.servers = [versa-kafka-gke-w-kafka-bootstrap:9093]
client.dns.lookup = use_all_dns_ips
client.id =
connections.max.idle.ms = 300000
default.api.timeout.ms = 60000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 2147483647
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = SSL
security.providers = null
send.buffer.bytes = 131072
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = /tmp/kafka/cluster.truststore.p12
ssl.truststore.password = [hidden]
ssl.truststore.type = PKCS12
(org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,848 WARN The configuration 'producer.ssl.truststore.password' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,849 WARN The configuration 'group.id' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,849 WARN The configuration 'rest.advertised.port' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,849 WARN The configuration 'plugin.path' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,850 WARN The configuration 'admin.security.protocol' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,850 WARN The configuration 'consumer.ssl.truststore.location' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,850 WARN The configuration 'producer.ssl.truststore.location' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,850 WARN The configuration 'status.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,850 WARN The configuration 'offset.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,850 WARN The configuration 'consumer.security.protocol' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,851 WARN The configuration 'value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,851 WARN The configuration 'key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,851 WARN The configuration 'admin.ssl.truststore.password' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,851 WARN The configuration 'consumer.ssl.truststore.password' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,851 WARN The configuration 'config.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,851 WARN The configuration 'producer.security.protocol' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,852 WARN The configuration 'rest.advertised.host.name' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,852 WARN The configuration 'status.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,852 WARN The configuration 'client.rack' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,852 WARN The configuration 'rest.port' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,853 WARN The configuration 'config.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,853 WARN The configuration 'offset.storage.replication.factor' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,855 WARN The configuration 'admin.ssl.truststore.location' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [main]
2023-02-17 06:43:27,856 INFO Kafka version: 3.0.0 (org.apache.kafka.common.utils.AppInfoParser) [main]
2023-02-17 06:43:27,857 INFO Kafka commitId: 8cb0a5e9d3441962 (org.apache.kafka.common.utils.AppInfoParser) [main]
2023-02-17 06:43:27,857 INFO Kafka startTimeMs: 1676616207856 (org.apache.kafka.common.utils.AppInfoParser) [main]
2023-02-17 06:43:30,857 INFO [AdminClient clientId=adminclient-1] Failed re-authentication with versa-kafka-gke-w-kafka-bootstrap/10.6.131.57 (Failed to process post-handshake messages) (org.apache.kafka.common.network.Selector) [kafka-admin-client-thread | adminclient-1]
2023-02-17 06:43:30,867 ERROR [AdminClient clientId=adminclient-1] Connection to node -1 (versa-kafka-gke-w-kafka-bootstrap/10.6.131.57:9093) failed authentication due to: Failed to process post-handshake messages (org.apache.kafka.clients.NetworkClient) [kafka-admin-client-thread | adminclient-1]
2023-02-17 06:43:30,870 WARN [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.SslAuthenticationException: Failed to process post-handshake messages
Caused by: javax.net.ssl.SSLException: Tag mismatch!
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:123)
at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681)
at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433)
at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637)
at org.apache.kafka.common.network.SslTransportLayer.read(SslTransportLayer.java:567)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:95)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:551)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1389)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1320)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.crypto.AEADBadTagException: Tag mismatch!
at java.base/com.sun.crypto.provider.GaloisCounterMode.decryptFinal(GaloisCounterMode.java:623)
at java.base/com.sun.crypto.provider.CipherCore.finalNoPadding(CipherCore.java:1116)
at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1053)
at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:941)
at java.base/com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:491)
at java.base/javax.crypto.CipherSpi.bufferCrypt(CipherSpi.java:779)
at java.base/javax.crypto.CipherSpi.engineDoFinal(CipherSpi.java:730)
at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2497)
at java.base/sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1903)
at java.base/sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:240)
at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:197)
at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:160)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
... 16 more
2023-02-17 06:43:30,946 INFO App info kafka.admin.client for adminclient-1 unregistered (org.apache.kafka.common.utils.AppInfoParser) [kafka-admin-client-thread | adminclient-1]
2023-02-17 06:43:30,947 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited. Call: fetchMetadata
2023-02-17 06:43:30,948 INFO [AdminClient clientId=adminclient-1] Metadata update failed (org.apache.kafka.clients.admin.internals.AdminMetadataManager) [kafka-admin-client-thread | adminclient-1]
org.apache.kafka.common.errors.TimeoutException: The AdminClient thread has exited. Call: fetchMetadata
2023-02-17 06:43:30,949 INFO [AdminClient clientId=adminclient-1] Timed out 2 remaining operation(s) during close. (org.apache.kafka.clients.admin.KafkaAdminClient) [kafka-admin-client-thread | adminclient-1]
2023-02-17 06:43:30,959 INFO Metrics scheduler closed (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2023-02-17 06:43:30,960 INFO Closing reporter org.apache.kafka.common.metrics.JmxReporter (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2023-02-17 06:43:30,961 INFO Metrics reporters closed (org.apache.kafka.common.metrics.Metrics) [kafka-admin-client-thread | adminclient-1]
2023-02-17 06:43:30,961 ERROR Stopping due to error (org.apache.kafka.connect.cli.ConnectDistributed) [main]
org.apache.kafka.connect.errors.ConnectException: Failed to connect to and describe Kafka cluster. Check worker's broker connection and security properties.
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:70)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:51)
at org.apache.kafka.connect.cli.ConnectDistributed.startConnect(ConnectDistributed.java:97)
at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:80)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.SslAuthenticationException: Failed to process post-handshake messages
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
at org.apache.kafka.connect.util.ConnectUtils.lookupKafkaClusterId(ConnectUtils.java:64)
... 3 more
Caused by: org.apache.kafka.common.errors.SslAuthenticationException: Failed to process post-handshake messages
Caused by: javax.net.ssl.SSLException: Tag mismatch!
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:123)
at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681)
at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433)
at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637)
at org.apache.kafka.common.network.SslTransportLayer.read(SslTransportLayer.java:567)
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:95)
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452)
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402)
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:551)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1389)
at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1320)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: javax.crypto.AEADBadTagException: Tag mismatch!
at java.base/com.sun.crypto.provider.GaloisCounterMode.decryptFinal(GaloisCounterMode.java:623)
at java.base/com.sun.crypto.provider.CipherCore.finalNoPadding(CipherCore.java:1116)
at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1053)
at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:941)
at java.base/com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:491)
at java.base/javax.crypto.CipherSpi.bufferCrypt(CipherSpi.java:779)
at java.base/javax.crypto.CipherSpi.engineDoFinal(CipherSpi.java:730)
at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2497)
at java.base/sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1903)
at java.base/sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:240)
at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:197)
at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:160)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
... 16 more
Pls note :
Strimzi Kafka version - 3.0.0, hence I've changed the version of kafkaconnect to - 3.0.0 as well.
You say you want to use MirrorMaker2? Then you should be using that kind, not KafkaConnect.
https://strimzi.io/blog/2020/03/30/introducing-mirrormaker2/
And, as commented, ensure the operator is watching the namespace where you install any resources. Just because you can get/describe the resource doesn't mean the operator knows about it, or is processing it (look at its logs)
i've a Structured Streaming code which reads data from a Kafka Topic (on a VM) & writes to another Kafka Topic on GKE (i should be using a Mirror Maker for this, but have not implemented that yet). it suddenly stopped working (been working fine for many months) giving following error :
22/10/18 19:02:35 WARN org.apache.spark.sql.streaming.StreamingQueryManager: spark.sql.adaptive.enabled is not supported in streaming DataFrames/Datasets and will be disabled.
22/10/18 19:03:42 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0) (stream2kafka2-w-1.c.versa-sml-googl.internal executor 2): org.apache.kafka.common.errors.TimeoutException: Topic syslog.ueba-us4.v1.versa.demo4 not present in metadata after 60000 ms.
22/10/18 19:03:42 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 0.1 in stage 0.0 (TID 1) (stream2kafka2-w-1.c.versa-sml-googl.internal executor 2): org.apache.spark.sql.execution.streaming.continuous.ContinuousTaskRetryException: Continuous execution does not support task retry
at org.apache.spark.sql.execution.streaming.continuous.ContinuousDataSourceRDD.compute(ContinuousDataSourceRDD.scala:76)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.sql.execution.streaming.continuous.ContinuousWriteRDD.$anonfun$compute$1(ContinuousWriteRDD.scala:53)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1473)
at org.apache.spark.sql.execution.streaming.continuous.ContinuousWriteRDD.compute(ContinuousWriteRDD.scala:84)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:498)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1439)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:501)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Suppressed: java.lang.NullPointerException
at org.apache.spark.sql.execution.streaming.continuous.ContinuousWriteRDD.$anonfun$compute$7(ContinuousWriteRDD.scala:84)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1495)
... 11 more
Code is very simple, and has been working for many months now :
class ReadFromKafka:
def readAndWrite(self):
df = spark \
.readStream \
.format('kafka') \
.option("kafka.bootstrap.servers", kafkaBrokersSrc) \
.option("subscribe", srcTopic) \
.option("startingOffsets", "latest") \
.option("failOnDataLoss", "false") \
.load()
query = df.selectExpr("CAST(value AS STRING)", "cast(key AS String)") \
.writeStream \
.format("kafka") \
.option("checkpointLocation", checkpoint) \
.option("outputMode", "append") \
.option("truncate", "false") \
.option("kafka.security.protocol", security_protocol) \
.option("kafka.ssl.truststore.location", ssl_truststore_location) \
.option("kafka.ssl.truststore.password", ssl_truststore_password) \
.option("kafka.ssl.keystore.location", ssl_keystore_location) \
.option("kafka.ssl.keystore.password", ssl_keystore_password) \
.option("kafka.bootstrap.servers", kafkaBrokersTgt) \
.option("topic", tgtTopic) \
.option("kafka.ssl.keystore.type", "PKCS12") \
.option("kafka.ssl.truststore.type", "PKCS12") \
.trigger(continuous='5 seconds') \
.start()
query.awaitTermination()
I'm running this on google dataproc
gcloud dataproc jobs submit pyspark /Users/karanalang/PycharmProjects/Kafka/versa-movedata2kafka/StructuredStreaming-readFromKafka-versa-sml-googl-v1.py --cluster stream2kafka --properties spark.jars.packages=org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2,spark.dynamicAllocation.enabled=true,spark.shuffle.service.enabled=true --files gs://kafka-certs/versa-kafka-gke-ca.p12,gs://kafka-certs/syslog-vani-noacl.p12 --region us-east1
any ideas on what the issue might be & how to debug this ?
tia!
Update :
I'm able to read & write in to the Kafka Topic when i use python Kafka Producer/Consumer- but Structured Streaming code is failing
Update :
Update :
I'm able to read the topic from GKE using spark-submit (batch & streaming mode), the SSL certs are stored on my local mac from where spark-submit is run
So, it seems like Spark is behaving correctly.
However, I tried reading from the kafka topic on GKE using - google cloud submit, and it gives error saying broker is not found (shown below) ..
the SSL certs are stored in the storage bucket, and i'm passing the certs as '--files gs://kafka-certs/versa-kafka-gke-ca.p12,gs://kafka-certs/syslog-vani-noacl.p12'
In the pyspark code, i access them using the file names - this has been working earlier, however i suspect - this might be causing the issue.
Question - is this the corect way to access the certs when I'm using Dataproc ?
commands :
gcloud dataproc jobs submit pyspark /Users/karanalang/PycharmProjects/Kafka/versa-movedata2kafka/StructuredStream-stream-readfrom-versa-sml-googl-certs-gs.py --cluster stream2kafka2 --properties spark.jars.packages=org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2,spark.dynamicAllocation.enabled=true,spark.shuffle.service.enabled=true --files gs://kafka-certs/versa-kafka-gke-ca.p12,gs://kafka-certs/syslog-vani-noacl.p12 --region us-east1
Code :
kafkaBrokersTgt='IP:port'
tgtTopic = "syslog.ueba-us4.v1.versa.demo3"
checkpoint='gs://versa-move2syslogdemo3/'
security_protocol="SSL"
ssl_truststore_location="versa-kafka-gke-ca.p12"
ssl_truststore_password='xxxx'
ssl_keystore_location = 'syslog-vani-noacl.p12'
ssl_keystore_password ='yyyy'
print(" reading from Kafka topic syslog-demo3 on versa-sml-googl, certs on gs storage ")
df_reader = spark.readStream.format('kafka')\
.option("kafka.bootstrap.servers",kafkaBrokersTgt)\
.option("kafka.security.protocol",security_protocol) \
.option("kafka.ssl.truststore.location",ssl_truststore_location) \
.option("kafka.ssl.truststore.password",ssl_truststore_password) \
.option("kafka.ssl.keystore.location", ssl_keystore_location)\
.option("kafka.ssl.keystore.password", ssl_keystore_password)\
.option("subscribe", tgtTopic) \
.option("startingOffsets", "earliest") \
.option("maxOffsetsPerTrigger", 20) \
.option("kafka.max.poll.records", 20) \
.option("kafka.ssl.keystore.type", "PKCS12") \
.option("kafka.ssl.truststore.type", "PKCS12") \
.load()
# .option("kafka.group.id", "ss.consumer1") \
query = df_reader.selectExpr("CAST(value AS STRING)", "cast(key AS String)") \
.writeStream \
.format("console") \
.option("numRows",500)\
.option("outputMode", "complete")\
.option("truncate", "false") \
.trigger(processingTime='3 minutes') \
.option("checkpointLocation", checkpoint) \
.start()
query.awaitTermination()
Error :
reading from Kafka topic syslog-demo3 on versa-sml-googl, certs on gs storage
22/10/20 04:37:48 WARN org.apache.spark.sql.streaming.StreamingQueryManager: spark.sql.adaptive.enabled is not supported in streaming DataFrames/Datasets and will be disabled.
22/10/20 04:37:50 INFO org.apache.kafka.clients.consumer.ConsumerConfig: ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = earliest
bootstrap.servers = [34.138.213.152:9094]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = consumer-spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0-1
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 1
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = SSL
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = syslog-vani-noacl.p12
ssl.keystore.password = [hidden]
ssl.keystore.type = PKCS12
ssl.protocol = TLSv1.2
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = versa-kafka-gke-ca.p12
ssl.truststore.password = [hidden]
ssl.truststore.type = PKCS12
value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
22/10/20 04:37:50 INFO org.apache.kafka.common.utils.AppInfoParser: Kafka version: 2.6.0
22/10/20 04:37:50 INFO org.apache.kafka.common.utils.AppInfoParser: Kafka commitId: 62abe01bee039651
22/10/20 04:37:50 INFO org.apache.kafka.common.utils.AppInfoParser: Kafka startTimeMs: 1666240670692
22/10/20 04:37:50 INFO org.apache.kafka.clients.consumer.KafkaConsumer: [Consumer clientId=consumer-spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0-1, groupId=spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0] Subscribed to topic(s): syslog.ueba-us4.v1.versa.demo3
22/10/20 04:40:01 WARN org.apache.kafka.clients.NetworkClient: [Consumer clientId=consumer-spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0-1, groupId=spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0] Connection to node -1 (/34.138.213.152:9094) could not be established. Broker may not be available.
22/10/20 04:40:01 WARN org.apache.kafka.clients.NetworkClient: [Consumer clientId=consumer-spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0-1, groupId=spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0] Bootstrap broker 34.138.213.152:9094 (id: -1 rack: null) disconnected
22/10/20 04:42:12 WARN org.apache.kafka.clients.NetworkClient: [Consumer clientId=consumer-spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0-1, groupId=spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0] Connection to node -1 (/34.138.213.152:9094) could not be established. Broker may not be available.
22/10/20 04:42:12 WARN org.apache.kafka.clients.NetworkClient: [Consumer clientId=consumer-spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0-1, groupId=spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0] Bootstrap broker 34.138.213.152:9094 (id: -1 rack: null) disconnected
22/10/20 04:44:23 WARN org.apache.kafka.clients.NetworkClient: [Consumer clientId=consumer-spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0-1, groupId=spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0] Connection to node -1 (/34.138.213.152:9094) could not be established. Broker may not be available.
22/10/20 04:44:23 WARN org.apache.kafka.clients.NetworkClient: [Consumer clientId=consumer-spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0-1, groupId=spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0] Bootstrap broker 34.138.213.152:9094 (id: -1 rack: null) disconnected
22/10/20 04:46:34 WARN org.apache.kafka.clients.NetworkClient: [Consumer clientId=consumer-spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0-1, groupId=spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0] Connection to node -1 (/34.138.213.152:9094) could not be established. Broker may not be available.
22/10/20 04:46:34 WARN org.apache.kafka.clients.NetworkClient: [Consumer clientId=consumer-spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0-1, groupId=spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0] Bootstrap broker 34.138.213.152:9094 (id: -1 rack: null) disconnected
22/10/20 04:48:45 WARN org.apache.kafka.clients.NetworkClient: [Consumer clientId=consumer-spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0-1, groupId=spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0] Connection to node -1 (/34.138.213.152:9094) could not be established. Broker may not be available.
22/10/20 04:48:45 WARN org.apache.kafka.clients.NetworkClient: [Consumer clientId=consumer-spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0-1, groupId=spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0] Bootstrap broker 34.138.213.152:9094 (id: -1 rack: null) disconnected
22/10/20 04:50:56 WARN org.apache.kafka.clients.NetworkClient: [Consumer clientId=consumer-spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0-1, groupId=spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0] Connection to node -1 (/34.138.213.152:9094) could not be established. Broker may not be available.
22/10/20 04:50:56 WARN org.apache.kafka.clients.NetworkClient: [Consumer clientId=consumer-spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0-1, groupId=spark-kafka-source-10bf0d29-761e-4b5a-95c6-308e036ca6f9-764682263-driver-0] Bootstrap broker 34.138.213.152:9094 (id: -1 rack: null) disconnected
Update :
per comment from #Daganag, when i use SparkFiles.get(filename) .. here is the error i get :
d-9fe7bb774985/syslog-vani-noacl.p12
java.nio.file.NoSuchFileException: /hadoop/spark/tmp/spark-19943d8b-d8c7-4406-b5cf-c352837ad71e/userFiles-32e5ebe3-7013-44f2-a0bd-9fe7bb774985/syslog-vani-noacl.p12
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.nio.file.Files.readAttributes(Files.java:1737)
at java.nio.file.Files.getLastModifiedTime(Files.java:2266)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$SecurityStore.lastModifiedMs(DefaultSslEngineFactory.java:312)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$SecurityStore.<init>(DefaultSslEngineFactory.java:284)
at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createKeystore(DefaultSslEngineFactory.java:255)
#Dagang, #OneCrickteer - i logged onto the worker nodes & i see the SSL certs uploaded (when i pass the certs as --files gs:// in google cloud submit.
How do i access them in code .. SparkFiles.get('cert') is not working, since the path SparkFiles gets is not the same
SSL certs on The Worker Node :
------------------------------
root#stream2kafka2-w-0:/# find . -name versa-kafka-gke-ca.p12
./hadoop/yarn/nm-local-dir/usercache/root/appcache/application_1666127856693_0016/container_e01_1666127856693_0016_01_000002/versa-kafka-gke-ca.p12
./hadoop/yarn/nm-local-dir/usercache/root/filecache/67/versa-kafka-gke-ca.p12
./hadoop/yarn/nm-local-dir/usercache/root/filecache/39/versa-kafka-gke-ca.p12
./hadoop/yarn/nm-local-dir/usercache/root/filecache/165/versa-kafka-gke-ca.p12
./hadoop/yarn/nm-local-dir/usercache/root/filecache/194/versa-kafka-gke-ca.p12
./hadoop/yarn/nm-local-dir/usercache/root/filecache/109/versa-kafka-gke-ca.p12
./hadoop/yarn/nm-local-dir/usercache/root/filecache/81/versa-kafka-gke-ca.p12
./hadoop/yarn/nm-local-dir/usercache/root/filecache/53/versa-kafka-gke-ca.p12
./hadoop/yarn/nm-local-dir/usercache/root/filecache/208/versa-kafka-gke-ca.p12
./hadoop/yarn/nm-local-dir/usercache/root/filecache/179/versa-kafka-gke-ca.p12
./hadoop/yarn/nm-local-dir/usercache/root/filecache/151/versa-kafka-gke-ca.p12
./hadoop/yarn/nm-local-dir/usercache/root/filecache/137/versa-kafka-gke-ca.p12
./hadoop/yarn/nm-local-dir/usercache/root/filecache/23/versa-kafka-gke-ca.p12
./hadoop/yarn/nm-local-dir/usercache/root/filecache/95/versa-kafka-gke-ca.p12
./hadoop/yarn/nm-local-dir/usercache/root/filecache/123/versa-kafka-gke-ca.p12
I am facing a very tough situation running MUnit on my project. I am using Mule 4.3.0 and Anypoint Studio 7.4.
Apparently, the point of error is occuring while loading a certs/cacerts file as a property used in some Apache Kakfa Connector TLS Configuration.
Its working absolutely fine when running the normal Mule code (having the TLS Context).
But fails to work, when running MUnit.
I have tried many ways to get this resolved with my team, but couldn't fix it. Although similar errors been reported by other developers occasionally, I couldn't concluded that this as a possible bug with the mule runtime or especially a Kafka Connector problem.
Finally, once removing the TLS context in Kafka Config, MUnit is working fine. But without TLS enabled, my project is basically useless
I need your help in resolving this and making my test work, but with TLS configurations present in its rightful place. Also, please take a look at this question in the forums: Mule Kafka Problem
Given below are the two snapshots of the Configuration used, and the error reported in the MUnit console:
Kafka Consumer TLS Configuration Snapshot:
Error Snapshot:
Compete error report given below:
INFO 2020-10-21 18:44:12,077 [main] org.mule.munit.remote.container.SuiteRunDispatcher: Suite errortopic-db-test-suite.xml will not be deployed: Suite was filtered from running
INFO 2020-10-21 18:44:12,078 [munit.01] org.mule.munit.runner.remote.api.server.RunnerServer: Waiting for client connection
INFO 2020-10-21 18:44:12,086 [munit.01] org.mule.munit.runner.remote.api.server.RunnerServer: Client connection received from 127.0.0.1 - true
WARN 2020-10-21 18:44:19,766 [munit.01] org.mule.runtime.core.internal.security.tls.TlsProperties: File tls-default.conf not found, using default configuration.
INFO 2020-10-21 18:44:19,767 [munit.01] org.mule.runtime.api.tls.AbstractTlsContextFactoryBuilderFactory: Loaded TlsContextFactoryBuilderFactory implementation 'org.mule.runtime.module.tls.api.DefaultTlsContextFactoryBuilderFactory' from classloader 'java.net.URLClassLoader#7fd8c559'
INFO 2020-10-21 18:44:21,684 [munit.01] org.mule.runtime.core.privileged.lifecycle.AbstractLifecycleManager: Initialising Bean: org.mule.runtime.module.extension.internal.runtime.config.ConfigurationProviderToolingAdapter-HTTP_Request_configuration_oauth
INFO 2020-10-21 18:44:21,736 [munit.01] org.mule.runtime.core.privileged.lifecycle.AbstractLifecycleManager: Initialising Bean: org.mule.runtime.module.extension.internal.runtime.config.ConfigurationProviderToolingAdapter-HTTP_Request_configuration-By
WARN 2020-10-21 18:44:21,800 [munit.01] org.mule.runtime.core.internal.security.tls.TlsProperties: File tls-default.conf not found, using default configuration.
INFO 2020-10-21 18:44:21,802 [munit.01] org.mule.runtime.core.privileged.lifecycle.AbstractLifecycleManager: Initialising Bean: org.mule.runtime.module.extension.internal.runtime.config.ConfigurationProviderToolingAdapter-Apache_Kafka_Consumer_configuration
WARN 2020-10-21 18:44:21,810 [munit.01] org.mule.runtime.core.internal.security.tls.TlsProperties: File tls-default.conf not found, using default configuration.
INFO 2020-10-21 18:44:21,872 [munit.01] org.mule.runtime.core.privileged.lifecycle.AbstractLifecycleManager: Initialising Bean: org.mule.runtime.module.extension.internal.runtime.config.ConfigurationProviderToolingAdapter-Apache_Kafka_Producer_configuration
WARN 2020-10-21 18:44:21,879 [munit.01] org.mule.runtime.core.internal.security.tls.TlsProperties: File tls-default.conf not found, using default configuration.
INFO 2020-10-21 18:44:21,917 [munit.01] org.apache.kafka.clients.producer.ProducerConfig: ProducerConfig values:
acks = -1
batch.size = 16384
bootstrap.servers = [hiding intentional]
buffer.memory = 1024000
client.dns.lookup = default
client.id = producer-1
compression.type = none
connections.max.idle.ms = 540000
delivery.timeout.ms = 120000
enable.idempotence = false
interceptor.classes = []
key.serializer = class com.mulesoft.connectors.kafka.internal.model.serializer.InputStreamSerializer
linger.ms = 0
max.block.ms = 60000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.max.age.ms = 300000
metadata.max.idle.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 1
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = [hidden]
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = PLAIN
security.protocol = SASL_SSL
security.providers = null
send.buffer.bytes = 131072
ssl.cipher.suites = [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
ssl.enabled.protocols = [TLSv1.2]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = SunJSSE
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = \C:/Users/xxx/AppData/Local/Temp/munit-temp-dir/munitworkingdir5345007588634892776/container/apps/app/cacerts
ssl.truststore.password = [hidden]
ssl.truststore.type = jks
transaction.timeout.ms = 60000
transactional.id = null
value.serializer = class com.mulesoft.connectors.kafka.internal.model.serializer.InputStreamSerializer
INFO 2020-10-21 18:44:21,985 [munit.01] org.apache.kafka.common.security.authenticator.AbstractLogin: Successfully logged in.
INFO 2020-10-21 18:44:21,996 [munit.01] org.apache.kafka.clients.producer.KafkaProducer: [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 0 ms.
org.mule.runtime.api.exception.MuleRuntimeException: org.mule.runtime.api.lifecycle.InitialisationException: The consumer has an invalid configuration
Caused by: org.mule.runtime.api.lifecycle.InitialisationException: The consumer has an invalid configuration
Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:434)
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:298)
at com.mulesoft.connectors.kafka.internal.connection.provider.ProducerConnectionProvider.initialise(ProducerConnectionProvider.java:437)
at com.mulesoft.connectors.kafka.internal.connection.provider.KafkaConnectionProvider.initialise(KafkaConnectionProvider.java:129)
at org.mule.runtime.core.api.lifecycle.LifecycleUtils.initialiseIfNeeded(LifecycleUtils.java:56)
at org.mule.runtime.core.api.lifecycle.LifecycleUtils.initialiseIfNeeded(LifecycleUtils.java:117)
at org.mule.runtime.core.internal.connection.AbstractConnectionProviderWrapper.initialise(AbstractConnectionProviderWrapper.java:113)
at org.mule.runtime.module.extension.internal.runtime.config.ClassLoaderConnectionProviderWrapper.initialise(ClassLoaderConnectionProviderWrapper.java:96)
at org.mule.runtime.core.api.lifecycle.LifecycleUtils.initialiseIfNeeded(LifecycleUtils.java:56)
at org.mule.runtime.core.api.lifecycle.LifecycleUtils.initialiseIfNeeded(LifecycleUtils.java:117)
at org.mule.runtime.core.internal.connection.AbstractConnectionProviderWrapper.initialise(AbstractConnectionProviderWrapper.java:113)
at org.mule.runtime.core.api.lifecycle.LifecycleUtils.initialiseIfNeeded(LifecycleUtils.java:56)
at org.mule.runtime.core.api.lifecycle.LifecycleUtils.initialiseIfNeeded(LifecycleUtils.java:117)
at org.mule.runtime.core.internal.connection.AbstractConnectionProviderWrapper.initialise(AbstractConnectionProviderWrapper.java:113)
at org.mule.runtime.core.api.lifecycle.LifecycleUtils.initialiseIfNeeded(LifecycleUtils.java:56)
at org.mule.runtime.core.api.lifecycle.LifecycleUtils.initialiseIfNeeded(LifecycleUtils.java:117)
at org.mule.runtime.module.extension.internal.runtime.config.LifecycleAwareConfigurationInstance.doInitialise(LifecycleAwareConfigurationInstance.java:297)
at org.mule.runtime.module.extension.internal.runtime.config.LifecycleAwareConfigurationInstance.initialise(LifecycleAwareConfigurationInstance.java:145)
at org.mule.runtime.core.api.lifecycle.LifecycleUtils.initialiseIfNeeded(LifecycleUtils.java:56)
at org.mule.runtime.core.api.lifecycle.LifecycleUtils.initialiseIfNeeded(LifecycleUtils.java:117)
at org.mule.runtime.module.extension.internal.runtime.config.LifecycleAwareConfigurationProvider.lambda$null$0(LifecycleAwareConfigurationProvider.java:83)
at org.mule.runtime.core.privileged.lifecycle.AbstractLifecycleManager.invokePhase(AbstractLifecycleManager.java:132)
at org.mule.runtime.core.internal.lifecycle.DefaultLifecycleManager.fireInitialisePhase(DefaultLifecycleManager.java:46)
at org.mule.runtime.module.extension.internal.runtime.config.LifecycleAwareConfigurationProvider.lambda$initialise$1(LifecycleAwareConfigurationProvider.java:81)
at org.mule.runtime.core.api.util.ExceptionUtils.tryExpecting(ExceptionUtils.java:224)
at org.mule.runtime.core.api.util.ClassUtils.withContextClassLoader(ClassUtils.java:966)
at org.mule.runtime.module.extension.internal.runtime.config.LifecycleAwareConfigurationProvider.initialise(LifecycleAwareConfigurationProvider.java:80)
at org.mule.runtime.core.api.lifecycle.LifecycleUtils.initialiseIfNeeded(LifecycleUtils.java:56)
at org.mule.runtime.core.api.util.func.CheckedConsumer.accept(CheckedConsumer.java:19)
at org.mule.runtime.core.internal.lifecycle.phases.DefaultLifecyclePhase.applyLifecycle(DefaultLifecyclePhase.java:115)
at org.mule.runtime.core.internal.lifecycle.phases.MuleContextInitialisePhase.applyLifecycle(MuleContextInitialisePhase.java:73)
at org.mule.runtime.config.internal.SpringRegistryLifecycleManager$SpringContextInitialisePhase.applyLifecycle(SpringRegistryLifecycleManager.java:128)
at org.mule.runtime.core.internal.lifecycle.RegistryLifecycleManager.doApplyLifecycle(RegistryLifecycleManager.java:175)
at org.mule.runtime.core.internal.lifecycle.RegistryLifecycleManager.applyPhase(RegistryLifecycleManager.java:146)
at org.mule.runtime.config.internal.SpringRegistry.applyLifecycle(SpringRegistry.java:289)
at org.mule.runtime.core.internal.registry.MuleRegistryHelper.applyLifecycle(MuleRegistryHelper.java:339)
at org.mule.runtime.config.internal.LazyMuleArtifactContext.initializeComponents(LazyMuleArtifactContext.java:287)
at org.mule.runtime.config.internal.LazyMuleArtifactContext.lambda$applyLifecycle$4(LazyMuleArtifactContext.java:250)
at org.mule.runtime.core.internal.context.DefaultMuleContext.withLifecycleLock(DefaultMuleContext.java:531)
at org.mule.runtime.config.internal.LazyMuleArtifactContext.applyLifecycle(LazyMuleArtifactContext.java:248)
at org.mule.runtime.config.internal.LazyMuleArtifactContext.initializeComponents(LazyMuleArtifactContext.java:329)
at org.mule.runtime.config.internal.LazyMuleArtifactContext.initializeComponents(LazyMuleArtifactContext.java:317)
at org.mule.munit.runner.config.TestComponentLocator.initializeComponents(TestComponentLocator.java:63)
at org.mule.munit.runner.model.builders.SuiteBuilder.build(SuiteBuilder.java:78)
at org.mule.munit.runner.remote.api.server.RunMessageHandler.buildSuite(RunMessageHandler.java:108)
at org.mule.munit.runner.remote.api.server.RunMessageHandler.parseSuiteMessage(RunMessageHandler.java:94)
at org.mule.munit.runner.remote.api.server.RunMessageHandler.parseAndRun(RunMessageHandler.java:81)
at org.mule.munit.runner.remote.api.server.RunMessageHandler.handle(RunMessageHandler.java:75)
at org.mule.munit.runner.remote.api.server.RunnerServer.handleClientMessage(RunnerServer.java:145)
at org.mule.munit.runner.remote.api.server.RunnerServer.run(RunnerServer.java:91)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.mule.service.scheduler.internal.AbstractRunnableFutureDecorator.doRun(AbstractRunnableFutureDecorator.java:111)
at org.mule.service.scheduler.internal.RunnableFutureDecorator.run(RunnableFutureDecorator.java:54)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.common.KafkaException: java.nio.file.InvalidPathException: Illegal char <:> at index 2: \C:/Users/xxx/AppData/Local/Temp/munit-temp-dir/munitworkingdir5345007588634892776/container/apps/app/cacerts
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:172)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:157)
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:73)
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
at org.apache.kafka.clients.producer.KafkaProducer.newSender(KafkaProducer.java:442)
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:423)
... 56 more
Caused by: java.nio.file.InvalidPathException: Illegal char <:> at index 2: \C:/Users/xxx/AppData/Local/Temp/munit-temp-dir/munitworkingdir5345007588634892776/container/apps/app/cacerts
at sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153)
at sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77)
at sun.nio.fs.WindowsPath.parse(WindowsPath.java:94)
at sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:255)
at java.nio.file.Paths.get(Paths.java:84)
at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.lastModifiedMs(SslEngineBuilder.java:298)
at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.<init>(SslEngineBuilder.java:275)
at org.apache.kafka.common.security.ssl.SslEngineBuilder.createTruststore(SslEngineBuilder.java:182)
at org.apache.kafka.common.security.ssl.SslEngineBuilder.<init>(SslEngineBuilder.java:100)
at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:95)
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:168)
... 61 more
This has been identified as a bug with the Anypoint Studio.
MuleSoft Support has suggested the below.
Upgrade the studio version to the latest patch 4.3.0-20200925 which has all the fixes included.
The patches are cumulative so all fixes from the august release will be available in the latest too.
The distributions are stored in the Mulesoft Nexus EE Repo repo so you will need to ensure you have the credentials configured in your settings.xml.
Follow this documentation Patching, which explains how to patch MUNITS using maven or studio.
Note: The first time execution may take a while because the Runtime artifacts will be downloaded to your local repo.
I have been trying to use Spark Structured Streaming API to connect to Kafka cluster with SASL_SSL. I have passed the jaas.conf file to the executors. It seems I couldn't set the values of keystore and truststore authentications.
I tried passing the values as mentioned in thisspark link
Also, tried passing it through the code as in this link
Still no luck.
Here is the log
20/02/28 10:00:53 INFO streaming.StreamExecution: Starting [id = e176f5e7-7157-4df5-93ce-1e267bae6125, runId = 03225a69-ec00-45d9-8092-1467da34980f]. Use flight/checkpoint to store the query checkpoint.
20/02/28 10:00:53 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0
20/02/28 10:00:53 INFO spark.SparkContext: Invoking stop() from shutdown hook
20/02/28 10:00:53 INFO server.AbstractConnector: Stopped Spark#46202f7b{HTTP/1.1,[http/1.1]}{0.0.0.0:0}
20/02/28 10:00:53 INFO consumer.ConsumerConfig: ConsumerConfig values:
metric.reporters = []
metadata.max.age.ms = 300000
partition.assignment.strategy = [org.apache.kafka.clients.consumer.RangeAssignor]
reconnect.backoff.ms = 50
sasl.kerberos.ticket.renew.window.factor = 0.8
max.partition.fetch.bytes = 1048576
bootstrap.servers = [broker1:9093, broker2:9093]
ssl.keystore.type = JKS
enable.auto.commit = false
sasl.mechanism = GSSAPI
interceptor.classes = null
exclude.internal.topics = true
ssl.truststore.password = null
client.id =
ssl.endpoint.identification.algorithm = null
max.poll.records = 1
check.crcs = true
request.timeout.ms = 40000
heartbeat.interval.ms = 3000
auto.commit.interval.ms = 5000
receive.buffer.bytes = 65536
ssl.truststore.type = JKS
ssl.truststore.location = null
ssl.keystore.password = null
fetch.min.bytes = 1
send.buffer.bytes = 131072
value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
group.id = spark-kafka-source-93d170e9-977c-40fc-9e5d-790d253fcff5-409016337-driver-0
retry.backoff.ms = 100
ssl.secure.random.implementation = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
ssl.trustmanager.algorithm = PKIX
ssl.key.password = null
fetch.max.wait.ms = 500
sasl.kerberos.min.time.before.relogin = 60000
connections.max.idle.ms = 540000
session.timeout.ms = 30000
metrics.num.samples = 2
key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
ssl.protocol = TLS
ssl.provider = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.keystore.location = null
ssl.cipher.suites = null
security.protocol = SASL_SSL
ssl.keymanager.algorithm = SunX509
metrics.sample.window.ms = 30000
auto.offset.reset = earliest
20/02/28 10:00:53 INFO ui.SparkUI: Stopped Spark web UI at http://<Server>:41037
20/02/28 10:00:53 ERROR streaming.StreamExecution: Query [id = e176f5e7-7157-4df5-93ce-1e267bae6125, runId = 03225a69-ec00-45d9-8092-1467da34980f] terminated with error
org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:702)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:557)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:540)
at org.apache.spark.sql.kafka010.SubscribeStrategy.createConsumer(ConsumerStrategy.scala:62)
at org.apache.spark.sql.kafka010.KafkaOffsetReader.createConsumer(KafkaOffsetReader.scala:297)
at org.apache.spark.sql.kafka010.KafkaOffsetReader.<init>(KafkaOffsetReader.scala:78)
at org.apache.spark.sql.kafka010.KafkaSourceProvider.createSource(KafkaSourceProvider.scala:88)
at org.apache.spark.sql.execution.datasources.DataSource.createSource(DataSource.scala:243)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$2$$anonfun$applyOrElse$1.apply(StreamExecution.scala:158)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$2$$anonfun$applyOrElse$1.apply(StreamExecution.scala:155)
at scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:194)
at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:80)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$2.applyOrElse(StreamExecution.scala:155)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anonfun$2.applyOrElse(StreamExecution.scala:153)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$2.apply(TreeNode.scala:267)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:70)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:266)
at org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:256)
at org.apache.spark.sql.execution.streaming.StreamExecution.logicalPlan$lzycompute(StreamExecution.scala:153)
at org.apache.spark.sql.execution.streaming.StreamExecution.logicalPlan(StreamExecution.scala:147)
at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runBatches(StreamExecution.scala:276)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:206)
Caused by: org.apache.kafka.common.KafkaException: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:86)
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:70)
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:83)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:623)
... 22 more
Caused by: javax.security.auth.login.LoginException: Could not login: the client is being asked for a password, but the Kafka client code does not currently support obtaining a password from the user. not available to garner authentication information from the user
at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Unknown Source)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Unknown Source)
at com.sun.security.auth.module.Krb5LoginModule.login(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at javax.security.auth.login.LoginContext.invoke(Unknown Source)
at javax.security.auth.login.LoginContext.access$000(Unknown Source)
at javax.security.auth.login.LoginContext$4.run(Unknown Source)
at javax.security.auth.login.LoginContext$4.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(Unknown Source)
at javax.security.auth.login.LoginContext.login(Unknown Source)
at org.apache.kafka.common.security.authenticator.AbstractLogin.login(AbstractLogin.java:69)
at org.apache.kafka.common.security.kerberos.KerberosLogin.login(KerberosLogin.java:110)
at org.apache.kafka.common.security.authenticator.LoginManager.<init>(LoginManager.java:46)
at org.apache.kafka.common.security.authenticator.LoginManager.acquireLoginManager(LoginManager.java:68)
at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:78)
... 25 more
20/02/28 10:00:53 INFO cluster.YarnClusterSchedulerBackend: Shutting down all executors
20/02/28 10:00:53 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
20/02/28 10:00:53 INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
services=List(),
started=false)
20/02/28 10:00:53 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
20/02/28 10:00:53 INFO memory.MemoryStore: MemoryStore cleared
20/02/28 10:00:53 INFO storage.BlockManager: BlockManager stopped
20/02/28 10:00:53 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
20/02/28 10:00:53 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
20/02/28 10:00:53 INFO spark.SparkContext: Successfully stopped SparkContext
20/02/28 10:00:53 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED
20/02/28 10:00:53 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
20/02/28 10:00:53 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://nameservice1/user/hasif.subair/.sparkStaging/application_1582866369627_0029
20/02/28 10:00:53 INFO util.ShutdownHookManager: Shutdown hook called
20/02/28 10:00:53 INFO util.ShutdownHookManager: Deleting directory /yarn/nm/usercache/hasif.subair/appcache/application_1582866369627_0029/spark-5addfec0-a99f-49e1-b9d1-671c331efb40
Code
val rawData = spark.readStream.format("kafka")
.option("kafka.bootstrap.servers", "broker1:9093, broker2:9093")
.option("subscribe", "hasif_test")
.option("spark.executor.extraJavaOptions", "-Djava.security.auth.login.config=jaas.conf")
.option("kafka.security.protocol", "SASL_SSL")
.option("ssl.truststore.location", "/etc/connect_ts/truststore.jks")
.option("ssl.truststore.password", "<PASSWORD>")
.option("ssl.keystore.location", "/etc/connect_ts/keystore.jks")
.option("ssl.keystore.password", "<PASSWORD>")
.option("ssl.key.password", "<PASSWORD>")
.load()
rawData.writeStream.option("path", "flight/output")
.option("checkpointLocation", "flight/checkpoint").format("csv").start()
spark-submit
spark2-submit --master yarn --deploy-mode cluster \
--conf spark.yarn.keytab=hasif.subair.keytab \
--conf spark.yarn.principal=hasif.subair#TEST.ABC \
--files /home/hasif.subair/jaas.conf \
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./jaas.conf" \
--conf "spark.driver.extraJavaOptions=-Djava.security.auth.login.config=./jaas.conf" \
--conf "spark.kafka.clusters.hasif.ssl.truststore.location=/etc/ts/truststore.jks" \
--conf "spark.kafka.clusters.hasif.ssl.truststore.password=testcluster" \
--conf "spark.kafka.clusters.hasif.ssl.keystore.location=/etc/ts/keystore.jks" \
--conf "spark.kafka.clusters.hasif.ssl.keystore.password=testcluster" \
--conf "spark.kafka.clusters.hasif.ssl.key.password=testcluster" \
--jars spark-sql-kafka-0-10_2.11-2.2.0.jar \
--class TestApp test_app_2.11-0.1.jar \
jaas.conf
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true
principal="hasif.subair#TEST.ABC"
useKeyTab=true
serviceName="kafka"
keyTab="hasif.subair.keytab"
client=true;
};
Any help will be deeply appreciated.
Kafka’s own configurations can be set via DataStreamReader.option with kafka. prefix, e.g.
val clusterName = "hasif"
stream.option(s"spark.kafka.clusters.${clusterName}.kafka.ssl.keystore.location", "/etc/connect_ts/keystore.jks")
Use kafka.ssl.truststore.location instead of ssl.truststore.location.
Similarly, you can prefix kafka for other properties and try.
I suspect the values for SSL is not getting picked up. As you can notice in your log the values are shown as null.
ssl.truststore.location = null
ssl.truststore.password = null
ssl.keystore.password = null
ssl.keystore.location = null
If the values are set properly it would reflect as
ssl.truststore.location = /etc/connect_ts/truststore.jks
ssl.truststore.password = [hidden]
ssl.keystore.password = [hidden]
ssl.keystore.location = /etc/connect_ts/keystore.jks
I am trying to run kafka connect in distributed mode to read from topic and write to HDFS.
The process is starting successfully with the required properties files given, but I am not able to commit the events to HDFS.
Please see the full logs below.
Note: Broker details have been masked and all other details are mentioned in the logs.
[2018-06-26 11:49:48,474] INFO ConsumerConfig values:
auto.commit.interval.ms = 5000
auto.offset.reset = earliest
bootstrap.servers = [****:****, ****:****]
check.crcs = true
client.id = consumer-3
connections.max.idle.ms = 540000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = connect-cluster
heartbeat.interval.ms = 3000
interceptor.classes = null
key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.ms = 50
request.timeout.ms = 305000
retry.backoff.ms = 100
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
session.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
(org.apache.kafka.clients.consumer.ConsumerConfig:180)
[2018-06-26 11:49:48,476] WARN The configuration 'config.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2018-06-26 11:49:48,476] WARN The configuration 'status.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2018-06-26 11:49:48,476] WARN The configuration 'internal.key.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2018-06-26 11:49:48,476] WARN The configuration 'rest.port' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2018-06-26 11:49:48,476] WARN The configuration 'value.converter.schema.registry.url' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2018-06-26 11:49:48,476] WARN The configuration 'internal.key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2018-06-26 11:49:48,476] WARN The configuration 'internal.value.converter.schemas.enable' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2018-06-26 11:49:48,476] WARN The configuration 'internal.value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2018-06-26 11:49:48,476] WARN The configuration 'offset.storage.topic' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2018-06-26 11:49:48,476] WARN The configuration 'value.converter' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2018-06-26 11:49:48,476] WARN The configuration 'key.converter' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2018-06-26 11:49:48,477] WARN The configuration 'key.converter.schema.registry.url' was supplied but isn't a known config. (org.apache.kafka.clients.consumer.ConsumerConfig:188)
[2018-06-26 11:49:48,477] INFO Kafka version : 0.10.1.0-IBM-8 (org.apache.kafka.common.utils.AppInfoParser:83)
[2018-06-26 11:49:48,477] INFO Kafka commitId : unknown (org.apache.kafka.common.utils.AppInfoParser:84)
[2018-06-26 11:49:48,682] INFO Discovered coordinator ****:**** (id: 2147483644 rack: null) for group connect-cluster. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:555)
[2018-06-26 11:49:48,686] INFO Finished reading KafkaBasedLog for topic connect-test-configs (org.apache.kafka.connect.util.KafkaBasedLog:148)
[2018-06-26 11:49:48,686] INFO Started KafkaBasedLog for topic connect-test-configs (org.apache.kafka.connect.util.KafkaBasedLog:150)
[2018-06-26 11:49:48,686] INFO Started KafkaConfigBackingStore (org.apache.kafka.connect.storage.KafkaConfigBackingStore:261)
[2018-06-26 11:49:48,687] INFO Herder started (org.apache.kafka.connect.runtime.distributed.DistributedHerder:171)
[2018-06-26 11:49:48,689] INFO Discovered coordinator ****:**** (id: 2147483644 rack: null) for group connect-cluster. (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:555)
[2018-06-26 11:49:48,691] INFO (Re-)joining group connect-cluster (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:381)
[2018-06-26 11:49:48,702] INFO Successfully joined group connect-cluster with generation 3 (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:349)
[2018-06-26 11:49:48,702] INFO Joined group and got assignment: Assignment{error=0, leader='connect-1-67ad9053-4b20-41dd-9a6a-ec8e4f4d622e', leaderUrl='http://****:****/', offset=-1, connectorIds=[], taskIds=[]} (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1009)
[2018-06-26 11:49:48,702] INFO Starting connectors and tasks using config offset -1 (org.apache.kafka.connect.runtime.distributed.DistributedHerder:745)
[2018-06-26 11:49:48,702] INFO Finished starting connectors and tasks (org.apache.kafka.connect.runtime.distributed.DistributedHerder:752)
[2018-06-26 11:49:55,748] INFO Reflections took 7022 ms to scan 225 urls, producing 15521 keys and 98977 values (org.reflections.Reflections:229)
but I am not able to commit the events to HDFS
When you start distributed mode, it doesn't take any individual connector properties, you load those later
Just want to know where should we give the properties of the topics,hdfs url,task in a distributed mode?
You POST JSON into it.
Make a connect-hdfs.json file, for example
{
"name": "quickstart-hdfs-sink"
"config": {
"store.url": "hdfs:///apps/kafka-connect",
"hadoop.conf.dir": "/etc/hadoop/conf",
"topics" : ...
}
}
Send it to Connect
curl -XPOST -d#connect-hdfs.json http://connect-server:8083/connectors