I'm trying to create kafka producer with ssl. I need information on how to set SSL parameters in the constructor, the information provided in kafka-python client is not descriptive enough.
What are the ssl_certfile, ssl_cafile, ssl_keyfile parameters. I'm not sure where to look for these files.
producer = KafkaProducer(bootstrap_servers=kafka_broker,
value_serializer=lambda v: json.dumps(v).encode('utf-8'),
security_protocol='SSL',
api_version=(0,10),
ssl_cafile='ca-certs.pem',ssl_certfile='server.pem',
ssl_keyfile='server.pem',ssl_password='xxx')
producer.send('rk976772_topic',{"test":0})
Traceback (most recent call last): File "", line 1, in
File
"/usr/lib/python2.7/site-packages/kafka/producer/kafka.py", line 543,
in send
self._wait_on_metadata(topic, self.config['max_block_ms'] / 1000.0) File "/usr/lib/python2.7/site-packages/kafka/producer/kafka.py", line 664,
in _wait_on_metadata
"Failed to update metadata after %.1f secs." % max_wait) kafka.errors.KafkaTimeoutError: KafkaTimeoutError: Failed to update
metadata after 60.0 secs.
I was having this issue as well as many other while trying to configure kafka with SSL or SASL_SSL. I'm posting a full tutorial here in case anyone else runs into the same issues. I am using kafka-python 1.4.6 with kafka 2.2.0 on CentOS 6.
Below are the configurations that worked for me for SASL_SSL using kafka-python client. These configurations can be used for PLAINTEXT and SSL security protocols along with SASL_SSL and SASL_PLAINTEXT.
Bash script to generate key files, CARoot, and self-signed cert for use with SSL:
#!/bin/bash
#Step 1
keytool -keystore server.keystore.jks -alias localhost -validity 365 -keyalg RSA -genkey
#Step 2
openssl req -new -x509 -keyout ca-key -out ca-cert -days 365
keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert
keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert
#Step 3
keytool -keystore server.keystore.jks -alias localhost -certreq -file cert-file
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:admin123
keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert
keytool -keystore server.keystore.jks -alias localhost -import -file cert-signed
You can then use the following command to extract the CARoot.pem:
keytool -exportcert -alias CARoot -keystore server.keystore.jks -rfc -file CARoot.pem
In my server.properties file I have:
listeners=PLAINTEXT://localhost:9091,SASL_PLAINTEXT://localhost:9092,SSL://localhost:9093,SASL_SSL://localhost:9094
security.protocol=SSL
sasl.enabled.mechanisms=PLAIN
ssl.truststore.location=/var/private/ssl/server.truststore.jks
ssl.truststore.password=admin123
ssl.keystore.location=/var/private/ssl/server.keystore.jks
ssl.keystore.password=admin123
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
advertised.listeners=PLAINTEXT://localhost:9091,SASL_PLAINTEXT://localhost:9092,SSL://localhost:9093,SASL_SSL://localhost:9094
In my JAAS configuration file(/etc/kafka/kafka_plain_jaas.conf):
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username=kafka
password=kafka-secret
user_username=password;
};
KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
username=username
password=password;
};
Before starting the Kafka server, need to run the following:
export KAFKA_OPTS="-Djava.security.auth.login.config=/etc/kafka/kafka_plain_jaas.conf"
Python consumer and producer:
The ssl_context and api_version are what caused SSL handshake errors to occur for me, leading to a timeout. So I commented those out. (There were some tutorials out there that mentioned to use those.)
from kafka import KafkaConsumer, KafkaProducer
import kafka
import ssl
import logging
logging.basicConfig(level=logging.DEBUG)
try:
topic = "sendMessage"
sasl_mechanism = "PLAIN"
username = "username"
password = "password"
security_protocol = "SASL_SSL"
#context = ssl.create_default_context()
#context.options &= ssl.OP_NO_TLSv1
#context.options &= ssl.OP_NO_TLSv1_1
consumer = KafkaConsumer(topic, bootstrap_servers='localhost:9094',
#api_version=(0, 10),
security_protocol=security_protocol,
#ssl_context=context,
ssl_check_hostname=True,
ssl_cafile='../keys/CARoot.pem',
sasl_mechanism = sasl_mechanism,
sasl_plain_username = username,
sasl_plain_password = password)
#ssl_certfile='../keys/certificate.pem',
#ssl_keyfile='../keys/key.pem')#,api_version = (0, 10))
producer = KafkaProducer(bootstrap_servers='localhost:9094',
#api_version=(0, 10),
security_protocol=security_protocol,
#ssl_context=context,
ssl_check_hostname=True,
ssl_cafile='../keys/CARoot.pem',
sasl_mechanism=sasl_mechanism,
sasl_plain_username=username,
sasl_plain_password=password)
#ssl_certfile='../keys/certificate.pem',
#ssl_keyfile='../keys/key.pem')#, api_version = (0,10))
# Write hello world to test topic
producer.send(topic, bytes("Hello World SSL"))
producer.flush()
for msg in consumer:
print(msg)
except Exception as e:
print e
Dudes, watch carefully and follow the instructions...
Step 1: Run all scripts (if necessary, set the values)
keytool -keystore kafka.server.keystore.jks -alias localhost -keyalg RSA -validity {validity} -genkey
openssl req -new -x509 -keyout ca-key -out ca-cert -days {validity}
keytool -keystore kafka.client.truststore.jks -alias CARoot -importcert -file ca-cert
keytool -keystore kafka.server.truststore.jks -alias CARoot -importcert -file ca-cert
keytool -keystore kafka.server.keystore.jks -alias localhost -certreq -file cert-file
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days {validity} -CAcreateserial -passin pass:{ca-password}
keytool -keystore kafka.server.keystore.jks -alias CARoot -importcert -file ca-cert
keytool -keystore kafka.server.keystore.jks -alias localhost -importcert -file cert-signed
keytool -exportcert -alias CARoot -keystore kafka.server.keystore.jks -rfc -file CARoot.pem
As a result, you will get:
kafka.server.keystore.jks, kafka.server.truststore.jks, kafka.client.truststore.jks, ca-cert, ca-cert.srl, ca-key, cert-file, cert-signed, CARoot.pem
Step 2: Copy kafka.server.keystore.jks and kafka.server.truststore.jks to the server and modify server.properties file (it is located in the config folder)
listeners=PLAINTEXT://MYSERVER:9092,SSL://MYSERVER:9093
advertised.listeners=PLAINTEXT://MYSERVER:9092,SSL://MYSERVER:9093
ssl.keystore.location=../store/kafka.server.keystore.jks
ssl.keystore.password=qwerty
ssl.truststore.location=../store/kafka.server.truststore.jks
ssl.truststore.password=qwerty
ssl.client.auth=required
ssl.endpoint.identification.algorithm=
Step 3: Create the python program
def kafka_consumer_ssl():
consumer = KafkaConsumer('test_topic',
bootstrap_servers=['MYSERVER:9093'],
auto_offset_reset='earliest',
enable_auto_commit=True,
value_deserializer=lambda x: x.decode('utf-8'),
security_protocol='SSL',
ssl_check_hostname=False,
ssl_cafile='CARoot.pem',
ssl_certfile='ca-cert',
ssl_keyfile='ca-key',
ssl_password='qwerty'
)
for event in consumer:
print(event.value)
kafka_consumer_ssl()
Step 4: Enjoy !!!
I had to publish the message over SASL_SSL
Used below code to create a producer with SASL_SSL protocol.
from kafka import KafkaProducer
security_protocol=environment_params.kafka_security_protocol
if env=='dev':
if security_protocol=='SASL_SSL':
producer = KafkaProducer(bootstrap_servers=environment_params.dev_kafka_broker,value_serializer=lambda v: json.dumps(v).encode('utf-8'),security_protocol=security_protocol,ssl_cafile='ca-certs.pem',sasl_mechanism='GSSAPI',api_version=environment_params.dev_kafka_api_version)
elif security_protocol=='PLAINTEXT':
producer = KafkaProducer(bootstrap_servers=environment_params.dev_kafka_broker,value_serializer=lambda v: json.dumps(v).encode('utf-8'))
Thanks Alot. I was having jks file and my kafka-producer was giving continuously error SSL Certification verify error 897
Though converted the CARoot.pem file but it was not working.
What helped is I converted using the below command and used on producer and it worked.
kafka.server.keystore.jks,
kafka.server.truststore.jks,
kafka.client.truststore.jks,
ca-cert,
ca-cert.srl,
ca-key, cert-file,
cert-signed,
CARoot.pem
Because when I run keytool in cmf.exe fill all fields and keep asking me back
keytool -genkey -v -keystore [keystore_name].keystore -alias [alias_name] -keyalg RSA -keysize 2048 -validity 10000
Do not generate the keystore, after asking if everything is correct, and I put yes becomes responsive again. Because? I'm doing wrong?
Not sure what is going wrong for you. And it works for me.
Try giving all the parameters in one go, like this :
keytool -genkey -v -keystore test.jks -alias test
-keyalg RSA -keysize 2048 -validity 10000 -storepass test-keypass test-dname "CN=test.com,OU=Test,O=Test,L=Test,S=Test,C=Test"
This should store the keystore without prompting for confirmation.
I had the same issue, and I realized that it was because of my OS language (I have spanish by default), so when I answered with "si" it worked
Question is around how to quickly generate a self signed certificate that you can use with Agg Cat services OR with other intuit services.
Is there a simple script.
Here is a simple script I created for this :
#!/bin/bash
if (( $# != 2 ))
then
echo "Usage: aliasname password"
exit 1
fi
keytool -genkey -keystore ./$1.p12 -deststoretype PKCS12 -storepass $2 -alias $1 -keyalg "RSA" -keysize 2048 -validity 9000
openssl pkcs12 -in $1.p12
keytool -v -importkeystore -srckeystore ./$1.p12 -srcstoretype PKCS12 -srcstorepass $2 -destkeystore ./$1.jks -deststoretype JKS -deststorepass $2
This will create an SSL certificate and print the private key and public cert.
You use the pub cert during app creation
Use the private key in the sample app of API Explorer.
I wanted to create a certificate into a PKCS12 keystore format with keytool program.
The keystore has extension .pfx.
How do I achieve this?
If the keystore is PKCS12 type (.pfx) you have to specify it with -storetype PKCS12 (line breaks added for readability):
keytool -genkey -alias <desired certificate alias>
-keystore <path to keystore.pfx>
-storetype PKCS12
-keyalg RSA
-storepass <password>
-validity 730
-keysize 2048
Additional answer to the key of the question.
With JDK 8 (1.8.0_121-b13) you don't get an exception if you remove -storetype pkcs12 but the keytool creates a JKS keystore instead, and the .pfx extension is ignored.
It also asks for a -keypass mykeypassword which the keytool doesn't support for PKCS12.
%JAVA_HOME%/bin/keytool -genkeypair -alias mykey -keyalg EC -dname "cn=CN, ou=OU, o=O, c=C" -validity 365 -keystore keystore.pfx -keypass mykeypassword -storepass mystorepassword -v
(translated)
Generating keypair (Type EC, 256 Bit) and self-signed certificate (SHA256withECDSA) with a validity of 365 days
for: CN=CN, OU=OU, O=O, C=C
[keystore.pfx saved]
List the contents:
%JAVA_HOME%/bin/keytool -list -keystore keystore.pfx -storepass mystorepassword
(translated)
Keystore-Type: JKS
Keystore-Provider: SUN
Keystore contains 1 entry.
mykey, 25.04.2017, PrivateKeyEntry,
Certificate-Fingerprint (SHA1): A1:6C:5F:8F:43:37:1A:B6:43:69:08:DE:6B:B9:4D:DB:05:C9:D5:84
You see it's a Java keystore.
The next problem is, that even if you specify -storetype pkcs12 when you -list the keystore, the keytool will still display the store as a JKS keystore!
Let's try that:
%JAVA_HOME%/bin/keytool -genkeypair -alias mykey -keyalg EC -dname "cn=CN, ou=OU, o=O, c=C" -validity 365 -storetype pkcs12 -keystore keystore.pkx -keypass mykeypassword -storepass mystorepassword -v
(translated)
Warning: No support for different keystore and key password for PKCS12 keystores. The value of -keypass will be ignored.
Generating keypair (Type EC, 256 Bit) and self signed certificate (SHA256withECDSA) with a validity of 365 Days
für: CN=CN, OU=OU, O=O, C=C
[keystore.pkx saved]
Now list the contents:
%JAVA_HOME%/bin/keytool -list -keystore keystore.pkx -storepass mystorepassword
(translated)
Keystore-Type: JKS // ??
Keystore-Provider: SUN
Keystore contains 1 entry
mykey, 25.04.2017, PrivateKeyEntry,
Certificate Fingerprint (SHA1): EA:C2:36:C6:55:69:CB:32:22:C7:14:83:67:47:D2:7E:06:8E:13:14
I used the -validity property of the keytool to generate a keystore and self sign a jar. During the process, I saw the debug message that says
[info] Warning:
[info] This jar contains entries whose signer certificate will expire within six
months.
I am using the following commands,
keytool -genkey -alias signFiles -keypass mypass -keystore mystore -storepass mypass-alias store -validity 365
keytool -selfcert -keystore mystore -alias store
Am I doing something wrong here? Actually I wanted to sign the jar with a certificate for 1 year.
Cheers,
J
Add -validity 365 to your -selfcert command.