I'm trying to create kafka producer with ssl. I need information on how to set SSL parameters in the constructor, the information provided in kafka-python client is not descriptive enough.
What are the ssl_certfile, ssl_cafile, ssl_keyfile parameters. I'm not sure where to look for these files.
producer = KafkaProducer(bootstrap_servers=kafka_broker,
value_serializer=lambda v: json.dumps(v).encode('utf-8'),
security_protocol='SSL',
api_version=(0,10),
ssl_cafile='ca-certs.pem',ssl_certfile='server.pem',
ssl_keyfile='server.pem',ssl_password='xxx')
producer.send('rk976772_topic',{"test":0})
Traceback (most recent call last): File "", line 1, in
File
"/usr/lib/python2.7/site-packages/kafka/producer/kafka.py", line 543,
in send
self._wait_on_metadata(topic, self.config['max_block_ms'] / 1000.0) File "/usr/lib/python2.7/site-packages/kafka/producer/kafka.py", line 664,
in _wait_on_metadata
"Failed to update metadata after %.1f secs." % max_wait) kafka.errors.KafkaTimeoutError: KafkaTimeoutError: Failed to update
metadata after 60.0 secs.
I was having this issue as well as many other while trying to configure kafka with SSL or SASL_SSL. I'm posting a full tutorial here in case anyone else runs into the same issues. I am using kafka-python 1.4.6 with kafka 2.2.0 on CentOS 6.
Below are the configurations that worked for me for SASL_SSL using kafka-python client. These configurations can be used for PLAINTEXT and SSL security protocols along with SASL_SSL and SASL_PLAINTEXT.
Bash script to generate key files, CARoot, and self-signed cert for use with SSL:
#!/bin/bash
#Step 1
keytool -keystore server.keystore.jks -alias localhost -validity 365 -keyalg RSA -genkey
#Step 2
openssl req -new -x509 -keyout ca-key -out ca-cert -days 365
keytool -keystore server.truststore.jks -alias CARoot -import -file ca-cert
keytool -keystore client.truststore.jks -alias CARoot -import -file ca-cert
#Step 3
keytool -keystore server.keystore.jks -alias localhost -certreq -file cert-file
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days 365 -CAcreateserial -passin pass:admin123
keytool -keystore server.keystore.jks -alias CARoot -import -file ca-cert
keytool -keystore server.keystore.jks -alias localhost -import -file cert-signed
You can then use the following command to extract the CARoot.pem:
keytool -exportcert -alias CARoot -keystore server.keystore.jks -rfc -file CARoot.pem
In my server.properties file I have:
listeners=PLAINTEXT://localhost:9091,SASL_PLAINTEXT://localhost:9092,SSL://localhost:9093,SASL_SSL://localhost:9094
security.protocol=SSL
sasl.enabled.mechanisms=PLAIN
ssl.truststore.location=/var/private/ssl/server.truststore.jks
ssl.truststore.password=admin123
ssl.keystore.location=/var/private/ssl/server.keystore.jks
ssl.keystore.password=admin123
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
advertised.listeners=PLAINTEXT://localhost:9091,SASL_PLAINTEXT://localhost:9092,SSL://localhost:9093,SASL_SSL://localhost:9094
In my JAAS configuration file(/etc/kafka/kafka_plain_jaas.conf):
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username=kafka
password=kafka-secret
user_username=password;
};
KafkaClient {
org.apache.kafka.common.security.plain.PlainLoginModule required
username=username
password=password;
};
Before starting the Kafka server, need to run the following:
export KAFKA_OPTS="-Djava.security.auth.login.config=/etc/kafka/kafka_plain_jaas.conf"
Python consumer and producer:
The ssl_context and api_version are what caused SSL handshake errors to occur for me, leading to a timeout. So I commented those out. (There were some tutorials out there that mentioned to use those.)
from kafka import KafkaConsumer, KafkaProducer
import kafka
import ssl
import logging
logging.basicConfig(level=logging.DEBUG)
try:
topic = "sendMessage"
sasl_mechanism = "PLAIN"
username = "username"
password = "password"
security_protocol = "SASL_SSL"
#context = ssl.create_default_context()
#context.options &= ssl.OP_NO_TLSv1
#context.options &= ssl.OP_NO_TLSv1_1
consumer = KafkaConsumer(topic, bootstrap_servers='localhost:9094',
#api_version=(0, 10),
security_protocol=security_protocol,
#ssl_context=context,
ssl_check_hostname=True,
ssl_cafile='../keys/CARoot.pem',
sasl_mechanism = sasl_mechanism,
sasl_plain_username = username,
sasl_plain_password = password)
#ssl_certfile='../keys/certificate.pem',
#ssl_keyfile='../keys/key.pem')#,api_version = (0, 10))
producer = KafkaProducer(bootstrap_servers='localhost:9094',
#api_version=(0, 10),
security_protocol=security_protocol,
#ssl_context=context,
ssl_check_hostname=True,
ssl_cafile='../keys/CARoot.pem',
sasl_mechanism=sasl_mechanism,
sasl_plain_username=username,
sasl_plain_password=password)
#ssl_certfile='../keys/certificate.pem',
#ssl_keyfile='../keys/key.pem')#, api_version = (0,10))
# Write hello world to test topic
producer.send(topic, bytes("Hello World SSL"))
producer.flush()
for msg in consumer:
print(msg)
except Exception as e:
print e
Dudes, watch carefully and follow the instructions...
Step 1: Run all scripts (if necessary, set the values)
keytool -keystore kafka.server.keystore.jks -alias localhost -keyalg RSA -validity {validity} -genkey
openssl req -new -x509 -keyout ca-key -out ca-cert -days {validity}
keytool -keystore kafka.client.truststore.jks -alias CARoot -importcert -file ca-cert
keytool -keystore kafka.server.truststore.jks -alias CARoot -importcert -file ca-cert
keytool -keystore kafka.server.keystore.jks -alias localhost -certreq -file cert-file
openssl x509 -req -CA ca-cert -CAkey ca-key -in cert-file -out cert-signed -days {validity} -CAcreateserial -passin pass:{ca-password}
keytool -keystore kafka.server.keystore.jks -alias CARoot -importcert -file ca-cert
keytool -keystore kafka.server.keystore.jks -alias localhost -importcert -file cert-signed
keytool -exportcert -alias CARoot -keystore kafka.server.keystore.jks -rfc -file CARoot.pem
As a result, you will get:
kafka.server.keystore.jks, kafka.server.truststore.jks, kafka.client.truststore.jks, ca-cert, ca-cert.srl, ca-key, cert-file, cert-signed, CARoot.pem
Step 2: Copy kafka.server.keystore.jks and kafka.server.truststore.jks to the server and modify server.properties file (it is located in the config folder)
listeners=PLAINTEXT://MYSERVER:9092,SSL://MYSERVER:9093
advertised.listeners=PLAINTEXT://MYSERVER:9092,SSL://MYSERVER:9093
ssl.keystore.location=../store/kafka.server.keystore.jks
ssl.keystore.password=qwerty
ssl.truststore.location=../store/kafka.server.truststore.jks
ssl.truststore.password=qwerty
ssl.client.auth=required
ssl.endpoint.identification.algorithm=
Step 3: Create the python program
def kafka_consumer_ssl():
consumer = KafkaConsumer('test_topic',
bootstrap_servers=['MYSERVER:9093'],
auto_offset_reset='earliest',
enable_auto_commit=True,
value_deserializer=lambda x: x.decode('utf-8'),
security_protocol='SSL',
ssl_check_hostname=False,
ssl_cafile='CARoot.pem',
ssl_certfile='ca-cert',
ssl_keyfile='ca-key',
ssl_password='qwerty'
)
for event in consumer:
print(event.value)
kafka_consumer_ssl()
Step 4: Enjoy !!!
I had to publish the message over SASL_SSL
Used below code to create a producer with SASL_SSL protocol.
from kafka import KafkaProducer
security_protocol=environment_params.kafka_security_protocol
if env=='dev':
if security_protocol=='SASL_SSL':
producer = KafkaProducer(bootstrap_servers=environment_params.dev_kafka_broker,value_serializer=lambda v: json.dumps(v).encode('utf-8'),security_protocol=security_protocol,ssl_cafile='ca-certs.pem',sasl_mechanism='GSSAPI',api_version=environment_params.dev_kafka_api_version)
elif security_protocol=='PLAINTEXT':
producer = KafkaProducer(bootstrap_servers=environment_params.dev_kafka_broker,value_serializer=lambda v: json.dumps(v).encode('utf-8'))
Thanks Alot. I was having jks file and my kafka-producer was giving continuously error SSL Certification verify error 897
Though converted the CARoot.pem file but it was not working.
What helped is I converted using the below command and used on producer and it worked.
kafka.server.keystore.jks,
kafka.server.truststore.jks,
kafka.client.truststore.jks,
ca-cert,
ca-cert.srl,
ca-key, cert-file,
cert-signed,
CARoot.pem
When I'm trying to convert pfx file, which was generated without password, to jks I get a WARNING WARNING etc... message from keytool, and an error afterwards
When I do the same with an password protected pfx, then everything is fine.
Can anyone suggest what I can do !? maybe a conversion from other formats or using other tools ?
ps. I did also conversion to pem, and pem to jks, but it failed, because it was not an x509 cert.
EDIT
keytool.exe -importkeystore -srckeystore "C:\Users\rodislav.moldovan\Projects
\ceva.pfx" -srcstoretype pkcs12 -destkeystore "C:\Users\rodislav.mol
dovan\Projects\ceva.jks" -deststoretype JKS
Enter destination keystore password: ******
Re-enter new password: ******
Enter source keystore password: // pressed enter, because there is no pass
***************** WARNING WARNING WARNING *****************
* The integrity of the information stored in the srckeystore*
* has NOT been verified! In order to verify its integrity, *
* you must provide the srckeystore password. *
***************** WARNING WARNING WARNING *****************
keytool error: java.security.UnrecoverableKeyException: Get Key failed: null
You can do it by making a p12 keystore first with OpenSSL and then convert it into JKS format with Keytool.
OpenSSL for CER & PVK file > P12
openssl pkcs12 -export -name servercert -in selfsignedcert.crt -inkey serverprivatekey.key -out myp12keystore.p12
Keytool for p12 > JKS
keytool -importkeystore -destkeystore mykeystore.jks -srckeystore myp12keystore.p12 -srcstoretype pkcs12 -alias servercert
Try to convert it to a p12 with a password before.
openssl pkcs12 -in in.pfx -out out.p12
If you just have a full PFX file that isn't password protected; for instance you downloaded the cert from Azure Key Vault like so:
az keyvault secret download -f mycert.pfx --encoding base64 --vault-name <vault name> --name <certificate name>
Then you can jump through a few hoops to add password protection (got this from here: http://www.1st-setup.nl/wordpress/howto-change-password-on-pfx-certificate-using-openssl/):
openssl pkcs12 -in mycert.pfx -out temppem.pem -nodes
openssl pkcs12 -export -out protectedcert.pfx -in temppem.pem
rm certs/mycert.pfx
rm certs/temppem.pem
Obviously you need to specify a password in the second openssl command to pw-protect the new PFX.
I wanted to create a certificate into a PKCS12 keystore format with keytool program.
The keystore has extension .pfx.
How do I achieve this?
If the keystore is PKCS12 type (.pfx) you have to specify it with -storetype PKCS12 (line breaks added for readability):
keytool -genkey -alias <desired certificate alias>
-keystore <path to keystore.pfx>
-storetype PKCS12
-keyalg RSA
-storepass <password>
-validity 730
-keysize 2048
Additional answer to the key of the question.
With JDK 8 (1.8.0_121-b13) you don't get an exception if you remove -storetype pkcs12 but the keytool creates a JKS keystore instead, and the .pfx extension is ignored.
It also asks for a -keypass mykeypassword which the keytool doesn't support for PKCS12.
%JAVA_HOME%/bin/keytool -genkeypair -alias mykey -keyalg EC -dname "cn=CN, ou=OU, o=O, c=C" -validity 365 -keystore keystore.pfx -keypass mykeypassword -storepass mystorepassword -v
(translated)
Generating keypair (Type EC, 256 Bit) and self-signed certificate (SHA256withECDSA) with a validity of 365 days
for: CN=CN, OU=OU, O=O, C=C
[keystore.pfx saved]
List the contents:
%JAVA_HOME%/bin/keytool -list -keystore keystore.pfx -storepass mystorepassword
(translated)
Keystore-Type: JKS
Keystore-Provider: SUN
Keystore contains 1 entry.
mykey, 25.04.2017, PrivateKeyEntry,
Certificate-Fingerprint (SHA1): A1:6C:5F:8F:43:37:1A:B6:43:69:08:DE:6B:B9:4D:DB:05:C9:D5:84
You see it's a Java keystore.
The next problem is, that even if you specify -storetype pkcs12 when you -list the keystore, the keytool will still display the store as a JKS keystore!
Let's try that:
%JAVA_HOME%/bin/keytool -genkeypair -alias mykey -keyalg EC -dname "cn=CN, ou=OU, o=O, c=C" -validity 365 -storetype pkcs12 -keystore keystore.pkx -keypass mykeypassword -storepass mystorepassword -v
(translated)
Warning: No support for different keystore and key password for PKCS12 keystores. The value of -keypass will be ignored.
Generating keypair (Type EC, 256 Bit) and self signed certificate (SHA256withECDSA) with a validity of 365 Days
für: CN=CN, OU=OU, O=O, C=C
[keystore.pkx saved]
Now list the contents:
%JAVA_HOME%/bin/keytool -list -keystore keystore.pkx -storepass mystorepassword
(translated)
Keystore-Type: JKS // ??
Keystore-Provider: SUN
Keystore contains 1 entry
mykey, 25.04.2017, PrivateKeyEntry,
Certificate Fingerprint (SHA1): EA:C2:36:C6:55:69:CB:32:22:C7:14:83:67:47:D2:7E:06:8E:13:14
I'm trying to convert x.PFX file to x.JKS file using keytool but I am getting following error:
keytool error: java.lang.Exception: Alias <2> does not exist
Actions that preceded this error are:
Listing x.PFX file content (just to read alias name):
keytool -v -list -storetype pkcs12 -keystore x.pfx
Enter keystore password: x
Keystore type: PKCS12
Keystore provider: SunJSSE
Your keystore contains 1 entry
Alias name: 2
Creation date: 11-nov-2012
Entry type: PrivateKeyEntry
Certificate chain length: 3
Certificate[1]:
Owner: CN=x, OU=x, C=x
Issuer: CN=x, O=x, C=x
Serial number: x
Valid from: Wed Oct 24 11:46:10 CEST 2012 until: Fri Dec 13 09:28:40 CET 2013
Certificate fingerprints:
etc.
Converting x.PFX file into x.JKS file using "2" as source alias name
keytool -importkeystore -srckeystore x.pfx -srcstoretype pkcs12 -srcalias 2 -destkeystore x.jks -deststoretype jks -destalias xyz
Enter destination keystore password: y
Re-enter new password: y
Enter source keystore password: x
keytool error: java.lang.Exception: Alias <2> does not exist
I am not sure what I am doing wrong? PFX file contain only one entry with just one alias (2). I also tried using these srcalias values: 2, "2", " 2". Is there any other way to convert PFX into JKS using keytool without knowing source alias name?
I had the exact same problem. I've solved using '1' instead of 2. Don't know why but it worked.
if set alias in pkcs12:
openssl pkcs12 -export -in certificate.pem -inkey private_key.pem -out keystore.p12 -name "myalias"
aftet alias setted successfully:
keytool -importkeystore -srckeystore keystore.p12 -srcstoretype pkcs12 -destkeystore keystore.jks -deststoretype JKS -alias myalias
your command should looks a bit more like this
keytool -importkeystore -srckeystore x.pfx -srcstoretype pkcs12 -***alias*** 2 -destkeystore x.jks -deststoretype jks -destalias xyz
Maybe "2" can't be found, because there are whitespaces included (e.g. "2 ")
If you don't wanna change the alias just remove the options -srcalias and -destalias and it will be imported with the original alias.