I have been reading a lot of Apache Kafka documentation, but am unable to find if Kafka supports secured communication between Producers-Brokers, Brokers-Consumers, and especially for inter-data center communication for broker replication.
Update: As of 0.9.0.0 the SSL implementation is added to Kafka. The SSL configuration should be added explicitly on the broker side.
It is even possible to enable SSL for inter-broker communication by adding following property to the broker's property file.
security.inter.broker.protocol=SSL
Regarding the producer and consumers the SSL is only supported for the new API.
For details regarding generation of key, certificate and configuration please check deploying SSL for Kafka
Previous Releases
Before 0.9.0.0
Kafka does not support SSL/authentication and as far as my understanding goes they do not have it in their near team road map. One way could be to use encryption at your end and send the encrypted data through producers. However they have this discussion regarding implementing security in future.
Similar discussion can be found here
UPDATE
Thanks to #ppearcy for his findings
Likely things changed since this was posted, but currently security is on the roadmap: https://cwiki.apache.org/confluence/display/KAFKA/Security
There have been patches to support client authentication and secure in transit message delivery but note this has not made it to any production release yet.
There are interesting discussions and future work that includes client authentication, authorization as well as encryption of data at rest https://www.mail-archive.com/dev#kafka.apache.org/msg11664.html
Related
I followed the tutorial of the answer of this question:
Kafka SASL zookeeper authentication
And i setted zookeeper.set.acl=true in the server.propeties, but i still can access the zookeeper on port 2181 and this is available for anyone through the: kafka-topics --zookeeper <server-name>:2181 --list
ps: instead of <server-name> i put the DN of my server.
Authentication enforcement feature has recently been submitted in the ZooKeeper codebase and afaik there's no stable version released yet which supports it.
When you turn on SASL authentication, it will be available, but clients are still able to connect without it. Hence the recommendation is to use ACLs side by side with authentication to prevent non-authenticated user from accessing sensitive data.
As per https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-custom-domain-tls-version.html it is possible to restrict aws api gateway to a particular version of TLS. Rather than just change this, I'm keen to log any existing connections that are being established over TLS 1.0 or 1.1 in order to identify and fix those apps generating insecure connections. Does anyone know if it's possible to have the service log this?
There is not currently a way to log this in API Gateway logging but it sounds like a good feature request. I'd recommend submitting this request in the forums if you can as the community there may make enough noise to push the feature request quicker.
I see that access logs allow you to log any context variable so I think the best feature request would be to ask for a context variable containing the negotiated TLS version so it can be logged, or for it to be logged by default in the execution logs when full request/response data is enabled for CloudWatch logging.
The REST API for Kafka Connect is not secured and authenticated.
Since its not authenticated, the configuration for a connector or Tasks are easily accessible by anyone. Since these configurations may contain about how to access the Source System [in case of SourceConnector] and destination system [in case of SinkConnector], Is there a standard way to restrict access to these APIs?
In Kafka 2.1.0, there is possibility to configure http basic authentication for REST interface of Kafka Connect without writing any custom code.
This became real due to implementation of REST extensions mechanism (see KIP-285).
Shortly, configuration procedure as follows:
Add extension class to worker configuration file:
rest.extension.classes = org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension
Create JAAS config file (i.e. connect_jaas.conf) for application name 'KafkaConnect':
KafkaConnect {
org.apache.kafka.connect.rest.basic.auth.extension.PropertyFileLoginModule required
file="/your/path/rest-credentials.properties";
};
Create rest-credentials.properties file in above-mentioned directory:
user=password
Finally, inform java about you JAAS config file, for example, by adding command-line property to java:
-Djava.security.auth.login.config=/your/path/connect_jaas.conf
After restarting Kafka Connect, you will be unable to use REST API without basic authentication.
Please keep in mind that used classes are rather examples than production-ready features.
Links:
Connect configuratin
BasicAuthSecurityRestExtension
JaasBasicAuthFilter
PropertyFileLoginModule
This is a known area in need of improvement in the future but for now you should use a firewall on the Kafka Connect machines and either an API Management tool (Apigee, etc) or a Reverse proxy (haproxy, nginx, etc.) to ensure that HTTPS is terminated at an endpoint that you can configure access control rules on and then have the firewall only accept connections from the secure proxy. With some products the firewall, access control, and SSL/TLS termination functions can be all done in a fewer number of products.
As of Kafka 1.1.0, you can set up SSL and SSL client authentication for the Kafka Connect REST API. See KIP-208 for the details.
Now you are able to enable certificate based authentication for client access to the REST API of Kafka Connect.
An example here https://github.com/sudar-path/kc-rest-mtls
I'm trying to connect broker with authorization( BlueMix IoT Foundation), but don't know how to provide credentials? How do I do it with Spark?
There is no way to connect to brokers with authorization using default MQTT connector from SparkStreaming. You have to redefine MQTTInputDstream in your own way to enable authentication.
I have customized the spark-streaming-mqtt-connector library from Apache Spark and added the following,
Added TLS v1.2 security such that the communication is always secured.
Stored topic along with the payload in the RDD.
The library is available here - https://github.com/sathipal/spark-streaming-mqtt-with-security_2.10-1.3.0. Also, added more detail in how to use the library to connect to IBM Watson IoT Platform. Let me know if you have any queries.
How are Cassandra clusters usually built in security way? Should they always be kept locally or are there any security functions that makes it reasonable to open up for external connections to the cluster? As far as I've understand I seems like Cassandra doesn't have any "inbuild security engine" for handling these kind of things. I'm planning on building a service to talk with the Cassandra from, should that connection be made locally (on the same net as the cluster) or from external using the DNS?
Cassandra supports builtin password authentication and authorisation since version 1.2.
User credentials and privileges are kept internally, in system auth tables. This can be viewed as its "inbuild security engine".
As for protecting connections (encryption), since version 1.2, there's SSL support for both internode and client-to-node communication. DataStax Enterprise platform additionally extends that with Kerberos/LDAP support to allow single-sign-on.
Configure a stateful firewall to allow incoming connections, but allow outgoing only if someone requested something from the server. Also C* has inbuilt SSL support, but not all APIs can use the SSL, so you'll have to pick a compatible one.