How to enable SASL mechanism with JAAS Authentication for kafka ? thus the consumer/producer have to provide username & password in order to be able to publish in the broker
The process of enabling SASL authentication in Kafka is extensively described in the Authentication using SASL section in the documentation. I suggest you follow the official documentation as it contains instructions for all the mechanisms and recommendations for production environments.
To give a bit of background, at a glance you need to:
Create a JAAS file for brokers with a KafkaServer block and the configuration for the specific mechanism.
Add -Djava.security.auth.login.config=<PATH_TO_JAAS_FILE> to your broker JVM command line argument.
Configure client to use SASL via the security.protocol, sasl.mechanism and sasl.jaas.config settings.
Related
Has anybody successfully established client connection to Amazon MSK Kafka cluster using JavaScript? No YouTube video or online example AFAIK is out there. Attempts to use KafkaJs npm module are not working for me, because the SASL AWS I am roles is not supported without installing IamAWSLogin plugin on the brokers which you can’t ssh into.
Trying to use plain SASL method doesn’t work on KafkaJs because aws doesn’t use username and password.
I am not finding kafka-node useful as well.
Any leads?
There is a new feature in development that permits to inject mechanisms for auth with AWS.
https://medium.com/#jm18457_46341/using-amazon-msk-with-iam-access-control-for-apache-kafka-and-node-js-with-kafkajs-71638912fe88
Maybe is necessary to add a branch dpendency for your project, and it is a risk for production builds, however the good news is was reviewd and shoudl be merged soon :)
https://github.com/tulios/kafkajs/pull/1101
We've battled with IAM too, and it seems to be for Java clients only.
We have got it working with username/password. Details for MSK config are here https://docs.aws.amazon.com/msk/latest/developerguide/msk-password.html. I recommend when you set up MSK using a custom security group and setting up appropriate inbound access for the MSK ports.
When the cluser is set up, use the "View client information" button to get the brokers/ports to use.
Then this is your KafkaJS client setup:
new Kafka({
clientId: 'my-app',
brokers: ['something.kafka.us-east-1.amazonaws.com:9096', 'somethingelse.kafka.us-east-1.amazonaws.com:9096'],
ssl: true,
sasl: {
mechanism: 'scram-sha-512',
username,
password,
}
})
I was able to connect and use Amazon MSK Kafka cluster, via kafkajs library.
Initially I followed instructions found in docs of kafkajs library on how to use aws mechanism for sasl.
Considering that by default MSK Kafka cluster is not accessible from internet, I created a VPN client first following this video: https://www.youtube.com/watch?v=Bv70DoHDDCY, made sure that the client authorized users to access subnets of my VPC and after that I simply removed the sasl part from configuration.
so... I used something like:
const kafkaClient = new Kafka({
clientId: 'local-client',
brokers: [
'b-2.xxx.xxx.xx.xxx.xx.eu-central-1.amazonaws.com:9094',
'b-3.xxx.xxx.xx.xxx.xx.eu-central-1.amazonaws.com:9094',
'b-1.xxx.xxx.xx.xxx.xx.eu-central-1.amazonaws.com:9094'
],
ssl: true,
})
If sasl: {...} part would be there, I would get weird errors like "[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Request is not valid given the current SASL state"
Most probably sasl is not needed anymore because of the VPN connection.
Maybe I miss something, if so forgive my ignorance.
Here what we have:
We use TLS authentication listeners in Kafka cluster (this can be changed, we can add new type of listeners).
When connect to Kafka topic from Java code I use SSL certificate generated for the Kafka user.
If I decide to avoid using SSL certificate, because of 2 reasons:
I will connect to Kafka topic only from trusted OpenShift cluster PODs
To avoid updating on producer/consumer side re-generareated yearly user's SSL certificate (because Kafka generates user certificate 1 year valid period)
Would be the SCRAM-SHA-512 authentication type for KafkaUser a better (and the only ?) choice for the two reasons above? Or SCRAM-SHA-512 also requires SSL certificates?
Another approach I saw was no authentication, but I am not sure how can ACL be used for such users? How I pass to server information which user is connecting. Is it possible to use both ACL and not authenticated by SSL certificate or by password Kafka user?
[UPD] Environment is built on Strimzi (Apache Kafka cluster in OpenShift)
Using SCRAM-SHA-512 does not require TLS. So you can just disable the TLS encryption in the Kafka custom resource (.spec.kafka.listeners -> set tls: false), enable he SCRAM-SHA-512 authentication (same place, in the authentication section). And then you just use the KafkaUser to create the user and get the password.
In general, TLS encryption is normally always recommended. But the SCRAM-SHA mechanisms do not send the password over the network directly, so using it without encryption should not leak the password. At the end, it is up to you to decide.
Also, just as a sidenote - the certificates are for 1 year by default. You can change it in the Kafka CR.
I followed the tutorial of the answer of this question:
Kafka SASL zookeeper authentication
And i setted zookeeper.set.acl=true in the server.propeties, but i still can access the zookeeper on port 2181 and this is available for anyone through the: kafka-topics --zookeeper <server-name>:2181 --list
ps: instead of <server-name> i put the DN of my server.
Authentication enforcement feature has recently been submitted in the ZooKeeper codebase and afaik there's no stable version released yet which supports it.
When you turn on SASL authentication, it will be available, but clients are still able to connect without it. Hence the recommendation is to use ACLs side by side with authentication to prevent non-authenticated user from accessing sensitive data.
I'm running a distributed Kafka-Broker where the inter-broker-communication is set up with SASL/SSL. For this I adapted the JAAS-Configuration given here. The finished file looks like this:
KafkaServer {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="admin-secret"
user_admin="admin-secret"
user_alice="alice-secret"
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN;
org.apache.kafka.common.security.scram.ScramLoginModule required;
};
Client {
org.apache.kafka.common.security.plain.PlainLoginModule required
username="admin"
password="admin-secret";
};
I noticed that the "KafkaServer"-section has 2 admin users. I also learned the hard way that I need both, but why is that? I have the feeling I've read (and forgot) the reason a few months ago, but I can't seem to find it anymore.
As per the Apache Kafka documentation, the KafkaServer section is used to configure authentication from this broker to other brokers, as well as for clients and other brokers connecting to this broker. The Client section is used for connecting to Zookeeper.
Since your question is about the KafkaServer section, and you are configuring a SASL/PLAIN authentication mechanism, refer to this part of the Apache Kafka documentation:
This configuration defines two users (admin and alice). The properties username and password in the KafkaServer section are used by the broker to initiate connections to other brokers. In this example, admin is the user for inter-broker communication. The set of properties user_userName defines the passwords for all users that connect to the broker and the broker validates all client connections including those from other brokers using these properties.
In other words, there are two separate cases configured here:
When this broker connects out to other brokers it will use the username and password defined in username and password.
When clients and other brokers connect to this broker, the user_userName entries are used to authenticate these connections, where the username is the userName part of the user_userName key, and the password is the value.
So, in your example, this broker will connect to other brokers with a username of admin and a password of admin-secret because of these two lines:
username="admin"
password="admin-secret"
And, clients and other brokers can connect to this broker either with the username password combo of admin / admin-secret or alice / alice-secret because of these two lines:
user_admin="admin-secret"
user_alice="alice-secret"
If you only are accepting connections from other brokers for inter-broker communication on this listener, they probably are using the user_admin="admin-secret" part of the configuration, and the user_alice="alice-secret" probably is superfluous.
Using the Message Hub Kafka interface, I've found that my Bluemix app that was using client.id authentication is getting connections refused by the Kafka brokers. How do I fix this?
The Message Hub service is switching off client.id authentication in mid-January 2016. It is replaced by SASL authentication. Documentation on authenticating with the service using SASL can be found in the documentation:
https://www.ng.bluemix.net/docs/services/MessageHub/index.html#messagehub063
https://developer.ibm.com/messaging/2016/01/25/message-hub-beta-plan-ending/
A Java sample using the Message Hub SASL login library can be found at:
https://github.com/ibm-messaging/message-hub-samples
SASL is currently available on both port 9093 and 9094 for a grace period, after which 9094 will be switched off. The timeline for this change is also available in the Message Hub docs, as linked above.