Has anybody successfully established client connection to Amazon MSK Kafka cluster using JavaScript? No YouTube video or online example AFAIK is out there. Attempts to use KafkaJs npm module are not working for me, because the SASL AWS I am roles is not supported without installing IamAWSLogin plugin on the brokers which you can’t ssh into.
Trying to use plain SASL method doesn’t work on KafkaJs because aws doesn’t use username and password.
I am not finding kafka-node useful as well.
Any leads?
There is a new feature in development that permits to inject mechanisms for auth with AWS.
https://medium.com/#jm18457_46341/using-amazon-msk-with-iam-access-control-for-apache-kafka-and-node-js-with-kafkajs-71638912fe88
Maybe is necessary to add a branch dpendency for your project, and it is a risk for production builds, however the good news is was reviewd and shoudl be merged soon :)
https://github.com/tulios/kafkajs/pull/1101
We've battled with IAM too, and it seems to be for Java clients only.
We have got it working with username/password. Details for MSK config are here https://docs.aws.amazon.com/msk/latest/developerguide/msk-password.html. I recommend when you set up MSK using a custom security group and setting up appropriate inbound access for the MSK ports.
When the cluser is set up, use the "View client information" button to get the brokers/ports to use.
Then this is your KafkaJS client setup:
new Kafka({
clientId: 'my-app',
brokers: ['something.kafka.us-east-1.amazonaws.com:9096', 'somethingelse.kafka.us-east-1.amazonaws.com:9096'],
ssl: true,
sasl: {
mechanism: 'scram-sha-512',
username,
password,
}
})
I was able to connect and use Amazon MSK Kafka cluster, via kafkajs library.
Initially I followed instructions found in docs of kafkajs library on how to use aws mechanism for sasl.
Considering that by default MSK Kafka cluster is not accessible from internet, I created a VPN client first following this video: https://www.youtube.com/watch?v=Bv70DoHDDCY, made sure that the client authorized users to access subnets of my VPC and after that I simply removed the sasl part from configuration.
so... I used something like:
const kafkaClient = new Kafka({
clientId: 'local-client',
brokers: [
'b-2.xxx.xxx.xx.xxx.xx.eu-central-1.amazonaws.com:9094',
'b-3.xxx.xxx.xx.xxx.xx.eu-central-1.amazonaws.com:9094',
'b-1.xxx.xxx.xx.xxx.xx.eu-central-1.amazonaws.com:9094'
],
ssl: true,
})
If sasl: {...} part would be there, I would get weird errors like "[BrokerPool] Failed to connect to seed broker, trying another broker from the list: Request is not valid given the current SASL state"
Most probably sasl is not needed anymore because of the VPN connection.
Related
Here i'm trying to use Debezium Connector to Read data from RDS and publish it to AWS MSK.
All i could see is using AWS IAM based authentication everywhere but unfortunately my cloud engineering team is not willing to add IAM but provided basic auth (username and password) with a topic to publish.
I'm planning to use debezium/connect docker image so i can deploy it to EKS.
Is there any way to connect to MSK using username and password ?
If you want Debezium to connect to your RDS database, you must include database credentials (username and password) in your Debezium config. For example, check out the 'Required Config' docs for the Postgres connector: https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-required-configuration-properties
You must supply database.user, database.password, and other related values.
If you want to connect to your MSK cluster, there are a couple of alternatives as described here: Connecting to an Amazon MSK Cluster.
For a username/password based authentication to MSK cluster best would be to use SASL SCRAM. See details here: https://docs.aws.amazon.com/msk/latest/developerguide/msk-password.html
Other authentication supported by MSK so far [JAN/31/2022] are:
IAM auth
mutual TLS - certificate based
SASL SCRAM - username/password based
Maybe I miss something, if so forgive my ignorance.
Here what we have:
We use TLS authentication listeners in Kafka cluster (this can be changed, we can add new type of listeners).
When connect to Kafka topic from Java code I use SSL certificate generated for the Kafka user.
If I decide to avoid using SSL certificate, because of 2 reasons:
I will connect to Kafka topic only from trusted OpenShift cluster PODs
To avoid updating on producer/consumer side re-generareated yearly user's SSL certificate (because Kafka generates user certificate 1 year valid period)
Would be the SCRAM-SHA-512 authentication type for KafkaUser a better (and the only ?) choice for the two reasons above? Or SCRAM-SHA-512 also requires SSL certificates?
Another approach I saw was no authentication, but I am not sure how can ACL be used for such users? How I pass to server information which user is connecting. Is it possible to use both ACL and not authenticated by SSL certificate or by password Kafka user?
[UPD] Environment is built on Strimzi (Apache Kafka cluster in OpenShift)
Using SCRAM-SHA-512 does not require TLS. So you can just disable the TLS encryption in the Kafka custom resource (.spec.kafka.listeners -> set tls: false), enable he SCRAM-SHA-512 authentication (same place, in the authentication section). And then you just use the KafkaUser to create the user and get the password.
In general, TLS encryption is normally always recommended. But the SCRAM-SHA mechanisms do not send the password over the network directly, so using it without encryption should not leak the password. At the end, it is up to you to decide.
Also, just as a sidenote - the certificates are for 1 year by default. You can change it in the Kafka CR.
I followed the tutorial of the answer of this question:
Kafka SASL zookeeper authentication
And i setted zookeeper.set.acl=true in the server.propeties, but i still can access the zookeeper on port 2181 and this is available for anyone through the: kafka-topics --zookeeper <server-name>:2181 --list
ps: instead of <server-name> i put the DN of my server.
Authentication enforcement feature has recently been submitted in the ZooKeeper codebase and afaik there's no stable version released yet which supports it.
When you turn on SASL authentication, it will be available, but clients are still able to connect without it. Hence the recommendation is to use ACLs side by side with authentication to prevent non-authenticated user from accessing sensitive data.
The REST API for Kafka Connect is not secured and authenticated.
Since its not authenticated, the configuration for a connector or Tasks are easily accessible by anyone. Since these configurations may contain about how to access the Source System [in case of SourceConnector] and destination system [in case of SinkConnector], Is there a standard way to restrict access to these APIs?
In Kafka 2.1.0, there is possibility to configure http basic authentication for REST interface of Kafka Connect without writing any custom code.
This became real due to implementation of REST extensions mechanism (see KIP-285).
Shortly, configuration procedure as follows:
Add extension class to worker configuration file:
rest.extension.classes = org.apache.kafka.connect.rest.basic.auth.extension.BasicAuthSecurityRestExtension
Create JAAS config file (i.e. connect_jaas.conf) for application name 'KafkaConnect':
KafkaConnect {
org.apache.kafka.connect.rest.basic.auth.extension.PropertyFileLoginModule required
file="/your/path/rest-credentials.properties";
};
Create rest-credentials.properties file in above-mentioned directory:
user=password
Finally, inform java about you JAAS config file, for example, by adding command-line property to java:
-Djava.security.auth.login.config=/your/path/connect_jaas.conf
After restarting Kafka Connect, you will be unable to use REST API without basic authentication.
Please keep in mind that used classes are rather examples than production-ready features.
Links:
Connect configuratin
BasicAuthSecurityRestExtension
JaasBasicAuthFilter
PropertyFileLoginModule
This is a known area in need of improvement in the future but for now you should use a firewall on the Kafka Connect machines and either an API Management tool (Apigee, etc) or a Reverse proxy (haproxy, nginx, etc.) to ensure that HTTPS is terminated at an endpoint that you can configure access control rules on and then have the firewall only accept connections from the secure proxy. With some products the firewall, access control, and SSL/TLS termination functions can be all done in a fewer number of products.
As of Kafka 1.1.0, you can set up SSL and SSL client authentication for the Kafka Connect REST API. See KIP-208 for the details.
Now you are able to enable certificate based authentication for client access to the REST API of Kafka Connect.
An example here https://github.com/sudar-path/kc-rest-mtls
We have a mondogDB deployment currently in our test environment. We have a 7 member Replica Set and no Arbiter.
We want to configure the data replication between the replica set members secure.
We don't want to configure SSL for the clients to our MongoDB cluster as the communication from the client to this MongoDB cluster is via Stunnel. So the client doesn't need to use SSL to connect.
Just curious to see if this possible i.e configure only the data being replicated between replica set members Secure but not the actual communication from the Client to this MongoDB cluster
_THanks much
I've not tried this personally but I do believe you can do this. In addition to compiling mongodb with ssl, or purchasing one of the MongoDb subscriptions that support SSL you will need to run with the following option:
--sslMode
set to preferSSL. This will use SSL for inter-server communications but allow both SSL and non-SSL for other connections:
http://docs.mongodb.org/manual/reference/program/mongod/
That of course is all in addition to the other configuration settings required for running with SSL:
http://docs.mongodb.org/manual/tutorial/configure-ssl/
Note that this is new for version 2.6. I don't have a version of mongodb compiled with ssl support so it's not been tested by me.
#DurgaDeep in MongoDB v2.6 you can run the MongoDB instance in mixed mode SSL while also specifying the x509 certificates for the cluster members to authenticate each other. Please note that SSL is not part of the default community build and you may need to build the binary on your own if you are using community build. x509 certificates is only available on subscription builds so that will not work for you if you are using a community build.
The easiest option to achieve what you want to do irrespective of the MongoDB version would be to setup stunnel between the replica-set nodes and let it take care of encrypted channel independently. This is the usual route a lot of applications take which do not have SSL embedded as an option.