Kafka connect mongoDB sink with TLS - mongodb

I set up my mongoDB cluster with TLS authentication.
I can successfully connect on a mongos instance using :
/opt/cluster/stacks/mongoDB/bin/mongosh --tls --host $(hostname).domain.name -tlsCAFile /opt/cluster/security/ssl/cert.pem --port 27017
I have a Kafka connect mongoDB sink that has the following configuration :
{
"name": "client-order-request-mongodb-sink",
"config": {
"connector.class": "com.mongodb.kafka.connect.MongoSinkConnector",
"database":"Trading",
"collection":"ClientOrderRequest",
"topics":"ClientOrderRequest",
"connection.uri":"mongodb://hostname1.domain.name:27017,pre-hostname2.domain.name:27017",
"mongo.errors.tolerance": "all",
"mongo.errors.log.enable": "true",
"errors.log.include.messages": "true",
"writemodel.strategy":"com.mongodb.kafka.connect.sink.writemodel.strategy.ReplaceOneBusinessKeyStrategy",
"document.id.strategy": "com.mongodb.kafka.connect.sink.processor.id.strategy.PartialValueStrategy",
"document.id.strategy.overwrite.existing": "true",
"document.id.strategy.partial.value.projection.type": "allowlist",
"document.id.strategy.partial.value.projection.list": "localReceiveTime,clientId,orderId"
}
}
It is working fine if I redeploy mongoDB without authentication, but now when I try to instantiate it with the following curl command :
curl -X POST -H "Content-Type: application/json" --data '#connect-task-sink-mongodb-client-order-request.json' $KAFKA_CONNECT_LEADER_NODE/connectors/
I have the following error:
{"error_code":400,"message":"Connector configuration is invalid and contains the following 1 error(s):\nUnable to connect to the server.\nYou can also find the above list of errors at the endpoint /connector-plugins/{connectorType}/config/validate"}
From the mongoDB kafka connect sink documentation I found that I needed to set up global variable of the KAFKA_OPTS so before starting the distributed connect server I do:
export KAFKA_OPTS="\
-Djavax.net.ssl.trustStore=/opt/cluster/security/ssl/keystore.jks \
-Djavax.net.ssl.trustStorePassword=\"\" \
-Djavax.net.ssl.keyStore=/opt/cluster/security/ssl/keystore.jks \
-Djavax.net.ssl.keyStorePassword=\"\""
Notice that I put an empty password because when I list the entry of my keystore with:
keytool -v -list -keystore key.jks
Then I just press enter when the password is prompted.

So the issue was that the ssl connection wasn't enabled on the client side.
If you want to do so with the mongoDB kafka connect plugin you need to state it in the connection.uri config parameter such as:
"connection.uri":"mongodb://hostname1.domain.name:27017,pre-hostname2.domain.name:27017/?ssl=true"

Related

I am trying to connect my local server database(Mssql Database) to different database (Mssql but different table) using kafka connect?

Creating the source-connection.
curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{
"name": "jdbc_source_mysql_01",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"connection.url": "jdbc:mysql://fulfillmentdbhost:3306/fulfillmentdb",
"connection.user": "fullfilment_user",
"connection.password": "<password>",
"topic.prefix": "order-status-update-",
"mode":"timestamp",
"table.whitelist" : "fulfullmentdb.status",
"timestamp.column.name": "LAST_UPDATED",
"validate.non.null": false
}
}'
Creating the sink-connection.
curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{
"name": "jdbc_sink_mysql_01",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"connection.url": "jdbc:mysql://crmdbhost:3306/crmdb",
"connection.user": "crm_user",
"connection.password": "<password>",
"topics": "order-status-update-status",
"table.name.format" : "crmdb.order_status"
}
}'
the connector.class is given from confluent community. But I want it from Apache Kafka
Which is open source
we were searching how we can replace that line with apache kafka one.
There is no JDBC Connector provided by Apache Kafka.
The Confluent one is open source.
There is also one from IBM and Aiven
Confluent (among other companies, as shown) simply write plugins for Kafka Connect, which you need to download/upgrade/install on your own Apache Kafka Connect servers.

Kafka : broker has no supported SASL mechanisms on some listener

I am trying to gradually enable ACLs on a existing cluster (3.1.0 bitnami helm chart) which is configured like this :
listeners=INTERNAL://:9093,CLIENT://:9092
listener.security.protocol.map=INTERNAL:PLAINTEXT,CLIENT:PLAINTEXT
advertised.listeners=CLIENT://$(MY_POD_NAME)-k8s.dev.host.com:4430,INTERNAL://$(MY_POD_NAME).message-broker-dev-kafka-headless.message-broker-dev.svc.cluster.local:9093
The kafka-k8s.dev.host.com:4430 is internally forwarded to the CLIENT listener on 9092
For now, we are doing TLS termination on the LB, hence the PLAINTEXT on the CLIENT listener but using SSL security.protocol :
kafkacat -b kafka-k8s.dev.host.com:4430 -X security.protocol=SSL -L
The plan is to add 2 new listeners that will require SASL auth, migrate the clients to the listeners & deprecate the existing listeners. The new configuration will look like this :
listeners=INTERNAL://:9093,CLIENT://:9092,SASL_INTERNAL://:9095,SASL_CLIENT://:9094
listener.security.protocol.map=INTERNAL:PLAINTEXT,CLIENT:PLAINTEXT,SASL_INTERNAL:SASL_PLAINTEXT,SASL_CLIENT:SASL_PLAINTEXT
advertised.listeners=CLIENT://$(MY_POD_NAME)-k8s.dev.host.com:4430,INTERNAL://$(MY_POD_NAME).message-broker-dev-kafka-headless.message-broker-dev.svc.cluster.local:9093,SASL_CLIENT://$(MY_POD_NAME)-sasl-k8s.dev.host.com:4430,SASL_INTERNAL://$(MY_POD_NAME).message-broker-dev-kafka-headless.message-broker-dev.svc.cluster.local:9095
allow.everyone.if.no.acl.found=true
authorizer.class.name=kafka.security.authorizer.AclAuthorizer
sasl.enabled.mechanisms=PLAIN,SCRAM-SHA-256,SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=PLAIN
After creating some SCRAM-SHA-512 users and applying ACLs to existing topics, everything is working fine on the SASL_INTERNAL listener but not on the SASL_CLIENT :
$ kafkacat -b message-broker-dev-kafka-headless.message-broker-dev:9095 -C -t protected-topic-v1 -X security.protocol=SASL_PLAINTEXT -X sasl.mechanisms=SCRAM-SHA-512 -X sasl.username=demo-user -X sasl.password=secret
{"userId":"1225"}
% Reached end of topic protected-topic-v1 [0] at offset 22
$ kafkacat -b kafka-sasl-k8s.dev.host.com:4430 -C -t protected-topic-v1 -X security.protocol=SASL_SSL -X sasl.mechanisms=SCRAM-SHA-512 -X sasl.username=demo-user -X sasl.password=secret
%3|1669825033.516|FAIL|rdkafka#consumer-1| [thrd:sasl_ssl://kafka-sasl-k8s.dev.host.com:4430/bootstrap]: sasl_ssl://kafka-sasl-k8s.dev.host.com:4430/bootstrap: SASL SCRAM-SHA-512 mechanism handshake failed: Broker: Request not valid in current SASL state: broker's supported mechanisms: (after 44ms in state AUTH_HANDSHAKE)
The kafka-sasl-k8s.dev.host.com:4430 is internally forwarded to the SASL_CLIENT listener on 9094 (and again using TLS termination on LB, so SASL_SSL instead of SASL_PLAINTEXT)
For now, I'm not totally sure if I missed a kafka configuration or messed a network configuration.
Thanks in advance.
Auto-answering, it was a network issue.
kafka-sasl-k8s.dev.host.com:4430 was sending traffic to 9092 & not 9094 as expeccted

GCP - How can I create an HL7v2 store configured with a Pub/Sub topic using curl?

In order to transmit HL7v2 messages over TCP/IP connections using the minimal lower layer protocol (MLLP) I'm following this guide. When I get to the part where I create an HL7v2 store configured with a Pub/Sub topic (here) I get an error.
This is what I typed in my terminal:
curl -X POST \
--data "{
'notificationConfigs': [
{
'pubsubTopic': 'projects/PROJECT_ID/topics/PUBSUB_TOPIC',
'filter': ''
}
]
}" \
-H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
-H "Content-Type: application/json; charset=utf-8" \
"https://healthcare.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/hl7V2Stores?hl7V2StoreId=HL7V2_STORE_ID"
This is the error I get:
{
"error": {
"code": 403,
"message": "Permission healthcare.hl7V2Stores.create denied on resource projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID (or it may be malformed or not exist)",
"status": "PERMISSION_DENIED"
}
}
The Dataset projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID exists, I double checked it.
So, is this somehow related to my permissions (IAM policy) ? I don't understand because I am Administer HL7v2 Stores.
How can I create my Datastore without getting this error ?
I found out that the command gcloud auth application-default print-access-token was not returning the correct token but gcloud auth print-access-token is.
So with the right auth token, the command works and I get the correct response:
{
"name": "projects/PROJECT_ID/locations/LOCATION/datasets/DATASET_ID/hl7V2Stores/HL7V2_STORE_ID",
"notificationConfigs": [
{
"pubsubTopic": "projects/PROJECT_ID/topics/PUBSUB_TOPIC"
}
]
}

Mongorestore problem Authentication failed

mongorestore -h ds121312.mlab.com:21312 -d heroku_jzxndzbk -u heroku_jzxndzbk -p xxxxxxxxxxxxxxx dump/Loc8r
Error:
2019-12-20T11:36:55.931-0500 error connecting to host: could not connect to server: connection() : auth error: sasl conversation error: unable to authenticate using mechanism "SCRA
M-SHA-1": (AuthenticationFailed) Authentication failed.
I am following along "Getting MEAN" text by Holmes/Harber - pg 152.
Any suggestions? Thank you
You must also specify which database you are using for authentication, e.g., --authenticationDatabase admin.
Reference: https://docs.mongodb.com/manual/reference/program/mongo/#cmdoption-mongo-authenticationdatabase

Keycloak: Admin-cli Add SMTP server details?

In our keycloak we have written admin-cli command to make things smooth after keycloak bought up .We will create some basic Realm/user/group and other details from the help of admin-cli .
Now we want to add the SMTP server details through admin-cli . How can be achieve this ?
Ok at last too many hit and try i found the solution . Here is the details how to add the SMTP server details via admin-cli
First Step Enable HTTPS(As in our case keycloak running on HTTPS)
/opt/keycloak/bin/kcadm.sh config truststore --trustpass keycloak /opt/keycloak/security/ssl/keycloak.jks
Second Step Login with client admin-cli
/opt/keycloak/bin/kcadm.sh config credentials --server https://{{keycloak_server_IP}}:{{keycloak_port}}/auth --realm master --user admin --password admin --client admin-cli
If realm not created create it via this
/opt/keycloak/bin/kcadm.sh create realms -s realm=SURE -s enabled=true
and then run below command
/opt/keycloak/bin/kcadm.sh update realms/CRUE_Realm -x -s 'smtpServer.host=Your Host Name' -s 'smtpServer.from=email#somemail.com' -s 'smtpServer.fromDisplayName=Mail Support' -s 'smtpServer.auth=false' -s 'smtpServer.ssl=false'
You can also use this command
/opt/keycloak/bin/kcadm.sh update realms/CRUE_Realm -f - << EOF
{"smtpServer" : {
"replyToDisplayName" : "...",
"starttls" : "",
"auth" : "true",
"envelopeFrom" : "...",
"ssl" : "true",
"password" : "...",
"port" : "...",
"host" : "...",
"replyTo" : "...",
"from" : "...",
"fromDisplayName" : "...",
"user" : "..."
}
}
EOF