I need to implement Kafka connect, so I have referred the video
https://www.youtube.com/watch?v=r7LUbtOFcQI
and implemented steps accordingly.
While starting the Kafka connect, I need to run the CURL command. while running the below command i am facing an issue as
curl -X POST -H "Content-Type: application/json" --data '{
"name": "jdbc_source_connector",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"tasks.max":2,
"connection.url": "jdbc:mysql://69.61.32.102:3306/ssptest",
"connection.user": "sspuser",
"connection.password": "xxx",
"topic.prefix": "",
"poll.interval.ms" : 3600000,
"table.whitelist" : "Persons",
"mode":"incrementing"
}
}'http://localhost:8083/connectors
I am facing an error as,
curl: no URL specified
Can anyone please suggest a solution to fix the error
Related
Creating the source-connection.
curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{
"name": "jdbc_source_mysql_01",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"connection.url": "jdbc:mysql://fulfillmentdbhost:3306/fulfillmentdb",
"connection.user": "fullfilment_user",
"connection.password": "<password>",
"topic.prefix": "order-status-update-",
"mode":"timestamp",
"table.whitelist" : "fulfullmentdb.status",
"timestamp.column.name": "LAST_UPDATED",
"validate.non.null": false
}
}'
Creating the sink-connection.
curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -d '{
"name": "jdbc_sink_mysql_01",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"connection.url": "jdbc:mysql://crmdbhost:3306/crmdb",
"connection.user": "crm_user",
"connection.password": "<password>",
"topics": "order-status-update-status",
"table.name.format" : "crmdb.order_status"
}
}'
the connector.class is given from confluent community. But I want it from Apache Kafka
Which is open source
we were searching how we can replace that line with apache kafka one.
There is no JDBC Connector provided by Apache Kafka.
The Confluent one is open source.
There is also one from IBM and Aiven
Confluent (among other companies, as shown) simply write plugins for Kafka Connect, which you need to download/upgrade/install on your own Apache Kafka Connect servers.
I set up my mongoDB cluster with TLS authentication.
I can successfully connect on a mongos instance using :
/opt/cluster/stacks/mongoDB/bin/mongosh --tls --host $(hostname).domain.name -tlsCAFile /opt/cluster/security/ssl/cert.pem --port 27017
I have a Kafka connect mongoDB sink that has the following configuration :
{
"name": "client-order-request-mongodb-sink",
"config": {
"connector.class": "com.mongodb.kafka.connect.MongoSinkConnector",
"database":"Trading",
"collection":"ClientOrderRequest",
"topics":"ClientOrderRequest",
"connection.uri":"mongodb://hostname1.domain.name:27017,pre-hostname2.domain.name:27017",
"mongo.errors.tolerance": "all",
"mongo.errors.log.enable": "true",
"errors.log.include.messages": "true",
"writemodel.strategy":"com.mongodb.kafka.connect.sink.writemodel.strategy.ReplaceOneBusinessKeyStrategy",
"document.id.strategy": "com.mongodb.kafka.connect.sink.processor.id.strategy.PartialValueStrategy",
"document.id.strategy.overwrite.existing": "true",
"document.id.strategy.partial.value.projection.type": "allowlist",
"document.id.strategy.partial.value.projection.list": "localReceiveTime,clientId,orderId"
}
}
It is working fine if I redeploy mongoDB without authentication, but now when I try to instantiate it with the following curl command :
curl -X POST -H "Content-Type: application/json" --data '#connect-task-sink-mongodb-client-order-request.json' $KAFKA_CONNECT_LEADER_NODE/connectors/
I have the following error:
{"error_code":400,"message":"Connector configuration is invalid and contains the following 1 error(s):\nUnable to connect to the server.\nYou can also find the above list of errors at the endpoint /connector-plugins/{connectorType}/config/validate"}
From the mongoDB kafka connect sink documentation I found that I needed to set up global variable of the KAFKA_OPTS so before starting the distributed connect server I do:
export KAFKA_OPTS="\
-Djavax.net.ssl.trustStore=/opt/cluster/security/ssl/keystore.jks \
-Djavax.net.ssl.trustStorePassword=\"\" \
-Djavax.net.ssl.keyStore=/opt/cluster/security/ssl/keystore.jks \
-Djavax.net.ssl.keyStorePassword=\"\""
Notice that I put an empty password because when I list the entry of my keystore with:
keytool -v -list -keystore key.jks
Then I just press enter when the password is prompted.
So the issue was that the ssl connection wasn't enabled on the client side.
If you want to do so with the mongoDB kafka connect plugin you need to state it in the connection.uri config parameter such as:
"connection.uri":"mongodb://hostname1.domain.name:27017,pre-hostname2.domain.name:27017/?ssl=true"
How can I delete a debezium connector. I am following this tutorial
https://debezium.io/documentation/reference/tutorial.html and I see the way to register a connector but couldn't figure out how to delete / update a connector.
curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d '{ "name": "inventory-connector", "config": { "connector.class": "io.debezium.connector.mysql.MySqlConnector", "tasks.max": "1", "database.hostname": "mysql", "database.port": "3306", "database.user": "debezium", "database.password": "dbz", "database.server.id": "184054", "database.server.name": "dbserver1", "database.include.list": "inventory", "database.history.kafka.bootstrap.servers": "kafka:9092", "database.history.kafka.topic": "dbhistory.inventory" } }'
Can you also please point to me to the documentation page where deleting and updating a connector is mentioned.
debezium connector is a standard connector which you plug in to kafka connet framework. The Kafka Connect framework support several REST commands in order to interact with it.
For delete you submit DELETE request
curl -i -X DELETE localhost:8083/connectors/inventory-connector/
To update configuration you submit PUT request with the new configuration
curl -i -X PUT -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/inventory-connector/config -d '{ "connector.class": "io.debezium.connector.mysql.MySqlConnector", "tasks.max": "1", "database.hostname": "mysql", "database.port": "3306", "database.user": "debezium", "database.password": "dbz", "database.server.id": "184054", "database.server.name": "dbserver1", "database.include.list": "inventory", "database.history.kafka.bootstrap.servers": "kafka:9092", "database.history.kafka.topic": "dbhistory.inventory" }'
Further REST API instructions
https://docs.confluent.io/platform/current/connect/references/restapi.html
I followed the documentation to create an external replica of cloud SQL here. I have a MySQL instance with version 8.0 on Google cloud. I successfully created the external read replica. Now, I want to demote this replica to master. I need to create a source representation instance for that. I called the following API as mentioned in the documentation.
gcloud auth login
ACCESS_TOKEN="$(gcloud auth print-access-token)"
curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \
--header 'Content-Type: application/json' \
--data '{
"name": "[SOURCE_REPRESENTATION_NAME]",
"region": "[REGION]",
"databaseVersion": "[EXTERNAL_SERVER_MYSQL_VERSION]",
"onPremisesConfiguration": {
"hostPort": "[EXTERNAL_SERVER_IP]:[EXTERNAL_SERVER_PORT]"
}
}' \
-X POST \
https://www.googleapis.com/sql/v1beta4/projects/[PROJECT-ID]/instances
The API works when I set databaseVersion to 5. But it fails when I try to set to version 8.
{
"error": {
"code": 400,
"message": "Missing parameter: DatabaseVersion.",
"errors": [
{
"message": "Missing parameter: DatabaseVersion.",
"domain": "global",
"reason": "required"
}
]
}
}
Is the MySQL version 8 not supported for representational server?
From documentation:
databaseVersion should be the MySQL version running on your source database server. The choices are MYSQL_5_5, MYSQL_5_6, MYSQL_5_7 or MYSQL_8_0.
Using "LIVE" credentials, I'm getting an INTERNAL_SERVICE_ERROR accepting a credit card payment, using this resource: https://api.paypal.com/v1/payments/payment. Same error happens on the web and via cURL. NOTE: This does not happen in SANDBOX only LIVE. See steps to recreate below. For this post, I had to add a space in "http s" as I cannot submit more than 2 links.
Get an OAuth token (substitute username/password in LIVE_USER_ID:LIVE_PASSWORD
curl http s://api.paypal.com/v1/oauth2/token -H "Accept: application/json" -H "Accept-Language: en_US" -u "LIVE_USER_ID:LIVE_PASSWORD" -d "grant_type=client_credentials"
No problem getting a token using LIVE credentials. Add the token to the following request to the resource:
curl -v http s://api.paypal.com/v1/payments/payment \
-H "Content-Type:application/json" \
-H "Authorization:Bearer YK.GUHkGhBAQgGgGkilvHoL7DdE9SVq.IDl-mRwAZeM" \
-d '{
"intent": "sale",
"payer": {
"payment_method": "credit_card",
"funding_instruments": [
{
"credit_card": {
"number": "5500005555555559",
"type": "mastercard",
"expire_month": 12,
"expire_year": 2018,
"cvv2": 111,
"first_name": "Joe",
"last_name": "Shopper"
}
}
]
},
"transactions": [
{
"amount": {
"total": "0.01",
"currency": "USD"
},
"description": "This is the payment transaction description."
}
]
}'
"name":"INTERNAL_SERVICE_ERROR","message":"An internal service error has occurred","information_link":"http_s://developer.paypal.com/webapps/developer/docs/api/#INTERNAL_SERVICE_ERROR","debug_id":"525bfb7a6382a"}johnruffin:~ johnruffin$
Thoughts???
Thanks for bringing this issue to our attention. I've opened an internal bug with the payments rest api development team. From the debug id provided I see that the credit card is being refused. In the future the error message from this scenario will fall under this error condition: https://developer.paypal.com/webapps/developer/docs/api/#CREDIT_CARD_REFUSED
Please try a another/valid credit card to verify.
Looks like the URL is not well formed. It's https and not http s as provided above.
Tried the calls now, it works fine. Please check and confirm.