I am trying to connect from local standalone Confluent Kafka topics to Snowflake tables. I am using the following connector config via ksqldb.
CREATE SINK CONNECTOR `snowflake_sink` WITH(
"name"='snowflake_sink',
"tasks.max"='1',
"connector.class"='com.snowflake.kafka.connector.SnowflakeSinkConnector',
"topics"='USERPROFILE',
"snowflake.url.name"='https://mybu.mycompany.us-east-1.aws.privatelink.snowflakecomputing.com',
"snowflake.user.name”=‘myuser’,
"snowflake.database.name"='DEMO_DB',
"snowflake.topic2table.map"='USERPROFILE:UK_SF_DB1_Table1',
"snowflake.schema.name"='PUBLIC',
"snowflake.private.key”=‘<valid private key>’,
"snowflake.private.key.passphrase”=‘<valid pass phrase>’,
"key.converter"='org.apache.kafka.connect.storage.StringConverter',
"value.converter"='com.snowflake.kafka.connector.records.SnowflakeJsonConverter',
"key.converter.schema.registry.url"='http://schema-registry:8081',
"value.converter.schema.registry.url"='http://schema-registry:8081');
Ksqldb throws error saying snowflake url, username, private key is not valid. They are all valid as I can login using the same via Snowflake web ui.
Some ideas on troubleshooting this:
it looks like you have some "smart quotes" in your config, please remove the smart single and double quotes and replace them with proper text quotes. This may have happened during your preparing of this question, so just double check them.
set your table name to all capital letters.
your snowflake account should not include the https:// prefix. Examples at these links:
https://community.snowflake.com/s/article/Docker-Compose-Setting-up-Kafka-using-Snowflake-Sink-Connector-and-testing-Streams
https://docs.snowflake.com/en/user-guide/kafka-connector-install.html
You should remove the smart quotes and use the normal one like
'snowflake.url.name'='my.domain.com:443',
Related
We have set our Kafka Connect to be able to read credentials from a file, instead of giving them directly in connector config. This is how a login part of connector config looks like:
"connection.user": "${file:/kafka/pass.properties:username}",
"connection.password":
"${file:/kafka/pass.properties:password}",
We also added these 2 lines to "connect-distributed.properties" file:
config.providers=file
config.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProvider
Mind that it works perfectly for JDBC connectors, so there is no problem with the pass.properties file. But for other connectors, such as couchbase, rabbitmq, s3 etc. it causes problems. All these connectors work fine when we give credentials directly but when we try to make Connect to read them from a file it gives some errors. What could be the reason? I don't see any JDBC specific configuration here.
EDIT:
An error about couchbase in connect.log:
[2021-12-02 11:50:19,580] ERROR [com.couchbase.io][SaslAuthenticationFailedEvent][20ms] Authentication Failure - Potential causes: invalid credentials or if LDAP is enabled ensure PLAIN SASL mechanism is exclusively used on the PasswordAuthenticator (insecure) or TLS is used (recommended) {"circuitBreaker":"DISABLED","coreId":"0xbf785c7500000001","remote":"10.30.142.109:11210","status":"UNKNOWN","type":"KV","xerror":{"ref":"ae3ce600-7097-4077-9231-8ced290cd399"}} (com.couchbase.io:533)
[2021-12-02 11:50:19,580] WARN [com.couchbase.endpoint][EndpointConnectionFailedEvent][23ms] Connect attempt 9 failed because of AuthenticationFailureException: Authentication Failure - Potential causes: invalid credentials or if LDAP is enabled ensure PLAIN SASL mechanism is exclusively used on the PasswordAuthenticator (insecure) or TLS is used (recommended) {"circuitBreaker":"DISABLED","coreId":"0xbf785c7500000001","remote":"10.30.142.109:11210","type":"KV"} (com.couchbase.endpoint:523)
com.couchbase.client.core.error.AuthenticationFailureException: Authentication Failure - Potential causes: invalid credentials or if LDAP is enabled ensure PLAIN SASL mechanism is exclusively used on the PasswordAuthenticator (insecure) or TLS is used (recommended) {"circuitBreaker":"DISABLED","coreId":"0xbf785c7500000001","remote":"10.30.142.109:11210","status":"UNKNOWN","type":"KV","xerror":{"ref":"ae3ce600-7097-4077-9231-8ced290cd399"}}
at com.couchbase.client.core.io.netty.kv.SaslAuthenticationHandler.failConnect(SaslAuthenticationHandler.java:488)
at com.couchbase.client.core.io.netty.kv.SaslAuthenticationHandler.maybeFailConnect(SaslAuthenticationHandler.java:293)
at com.couchbase.client.core.io.netty.kv.SaslAuthenticationHandler.channelRead(SaslAuthenticationHandler.java:250)
at com.couchbase.client.core.io.netty.kv.MemcacheProtocolVerificationHandler.channelRead(MemcacheProtocolVerificationHandler.java:84)
at java.lang.Thread.run(Thread.java:748)
It says something about authentication but works fine when credentials are given directly. If the masking is not working correctly, how does it work for JDBC connectors?
Looks like the problem was quote marks in pass.properties file. The interesting thing is, even if credentials are typed with or without quote marks, JDBC connectors work well. Maybe the reason is it is the first line in the file but just a small possibility.
So, do NOT use quote marks in your password files, even if some of the connectors work this way.
We use Kafka, Kafka connect and Schema-registry in our stack. Version is 2.8.1(Confluent 6.2.1).
We use Kafka connect's configs(key.converter and value.converter) with value: io.confluent.connect.avro.AvroConverter.
It registers a new schema for topics automatically. But there's an issue, AvroConverter doesn't specify subject-level compatibility for a new schema
and the error appears when we are trying to get config for the schema via REST API /config: Subject 'schema-value' does not have subject-level compatibility configured
If we specify the request parameter defaultToGlobal then global compatibility is returned. But it doesn't work for us because we cannot specify it in the request. We are using 3rd party UI: AKHQ.
How can I specify subject-level compatibility when registering a new schema via AvroConverter?
Last I checked, the only properties that can be provided to any of the Avro serializer configs that affect the Registry HTTP client are the url, whether to auto-register, and whether to use the latest schema version.
There's no property (or even method call) that sets either the subject level or global config during schema registration
You're welcome to check out the source code to verify this
But it doesn't work for us because we cannot specify it in the request. We are using 3rd party UI: AKHQ
Doesn't sound like a Connect problem. Create a PR for AKHQ project to fix the request
As of 2021-10-26, I used akhq 0.18.0 jar and confluent-6.2.0, the schema registry in akhq is working fine.
Note: I also used confluent-6.2.1, seeing exactly the same error. So, you may want to switch back to 6.2.0 to give a try.
P.S: using all only for my local dev env, VirtualBox, Ubuntu.
#OneCricketeer is correct.
There is no possibility to specify subject-level compatibility in AvroConverter unfortunately.
I see only two solutions:
Override AvroConverter to add property and functionality to send an additional request to API /config/{subject} after registering the schema.
Contribute to AKHQ to support defaultToGlobal parameter. But in this case, we also need to backport schema-registry RestClient. Github issue
The second solution is more preferable till the user would specify the compatibility level in the settings of the converter. Without this setting in the native AvroConverter, we have to use the custom converter for every client who writes a schema. And it makes a lot of effort.
For me, it looks strange why the client cannot set up the compatibility at the moment of registering the schema and has to use a different request for it.
I'm trying to get Debezium Server running so that I can use GCP (Google) PubSub, and not have to use Kafka and the Kafka connectors. I have it mostly running, however, I'm having trouble with the using variables in the tranforms section to define a Topic name.
According to the documentation, when using the Outbox transformation, I can choose the Topic name by using the variable ${routedByValue} for the setting route.topic.replacement and this will use the value that is determined by the setting route.by.field. If the replacement setting is omitted, it will use a default topic name of outbox.event.<route.by.field value>.
When I try to use this variable in the 'application.properties' file ...
debezium.transforms.outbox.route.by.field=aggregate_type
debezium.transforms.outbox.route.topic.replace=${routedByValue}
... the Debezium Server stops with a NoSuchElementException, saying it cannot expand routedByValue. If I omit that setting, it works fine and defines the topic name as outbox.event.<route.by.field value>.
How can I use this variable correctly in the 'applications.properties' file so I can customise the topic name (e.g. route.topic.replace=myservice.${routedByValue})?
The way I got this to work was to do the following ...
debezium.transforms.outbox.route.by.file=aggregate_type
debezium.transforms.outbox.route.topic.replacement=$1
I believe this works because omitted from the config is another setting - debezium.transforms.outbox.route.topic.regex - and this has a default value of - (?<routedByValue>.*).
If I understand the documentation correctly, the $1 refers to the first group in the regex expression. In my case, this will return whatever the value of aggregrate_type equates to.
I'm using Debezium Server 2.1 with Pulsar as sink type and the #Dazfl answer solve my issue !
debezium.transforms=outbox
debezium.transforms.outbox.type=io.debezium.transforms.outbox.EventRouter
debezium.transforms.outbox.route.topic.replacement=outbox.event.transactions.$1
Although the Debezium Server docs says to use $routedByValue, this do not works as expeceted...
I am very new to Apache Nifi. I am trying to Migrate data from Oracle to Mongo DB as per the screenshot in Apache NiFi. I am failing with the reported error. Pls help.
Till PutFile i think its working fine, as i can see the below Json format file in my local directory.
Simple setup direct from Oracle Database to MongoDb without SSL or username and password (not recommended for Production)
Just keep tinkering on PutMongoRecord Processor until you resolve all outstanding issues and exclamation mark is cleared
I am first using an ExecuteSQL processor which is resulting the dataset in Avro, I need the final data in JSON. In DBconnection pooling Service, you need to create a controller with the credentials of your Orcale database. Post that I am using Split Avro and then Transform XML to convert it into JSON. In Transform XML, you need to use XSLT file. After that, I use PutMongo Processor for ingestion in Json which gets automatically converted in BSON
I'm trying to create a service-connector to my s3 instance like this:
cf service-connector 13001 mybucketname.ds31s3.swisscom.com:443
But I get the following error:
Server-Error 403: Check of security groups failed (no access)
I have created my service key according to this documentation.
Connecting to my MongoDB works perfectly using a service connector.
You can access Swisscom's S3 directly without the service connector.
The error message suggests that your current org and space do no have access to the S3. This is usually the case is there is no app-binding for that service in the current space. Please check whether you created your service key in the right org and space.
There was a misconfiguration due to security changes. We fixed the issue, so connecting to s3 with the service-connector should now work.