I was working with Kafka and try to connect snowflake with it but it's not connecting.
It gives some kind of regional error that the server and cluster should be on the same region, and I tried it on local Kafka and did not accept the "snowflake.url.name".
here is the config file in distributed mode.
curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d '
{
"name":"XYZCompanySensorData",
"config":{
"connector.class":"com.snowflake.kafka.connector.SnowflakeSinkConnector",
"tasks.max":"2",
"topics":"MysqlJConfluent.source.user_details",
"buffer.count.records":"10000",
"buffer.flush.time":"60",
"buffer.size.bytes":"5000000",
"snowflake.url.name":"fk79721.europe-west2.gcp.snowflakecomputing.com",
"snowflake.user.name":"********",
"snowflake.private.key":"********************************************",
"snowflake.database.name":"DEMO_DB",
"snowflake.schema.name":"PUBLIC",
"key.converter":"org.apache.kafka.connect.storage.StringConverter",
"value.converter":"com.snowflake.kafka.connector.records.SnowflakeAvroConverter"
}
}'
Log file :
HTTP/1.1 100 Continue
HTTP/1.1 400 Bad Request
Date: Wed, 28 Apr 2021 11:50:41 GMT
Content-Type: application/json
Content-Length: 368
Server: Jetty(9.4.33.v20201020)
{"error_code":400,"message":"Connector configuration is invalid and contains the following 3 error(s):\nsnowflake.url.name: Cannot connect to Snowflake\nsnowflake.user.name: Cannot connect to Snowflake\nsnowflake.private.key: Cannot connect to Snowflake\nYou can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`"}
Please let me know how can I connect.
This line:
"snowflake.url.name":"fk79721.europe-west2.gcp.snowflakecomputing.com
shows that you're not closing the quotes, and it should be like this:
"snowflake.url.name":"fk79721.europe-west2.gcp.snowflakecomputing.com:443"
Related
I'm trying to connect apache kafka 2.x to eclipse ditto in locally with default configurations.
I'm following this https://www.eclipse.org/ditto/connectivity-protocol-bindings-kafka2.html, so I tried to create a new connection with a POST request, as follows:
curl -X POST -i -u devops:foobar -H 'Content-Type: application/json' -d '{
"targetActorSelection":"/system/sharding/connection",
"headers":{
"aggregate":false
},
"piggybackCommand":{
"type":"connectivity.commands:createConnection",
"connection":{
"id":"kafka123",
"connectionType":"kafka",
"connectionStatus":"open",
"failoverEnabled":true,
"uri":"tcp://localhost:9092",
"specificConfig":{
"bootstrapServers":"localhost:9092"
},
"sources":[
{
"addresses":["topic01"],
"consumerCount":1,
"qos":0,
"authorizationContext":["nginx:ditto"],
"enforcement":{
"input":"{{ header:device_id }}",
"filters":["{{ entity:id }}"]
},
"headerMapping":{
"topic01":"{{ header:topic01 }}",
"the-key":"{{ header:kafka.key }}"
},
"payloadMapping":["Ditto"],
"replyTarget":{
"enabled":true,
"address":"topic01",
"headerMapping":{
},
"expectedResponseTypes":[
"response",
"error",
"nack"
]
},
"acknowledgementRequests":{
"includes":[]
},
"declaredAcks":[]
}
]
}
}
}' 'http://localhost:8080/devops/piggyback/connectivity?timeout=10'
This is the result:
HTTP/1.1 408 Request Timeout
Server: nginx/1.21.6
Date: Wed, 18 May 2022 14:11:33 GMT
Content-Type: application/json
Content-Length: 4
Connection: keep-alive
correlation-id: 16231ebe-3cd1-4ec1-9f42-ad58b9b82d14
timeout: 10
response-required: false
Access-Control-Allow-Methods: GET, POST, PUT, PATCH, DELETE, OPTIONS
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: *
null%
I can't understand what I'm doing wrong. how can i solve?
I solved: Checking in the ditto-connectivity logs (on docker), I found an error regarding a java library. To solve the problem I followed the installation of ditto again (https://github.com/eclipse/ditto#getting-started).
After that, the command written in my question worked and I got a 200 response.
Seems nginx could not send your request over to ditto services. Maybe some of ditto services are not running or you are trying to use a wrong port 8080 (if you followed the instructions for ditto setup, it should be 30080).
Please check if all ditto pods are running (use 'kubectl get pods') and which port your ditto listen on (use 'kubectl get svc')
I am using cURL to execute commands using the REST interface of BaseX like this:
curl http://localhost:8984/rest/?command=repo+list
There are also commands to manage the XQuery module repository. I am especially interested in REPO INSTALL to install a package. Is it somehow possible to execute this command using cURL and the REST interface but without having the package already on the target server? I want to provide the file in the body of the cURL request, similar to adding a resource to a database (source) which goes like this:
curl -i -X PUT -T "etc/xml/factbook.xml" "http://localhost:8984/rest/factbook"
Trying
curl -i -X PUT -T "/tmp/foo.xar" http://localhost:8984/rest/?command=repo+install
Gives me
HTTP/1.1 404 Not Found
Content-Type: text/plain;charset=UTF-8
Content-Length: 18
Connection: close
Server: Jetty(9.4.18.v20190429)
No path specified.
Adding -H "Content-Type: application/x-xar" does not help either.
And replacing PUT with POST gives me
HTTP/1.1 100 Continue
HTTP/1.1 400 Bad Request
Date: Tue, 03 Mar 2020 09:53:21 GMT
Content-Type: text/plain;charset=utf-8
Content-Length: 46
Server: Jetty(9.4.18.v20190429)
"" (Line 1): Content is not allowed in prolog.
The following works in case of standard modules (replace user/pass/server if needed):
$ curl http://admin:admin#localhost:8984/rest/?command=repo+install+http://www.xqueryfunctions.com/xq/functx-1.0.1-doc.xq
I am trying to create rest-api instance with the following configuration:
rest-api.json
{
"rest-api": {
"name": "restdb-api",
"database": "restdb",
"port": "8003",
"xdbc-enabled": true,
"forests-per-host": 1,
"error-format": "json"
}
}
curl --anyauth --user admin:admin -i -X POST -d #"./REST/rest-api.json" -H "Content-type: application/json" http://localhost:8002/LATEST/rest-apis
The endpoint returns 201 created, but I am unable to access the created endpoint at http://localhost:8003. I have tried using other ports, but the same thing is happening. The port 8003 is not listening. Please help me solve this problem.
HTTP/1.1 401 Unauthorized
Server: MarkLogic
WWW-Authenticate: Digest realm="public", qop="auth", nonce="36473d01f5e45a:ND9/6NHD0sw9o2y/xad/uQ==", opaque="e9594a1b7e019a97"
Content-Type: text/html; charset=utf-8
Content-Length: 209
Connection: Keep-Alive
Keep-Alive: timeout=5
HTTP/1.1 201 Created
Server: MarkLogic
Content-Length: 0
Connection: Keep-Alive
Keep-Alive: timeout=5
Since you said you are running on a local docker container, you might need to publish the port.
See Docker Expose
Please note from the link that - "The EXPOSE instruction does not actually publish the port. It functions as a type of documentation between the person who builds the image and the person who runs the container, about which ports are intended to be published. To actually publish the port when running the container, use the -p flag on docker run to publish and map one or more ports, or the -P flag to publish all exposed ports and map them to high-order ports."
Hope that helps!
I make a POST request to REST API to upload a file. In Postman everything works fine. I add Basic authorization and custom CSRF (XSRF) token which I get from the server.
I want to make the same using cURL. I copied the code from Postman, and it does not seem to work.
I believe that the error is related to CSRF because if I turn off CSRF on server and make the same cURL call without CSRF token, everything works fine.
Now some more details:
That's what the command for cURL which Postman gives:
curl -X POST -H "XSRF: 79f51981-8e85-4e26-be1b-bf63aed92a42" -H "Authorization: Basic bbhjbjb=" -H "Cache-Control: no-cache" -H "Postman-Token: 76a7a43b-f407-15a2-aaff-5242b44d0f47" -H "Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW" -F "package=#C:\Downloads\hello-world.zip" "http://host:port/api/import"
And that's the reply I get with --verbose
timeout on name lookup is not supported
Trying ::1...
Connected to localhost (::1) port 7777 (#0)
POST /api/import HTTP/1.1
Host: localhost:7777
User-Agent: curl/7.47.1
Accept: /
XSRF: 79f51981-8e85-4e26-be1b-bf63aed92a42
Authorization: Basic bbhjbjb=
Cache-Control: no-cache
Postman-Token: 76a7a43b-f407-15a2-aaff-5242b44d0f47
Content-Length: 31281
Expect: 100-continue
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW;
boundary=------------------------742d3475ac5f6aba
< HTTP/1.1 302 Found
< Set-Cookie: JSESSIONID=1qfjmbntrthxll;Path=/api < Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Set-Cookie: XSRF=b29bd143-cc80-49ad-b495-711125678o;Path=/;Expires=Thu, 15-Dec-2016 10:28:46 GMT
< XSRF: b29bd143-cc80-49ad-b495-711125678o < Location:
http://localhost:7777/api/login/error.jsp?errorMessage=Access Denied
< Content-Length: 0
< Server: Jetty(9.2.17.v20160517)
HTTP error before end of send, stop sending
Closing connection 0
I am probably missing something very obvious here, but don't know what yet.
Looks like I am redirected to login page, not being authenticated correctly, but do not know why (I do send XSRF in cURL). I tried also adding sessionid in cURL - also didn't work.
Any ideas and directions about where to search would be very appreciated!!!
As mentioned in this post, add following option
--cookie "csrftoken=XXXXXX;sessionid=YYYYYYY"
along with
-H "X-CSRFToken: XXXXX"
It is unclear how your server side code is implemented. One visible difference can be seen here is the UserAgent string in request header User-Agent: curl/7.47.1. You may try with adding -A "Mozilla/5.0" with your curl request.
About the comment above regarding XSRF 1-time token; Your server is returning Set-Cookie header in response. It can happen that the postman is using that as cookie for second time request, and that's why it works for it over and over. You can try adding -H "Cookie: XSRF=b29bd143-cc80-49ad-b495-711125678o" at the end of your curl and see if that makes any difference.
Those are all wild guess. Better you add some code at your server side that can print the request-headers. Then make two requests, one from curl and other one from postman. After that check the difference between the request headers. That will give you some clue.
In the end it turned out that the session id was required (adding JSESSIONID in cURL solved the problem).
Without more info on the server side code, I'm not sure either. If you're making your call from cURL, and not Postman, do you really need the Postman-Token header? Maybe it will work if you remove -H "Postman-Token: 76a7a43b-f407-15a2-aaff-5242b44d0f47" from the code.
curl -X POST \
-H "XSRF: 79f51981-8e85-4e26-be1b-bf63aed92a42" \
-H "Authorization: Basic bbhjbjb=" \
-H "Cache-Control: no-cache" \
-H "Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW" \
-F "package=#C:\Downloads\hello-world.zip" \
"http://host:port/api/import"
I've build a demo project on Grails 3.0.10
I can read the data with a REST request but I don't know how to insert data.
Here's how my project is built:
grails create-app bookstore
cd bookstore
grails create-domain-class bookstore.Book
edit file grails-app/domain/bookstore/Book.groovy :
package bookstore
class Book {
String title
String author
Date publicationDate
static constraints = {
}
}
Scaffold:
grails generate-all bookstore.Book
Run the app:
grails run-app
Browse:
http://localhost:8080
From here I can insert and list items from the browser ;o)
Now I want to use the REST API !
Reading data is OK:
curl -H "Accept: application/json" -i http://localhost:8080
/book.jsonHTTP/1.1 200 OK
Server: Apache-Coyote/1.1
X-Application-Context: application:development
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
Date: Mon, 14 Dec 2015 12:14:01 GMT
[{"class":"bookstore.Book","id":1,"author":"James","publicationDate":"2015-12-13T23:00:00Z","title":"Hello"}]
But when I try to insert data:
curl -i -X POST -H "Content-Type: application/json" -H "Accept: application/json" -d '{"title":"Test", "author":"Franck"}'
http://localhost:8080/book/create.jsonHTTP/1.1 200 OK
Server: Apache-Coyote/1.1
X-Application-Context: application:development
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
Date: Mon, 14 Dec 2015 12:16:03 GMT
{"class":"bookstore.Book","id":null,"author":null,"publicationDate":null,"title":null}
And nothing is inserted.
The default create action doesn't save anything. It is used to populate the initial form when creating an instance using a browser form. You probably want to be invoking the save action instead.
This isn't really related to your question but FYI... if you are building a REST interface, you should look at RestfulController and/or the #Resource annotation.