Unable to create marklogic rest api instance - rest

I am trying to create rest-api instance with the following configuration:
rest-api.json
{
"rest-api": {
"name": "restdb-api",
"database": "restdb",
"port": "8003",
"xdbc-enabled": true,
"forests-per-host": 1,
"error-format": "json"
}
}
curl --anyauth --user admin:admin -i -X POST -d #"./REST/rest-api.json" -H "Content-type: application/json" http://localhost:8002/LATEST/rest-apis
The endpoint returns 201 created, but I am unable to access the created endpoint at http://localhost:8003. I have tried using other ports, but the same thing is happening. The port 8003 is not listening. Please help me solve this problem.
HTTP/1.1 401 Unauthorized
Server: MarkLogic
WWW-Authenticate: Digest realm="public", qop="auth", nonce="36473d01f5e45a:ND9/6NHD0sw9o2y/xad/uQ==", opaque="e9594a1b7e019a97"
Content-Type: text/html; charset=utf-8
Content-Length: 209
Connection: Keep-Alive
Keep-Alive: timeout=5
HTTP/1.1 201 Created
Server: MarkLogic
Content-Length: 0
Connection: Keep-Alive
Keep-Alive: timeout=5

Since you said you are running on a local docker container, you might need to publish the port.
See Docker Expose
Please note from the link that - "The EXPOSE instruction does not actually publish the port. It functions as a type of documentation between the person who builds the image and the person who runs the container, about which ports are intended to be published. To actually publish the port when running the container, use the -p flag on docker run to publish and map one or more ports, or the -P flag to publish all exposed ports and map them to high-order ports."
Hope that helps!

Related

How to connect apache kafka to eclipse ditto?

I'm trying to connect apache kafka 2.x to eclipse ditto in locally with default configurations.
I'm following this https://www.eclipse.org/ditto/connectivity-protocol-bindings-kafka2.html, so I tried to create a new connection with a POST request, as follows:
curl -X POST -i -u devops:foobar -H 'Content-Type: application/json' -d '{
"targetActorSelection":"/system/sharding/connection",
"headers":{
"aggregate":false
},
"piggybackCommand":{
"type":"connectivity.commands:createConnection",
"connection":{
"id":"kafka123",
"connectionType":"kafka",
"connectionStatus":"open",
"failoverEnabled":true,
"uri":"tcp://localhost:9092",
"specificConfig":{
"bootstrapServers":"localhost:9092"
},
"sources":[
{
"addresses":["topic01"],
"consumerCount":1,
"qos":0,
"authorizationContext":["nginx:ditto"],
"enforcement":{
"input":"{{ header:device_id }}",
"filters":["{{ entity:id }}"]
},
"headerMapping":{
"topic01":"{{ header:topic01 }}",
"the-key":"{{ header:kafka.key }}"
},
"payloadMapping":["Ditto"],
"replyTarget":{
"enabled":true,
"address":"topic01",
"headerMapping":{
},
"expectedResponseTypes":[
"response",
"error",
"nack"
]
},
"acknowledgementRequests":{
"includes":[]
},
"declaredAcks":[]
}
]
}
}
}' 'http://localhost:8080/devops/piggyback/connectivity?timeout=10'
This is the result:
HTTP/1.1 408 Request Timeout
Server: nginx/1.21.6
Date: Wed, 18 May 2022 14:11:33 GMT
Content-Type: application/json
Content-Length: 4
Connection: keep-alive
correlation-id: 16231ebe-3cd1-4ec1-9f42-ad58b9b82d14
timeout: 10
response-required: false
Access-Control-Allow-Methods: GET, POST, PUT, PATCH, DELETE, OPTIONS
Access-Control-Allow-Credentials: true
Access-Control-Expose-Headers: *
null%
I can't understand what I'm doing wrong. how can i solve?
I solved: Checking in the ditto-connectivity logs (on docker), I found an error regarding a java library. To solve the problem I followed the installation of ditto again (https://github.com/eclipse/ditto#getting-started).
After that, the command written in my question worked and I got a 200 response.
Seems nginx could not send your request over to ditto services. Maybe some of ditto services are not running or you are trying to use a wrong port 8080 (if you followed the instructions for ditto setup, it should be 30080).
Please check if all ditto pods are running (use 'kubectl get pods') and which port your ditto listen on (use 'kubectl get svc')

Snowflake connector "Cannot connect to Snowflake"

I was working with Kafka and try to connect snowflake with it but it's not connecting.
It gives some kind of regional error that the server and cluster should be on the same region, and I tried it on local Kafka and did not accept the "snowflake.url.name".
here is the config file in distributed mode.
curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d '
{
"name":"XYZCompanySensorData",
"config":{
"connector.class":"com.snowflake.kafka.connector.SnowflakeSinkConnector",
"tasks.max":"2",
"topics":"MysqlJConfluent.source.user_details",
"buffer.count.records":"10000",
"buffer.flush.time":"60",
"buffer.size.bytes":"5000000",
"snowflake.url.name":"fk79721.europe-west2.gcp.snowflakecomputing.com",
"snowflake.user.name":"********",
"snowflake.private.key":"********************************************",
"snowflake.database.name":"DEMO_DB",
"snowflake.schema.name":"PUBLIC",
"key.converter":"org.apache.kafka.connect.storage.StringConverter",
"value.converter":"com.snowflake.kafka.connector.records.SnowflakeAvroConverter"
}
}'
Log file :
HTTP/1.1 100 Continue
HTTP/1.1 400 Bad Request
Date: Wed, 28 Apr 2021 11:50:41 GMT
Content-Type: application/json
Content-Length: 368
Server: Jetty(9.4.33.v20201020)
{"error_code":400,"message":"Connector configuration is invalid and contains the following 3 error(s):\nsnowflake.url.name: Cannot connect to Snowflake\nsnowflake.user.name: Cannot connect to Snowflake\nsnowflake.private.key: Cannot connect to Snowflake\nYou can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`"}
Please let me know how can I connect.
This line:
"snowflake.url.name":"fk79721.europe-west2.gcp.snowflakecomputing.com
shows that you're not closing the quotes, and it should be like this:
"snowflake.url.name":"fk79721.europe-west2.gcp.snowflakecomputing.com:443"

How to add a package to the XQuery repository?

I am using cURL to execute commands using the REST interface of BaseX like this:
curl http://localhost:8984/rest/?command=repo+list
There are also commands to manage the XQuery module repository. I am especially interested in REPO INSTALL to install a package. Is it somehow possible to execute this command using cURL and the REST interface but without having the package already on the target server? I want to provide the file in the body of the cURL request, similar to adding a resource to a database (source) which goes like this:
curl -i -X PUT -T "etc/xml/factbook.xml" "http://localhost:8984/rest/factbook"
Trying
curl -i -X PUT -T "/tmp/foo.xar" http://localhost:8984/rest/?command=repo+install
Gives me
HTTP/1.1 404 Not Found
Content-Type: text/plain;charset=UTF-8
Content-Length: 18
Connection: close
Server: Jetty(9.4.18.v20190429)
No path specified.
Adding -H "Content-Type: application/x-xar" does not help either.
And replacing PUT with POST gives me
HTTP/1.1 100 Continue
HTTP/1.1 400 Bad Request
Date: Tue, 03 Mar 2020 09:53:21 GMT
Content-Type: text/plain;charset=utf-8
Content-Length: 46
Server: Jetty(9.4.18.v20190429)
"" (Line 1): Content is not allowed in prolog.
The following works in case of standard modules (replace user/pass/server if needed):
$ curl http://admin:admin#localhost:8984/rest/?command=repo+install+http://www.xqueryfunctions.com/xq/functx-1.0.1-doc.xq

POST request with CSRF works in Postman but fails in cURL

I make a POST request to REST API to upload a file. In Postman everything works fine. I add Basic authorization and custom CSRF (XSRF) token which I get from the server.
I want to make the same using cURL. I copied the code from Postman, and it does not seem to work.
I believe that the error is related to CSRF because if I turn off CSRF on server and make the same cURL call without CSRF token, everything works fine.
Now some more details:
That's what the command for cURL which Postman gives:
curl -X POST -H "XSRF: 79f51981-8e85-4e26-be1b-bf63aed92a42" -H "Authorization: Basic bbhjbjb=" -H "Cache-Control: no-cache" -H "Postman-Token: 76a7a43b-f407-15a2-aaff-5242b44d0f47" -H "Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW" -F "package=#C:\Downloads\hello-world.zip" "http://host:port/api/import"
And that's the reply I get with --verbose
timeout on name lookup is not supported
Trying ::1...
Connected to localhost (::1) port 7777 (#0)
POST /api/import HTTP/1.1
Host: localhost:7777
User-Agent: curl/7.47.1
Accept: /
XSRF: 79f51981-8e85-4e26-be1b-bf63aed92a42
Authorization: Basic bbhjbjb=
Cache-Control: no-cache
Postman-Token: 76a7a43b-f407-15a2-aaff-5242b44d0f47
Content-Length: 31281
Expect: 100-continue
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW;
boundary=------------------------742d3475ac5f6aba
< HTTP/1.1 302 Found
< Set-Cookie: JSESSIONID=1qfjmbntrthxll;Path=/api < Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Set-Cookie: XSRF=b29bd143-cc80-49ad-b495-711125678o;Path=/;Expires=Thu, 15-Dec-2016 10:28:46 GMT
< XSRF: b29bd143-cc80-49ad-b495-711125678o < Location:
http://localhost:7777/api/login/error.jsp?errorMessage=Access Denied
< Content-Length: 0
< Server: Jetty(9.2.17.v20160517)
HTTP error before end of send, stop sending
Closing connection 0
I am probably missing something very obvious here, but don't know what yet.
Looks like I am redirected to login page, not being authenticated correctly, but do not know why (I do send XSRF in cURL). I tried also adding sessionid in cURL - also didn't work.
Any ideas and directions about where to search would be very appreciated!!!
As mentioned in this post, add following option
--cookie "csrftoken=XXXXXX;sessionid=YYYYYYY"
along with
-H "X-CSRFToken: XXXXX"
It is unclear how your server side code is implemented. One visible difference can be seen here is the UserAgent string in request header User-Agent: curl/7.47.1. You may try with adding -A "Mozilla/5.0" with your curl request.
About the comment above regarding XSRF 1-time token; Your server is returning Set-Cookie header in response. It can happen that the postman is using that as cookie for second time request, and that's why it works for it over and over. You can try adding -H "Cookie: XSRF=b29bd143-cc80-49ad-b495-711125678o" at the end of your curl and see if that makes any difference.
Those are all wild guess. Better you add some code at your server side that can print the request-headers. Then make two requests, one from curl and other one from postman. After that check the difference between the request headers. That will give you some clue.
In the end it turned out that the session id was required (adding JSESSIONID in cURL solved the problem).
Without more info on the server side code, I'm not sure either. If you're making your call from cURL, and not Postman, do you really need the Postman-Token header? Maybe it will work if you remove -H "Postman-Token: 76a7a43b-f407-15a2-aaff-5242b44d0f47" from the code.
curl -X POST \
-H "XSRF: 79f51981-8e85-4e26-be1b-bf63aed92a42" \
-H "Authorization: Basic bbhjbjb=" \
-H "Cache-Control: no-cache" \
-H "Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW" \
-F "package=#C:\Downloads\hello-world.zip" \
"http://host:port/api/import"

using wget against protected site with NTLM

Trying to mirror a local intranet site and have found previous questions using 'wget'. It works great with sites that are anonymous, but I have not been able to use it against a site that is expecting username\password (IIS with Integrated Windows Authentication).
Here is what I pass in:
wget -c --http-user='domain\user' --http-password=pwd http://local/site -dv
Here is the debug output (note I replaced some with dummy values obviously):
Setting --verbose (verbose) to 1
DEBUG output created by Wget 1.11.4 on Windows-MSVC.
--2009-07-14 09:39:04-- http://local/site
Host `local' has not issued a general basic challenge.
Resolving local... seconds 0.00, x.x.x.x
Caching local => x.x.x.x
Connecting to local|x.x.x.x|:80... seconds 0.00, connected.
Created socket 1896.
Releasing 0x003e32b0 (new refcount 1).
---request begin---
GET /site/ HTTP/1.0
User-Agent: Wget/1.11.4
Accept: */*
Host: local
Connection: Keep-Alive
---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 401 Access Denied
Server: Microsoft-IIS/5.1
Date: Tue, 14 Jul 2009 13:39:04 GMT
WWW-Authenticate: Negotiate
WWW-Authenticate: NTLM
Content-Length: 4431
Content-Type: text/html
---response end---
401 Access Denied
Closed fd 1896
Unknown authentication scheme.
Authorization failed.
NTLM authentication is broken in wget 1.11, use 1.10 instead.
Curl is actually probably a better tool for fetching content from NTLM-authenticated web servers. You can get an equivalent function to your proposed wget command line by using:
curl --anyauth --user username:password http://someserver/site
I've seen references to being able to use the NTLM Authorization Proxy Server to get around these types of problems.
use --auth-no-challenge option (wget 1.11+) (it's now considered unsafe)
I found solution.
It is work-around for Basic auth IIS7.
When auth is successeful it send next http header:
'Authorization: < type > < credentials >'.
So we able to do authorization in browser and
copy this header params from browser (firebug addon) or generate:
$ echo -en 'username:password' | base64
dXNlcm5hbWU6cGFzc3dvcmQK
$ echo 'dXNlcm5hbWU6cGFzc3dvcmQK' | base64 -d
username:password
example:
$ wget --header="Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQK" http://example.com/