Curl error while trying to automate file upload - http no such file - mongodb

I've searched through the various questions and tried different permutations of curl commands but yet to find one the works so posting the question as I'm likely missing something obvious and cannot see it for looking.
I'm running a curl command to try and upload a file for parsing.
S-MBP:project SG$ curl -i -X POST -F "data=#/Users/SG/Desktop/project/mongodb-cluster-shard-00-02.lrqcr.mongodb.net_27017-cluster.bson.gz" 'http://localhost:3030/upload/'
HTTP/1.1 100 Continue
HTTP/1.1 200 OK
Access-Control-Allow-Headers: accept, content-type
Access-Control-Allow-Methods: POST
Access-Control-Allow-Origin: *
Date: Sat, 25 Jul 2020 14:56:05 GMT
Content-Length: 19
Content-Type: text/plain; charset=utf-8
http: no such file
Some of the permutations tried based on previous answers:
curl -i -X POST -F "filename=#/Users/SG/Desktop/project/mongodb-cluster-shard-00-02.lrqcr.mongodb.net_27017-cluster.bson.gz" http://localhost:3030/upload/
curl -i -X POST -F "filename=#mongodb-cluster-shard-00-02.lrqcr.mongodb.net_27017-cluster.bson.gz" http://localhost:3030/upload
curl -i -X POST -F filename=#/Users/SG/Desktop/project/mongodb-cluster-shard-00-02.lrqcr.mongodb.net_27017-cluster.bson.gz http://localhost:3030/upload/
curl -i -X POST -F "filename=#/Users/SG/Desktop/project/mongodb-cluster-shard-00-02.lrqcr.mongodb.net_27017-cluster.bson.gz" "http://localhost:3030/upload/"
Interestingly, if I pass in the name of a file that doesn't exist I get the same error but if I change the name of the directory to one that doesn't exist the error is a curl(26) making me think that the command couldn't care less about the file at the moment. I am running this on a mac if that's of nay relevance, I saw a post that implied there may be an issue with brew curl.
The form that I'm trying to target is part of the docker image https://hub.docker.com/repository/docker/simagix/maobi
The form with some values omitted
<form action="/upload" id="maobi" class="dropzone dz-clickable dz-started">
<div class="dz-message">
Drop files here or click to upload.<br>
</div>
<div class="dz-filename"><span data-dz-name="">mongodb-cluster-shard-00-02.lrqcr.mongodb.net_27017-cluster.bson.gz</span></div></div></form>
and then the script I see in the page that I believe is used to upload and then parse the document generating the output.
<script>
Dropzone.options.maobi = {
timeout: 300000,
init: function() {
this.on("success", function(file, responseText) {
blob = new Blob([responseText], { type: 'text/html' }),
anchor = document.createElement('a');
filename = file.upload.filename;
if ((n = filename.indexOf("mdiag-")) == 0 ) {
n = filename.lastIndexOf(".json")
filename = filename.substring(0, n) + ".html";
} else if ((n = filename.lastIndexOf(".json")) > 0 ) {
//...
//...
} else if ((n = filename.indexOf(".log")) > 0 && (n = filename.lastIndexOf(".gz")) > 0) {
filename = filename.substring(0, n) + ".html";
} else if ((n = filename.lastIndexOf(".bson")) > 0 ) {
filename = filename.substring(0, n) + "-cluster.html";
}
anchor.download = filename;
anchor.href = (window.webkitURL || window.URL).createObjectURL(blob);
anchor.dataset.downloadurl = ['text/html', anchor.download, anchor.href].join(':');
anchor.click();
});
this.on("error", function(file, responseText) {
alert(responseText);
});
}
};
</script>

It seems to me that you are not passing the file in the right form field.
From looking at the Dropzone.js documentation, it seems the right field name is file (since that's the default for the paramName configuration), not data or filename. But to be entirely sure, it'd be best to look at the network request in your browser's devtools and see what POST field name is used there for passing the file.
curl -i -X POST -F "file=#/Users/SG/Desktop/project/mongodb-cluster-shard-00-02.lrqcr.mongodb.net_27017-cluster.bson.gz" 'http://localhost:3030/upload/'

Related

Translating curl into Matlab/Webwrite

I have the following curl command I need to sent to a web server using Matlab and webwrite using POST. My problem is that I always get a "Bad request" answer so my syntax must be wrong somehow. Does anybody have an idea how this curl command, sending the body could look like in Matlab using webwrite in a correct way ?
body=$(cat << EOF
{
"order": {
"units": "100",
"instrument": "EUR_USD",
"timeInForce": "FOK",
"type": "MARKET",
"positionFill": "DEFAULT"
}
}
EOF
)
curl \
-X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <AUTHENTICATION TOKEN>" \
-d "$body" \
"https://api-fxtrade.oanda.com/v3/accounts/<ACCOUNT>/orders"
I have just asked a potentially similar question so this may not work first time. However I cannot test without knowing some login details so I can but hope this helps.
data_InputValues = struct ('units',100,'instrument','EUR_USD','timeInForce','FOK','type','MARKET','positionFill','DEFAULT');
MyBody = matlab.net.http.MessageBody(struct('order',data_InputValues));
MyHTTPOptions = matlab.net.http.HTTPOptions(); % use this to change the options if necessary (e.g. extend timeout)
Request = matlab.net.http.RequestMessage;
Request.Method = 'POST';
Request.Header = matlab.net.http.HeaderField('Content-Type','application/json','Authorization: Bearer',AUTHENTICATION TOKEN);
Request.Body = MyBody;
uri = matlab.net.URI('https://api-fxtrade.oanda.com/v3/accounts/<ACCOUNT>/orders');
[response a ~] = Request.send(uri,MyHTTPOptions);
The part I struggle with is generating the MyBody part (in your case this is parsing the order variable's sub-variables). If you get this to work I would be keen to know how! P.S. my question in case it helps: Matlab RESTful PUT Command - net.http - nesting body values
The correct format for the body is as follows:
body = struct('units',100,'instrument','EUR_USD','timeInForce','FOK',...
'type','MARKET','positionFill','DEFAULT');
As for the HTTP headers that you require you can specify them with weboptions when using webwrite.
The syntax for an additional header:
options = weboptions('KeyName','Name','KeyValue','Value')
Where Name and Value are the name of the header and its value respectively.
You must add the headers that you require in weboptions.
For the code you provided, the correct syntax would be as follows:
options = weboptions('MediaType','application/json',...
'KeyName','Authorization: Bearer','KeyValue','Token');
You can then perform the POST request at the URL of interest.
response = webwrite(url,body,options);

Matlab RESTful PUT Command - net.http - nesting body values

I am using Matlab's matlab.net.http library to launch get, put and post commands to a website. I can successfully launch get and post commands.
For example:
MyBody = matlab.net.http.MessageBody(struct('Id',YYYYYY,'WindfarmId',XXX,'Month','YYYY-MM-DD'));
Request = matlab.net.http.RequestMessage;
Request.Method = 'POST';
Request.Header = matlab.net.http.HeaderField('Content-Type','application/json','Authorization',['Basic ' matlab.net.base64encode([Username ':' Password])]);
Request.Body = MyBody;
uri = matlab.net.URI(ENTERURLHERE);
Response = Request.send(uri,MyHTTPOptions);
This works well. However using a PUT command I have to enter the equiavlent of this body (written in curl syntax):
-d '{ "InputValues": [ {"MetricLevelAId": 1, "MetricLevelBId": 1, "InputMetricId": 7, "Value": 56 } ] }'
I tried this:
data_InputValues = struct ('MetricLevelAId',1,'MetricLevelBId',1,'InputMetricId',7,'Value',56);
MyBody = matlab.net.http.MessageBody(struct('InputValues',dataInputValues));
However I keep receiving the following 'Bad Request' response from the server:
"Input values required"
I think this is linked to the way Matlab interprets the body part of the request and passes it to the server, i.e. it cannot pass the nested struct correctly. Anyone got any ideas how to solve this?
N.B. potentially linked to Translating curl into Matlab/Webwrite (it is dealing with a nested value)

Can't able to view a transform in browser using REST in Marklogic 9?

I tried to install below Server-Side JavaScript using this documentation and saved below as rest-sjs
function insertTimestamp(context, params, content)
{
if (context.inputType.search('json') >= 0) {
var result = content.toObject();
if (context.acceptTypes) { /* read */
result.readTimestamp = fn.currentDateTime();
} else { /* write */
result.writeTimestamp = fn.currentDateTime();
}
return result;
} else {
/* Pass thru for non-JSON documents */
return content;
}
};
exports.transform = insertTimestamp;
I tried to push this using below curl cmd:
curl --anyauth --user public\admin:admin -X PUT -i --data-binary #"C:/Users/name/Desktop/rest.sjs" -H "Content-type: application/vnd.marklogic-javascript" 'http://localhost:9963/v1/config/transforms/js-example'
When I used localhost:9963 and went to /v1/config/transforms I can see:
<rapi:transforms xmlns:rapi="http://marklogic.com/rest-api">
<rapi:transform>
<rapi:name>rest-tsm</rapi:name>
<rapi:source-format>javascript</rapi:source-format>
<rapi:transform-parameters/>
<rapi:transform-source>/v1/config/transforms/rest-tsm</rapi:transform-source>
</rapi:transform>
</rapi:transforms>
But when I went though the module /v1/config/transforms/rest-tsm I am seeing an error response:
<error-response xmlns="http://marklogic.com/xdmp/error">
<status-code>406</status-code>
<status>Unacceptable Type</status>
<message-code>REST-UNACCEPTABLETYPE</message-code>
<message>
REST-UNACCEPTABLETYPE: (err:FOER0000) No acceptable content type: None of the requested types text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8 can be provided
</message>
</error-response>
I can see the the module in Modules db. Which worked fine when I try to insert a document by using the transform.
Why can't I view the transform in the browser?
Unfortunately, that rest endpoint isn't very browser friendly. The required/acceptable Accept header values do not match what browsers will normally send.
When you made the GET request through your browser, it was sending the following Accept header:
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept
This field contains a semicolon-separated list of representation schemes ( Content-Type metainformation values) which will be accepted in the response to this request.
Unfortunately, the v1/config/transform/{name} (GET) REST endpoint is strict in what it will accept for the Accept header and expects a specific value:
The MIME type of the data expected in the response, either application/xslt+xml or application/xquery.
If you use the example CURL command from the documentation, and customize for your transform URI, it will return the expected response.
curl --anyauth --user public\admin:admin -X GET -i \
-H "Accept: application/xquery" \
http://localhost:9963/v1/config/transforms/rest-tsm

MSL Socket Script displaying html and body tag

When ever the mirc radio bot announces the song playing and the ammount of listeners a stray html and body tag show up as seen below.
The code for the radio bot is as follows
#announcer on
ctcp *:*:*:{
if ($1 == SRstats) {
set %sctat.chan $chan
sockclose scstat
sockopen scstat 149.202.90.221 8132
}
}
on *:SOCKOPEN:scstat:{
sockwrite -n $sockname GET /7.html HTTP/1.0
sockwrite -n $sockname User-Agent: Mozilla
sockwrite -n $sockname $crlf
}
on *:sockread:scstat:{
if ($sockerr > 0) return
:nextread
sockread -f %scasttemp
if ($sockbr == 0) return
if (%scasttemp == $null) %scasttemp = empty
set %scasttemp $remove(%scasttemp,<html><head><meta http-equiv="Pragma" content="no-cache"></head><body>,</body></html>)
if ((HTTP/1.* !iswm %scasttemp) && (content-type* !iswm %scasttemp) && (%scasttemp != empty)) {
set %scstat.song.temp $gettok(%scasttemp,7-,44)
set %scstat.bitrate $gettok(%scasttemp,6,44)
set %scstat.listeners $gettok(%scasttemp,1,44)
set %scstat.maxlist $gettok(%scasttemp,4,44)
set %scstat.peak $gettok(%scasttemp,3,44)
if ($gettok(%scasttemp,2,44) == 1) set %scstat.livedj connected
else set %scstat.livedj not connected
; changing some of the html codes back to regular characters
set %scast.song $replace(%scast.song,&,$chr(38),',$chr(39))
}
goto nextread
}
on *:sockclose:scstat:{
if (( %scstat.song.temp == %scstat.song ) || ( %scstat.song.temp == Line Recording )) { goto scstat.end }
else {
set %scstat.song %scstat.song.temp
set %song.msg 6,0 $+ %dj_nick is playing 6 : 12 %scstat.song $+ . 0,1 Tune into Radio-Airwaves, type !radiohelp/4 %scstat.listeners $+ --listeners are tuned in.
; set %chans $chan(0)
; while %chans {
/scid -a msg #Radio-Airwaves-Lounge %song.msg
; dec %chans
; }
:scstat.end
}
}
on *:TEXT:!playing:#: msg $chan %song.msg
#announcer end
I thought the first fix should be changing the , between the body tags to html number code but that just displayed the number code and not the actually comma. I also maybe there were mismatched tags/stray tags so I check for that. I didn't find any. I've yet to see the cause of the stray tags appearing when the accouner is on. Any help would be greatly appericated.
The line your trying to extract the information from, <html><body> at the start and end of the retrieved text.
You can solve this by setting %scasttemp using several techniques.
Strip the Html tags using $nohtml a like script. - Recomended
Staticly Substring from $right(%text, -12)
Dynamically find the first occurrence after body**>** and substring the rest of the text.
Use Regular Expressions
And many more..
$nohtml
alias nohtml { var %x,%y = $regsub($1-,/(<[^>]+>)/g,$null,%x) | return %x }
Additionally, when handling sockread i would use Tokenize to handle $1.. identifier instead of tokens.
if (!$sockbr || !%scasttemp) {
return
}
tokenize 32 $nohtml(%scasttemp)
;;; Comment out the below line if you still want to use the old variable, otherwise you should change the rest of the code.
;;;set %scasttemp $1-
;;; Identify the data we wish to extract the information, else return.
if ($numtok($1-, 44) < 7) {
return
}
;;; Rest of the code here..
Sending to server header request suggesting to close the connection after the information received is a good practice.
sockwrite -n $sockname Connection: close
Also adding the sockclose after all information received is a good convention instead of letting the sockets to hang around. (If Connection: close was not requested)
goto nextread
sockclose $sockname

Creating command line strings with groovy for cURL - cURL ignores options

I need help figuring out why the last two parameters of my cURL query are ignored.
Please refrain on comment on how this is not the best way to do a rest call. I KNOW. This is going to be a kind of fall back method / work around for another issue.
I manyl handle my rest-work with the wslite (1.1.2) API.
Now let me explain what i do:
I am using the groovy shell executor to make a command line call for a rest service via cURL.
I have built a little class to build the query string and handle the command line:
class Curl {
def static getUserLogin(){
def url = '"https://some-login.someSystem-dev.someHost.com/someResource.beyond.foobar/login/LoginAUser '
def requestFilePath = '-d #temp/LoginPayload.json '
def heads = "-H 'Content-Type: application/json' -H 'Accept: text/plain' "
def params = '-k -v' //-k = ignore unsecure -v = more verbose output
def fullurl = url+requestFilePath+heads+params
return ex(fullurl)
}
/**
*
* #param _command The command you want to execute on your shell.
* #param _workingDir Optional: You may specify the directory where the command will be executed. Default is user dir.
* #return Exit value for the process. 0 = normal termination.
*/
def static ex(String _command, File _workingDir = new File(System.properties.'user.dir')) {
println "Executing command> $_command \n"
def process = new ProcessBuilder(addShellPrefix(_command))
.directory(_workingDir)
.redirectErrorStream(true)
.start()
process.inputStream.eachLine {println it}
process.waitFor();
return process.exitValue().value
}
private static addShellPrefix(String _command) {
def commandArray = new String[2]
commandArray[0] = "curl "
commandArray[1] = _command
return commandArray
}
}
Curl.getUserLogin() //to execute
I hope the code is self-explenatory enough. It all works fine with simple URLs respectively with less parameters.
Executing this will yield the following response (excerpt from the full debug output):
Executing command>
"https://some-login.someSystem-dev.someHost.com/someResource.beyond.foobar/login/LoginAUser"
-d #temp/LoginPayload.json -H 'Content-Type: application/json' -H 'Accept: text/plain' -k -v
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:--
--:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (60) SSL certificate problem: self signed certificate in certificate chain More details here:
http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a
"bundle" of Certificate Authority (CA) public keys (CA certs). If the
default bundle file isn't adequate, you can specify an alternate file
using the --cacert option. If this HTTPS server uses a certificate
signed by a CA represented in the bundle, the certificate
verification probably failed due to a problem with the certificate
(it might be expired, or the name might not match the domain name in
the URL). If you'd like to turn off curl's verification of the
certificate, use the -k (or --insecure) option.
Now, as you can see I have attached the required option "-k" to the query string but somehow it is ignored. Using this string directly in the windows command line tool (if you try this make sure you escape potential double quotes) works perfectly fine though.
Any ideas why this happens or how I could accquire more debug information?
Thx in advance!
UPDATE:
Solution:
Passing ever option as a single argument (via a list) fixed the issue.
New Issue:
After that i wante curl to output the response to a file using '-o C:\Temp\response.txt' to the argument list. This works fine when used from a command line tool. Executing it from the groovy script results in:
curl: (23) Failed writing body (0 != 386)
I can get around this by just writing the stream to a file. What is really bugging me is that fact that the response does not seem to contain any information in the body. Executing the curl command from windows command line tool returns me a pretty long token as expected.
Andy ideas?
If you use ProcessBuilder, you have to give each parameter as own argument. You give two arguments to the constructor, the program name and the remaining parameters which are taken as one argument, just like if you put quotes around the whole string in the command line. Make fullurl a list instead where each parameter is its own list element and it should work as expected. You can and should leave out any other quoting like you have around the URL though.
Your code can be greatly improved. You shouldn't concatenate the command parts into a single String, just use a List.
Also, the _ prefix on variables is commonly used for private fields or just internals, not method parameters which are clearly not internals.
Using String arrays in Groovy is quite strange, you should definitely learn some Groovy!
Anyways, here's a better version of this code:
def static getUserLogin() {
def url = '"https://some-login.someSystem-dev.someHost.com/someResource.beyond.foobar/login/LoginAUser'
def requestFilePath = '-d #temp/LoginPayload.json'
def heads = "-H 'Content-Type: application/json' -H 'Accept: text/plain' "
def insecure = '-k'
def verbose = '-v'
return ex( [ url, requestFilePath, heads, insecure, verbose ] )
}
/**
*
* #param commands The command + args you want to execute on your shell.
* #param _workingDir Optional: You may specify the directory where the command will be executed. Default is user dir.
* #return Exit value for the process. 0 = normal termination.
*/
static ex( List<String> commands, File _workingDir = new File( System.properties.'user.dir' ) ) {
println "Executing command> $commands \n"
def process = new ProcessBuilder( addShellPrefix( commands ) )
.directory( _workingDir )
.inheritIO()
.start()
process.waitFor()
return process.exitValue().value
}
private static addShellPrefix( List<String> commands ) {
[ 'curl' ] + commands
}