Using scala sys.process with single quotes, white-space, pipes etc - scala

I am trying to use scala.sys.process._ to submit a POST request to my chronos server with curl. Because there is white space in the command's arguments, I am using the Seq[String] variant of cmd.!!
I am building the command like so:
val cmd = Seq("curl", "-L", "-X POST", "-H 'Content-Type: application/json'", "-d " + jsonHash, args.chronosHost + "/scheduler/" + jobType)
which produces, as expected,
cmd: Seq[String] = List(curl, -L, -X POST, -H 'Content-Type: application/json', -d '{"schedule":"R/2014-02-02T00:00:00Z/PT24H", "name":"Scala-Post-Test", "command":"which scalac", "epsilon":"PT15M", "owner":"myemail#thecompany.com", "async":false}', localhost:4040/scheduler/iso8601)
however, running this appears to mangle the 'Content-Type: application/json' argument:
scala> cmd.!!
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 264 0 100 100 164 2157 3538 --:--:-- --:--:-- --:--:-- 54666
res21: String =
"The HTTP header field "Accept" with value "*/* 'Content-Type:application/json'" could not be parsed.
"
which I don't understand. By contrast, calling cmd.mkString(" ") and copy+pasting into the terminal works as expected.
curl -L -X POST -H 'Content-Type:application/json' -d '{"schedule":"R/2014-02-02T00:00:00Z/PT24H", "name":"Scala-Post-Test", "command":"which scalac", "epsilon":"PT15M", "owner":"austin#quantifind.com", "async":false}' mapr-01.dev.quantifind.com:4040/scheduler/iso8601
I have tried numerous variations on the -H argument to no avail, any insight into using single quotes in sys.process._'s !! would be greatly appreciated.
I have tried variations on this as well, which generates a slew of errors, including
<h2>HTTP ERROR: 415</h2>
<p>Problem accessing /scheduler/iso8601. Reason:
<pre> Unsupported Media Type</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
(in addition to butchering the jsonHash, ie:
[1/6]: '"schedule":"R/2014-02-02T00:00:00Z/PT24H"' --> <stdout>
curl: (6) Couldn't resolve host ''"schedule"'
Which makes me think it is not interpreting the -H argument correctly

You need to split each argument into a separate element of a sequence.
Instead of this:
val cmd = Seq("curl", "-L", "-X POST", "-H 'Content-Type: application/json'", "-d " + jsonHash, args.chronosHost + "/scheduler/" + jobType)
you need to write this:
val cmd = Seq("curl", "-L", "-X", "POST", "-H", "'Content-Type: application/json'", "-d " + jsonHash, args.chronosHost + "/scheduler/" + jobType)
It puts each element of a sequence as an argument on a command line. So "-H 'Content-Type... looks like a single argument to curl while it should be 2.
Here is a simple way to test:
import scala.sys.process._
val cmd = Seq("find", "/dev/null", "-name", "null") // works
// does not work: val cmd = Seq("find", "/dev/null", "-name null")
val res = cmd.!!
println(res)

Most of the other answers are a bit faffy, use the following trick to get Bash to do all the quote handling etc etc for you
import scala.sys.process._
Seq("bash", "-c", bashLine) !
For easier Googling myself: bash execute process sys.process pipe string scala
As a String pimp method (easy copy pasting)
import scala.sys.process._
implicit class PimpedString(bashLine: String) {
def !!!: Int = Seq("bash", "-c", bashLine) !
}

I also had a hard time to make this work. Enlighten by Aleksey, below worked for me. Note that, surprisingly, there is no extra quote (or double quote) for Content-type:
val deploy = Seq("curl", "-X", "POST", s"$ip/v2/apps", "-H", "Content-Type: application/json", "-d", s"#$containerJson")
println(deploy.!!)

Related

Use groovy-wslite make a post request from CURL request

Hi i have the following CURL request that i'd like to include in a groovy script using the groovy-wslite library put struggling to get the request working.
curl -s -X POST -k -u user:password https://artifactory_url/api/search/aql -H 'Content-Type: text/plain' -d 'items.find({"type":"file","repo":{"$eq": "my-repo-name"},"path":{"$match":"com/mycompany/product1/subcat/mob/*"},"name":{"$match":"*apk"}}).sort({"$desc":["path"]}).limit(1)'
You can use http-builder-ng and your code could look like
compile 'io.github.http-builder-ng:http-builder-ng-CLIENT:1.0.4'
HttpBuilder.configure {
request.uri = 'https://artifactory_url/api/search/aql'
request.auth.basic 'un', 'pw'
request.contentType = 'text/plain'
}.post {
request.body = 'items.find({"type":"file","repo":{"$eq": "my-repo-name"},"path":{"$match":"com/mycompany/product1/subcat/mob/*"},"name":{"$match":"*apk"}}).sort({"$desc":["path"]}).limit(1)'
response.success { FromServer fs, Object body ->
println body
}
}

MultiPart PUT request not working in spring boot

I have the following curl request
curl -X PUT http://localhost:50005/did:corda:tcn:77ccbf5e-4ddd-4092-b813-ac06084a3eb0 -H 'content-type:multipart/form-data' -F 'instruction=hgfhhf'
I am trying to read the instruction in my spring boot controller as seen below
#PutMapping(value = "{id}",
produces = arrayOf(MediaType.APPLICATION_JSON_VALUE),
consumes = arrayOf(MediaType.MULTIPART_FORM_DATA_VALUE))
fun createID(#PathVariable(value = "id") id: String,
#RequestParam("instruction") instruction: String ) : ResponseEntity<Any?>
But the code above returns
"status":400,"error":"Bad Request","exception":"org.springframework.web.multipart.support.MissingServletRequestPartException","message":"Required request part 'instruction' is not present"
remove unused:
consumes = arrayOf(MediaType.MULTIPART_FORM_DATA_VALUE)
you have missed request param instruction (in reqeust), try this:
curl -X PUT -G 'http://localhost:50005/did:corda:tcn:77ccbf5e-4ddd-4092-b813-ac06084a3eb0' -d 'instruction=hgfhhf'
also take a look at CURL Command Line URL Parameters

Translating curl into Matlab/Webwrite

I have the following curl command I need to sent to a web server using Matlab and webwrite using POST. My problem is that I always get a "Bad request" answer so my syntax must be wrong somehow. Does anybody have an idea how this curl command, sending the body could look like in Matlab using webwrite in a correct way ?
body=$(cat << EOF
{
"order": {
"units": "100",
"instrument": "EUR_USD",
"timeInForce": "FOK",
"type": "MARKET",
"positionFill": "DEFAULT"
}
}
EOF
)
curl \
-X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <AUTHENTICATION TOKEN>" \
-d "$body" \
"https://api-fxtrade.oanda.com/v3/accounts/<ACCOUNT>/orders"
I have just asked a potentially similar question so this may not work first time. However I cannot test without knowing some login details so I can but hope this helps.
data_InputValues = struct ('units',100,'instrument','EUR_USD','timeInForce','FOK','type','MARKET','positionFill','DEFAULT');
MyBody = matlab.net.http.MessageBody(struct('order',data_InputValues));
MyHTTPOptions = matlab.net.http.HTTPOptions(); % use this to change the options if necessary (e.g. extend timeout)
Request = matlab.net.http.RequestMessage;
Request.Method = 'POST';
Request.Header = matlab.net.http.HeaderField('Content-Type','application/json','Authorization: Bearer',AUTHENTICATION TOKEN);
Request.Body = MyBody;
uri = matlab.net.URI('https://api-fxtrade.oanda.com/v3/accounts/<ACCOUNT>/orders');
[response a ~] = Request.send(uri,MyHTTPOptions);
The part I struggle with is generating the MyBody part (in your case this is parsing the order variable's sub-variables). If you get this to work I would be keen to know how! P.S. my question in case it helps: Matlab RESTful PUT Command - net.http - nesting body values
The correct format for the body is as follows:
body = struct('units',100,'instrument','EUR_USD','timeInForce','FOK',...
'type','MARKET','positionFill','DEFAULT');
As for the HTTP headers that you require you can specify them with weboptions when using webwrite.
The syntax for an additional header:
options = weboptions('KeyName','Name','KeyValue','Value')
Where Name and Value are the name of the header and its value respectively.
You must add the headers that you require in weboptions.
For the code you provided, the correct syntax would be as follows:
options = weboptions('MediaType','application/json',...
'KeyName','Authorization: Bearer','KeyValue','Token');
You can then perform the POST request at the URL of interest.
response = webwrite(url,body,options);

Creating command line strings with groovy for cURL - cURL ignores options

I need help figuring out why the last two parameters of my cURL query are ignored.
Please refrain on comment on how this is not the best way to do a rest call. I KNOW. This is going to be a kind of fall back method / work around for another issue.
I manyl handle my rest-work with the wslite (1.1.2) API.
Now let me explain what i do:
I am using the groovy shell executor to make a command line call for a rest service via cURL.
I have built a little class to build the query string and handle the command line:
class Curl {
def static getUserLogin(){
def url = '"https://some-login.someSystem-dev.someHost.com/someResource.beyond.foobar/login/LoginAUser '
def requestFilePath = '-d #temp/LoginPayload.json '
def heads = "-H 'Content-Type: application/json' -H 'Accept: text/plain' "
def params = '-k -v' //-k = ignore unsecure -v = more verbose output
def fullurl = url+requestFilePath+heads+params
return ex(fullurl)
}
/**
*
* #param _command The command you want to execute on your shell.
* #param _workingDir Optional: You may specify the directory where the command will be executed. Default is user dir.
* #return Exit value for the process. 0 = normal termination.
*/
def static ex(String _command, File _workingDir = new File(System.properties.'user.dir')) {
println "Executing command> $_command \n"
def process = new ProcessBuilder(addShellPrefix(_command))
.directory(_workingDir)
.redirectErrorStream(true)
.start()
process.inputStream.eachLine {println it}
process.waitFor();
return process.exitValue().value
}
private static addShellPrefix(String _command) {
def commandArray = new String[2]
commandArray[0] = "curl "
commandArray[1] = _command
return commandArray
}
}
Curl.getUserLogin() //to execute
I hope the code is self-explenatory enough. It all works fine with simple URLs respectively with less parameters.
Executing this will yield the following response (excerpt from the full debug output):
Executing command>
"https://some-login.someSystem-dev.someHost.com/someResource.beyond.foobar/login/LoginAUser"
-d #temp/LoginPayload.json -H 'Content-Type: application/json' -H 'Accept: text/plain' -k -v
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:--
--:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (60) SSL certificate problem: self signed certificate in certificate chain More details here:
http://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a
"bundle" of Certificate Authority (CA) public keys (CA certs). If the
default bundle file isn't adequate, you can specify an alternate file
using the --cacert option. If this HTTPS server uses a certificate
signed by a CA represented in the bundle, the certificate
verification probably failed due to a problem with the certificate
(it might be expired, or the name might not match the domain name in
the URL). If you'd like to turn off curl's verification of the
certificate, use the -k (or --insecure) option.
Now, as you can see I have attached the required option "-k" to the query string but somehow it is ignored. Using this string directly in the windows command line tool (if you try this make sure you escape potential double quotes) works perfectly fine though.
Any ideas why this happens or how I could accquire more debug information?
Thx in advance!
UPDATE:
Solution:
Passing ever option as a single argument (via a list) fixed the issue.
New Issue:
After that i wante curl to output the response to a file using '-o C:\Temp\response.txt' to the argument list. This works fine when used from a command line tool. Executing it from the groovy script results in:
curl: (23) Failed writing body (0 != 386)
I can get around this by just writing the stream to a file. What is really bugging me is that fact that the response does not seem to contain any information in the body. Executing the curl command from windows command line tool returns me a pretty long token as expected.
Andy ideas?
If you use ProcessBuilder, you have to give each parameter as own argument. You give two arguments to the constructor, the program name and the remaining parameters which are taken as one argument, just like if you put quotes around the whole string in the command line. Make fullurl a list instead where each parameter is its own list element and it should work as expected. You can and should leave out any other quoting like you have around the URL though.
Your code can be greatly improved. You shouldn't concatenate the command parts into a single String, just use a List.
Also, the _ prefix on variables is commonly used for private fields or just internals, not method parameters which are clearly not internals.
Using String arrays in Groovy is quite strange, you should definitely learn some Groovy!
Anyways, here's a better version of this code:
def static getUserLogin() {
def url = '"https://some-login.someSystem-dev.someHost.com/someResource.beyond.foobar/login/LoginAUser'
def requestFilePath = '-d #temp/LoginPayload.json'
def heads = "-H 'Content-Type: application/json' -H 'Accept: text/plain' "
def insecure = '-k'
def verbose = '-v'
return ex( [ url, requestFilePath, heads, insecure, verbose ] )
}
/**
*
* #param commands The command + args you want to execute on your shell.
* #param _workingDir Optional: You may specify the directory where the command will be executed. Default is user dir.
* #return Exit value for the process. 0 = normal termination.
*/
static ex( List<String> commands, File _workingDir = new File( System.properties.'user.dir' ) ) {
println "Executing command> $commands \n"
def process = new ProcessBuilder( addShellPrefix( commands ) )
.directory( _workingDir )
.inheritIO()
.start()
process.waitFor()
return process.exitValue().value
}
private static addShellPrefix( List<String> commands ) {
[ 'curl' ] + commands
}

Unable to upload file via REST request in GRAILS

I am trying to upload a file via REST using GRAILS
curl -X POST -H "Cache-Control: no-cache" -H "Postman-Token: d5d7aef8-3964-311b-8b64-4a7a82c52323" -H "Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW" -F "file1=myfile.jpg" -F "fname=swateek" -F "lname=jena" 'http://localhost:8081/sampleFileREST/document/upload'
Here's how my controller looks like:
class DocumentController extends RestfulController<Document> {
static responseFormats = ['json', 'xml']
def upload() {
def fileLocation="<xml>empty</xml>"
def file = request.getParameter('file1')
def f1 = request.getParameter('fname')
def f2 = "<abc>"+request.getParameter('lname')+"</abc>"
def params = "gf"
if(file.empty) {
fileLocation = "<xml>"+"File cannot be empty"+"</xml><allprm>"+params+"</allprm>"
} else {
def documentInstance = new Document()
documentInstance.filename = file.originalFilename
documentInstance.fullPath = grailsApplication.config.uploadFolder + documentInstance.filename
file.transferTo(new File(documentInstance.fullPath))
documentInstance.save()
fileLocation = "<xml>"+documentInstance.fullPath+"</xml>"
}
/* return "File uploaded to: "+documentInstance.fullPath */
render(text: fileLocation, contentType: "text/xml", encoding: "UTF-8")
}
}
I am able to access the parameters of the request, anything except the file I am sending in the request.
Unable to figure out what's wrong here.
UPDATE
I had used .getParameter() to fetch a file. That's incorrect, the correct way is as below:
request.getFile('<filename>') // without the <>
This might raise an error in IntelliJ as "Symbol Not Found" or "Cannot Resolve Method", please follow the procedure in the answer below.
Damn the IDE that I was using, IntelliJ.
Also, this piece of code while getting the file:
def file = request.getParameter('file1')
should be replaced as
def file = request.getFile('file1')
Now previously, when I was using the request.getFile() method I was getting an "Symbol Not Found" error and it was failing to execute the request.
Solution:
Open IntelliJ
Click on "File"
Find the option "Invalidate Caches/Restart" and wait for IntelliJ to come back again.
If this doesn't work, the other way is mentioned in this answer:
IntelliJ IDEA JDK configuration on Mac OS