I am trying to execute the curl command through scala for scalatest,
CURL :
curl -H "Content-Type: application/json" --data #/home/examples/demo/demo.json http://localhost:9090/job
which works aas expected while I tried doing it with scala ,like
import scala.sys.process._
val json = getClass.getClassLoader.getResource(arg0).getPath
val cmd = Seq("curl", "-H", "'Content-Type: application/json'","-d", s"#$json","http://localhost:9090/job")
cmd.!
and it produces following error
Expected 'application/json'
You're quoting too much:
Seq("curl", "-H", "'Content-Type: application/json'","-d", s"#$json","http://localhost:9090/job")
// !^! !^!
Should be:
Seq("curl", "-H", "Content-Type: application/json", "-d", s"#$json","http://localhost:9090/job")
// ! !
The reason you need to quote the content-type in the shell is because the shell will break it up if it isn't quoted. Curl doesn't know how to deal with quotes, because quotes aren't its business. Scala won't do any word-splitting, either, because that's your job.
Related
I want to query a REST API with HTTPie. I am usuale to do so with curl, with which I am able to specify maxKeys and startAfterFilename e.g.
curl --location --request GET -G \
"https://some.thing.some.where/data/v1/datasets/mydataset/versions/2/files" \
-d maxKeys=100 \
-d startAfterFilename=YYYMMDD_HHMMSS.file \
--header "Authorization: verylongtoken"
How can I use those -d options in HTTPie?
In your case the command looks like this:
http -F https://some.thing.some.where/data/v1/datasets/mydataset/versions/2/files \
Authorization:verylongtoken \
startAfterFilename=="YYYMMDD_HHMMSS.file" \
maxKeys=="100"
Although, there is a bunch of methods to pass some data with httpie. For example
http POST http://example.com/posts/3 \
Origin:example.com \ # : HTTP headers
name="John Doe" \ # = string
q=="search" \ # == URL parameters (?q=search)
age:=29 \ # := for non-strings
list:='[1,3,4]' \ # := json
file#file.bin \ # # attach file
token=#token.txt \ # =# read from file (text)
user:=#user.json # :=# read from file (json)
Or, in the case of forms
http --form POST example.com \
name="John Smith" \
cv=#document.txt
I have a json file like below
[
{
"field": {
"empID": "sapid",
"location": "India",
}
},
{
"field": {
"empID": "sapid",
"location": "India",
}
},
{
"field": {
"empID": "sapid",
"location": "India",
}
}
{
"field": {
"empID": "sapid",
"location": "India",
}
},
{
"field": {
"empID": "sapid",
"location": "India",
}
}
{
"field": {
"empID": "sapid",
"location": "India",
}
}
.... upto 1 million
]
I have to use this json as an input for a rest request For example
curl <REST Server URL with temp.json as input> "Content-Type: application/json" -d #temp.json
My server will not accept 1 million json object at a time.
I am looking for an approach where i have to extract the first 500 objects from the main json and send it in one rest query and then next 500 object in second rest query and so on.
Can you please suggest how can i achieve this by jq?
There's an intrinsic tradeoff here between space and time efficiency. In the following, the focus is on the latter.
Assuming that each call to curl must send a JSON array, a time-efficient solution can be constructed along the following lines:
< array.json jq -c '
def batch($n): length as $l | range(0;length;$n) as $i | .[$i: $i+n];
batch(500)
' | while read -r json
do
echo "$json" | curl -X POST -H "Content-Type: application/json" -d -# ....
done
Here .... signifies additional appropriate curl arguments.
GNU parallel
You might also want to consider using GNU parallel, e.g.:
< array.json jq -c '
def batch($n):
length as $l
| range(0;length;$n) as $i
| .[$i: $i+n];
batch(500)
' | parallel --pipe -N1 curl -X POST -H "Content-Type: application/json" -d #- ....
You have not shared any HW information of the system you are running this on. At the minimum you need to do some sort of multiprocessing to make this faster instead of running (1000000/500) curl requests altogether.
One way, would be to use GNU xargs which has a built-in to run number of parallel instances of a given process using the -P flag and number of lines of input to read from at any time with the L flag.
To start with you can do something like below to instruct curl to run on 500 lines at a time and invoke 20 such invocations in parallel. So at a given tick, approximately (500 *20) lines of input are processed. You can tune the numbers depending on your HW capability both on the host and the server side.
xargs -L 500 -P 20 curl -X POST -H "Content-Type: application/json" http://sample-url -d #- < <(jq -c 'range(0;length;500) as $i | .[$i: $i+500]' json)
Modified jq filter to pack the JSON payload as an array of objects (credit peak's answer). The earlier version jq -c '.[]' json might not work as the individual chunk of lines passed at a time doesn't represent a valid JSON.
Note: Not tested due to performance constraints.
Assuming you have this formatting, splitting can be done by unpacking the array and saving the desired number of objects to separate files, e.g.:
<input.json jq -c '.[]' | split -l500
Which creates xaa with the first 500 objects, xab with the next 500 objects, etc. If you want to repackage the objects in an array, use the -s option to jq, e.g.: jq -s . xaa.
If you want to do this from the shell, you could use jq to split your JSON and pass it to xargs to call curl for each object returned.
jq -c '.[]' temp.json | xargs -I {} curl <REST Server URL with temp.json as input> "Content-Type: application/json" -d '{}'
This will send one curl request for each object. However, if you e.g. want to send the first 500 objects in a single curl request, you can specify a subarray in the jq filter. To send all of your JSON objects you will then somehow need to repeat the command, as afaik jq has no built-in way to split the input into chunks of objects.
jq -c '.[0:500]' temp.json | xargs -I {} curl <REST Server URL with temp.json as input> "Content-Type: application/json" -d '{}'
jq -c '.[500:1000]' temp.json | xargs -I {} curl <REST Server URL with temp.json as input> "Content-Type: application/json" -d '{}'
jq -c '.[1000:1500]' temp.json | xargs -I {} curl <REST Server URL with temp.json as input> "Content-Type: application/json" -d '{}'
[...]
I am trying to send a message to slack from Scala using CURL command. It gives error invalid payload
val slackurl = "https://hooks.slack.com/services/<something here>
val cmd = Seq(
"curl",
"-d",
"testdata",
"-H",
"Content-Type: application/x-www-form-urlencoded",
"-X",
"POST",
slackurl
)
cmd.!
i am getting the error "invalid_payload"
or
val cmd = Seq(
"curl",
"-d",
s"#$testData",
"-H",
"Content-Type: application/x-www-form-urlencoded",
"-X",
"POST",
slackurl
)
when I am trying to send some data, the error is : "cant read from the file"
i used the following: Runtime.getRuntime.exec( Array[String]( "curl", "-X", "POST", "-H", "Content-type: application/json", "--data", actData, slackurl ) ) Its working for me now. [space issue in CURL with Runtime
But when i send the a json to slack, message was displayed as "Array".
Is it possible to create a new Class programmatically (i.e. not from the dashboard) via any of the API's or the Parse CLI?
The REST API appears to have functionality to fetch, modify and delete individual Schemas (classes) but not to add them. (https://parse.com/docs/rest/guide#schemas).
Hoping for something like the following:
curl -X ADD \
-H "X-Parse-Application-Id: XXXXXX" \
-H "X-Parse-Master-Key: XXXXXXXX" \
-H "Content-Type: application/json" \
https://api.parse.com/1/schemas/City
You seem to have skipped the part which deals with adding schema in the documentation. To create a new class, according to documentation, You use following method in cURL:
curl -X POST \
-H "X-Parse-Application-Id: Your APP Id" \
-H "X-Parse-Master-Key: Your master key" \
-H "Content-Type: application/json" \
-d '
{
"className": "Your Class name goes here",
"fields": {
"Your field name here": {
"type": "Your field's data type e.g. String, Int etc. Add multiple fields if you want"
}
}
}' \
https://api.parse.com/1/schemas/[Your class name]
Or in Python:
import json,httplib
connection = httplib.HTTPSConnection('api.parse.com', 443)
connection.connect()
connection.request('POST', '/1/schemas/Game', json.dumps({
"className":"[Your class name]","fields":{"Your field name":{"type":"your field's data type"} }
}), {
"X-Parse-Application-Id": "7Lo3U5Ei75dragCphTineRMoCfwD7UJjd1apkPKX",
"X-Parse-Master-Key": "ssOXw9z1ni1unx8tW5iuaHCmhIObOn4nSW9GHj5W",
"Content-Type": "application/json"
})
result = json.loads(connection.getresponse().read())
print result
How to specify parameters in something like this - WHERE a.name IN ["Peter", "Tobias"]. I am trying to pass the collection after IN operator as a parameter in Cypher. I am using Cypher through REST API.
This is my example:
curl -X POST http://localhost:7474/db/data/ext/CypherPlugin/graphdb/execute_query -H "Content-Type: applicatio/json" --data-binary '{
"query": "start ca=node:ca({search_ca_query}) MATCH ca_club-[:has]-ca WHERE (ca_club.CA_CLUB IN {CA_CLUB}) RETURN distinct ca.NUM_OFC_CA, ca.NME_CA, ca_club.CA_CLUB",
"params": {
"search_ca_query": "NUM_OFC_CA:(\"000333\", \"111033\", \"222197\")",
"CA_CLUB": "[\"Driad\", \"No-Club\"]"
}
}'
I have also tried swapping square brackets in query, but even that didn't worked. (i.e. i am not getting any error but getting an empty list - "data" : [ ].
Any suggestions on how to do this?
Your in parameter needs to be a list:
curl -X POST http://localhost:7474/db/data/ext/CypherPlugin/graphdb/execute_query -H "Content-Type: applicatio/json" --data-binary '{
"query": "start ca=node:ca({search_ca_query}) MATCH ca_club-[:has]-ca WHERE (ca_club.CA_CLUB IN {CA_CLUB}) RETURN distinct ca.NUM_OFC_CA, ca.NME_CA, ca_club.CA_CLUB",
"params": {
"search_ca_query": "NUM_OFC_CA:(\"000333\", \"111033\", \"222197\")",
"CA_CLUB": ["Driad", "No-Club"]
}
}'