I have this cURL command to run:
curl -X POST -H "Content-Type: application/json"
-u "{username}":"{password}"
"https://gateway.watsonplatform.net/retrieve-and-rank/api/v1/solr_clusters/{solr_cluster_id}/solr/example_collection/update"
--data-binary #./data.json
Where data.json is a 60 GB file. I ran it, and my computer bluescreened after reaching 100% memory usage. I assume by that that cURL reads the entire file into memory before sending it.
Obviously, with 16GB of RAM I won't be able to do that. So, my question is: is there a way to use cURL where I could send a huge file like that? And if not, is there an alternative way to accomplish the same thing that cURL does?
Related
im looking at apache Ranger rest API to add an existing internal user/users to an existing internal group.
I have been looking at the docs and cant seem to find something useful,is there an API for that?
Try following api call:-
curl -ivk -u admin:admin -H "Content-Type: application/json" -d '{"id":13,"createDate":"2020-12-23T07:55:04Z","updateDate":"2020-12-23T07:55:04Z","owner":"rangerusersync","updatedBy":"rangerusersync","name":"atlas","password":"*****","description":"atlas - add from Unix box","groupIdList":[6,59,4,131,133],"groupNameList":["atlas","hadoop","shadow"],"status":0,"isVisible":1,"userSource":1,"userRoleList":["ROLE_USER"],"otherAttributes":"{\"full_name\":\"atlas\",\"original_name\":\"atlas\"}"}' -X PUT https://RANGER_HOST:6182/service/xusers/users
Its little bit big but if user is already added to ranger then run following api to get the id information for users:-
curl -ivk -u admin:admin -H "Accept : application/json" -X GET https://RANGER_HOST:6182/service/xusers/users
Once you have IDs for all users, you can run following curl api to get the json formatted data which you can use to modify and then use PUT method in first API I mentioned:-
curl -ivk -u admin:admin -H "Accept: application/json" -X GET https://RANGER_HOST:6182/service/xusers/users/13
Above Curl api should return something like following:-
{"id":13,"createDate":"2020-12-23T07:55:04Z","updateDate":"2020-12-23T16:45:14Z","owner":"rangerusersync","updatedBy":"admin","name":"atlas","password":"*****","description":"atlas - add from Unix box","groupIdList":[133,6],"groupNameList":["apitest","atlas","hadoop","shadow","ssb"],"status":0,"isVisible":1,"userSource":1,"userRoleList":["ROLE_USER"],"otherAttributes":"{\"full_name\":\"atlas\",\"original_name\":\"atlas\"}"}
you have to modify "groupIdList":[133,6] from the above output copy entire output and pass it with PUT method as shown in the first api call mentioned above.
I'm trying to download the latest snapshot dependency of a zip on Nexus 3 (version 3.22.1-02) from a command line using curl:
curl -u username:password -X GET "https://mynexusserver/service/rest/v1/search/assets/download?sort=version&repository=snapshotsrepo&maven.groupId=mygroup&maven.artefactId=myartefact&maven.extension=zip" -H "accept: application/json" -o myartefact.zip
This request is similar to this example: http://help.sonatype.com/repomanager3/rest-and-integration-api/search-api#SearchAPI-DownloadingtheLatestVersionofanAsset but no result is returned, myartefact.zip is empty.
However with the same URL, my artefact is downloaded from a web browser or with gradle.
With curl the following command line is working fine, returning the list of all snapshot versions of my artefact:
curl -u username:password -X GET "https://mynexusserver/service/rest/v1/search/assets?sort=version&repository=snapshotsrepo&maven.groupId=mygroup&maven.artefactId=myartefact&maven.extension=zip" -H "accept: application/json" -o myartefact.zip
Downloading the artefact directly is working fine as well with a command line like:
curl -u username:password "https://mynexusserver/repository/snapshotsrepo/mygroup/batchfactory/myversion-SNAPSHOT/myartefact-myversion-mytimestamp.zip" -H "accept: application/json" -o myartefact.zip
Verbose logs (-v option) show the artefact is found (I get HTTP/1.1 302 Found message) but nothing is downloaded.
Using wget doesn't work any better, I can't even query the list of snapshot version of the artefact.
Am I missing something?
Thanks #Zeitounator, after adding "-L" the command line works fine. Considering code 302 this feels obvious now...
Nexus documentation should also probably be updated to add the "-L" option.
I have a POST method in my API which uses multipart encoded form data. I have set up the correct header and data settings so that the mashape web interface generated the following curl:
curl -X POST --include 'https://sslavov-text-analytics-v1.p.mashape.com/news' \
-H 'Authorization: Basic ***********' \
-H 'X-Mashape-Key: ************' \
-H 'Content-Type: multipart/form-data' \
-F 'file=#sample.docx' \
-F 'meta={"documentType": "application/vnd.openxmlformats-officedocument.wordprocessingml.document"};type=application/json'
Basically i'm trying to upload a file with a simple paragraph of text for processing. The curious part is that when I run this exact curl in a bash script, everything works smoothly, but when I try to run it through mashape, it says either 400 Bad Request or 500 Internal Server Error
In my particular case, these errors are generated when I don't pass correct form or headers. So my question is: Is there an error in the curl syntax or should I keep looking for the error on server side?
EDIT: I figured out what the problem was. -F 'file=#sample.docx' was passed before -F 'meta....' and that was causing the 500 Internal Server Error So now the question is: Is there any way to specifically arrange the order of the form fields (because mashape rearranges them aplhabetically)?
Scenario
Among other things, Powershell 2.0 doesn't have the useful cmdlet Invoke-RestMethod.
I can't upgrade to version 3 and most examples I've found use version 3.
I have found this article, which seems, however, too complicated for my simple scenario.
I need to write a Powershell script that POSTs data in Json format, e.g.
{"Id":5,"Email":"test#com","DataFields":null,"Status":0}
What I've tried
I am able to GET data. This is one of the scripts I have tried.
curl -v --user username:password https://api.dotmailer.com/v2/account-info
But, when I try to POST, I can't figure out where to put the body of the message in the script. This is what I've got so far:
curl -v -X POST -H "Accept: application/json" -H "Content-Type: application/json" -u username:password -d '{"Id":5,"Email":"test#com","OptInType":0,"EmailType":0, "DataFields":null,"Status":0}' https://api.dotmailer.com/v2/contacts
which returns the following error:
{"message":"Could not parse the body of the request based on the content type \"application/json\" ERROR_BODY_DOES_NOT_MATCH_CONTENT_TYPE"}*
Question
Can anyone advise on how to POST Json data from Powershell using cURL?
Any pointers to why I get the error I mentioned in the Waht I've tried section would be much appreciated.
Thanks.
Note that the question is about the curl.exe external program, not about PowerShell's Invoke-WebRequest cmdlet (which, unfortunately, is aliased to curl in later PowerShell versions, preempting calls to the external program unless the .exe extension is explicitly specified (curl.exe ...).
Unfortunately and unexpectedly, you have to \-escape embedded " instances in a string you pass as an argument to an external program.
Therefore, even though:
'{"Id":5,"Email":"test#com","DataFields":null,"Status":0}'
should work, it doesn't, due to a long-standing bug; instead, you must use:
'{\"Id\":5,\"Email\":\"test#com\",\"DataFields\":null,\"Status\":0}'
See this answer for more information.
From curl's man page it appears you need to use -d switch:
curl -v --user username:password -H "Content-Type: application/json" -d '{"Id":5,"Email":"test#com","DataFields":null,"Status":0}' https://api.dotmailer.com/v2/contacts
I'm writing unit tests for an ElasticSearch datasource, however, I am getting mixed results. The problem is that a match_all query isn't finding records that I submitted, however, when I run the commands by hand using CURL in the same order the unit test does I am able to find the records.
I believe that perhaps the index isn't refreshed, so, I started running the "refresh" api command after submitting records, however, this didn't work either. Here is my list of commands - it would be helpful if anyone had any suggestions on how to make sure these commands worked even if they were run in immediate succession.
Commands the unit test runs:
curl -XGET 'http://localhost:9200/test_index/_mapping'
curl -XDELETE 'http://localhost:9200/test_index/test_models'
curl -XPOST 'http://localhost:9200/test_index/test_models/_refresh' -d '{}'
curl -XPUT 'http://localhost:9200/test_index/test_models/_mapping' -d '{"test_models":{"properties":{"TestModel":{"properties":{"id":{"type":"string","index":"not_analyzed"},"string":{"type":"string"},"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"},"modified":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}},"type":"object"}}}}'
curl -XPOST 'http://localhost:9200/test_index/test_models/_bulk' -d '{"index":{"_index":"test_index","_type":"test_models","_id":"test-model"}}
{"TestModel":{"id":"test-model","string":"Analyzed for terms","created":"2012-01-01 00:00:00","modified":"2012-02-01 00:00:00"}}
'
curl -XPOST 'http://localhost:9200/test_index/test_models/_refresh' -d '{}'
curl -XGET 'http://localhost:9200/test_index/_mapping'
curl -XGET 'http://localhost:9200/test_index/test_models/_search' -d '{"query":{"match_all":{}},"size":10}'
This question has also been posted to the (super awesome) ElasticSearch mailing list:
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/Nxv0XpLDY4k
-DK
The problem is with the _refresh command.
You can't refresh a type, only an index. I changed the refresh command to:
curl -XPOST 'http://localhost:9200/test_index/_refresh'
And it is now fixed!