I'm writing unit tests for an ElasticSearch datasource, however, I am getting mixed results. The problem is that a match_all query isn't finding records that I submitted, however, when I run the commands by hand using CURL in the same order the unit test does I am able to find the records.
I believe that perhaps the index isn't refreshed, so, I started running the "refresh" api command after submitting records, however, this didn't work either. Here is my list of commands - it would be helpful if anyone had any suggestions on how to make sure these commands worked even if they were run in immediate succession.
Commands the unit test runs:
curl -XGET 'http://localhost:9200/test_index/_mapping'
curl -XDELETE 'http://localhost:9200/test_index/test_models'
curl -XPOST 'http://localhost:9200/test_index/test_models/_refresh' -d '{}'
curl -XPUT 'http://localhost:9200/test_index/test_models/_mapping' -d '{"test_models":{"properties":{"TestModel":{"properties":{"id":{"type":"string","index":"not_analyzed"},"string":{"type":"string"},"created":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"},"modified":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}},"type":"object"}}}}'
curl -XPOST 'http://localhost:9200/test_index/test_models/_bulk' -d '{"index":{"_index":"test_index","_type":"test_models","_id":"test-model"}}
{"TestModel":{"id":"test-model","string":"Analyzed for terms","created":"2012-01-01 00:00:00","modified":"2012-02-01 00:00:00"}}
'
curl -XPOST 'http://localhost:9200/test_index/test_models/_refresh' -d '{}'
curl -XGET 'http://localhost:9200/test_index/_mapping'
curl -XGET 'http://localhost:9200/test_index/test_models/_search' -d '{"query":{"match_all":{}},"size":10}'
This question has also been posted to the (super awesome) ElasticSearch mailing list:
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/Nxv0XpLDY4k
-DK
The problem is with the _refresh command.
You can't refresh a type, only an index. I changed the refresh command to:
curl -XPOST 'http://localhost:9200/test_index/_refresh'
And it is now fixed!
Related
im looking at apache Ranger rest API to add an existing internal user/users to an existing internal group.
I have been looking at the docs and cant seem to find something useful,is there an API for that?
Try following api call:-
curl -ivk -u admin:admin -H "Content-Type: application/json" -d '{"id":13,"createDate":"2020-12-23T07:55:04Z","updateDate":"2020-12-23T07:55:04Z","owner":"rangerusersync","updatedBy":"rangerusersync","name":"atlas","password":"*****","description":"atlas - add from Unix box","groupIdList":[6,59,4,131,133],"groupNameList":["atlas","hadoop","shadow"],"status":0,"isVisible":1,"userSource":1,"userRoleList":["ROLE_USER"],"otherAttributes":"{\"full_name\":\"atlas\",\"original_name\":\"atlas\"}"}' -X PUT https://RANGER_HOST:6182/service/xusers/users
Its little bit big but if user is already added to ranger then run following api to get the id information for users:-
curl -ivk -u admin:admin -H "Accept : application/json" -X GET https://RANGER_HOST:6182/service/xusers/users
Once you have IDs for all users, you can run following curl api to get the json formatted data which you can use to modify and then use PUT method in first API I mentioned:-
curl -ivk -u admin:admin -H "Accept: application/json" -X GET https://RANGER_HOST:6182/service/xusers/users/13
Above Curl api should return something like following:-
{"id":13,"createDate":"2020-12-23T07:55:04Z","updateDate":"2020-12-23T16:45:14Z","owner":"rangerusersync","updatedBy":"admin","name":"atlas","password":"*****","description":"atlas - add from Unix box","groupIdList":[133,6],"groupNameList":["apitest","atlas","hadoop","shadow","ssb"],"status":0,"isVisible":1,"userSource":1,"userRoleList":["ROLE_USER"],"otherAttributes":"{\"full_name\":\"atlas\",\"original_name\":\"atlas\"}"}
you have to modify "groupIdList":[133,6] from the above output copy entire output and pass it with PUT method as shown in the first api call mentioned above.
I'm trying to download the latest snapshot dependency of a zip on Nexus 3 (version 3.22.1-02) from a command line using curl:
curl -u username:password -X GET "https://mynexusserver/service/rest/v1/search/assets/download?sort=version&repository=snapshotsrepo&maven.groupId=mygroup&maven.artefactId=myartefact&maven.extension=zip" -H "accept: application/json" -o myartefact.zip
This request is similar to this example: http://help.sonatype.com/repomanager3/rest-and-integration-api/search-api#SearchAPI-DownloadingtheLatestVersionofanAsset but no result is returned, myartefact.zip is empty.
However with the same URL, my artefact is downloaded from a web browser or with gradle.
With curl the following command line is working fine, returning the list of all snapshot versions of my artefact:
curl -u username:password -X GET "https://mynexusserver/service/rest/v1/search/assets?sort=version&repository=snapshotsrepo&maven.groupId=mygroup&maven.artefactId=myartefact&maven.extension=zip" -H "accept: application/json" -o myartefact.zip
Downloading the artefact directly is working fine as well with a command line like:
curl -u username:password "https://mynexusserver/repository/snapshotsrepo/mygroup/batchfactory/myversion-SNAPSHOT/myartefact-myversion-mytimestamp.zip" -H "accept: application/json" -o myartefact.zip
Verbose logs (-v option) show the artefact is found (I get HTTP/1.1 302 Found message) but nothing is downloaded.
Using wget doesn't work any better, I can't even query the list of snapshot version of the artefact.
Am I missing something?
Thanks #Zeitounator, after adding "-L" the command line works fine. Considering code 302 this feels obvious now...
Nexus documentation should also probably be updated to add the "-L" option.
I am using Marklogic 9 and try to ingest data from external source into MarkLogic. I made an REST API on port 8031. When I try to execute the following curl command:
curl --anyauth --user admin:admin -i -X POST -d https://services7.arcgis.com/21GdwfcLrnTpiju8/arcgis/rest/services/Geluidsbelasting/FeatureServer/0/query?where=1%3D1&outFields=*&outSR=4326&f=json
-H "Content-type: application/json" -H "Accept: application/json" \
'http://localhost:8031
After executing this statement I receive the error:
Curl: URL is not specified
Can you please help me out!
Many thanks
Erik
Your -d parameter has special characters that are not escaped. Try putting quotes around your -d url. It will prevent your command from getting truncated and misinterpreted at & signs..
HTH!
Scenario
Among other things, Powershell 2.0 doesn't have the useful cmdlet Invoke-RestMethod.
I can't upgrade to version 3 and most examples I've found use version 3.
I have found this article, which seems, however, too complicated for my simple scenario.
I need to write a Powershell script that POSTs data in Json format, e.g.
{"Id":5,"Email":"test#com","DataFields":null,"Status":0}
What I've tried
I am able to GET data. This is one of the scripts I have tried.
curl -v --user username:password https://api.dotmailer.com/v2/account-info
But, when I try to POST, I can't figure out where to put the body of the message in the script. This is what I've got so far:
curl -v -X POST -H "Accept: application/json" -H "Content-Type: application/json" -u username:password -d '{"Id":5,"Email":"test#com","OptInType":0,"EmailType":0, "DataFields":null,"Status":0}' https://api.dotmailer.com/v2/contacts
which returns the following error:
{"message":"Could not parse the body of the request based on the content type \"application/json\" ERROR_BODY_DOES_NOT_MATCH_CONTENT_TYPE"}*
Question
Can anyone advise on how to POST Json data from Powershell using cURL?
Any pointers to why I get the error I mentioned in the Waht I've tried section would be much appreciated.
Thanks.
Note that the question is about the curl.exe external program, not about PowerShell's Invoke-WebRequest cmdlet (which, unfortunately, is aliased to curl in later PowerShell versions, preempting calls to the external program unless the .exe extension is explicitly specified (curl.exe ...).
Unfortunately and unexpectedly, you have to \-escape embedded " instances in a string you pass as an argument to an external program.
Therefore, even though:
'{"Id":5,"Email":"test#com","DataFields":null,"Status":0}'
should work, it doesn't, due to a long-standing bug; instead, you must use:
'{\"Id\":5,\"Email\":\"test#com\",\"DataFields\":null,\"Status\":0}'
See this answer for more information.
From curl's man page it appears you need to use -d switch:
curl -v --user username:password -H "Content-Type: application/json" -d '{"Id":5,"Email":"test#com","DataFields":null,"Status":0}' https://api.dotmailer.com/v2/contacts
I am trying to create a WCF REST client that will communicate to Jenkins and create a job from an XML file and then build the job. My understanding is that you can do that with Jenkins.
Can some one please provide some commands that you can type on a browser's address bar to create and build jobs? ie: http:localhost/jenkins/createItem?name=TESTJOB something along those lines.
Usually, when parsing through the documentation, it can take one or two days. It is helpful to be able to access code or curl commands to get you up and running in one hour. That is my objective with a lot of third party software.
See the post at http://scottizu.wordpress.com/2014/04/30/getting-started-with-the-jenkins-api/ which lists several of the curl commands. You will have to replace my.jenkins.com (ie JENKINS_HOST) with the your own url.
To create a job, for instance, try:
curl -X POST -H "Content-Type:application/xml" -d "<project><builders/><publishers/><buildWrappers/></project>" "http://JENKINS_HOST/createItem?name=AA_TEST_JOB2"
This uses a generic config. You can also download a config from a manually created job and then use that as a template.
curl "http://JENKINS_HOST/job/MY_JOB_NAME/config.xml" > config.xml
curl -X POST -H "Content-Type:application/xml" -d #config.xml "http://JENKINS_HOST/createItem?name=AA_TEST_JOB3"
To execute the job (and set string parameters), use:
curl "http://JENKINS_HOST/job/MY_JOB_NAME/build"
curl "http://JENKINS_HOST/job/MY_JOB_NAME/buildWithParameters?PARAMETER0=VALUE0&PARAMETER1=VALUE1"
See the Jenkins API Wiki page (including the comments at the end). You can fill in the gaps using the documentation provided by Jenkins itself; for example, http://JENKINS_HOST/api will give you the URL for creating a job and http://JENKINS_HOST/job/JOBNAME/api will give you the URL to trigger a build.
I highly recommend avoiding the custom creation of job configuration XML files and looking at something like the Job DSL plugin instead. This gives you a nice Groovy-based DSL to create jobs programmatically - much more concise and less error-prone.
Thanks to a GIST - https://gist.github.com/stuart-warren/7786892
Check if job exists
curl -XGET 'http://jenkins/checkJobName?value=yourJobFolderName' --user user.name:YourAPIToken
With folder plugin
curl -s -XPOST 'http://jenkins/job/FolderName/createItem?name=yourJobName' --data-binary #config.xml -H "Content-Type:text/xml" --user user.name:YourAPIToken
Without folder plugin
curl -s -XPOST 'http://jenkins/createItem?name=yourJobName' --data-binary #config.xml -H "Content-Type:text/xml" --user user.name:YourAPIToken
Create folder
curl -XPOST 'http://jenkins/createItem?name=FolderName&mode=com.cloudbees.hudson.plugins.folder.Folder&from=&json=%7B%22name%22%3A%22FolderName%22%2C%22mode%22%3A%22com.cloudbees.hudson.plugins.folder.Folder%22%2C%22from%22%3A%22%22%2C%22Submit%22%3A%22OK%22%7D&Submit=OK' --user user.name:YourAPIToken -H "Content-Type:application/x-www-form-urlencoded"
If you want to create a job into a view given the view exists.
curl -X POST -H "Content-Type:application/xml" -d #build.xml "http://jenkins_host/view/viewName/createItem?name=itemName"
the build.xml filetemplate could be found in the root directory of a job's workspace
if you want to create a view:
curl -X POST -H "Content-Type:application/xml" -d #view.xml "http://jenkins_host/createView?name=viewName"
the content of the file view.xml could be:
<?xml version="1.0" encoding="UTF-8"?>
<hudson.model.ListView>
<name>viewName</name>
<filterExecutors>false</filterExecutors>
<filterQueue>false</filterQueue>
<properties class="hudson.model.View$PropertyList"/>
<jobNames>
<comparator class="hudson.util.CaseInsensitiveComparator"/>
</jobNames>
<jobFilters/>
<columns>
<hudson.views.StatusColumn/>
<hudson.views.WeatherColumn/>
<hudson.views.JobColumn/>
<hudson.views.LastSuccessColumn/>
<hudson.views.LastFailureColumn/>
<hudson.views.LastDurationColumn/>
<hudson.views.BuildButtonColumn/>
</columns>
</hudson.model.ListView>
and to check if a view exists:
curl -X POST -H "Content-Type:application/xml" "http://jenkins_host/checkViewName?value=viewName"
to check if a job exists:
curl -X POST -H "Content-Type:application/xml" "http://jenkins_host/checkJobName?value=jobName"
To create a job:
curl -X POST -H "Content-Type:application/xml" -d "<project><builders/><publishers/><buildWrappers/></project>" -u username: API_Token http://JENKINS_HOST/createItem?name=AA_TEST_JOB2
To build a job:
curl -X POST -u username:API_TOKEN http://JENKINS_HOST/job/MY_JOB_NAME/build
In case you need to make the same HTTP calls using the Python requests library, instead of CURL...
Download a job config:
import requests
auth = ("username", "api_token")
url = "http://" + JENKINS_HOST + "/job/" + JOB_NAME + "/config.xml"
response = requests.get(url, auth=auth)
open('config.xml', 'wt').write(response.text)
Create a new job using same config:
url = "http://" + JENKINS_HOST + "/createItem?name=" + NEW_JOB_NAME
headers = {'content-type': 'text/xml'}
data = response.text
response = requests.post(url, auth=auth, headers=headers, data=data)
Omit auth parameter when not needed.