Where do I retrieve the NextToken from when using the AWS Powershell CmdLets?
For example when I call Get-CDDeploymentList I need to supply the NextToken to retrieve the next set of deployment IDs. However the Get-CDDeploymentList command only returns an array of deployment IDs and not a NextToken.
The NextToken is contained in the $AWSHistory.LastServiceResponse variable.
In the case of the Get-CDDeploymentList command the LastServiceResponse will contain the properties Deployments and the NextToken, so the NextToken can be retrieved using:
$AWSHistory.LastServiceResponse.NextToken
For more information on the $AWSHistory object see http://docs.aws.amazon.com/powershell/latest/userguide/pstools-pipelines.html.
Actually you don't need to use NextToken unless you want or need to take manual control of pagination. By default, if NextToken isn't supplied to the vast majority of the cmdlets, they will automatically handle pagination for you internally and make multiple calls to the underlying service api to emit the full data set to the pipeline.
There are a couple of service apis where the response data from the api call contains more than one field that we would emit to the pipeline (imagine a call that returned a list of 'success' elements as well as a list of 'failed' elements). In these scenarios the cmdlets will emit the entire response object to the pipeline and it will contain the next token element -- for these you (the user) have to manually paginate.
I'm sure we used to note when cmdlets auto-paginate (and when they don't) in the cmdlet documentation but in looking at the linked cmdlet documentation it seems we've dropped this somewhere along the way - I'll investigate and get this fixed.
Related
Hi i am processing a set of ~50K records from a pipe delimeted flat kn azure data factory and need to invoke a rest API call for each input record. So, I am using a foreach loop to access each record and inside the loop, I am using a copy activity to invoke a rest API call.
My question is, can I invoke the rest API call in bulk for all the records at once, as the foreach loop is slowing the pipeline execution. I want to remove the foreach loop and also process the API json response and store it in azure sql database.
Thanks
You will have to check the Pagination properties so that you can decide how much payload you need to return from source API:
https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#pagination-support
Also, if you need to store the API JSON response in Azure SQL, then you can do so with many built in functions like JSON_PATH
More details can be found in this link:
https://learn.microsoft.com/en-us/azure/azure-sql/database/json-features
I am trying to collect some metrics on releases in Azure Devops via a Powershell script.
I have very limited dev experience and am new to PowerShell. And this is the first time I have worked with an API. So far I have been able to authenticate, return a list of releases, loop through them and export the data to a file. Now I need to filter the releases based on a substring of the release name. For the record I have been doing my initial testing in Postman to make sure my syntax and results are correct. Then I migrated working syntax to Powershell.
https://{{organization}}.vsrm.visualstudio.com/{{project}}/_apis/release/releases?api-version=5.0
If I add the id filter as shown here:
https://{{organization}}.vsrm.visualstudio.com/{{project}}/_apis/release/releases?api-version=5.0&releaseId=34567
I get this result:
"id": 34567,
"name": "Test-Release-MyService",
But if use the same filter format for Release Name,
https://{{organization}}.vsrm.visualstudio.com/{{project}}/_apis/release/releases?api-version=5.0&releaseName="Test-Release-MyService"
I get back 50 results of which none match that criteria, whether I wrap the string in quotes or not. Furthermore, what i really want to do is to have the response only include records where the releaseName contains "XYZ".
So the question: Is there a filter operator for "contains" so I only get back records where the release name contains the "XYZ" substring?
Thanks in advance for your advice.
Every parameter you used in Azure DevOps REST API needs to be consistent with the description in the document, Azure DevOps REST API does not support custom parameters. For your question, the parameter searchText is used to filter the the searching result with the release name containing the keyword. I have tested with POSTMAN to call the api, it works fine. In addition, the value of parameter searchText is not case-sensitive. Filter release name
If you want to do more filter, in fact you can use powershell or other client library to deserialize the json response to an object, and do some convert or filter. Following documents may be helpful for you:
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/convertfrom-json?view=powershell-6
https://devblogs.microsoft.com/scripting/playing-with-json-and-powershell/
I'm attempting to pull data from the Square Connect v1 API using ADF. I'm utilizing a Copy Activity with a REST source. I am successfully pulling back data, however, the results are unexpected.
The endpoint is /v1/{location_id}/payments. I have three parameters, shown below.
I can successfully pull this data via Postman.
The results are stored in a Blob and are as if I did not specify any parameters whatsoever.
Only when I hardcode the parameters into the relative path
do I get correct results.
I feel I must be missing a setting somewhere, but which one?
You can try setting the values you want into a setVariable activity, and then have your copyActivity reference those variables. This will tell you whether it is an issue with the dynamic content or not. I have run into some unexpected behavior myself. The benefit of the intermediate setVariable activity is twofold. Firstly it coerces the datatype, secondly, it lets you see what the value is.
My apologies for not using comments. I do not yet have enough points to comment.
I have an hcm cloud instance. I'm working on the rest api which are provided by the cloud.
I want to get an employee by matching both PersonNumber as well as DateOfBirth.
But whatever I tried based on the first parameter, I'm getting the output. Second is not even checked.
Can anyone help?
This is the rest url I'm using
https://host:port/hcmCoreApi/resources/11.12.1.0/emps/?q=DateOfBirth=1991-09-19&PersonNumber=240
For passing multiple search items in a query parameter in a rest call, the structure should be as following
https://host:port/hcmCoreApi/resources/11.12.1.0/emps/?q=DateOfBirth='1991-09-19'&PersonNumber=240
Basically quotes '' are required around String inputs and integer can be passed directly.
I have a scenario in which I have REST API which manages a Resource which we will call Group. A Group contains members and the group resource is dynamic - whenever you retrieve it, you get the latest data (so a query must run server side to update the number of members in a group - in other words, the result of the request is to modify the data, since the results of running the query are stored).
Given a *group_id* it should return a minimal amount of information like
{
group_id: "5t7yu8i9io0op",
group_name: "That's my name",
size: 34
}
So a GET to this resource causes the resource to change, since a subsequent GET could return a new value for 'size'. This tells me it is not idempotent and so you should use POST to retrieve this resource. Am I correct in this conclusion?
If I am correct, do you think it is advisable to also provide a GET method that only returns the currently stored data for the group (eg. so the size could be out of date, even the name too). I suppose in this case I should return a last-modified date as one of the fields so that the user knows how up-to-date the resource is and can then elect to use the POST method...but then I am left wondering why would anyone do that, so why not ONLY provide the POST method and forget about GET?
Confused I am!
Thanks in advance.
[EDIT]
#Satish posted a link in his/her answer to the HTTP specs. In section 9.1.1. it ends with this sentence:
Naturally, it is not possible to ensure that the server does not generate side-effects as a result of performing a GET request; in fact, some dynamic resources consider that a feature. The important distinction here is that the user did not request the side-effects, so therefore cannot be held accountable for them.
So in my scenario, the requester does not really care about the side-effect that the value for 'size' is recomputed as a direct result of making the request. They want the group information and it just so happens that to provide accurate, up-to-date group data, the size query must be run in order to update that value. Whilst making the request causes data to change implies this should be a POST, the user did not request that side-effect and so therefore a GET request would be acceptable and more intuitive, would it not? And therefore still be restful according to this sentence.
[2nd EDIT]
#Satish asks a very important question in the comments. So for others who read this I'll explain further about this problem:
Normally you would not run the group query to update its size from a REST request. As members are added or removed from a group, you would update the computed size of that group, store it and then a simple GET request would always return the correct size. However, our situation is more complicated in that a group is only stored as a query definition in ElasticSearch (kind of like a view in an RDBMS). Members do not get added/removed to and from groups. They get added to a much larger set of data (a collection in MongoDB). There are hundreds, potentially thousands, of different 'group definitions' so it is not practical to recompute size for every group when the collection changes. We cannot know when an item is added/removed to/from the collection which groups might change size - you only know by running the group definition who is in that group and what the size is. I hope that clears things up. :)
You should use GET. Even if dynamic resource is changing, you did not request for that change through your request and you are not accountable for that change. Ref: http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html
In your case when you do a GET you retrieve some information about the Group. You don't modify the group structure. Ok, the group can be changed by an external entity so your next GET may bring you another data. Am I right? Who modifies the structure of the group and when?
So you should use GETbecause the resource it will be modified from somewhere else and not by your call that tries to do a read operation.
EDIT
After your edited the question I just want to add that I agree about the side effects.
It matters if you sent data or a change command explicitly to the server or you just read something and you don't have to pay attention for what the server side is doing to gave you the response. More intuitively:
GET - Requests data from a specified resource
POST - Submits data to be processed to a specified resource
It is the combination of GET and POST. So you should use POST.
Refer : http://adarshdchaurasia.wordpress.com/2013/09/26/http-get-vs-post/
You should not use GET because if you use GET method then search engines may cache the responses. It may cause unintentional data update at your server side, which you do not want. GET method is meant to return content without updating anything on server. POST is meant to updated the things at server and return result against that operation.