I would like to chain 2 Azure Data Factory pipelines A -> B so that the pipeline B consumes the output of the Pipe line A.
So, basically A is a authorization pipeline that has few activities, the last of which produces the authentication token. Now that token I want use in the Pipeline B (and C, and D.. etc...). So, basically I want to reuse pipe line A.
I used Execute Pipleline activity but, its output is NOT what I expect .. How can use the actual output of A in B ?
Create Pipeline B
Click on Parameters
Add a parameter value ->inputfromA and give Default value as blank
Create a variable fromA and in the value,click on add dynamic content->#pipeline().parameters.inputfromA
Create Pipeline A
Store the value of authorization token in some variable
Add a execute pipeline and on InvokedPipeline->add pipelineB
Then it will ask to input the value of inputfromA ->click add dynamic content and then you can add the token value there.
So, I figured out how to solve this, for my case at least.
Since, I am dealing with (in the particular case) with the authentication token, I made use of the Azure Key Vault. Where, in the last activity of a pipe line A I POST the acquired token to the Azure key vault as a named secret.
In the pipeline B - what I do is I call 'execute Pipeline' activity - executing pipe line A. After that on completion, calling the web activity to read the token acquired in 'A'. And as a next activity in "B" I am using that token in API.
Related
My question is basically the title. I couldn't find such information reading this page and searching the web.
My scenario is the following: whenever I create a PR to master I add a tag with some information, like the lib version seen here:
Then, during the build process, I generate the release notes and would like to access that PR tag inside a task like I do with $(Build.BuildId) here:
How can I accomplish that? Something like $(PullRequest.Tag)
Thanks!
How can I accomplish that? Something like $(PullRequest.Tag) Thanks!
There's no predefined variable for Pull Request tag. (I use printenv command in CMD task to confirm this!)
Here're my workaround:
Use Powershell task to call this rest api, the response would contain the tag of specific PR
$url = "$($env:SYSTEM_TEAMFOUNDATIONSERVERURI)$env:SYSTEM_TEAMPROJECTID/_apis/git/repositories/$($env:BUILD_REPOSITORY_NAME)/pullRequests/$($env:SYSTEM_PULLREQUEST_PULLREQUESTID)/labels?api-version=5.1-preview.1"
$response = Invoke-RestMethod -Uri $url -Method Get -Headers #{
Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
Write-Host "##vso[task.setvariable variable=PullRequestTag;isOutput=true]$($response.value.name)"
Then pass the variable $response.value.name(come from rest api response, the name represents the tag name) to output variable PullRequestTag(custom variable) so that next tasks can access the returned tag name.
Notes:
1.Make sure the job which contains the Powershell task allow scripts to access the OAuth token cause my script uses OAuth token instead of PAT to call rest api. (Click Agent Job Name=>Additional options)
2.Since it's a output variable, we should use format $(referenceName.variablename).
After that, we can use $(PS.PullRequestTag) in next tasks to access the tag name.
3.Since your scenario is in a pipeline run triggered by PR, so actually the Powershell task should only run when current pipeline is triggered by PR instead of manual run/CI.
Use and(succeeded(), eq(variables['Build.Reason'], 'PullRequest')) in PS task's control options. See conditions and Build.Reason variable.
Update:
If I added several tags when creating the PR:
The $(PS.PullRequestTag)'s value would be:
I doubt there's a predefined variable for a pull request tag, but you can achieve the goal with the REST API. For example, you can have a build task (e.g. a PowerShell script) which:
gets all tags (aka labels) of a pull request
parses the list of tags to find out the necessary one
set the build variable to the name of the necessary tag
We have a lot of triggered Tasks that run on the same pipelines, but with different parameters.
My question regarding this - is there a possible way, like a function or an expression to capture the name of the triggered task so that we could use the information while writing the reports and e-mails of which Task started the error pipeline. I can't find anything even close to it.
Thank you in advance.
This answer addresses the requirement of uniquely identify the invoker task in the invoked pipeline
For triggered tasks, there isn't anything provided out of the box by SnapLogic. Although, in case of ULTRA tasks you can get $['task_name'] from the input to the pipeline.
Out of the box, SnapLogic provides the following headers that can be captured and used in the pipeline being initiated by the triggered task. These are as follows.
PATH_INFO - The path elements after the Task part of the URL.
REMOTE_USER - The name of the user that invoked this request.
REMOTE_ADDR - The IP address of the host that invoked this request.
REQUEST_METHOD - The method used to invoke this request.
None of these contains the task-name.
In your case, as a workaround, to uniquely identify the invoker task in the invoked pipeline you could do one of the following three things.
Pass the task-name as a parameter
Add the task-name in the URL like https://elastic.snaplogic.com/.../task-name
Add a custom header from the REST call
All the three above methods can help you capture the task-name in the invoked pipeline.
In your case, I would suggest you go for a custom header because the parameters you pass in the pipeline could be task-specific and it is redundant to add the task-name again in the URL.
Following is how you can add a custom header in your triggered task.
From SnapLogic Docs -
Custom Headers
To pass a custom HTTP header, specify a header and its value through the parameter fields in Pipeline Settings. The
request matches any header with Pipeline arguments and passes those to
the Task, while the Authorization header is never passed to the
Pipeline.
Guidelines
The header must be capitalized in its entirety. Headers are case-sensitive.
Hyphens must be changed to underscores.
The HTTP custom headers override both the Task and Pipeline parameters, but the query string parameter has the highest precedence.
For example, if you want to pass a tenant ID (X-TENANT-ID) in a
header, add the parameter X_TENANT_ID and provide a default or leave
it blank. When you configure the expression, refer to the Pipeline
argument following standard convention: _X_TENANT_ID. In the HTTP
request, you add the header X-TENANT-ID: 123abc, which results in the
value 123abc being substituted for the Pipeline argument X_TENANT_ID.
Creating a task-name parameter in the pipeline settings
Using the task-name parameter in the pipeline
Calling the triggered task
Note: Hyphens must be changed to underscores.
References:
SnapLogic Docs - Triggered Tasks
I'm adding this as a separate answer because it addresses the specific requirement of logging an executed triggered task separate from the pipeline. This solution has to be a separate process (or pipeline) instead of being part of the triggered pipeline itself.
The Pipeline Monitoring API doesn't have any explicit log entry for the task name of a triggered task. invoker is what you have to use.
However, the main API used by SnapLogic to populate the Dashboard is more verbose. Following is a screenshot of how the response looks on Google Chrome Developer Tools.
You can use the invoker_name and pipe_invoker fields for identifying a triggered task.
Following is the API that is being used.
POST https://elastic.snaplogic.com/api/2/<org snode
id>/rest/pm/runtime
Body:
{
"state": "Completed,Stopped,Failed,Queued,Started,Prepared,Stopping,Failing,Suspending,Suspended,Resuming",
"offset": 0,
"limit": 25,
"include_subpipelines": false,
"sort": {
"create_time": -1
},
"start_ts": null,
"end_ts": null,
"last_hours": 1
}
You can have a pipeline that periodically fires this API then parses the response and populates a log table (or creates a log file).
I am trying to make a call to an authorization endpoint using OAuth2 with grant type Client Credentials - my call is successful - that is not an issue. However, I, now, want to take the access token that is returned and put it in a variable so I may use it in subsequent calls without having to manually cut-and-paste to my other calls.
When the call returns I see the token I desire to copy in the Access Token field at the bottom of the OAuth2 window (the one shown below that says expires in 42 minutes) AND I see it in the Authorization field on the Timeline tab of the results. I just can't figure out how to get access to it so I may dump it into variable.
The gif on the FAQ goes really fast, and does not provide step by step. Also, I didnt find any answer on YouTube or other websites, so I thought to share step by step for chaining requests on Insomnia.
Create a POST query to obtain your access token. Notice that my access token is returned in the field called "access_token", we will use this in step 3. Your return field may be different.
Create a second GET request for the API that would return the data for you. In my case, I wanted to get all users from a SCIM interface. In the Bearer tab, type in Response => Body Attribute (Insomnia will autofill).
Mouse click on the Request => Body Attribute (the one you just typed in), and select the authentication post in the dropdown "Request" (this is the one you created in step 1), and in the "Filter (JSONPath)" field, type in the $.[attribute name] - where attribute name is the response that returns from authentication call. In my case, it was access_token, see step 1 for yours.
Enjoy!!
Click No Environment > Manage Environments and you will see a base environment in JSON.
Since this is in JSON, create a { "jwt_token": "Response => Body Attribute" }" pair for your token variable. Please note that "Response => Body Attribute" needs to be configured. When you type response, hit space and this option should be available.
Once done choosing "Response => Body Attribute", it will show with some gibberish content and with red background, no worries... just click it to configure. Make sure you have the same setup.
However... you need to change your request to the route where you get the token from the server and another thing is the Filter (JSONPath or XPath) change it depending on your setup.
You should have the token, stored in jwt_token variable and can use the variable on a route that you like.
Example:
If you want to save a token that is returned in a response into an environment variable, you can use request chaining in your environment variable. Take a look at this url for more details on that https://support.insomnia.rest/article/43-chaining-requests...
Here is what you could do (what I did)
Create an environment variable
For the value of the variable, use the Response => Body Attribute and under Filter (JSONPath or XPath), choose the attribute of the token in your response body (if it is "token" then put $.token).
After that just put the token environment variable wherever you need it in the following requests.
I was not able to resolve this question but was able to get around it by defining the fields in the body of the request and bypassing the OAuth2 tab completely.
You can add it as a header, by referencing the outputs of the OAuth2 request:
So as a part of some tests to automatically accept / merge successful pipelines in our git repository i was running some tests to flag the "merge when pipeline succeeds" feature when the pipeline is still running:
So this button is available when the pipeline is still running and will convert to a green 'Accept merge' button when the pipeline succeeds:
(note that this picture was taken afterwards not to confuse the use-case)
additionally i have set these general settings:
So when checking the Gitlab API Documentation it says i should use the following endpoint:
PUT /projects/:id/merge_requests/:merge_request_iid/merge
when using the parameter ?merge_when_pipleline_succeeds=true it should flag the button.
However when i call this endpoint when the pipeline is still running (i built in a wait for 10 mins while testing this) i get the following result:
i am getting a Method Not Allowed. My assumption is that the endpoint i am using is correct because otherwise i would've gotten a bad request / not found return code.
when checking the gitlab merge request i am seeing that indeed the flag is not set to true:
However, when i manually click the blue button the mergerequest looks like this:
Also if i let the pipeline finish and then proceed to call the merge api (w/ or w/o the merge when pipeline succeeds flag) it will accept the merge. It just does not work when the pipeline is running (which is odd because even the button itself only shows when the pipeline is running)
so i am wondering what I am doing wrong here.
I am using a Powershell module to call the GitLab API. The Accept part of the module is not official and was made by myself because i found this feature missing.
I am using the same credentials for the API w/ a personal access token to authenticate to the API. Other functionality of the API work with this token like creating merge requests, retrieving status of a current MR and accepting a MR when the pipeline is finished.
I have tried the following variants :
Use the V3 api with merge_when_build_succeeds=true --> nets the same
result
Uncheck the "Only allow merge request to be merged if the
pipeline succeeds" --> nets the same result
Use ID of the merge request instead of IID
use /merge_when_pipeline_succeeds instead of ?merge_when_pipeline_succeeds=true
use True instead of true --> nets the same result
I get a similar issue with the python-gitlab library on v4. It works sometimes when I use:
mr.merge(merge_when_pipeline_succeeds=True)
Where mr is a ProjectMergeRequest object. However, if the MR has a merge conflict in it I get that 405 Method Not Allowed error back.
My best guess is to see if I can apply logic before calling mr.merge() to check for problems. Will update this if that works.
UPDATE: Looks like there is no feature to check for conflicts as of today. https://gitlab.com/gitlab-org/gitlab-ce/issues/41762
UPDATE 2: You can check merge_status when looking at the MR information, so either that attribute or an exception then mr.merge() fails would let you identify when it won't work.
I'm creating a test case for a REST API in soapUI 4.5 where I'm going to use the content from step X to make a new call in step Y.
Ideally I'd create the REST request with some parameters, say A and B, and say that these parameters should be used in the URL:
http://myapi.com/v1/stuff/A/B
Then I'd do a property transfer step and simply set values extracted from step X into A and B.
It looks as if soapUI only lets me create querystring parameters, like this:
http://myapi.com/v1/stuff?ParamA=A&ParamB=B
This works of course, but I'd like to be able to call it both ways, to verify they're both working.
Am I missing something?
I am not a soapui expert by any means, but have just worked through a very similar scenario, so this might help you out.
Part 1: Create a paramatized resource
In my service, I have a resource called stuff:
http://{host}/stuff
I create a child resource with the below values:
Resource Name: Get stuff by ID
Resource Path/Endpoint: {stuffId}
and before clicking ok, click Extract Params - this will populate the Parameters table with an entry like:
Name | Default value | Style | Location
stuffId | stuffId | TEMPLATE | RESOURCE
then click ok. You now have a resource that allows you to dynamically supply an id:
http://{host}/stuff/{id}
you would need to repeat this to create the B parameter above (or you could create A and B as two parameters to the single resource if you never call /stuff/A without also supplying B).
Part 2: Create the test case
Now in the test case, you need to retrieve A, transfer the property, and then send a request to the above resource using the property:
In the test case, create the request to retrieve the response containing A
Right click the testcase and add a Properties step. Add a property to store the value of A.
From the response in the Outline view, right click the value of A and select "Transfer to > Property", select the property you just created and hit ok
Create a new request, using the new paramatized resource created in the first part. In the place of the id, put a reference to the property which is holding the value of A in this format:
${propertyName}
I might have done something wrong, but my test initially fails on the property transfer with "Missing source property". In the Source are of the PropertyTransfer step, I needed to set the property to ResponseAsXml
Hope this helps!