Azure Datafactory Pipeline execution status - azure-data-factory

It is kind of annoying we cannot change the logical order(AND/OR) of the Activity dependencies. however, I have got another issue. having said that I have activities for on failure to log the error messages in DB, since the logging activity succeeds, the entire pipeline succeeds too! is there any workaround to say if any activities failed the entire pipeline and the parent pipeline, if it is called from another pipeline, should be failed either?
In my screenshot, i have selected the on completion dependencies to log the successful or error.

I see that you defined "On Success" of the copy activity to run "usp_postexecution" . Please define a "On failure" of the copy activity and add any activity ( may be a set variable for testing ) and execute the pipeline . The pipeline will fail .
Just to give you more context what i tried .
I have a variable name "test" of the type boolean and I am failing it deliberately ( by assigning to a non-boolean value of true1 )
Pipeline will fail when I define both success and failure scenarios .
The pipeline will succeed when you have only "Failure" defined

Related

Azure DevOps deployment fails with the error message "The operation was canceled."

My Azure DevOps pipeline tasks successfully complete without issues except for the final deployment step:
Job Issues - 1 Error
The job running on agent XXXX ran longer than the maximum time of 00:05:00 minutes. For more information, see https://go.microsoft.com/fwlink/?linkid=2077134
The build logs state the operation was canceled:
021-03-02T20:50:00.4223027Z Folders: 695
2021-03-02T20:50:00.4223319Z Files: 10645
2021-03-02T20:50:00.4223589Z Size: 672611102
2021-03-02T20:50:00.4223851Z Compressed: 249144045
2021-03-02T20:50:03.6023001Z ##[warning]Unable to apply transformation for the given package. Verify the following.
2021-03-02T20:50:03.6032907Z ##[warning]1. Whether the Transformation is already applied for the MSBuild generated package during build. If yes, remove the <DependentUpon> tag for each config in the csproj file and rebuild.
2021-03-02T20:50:03.6034584Z ##[warning]2. Ensure that the config file and transformation files are present in the same folder inside the package.
2021-03-02T20:50:04.5268038Z Initiated variable substitution in config file : C:\azagent\A2\_work\_temp\temp_web_package_3012195912183888\Areas\Admin\sitemap.config
2021-03-02T20:50:04.5552027Z Skipped Updating file: C:\azagent\A2\_work\_temp\temp_web_package_3012195912183888\Areas\Admin\sitemap.config
2021-03-02T20:50:04.5553082Z Initiated variable substitution in config file : C:\azagent\A2\_work\_temp\temp_web_package_3012195912183888\web.config
2021-03-02T20:50:04.5642868Z Skipped Updating file: C:\azagent\A2\_work\_temp\temp_web_package_3012195912183888\web.config
2021-03-02T20:50:04.5643366Z XML variable substitution applied successfully.
2021-03-02T20:51:00.8934630Z ##[error]The operation was canceled.
2021-03-02T20:51:00.8938641Z ##[section]Finishing: Deploy IIS Website/App:
When I examine the deployment states, I notice one of my tasks takes quite a while for what should be a fairly simple operation:
The file transform portion takes over half of the allotted 5 minutes? Could this be the issue?
steps:
- task: FileTransform#1
displayName: 'File Transform: '
inputs:
folderPath: '$(System.DefaultWorkingDirectory)/_site.com/drop/Release/Nop.Web.zip'
fileType: json
targetFiles: '**/dataSettings.json'
It may be inefficient but FileTransform log shows a significant amount of time spent after the variable has been substituted. Not sure what's causing the long delay, but the logs don't account for the time after the variable has been successfully substituted:
2021-03-02T23:04:44.3796910Z Folders: 695
2021-03-02T23:04:44.3797285Z Files: 10645
2021-03-02T23:04:44.3797619Z Size: 672611002
2021-03-02T23:04:44.3797916Z Compressed: 249143976
2021-03-02T23:04:44.3970596Z Applying JSON variable substitution for **/App_Data/dataSettings.json
2021-03-02T23:04:45.2396016Z Applying JSON variable substitution for C:\azagent\A2\_work\_temp\temp_web_package_0182869515217865\App_Data\dataSettings.json
2021-03-02T23:04:45.2399264Z Substituting value on key DataConnectionString with (string) value: ***
**2021-03-02T23:04:45.2446986Z JSON variable substitution applied successfully.
2021-03-02T23:07:25.4881687Z ##[section]Finishing: File Transform:**
The job running on agent XXXX ran longer than the maximum time of 00:05:00 minutes.
Based on the error message, the cause of this issue is that the running time of the Agent Job has reached the maximum value set.
If you are using the release pipeline, you could set the timeout in Agent Job -> Execution plan ->timeout.
If you are using the build pipeline, you could set the timeout in Agent Job -> Agent Job -> Execution plan -> timeout (For An Agent Job) and Options -> Build job -> Build job timeout in minutes (For whole Build pipeline).
The file transform portion takes over half of the allotted 5 minutes
From the task log, the zip package contains many files and folders. Therefore, transform task will take more time to traverse the file to find the target file.

Powershell to fetch the Component Governance result from azure devops build pipeline

We have a requirement to fail the azure devops Build pipeline if you get a Component goverance alert om missing license information on that particular build. Is there any powersehll command topull the component goveranace build result from pipeline and fail the build
When you get a Component goverance alert om missing license information, you could write an error with the following code:
# Writes an error to build summary and to log in red text
Write-Host "##vso[task.LogIssue type=error;]This is the error"
If you want this error to fail the build, then add this line:
exit 1

How to refer previous task and stop the build in azure devops if there is no new data to publish an artifact

Getsolution.exe will give New data available or no new data available, if new data available then next jobs should be executed else nothing should be executed. How should i do it? (i am working on classic editor)
example: i have set of tasks, consider 4 tasks:
task-1: builds the solution
task-2: runs the Getstatus.exe which get the status of data available or no data available
task-3: i should be able to use the above task and make a condition/use some api query and to proceed to publish an artifact if data is available if not cleanly break out of the task and stop the build. it Shouldn't proceed to publish artifact or move to the next available task
task-4:publish artifact
First what you need is to set a variable in your task where you run Getstatus.exe:
and then set condition in next tasks:
If you set doThing to different valu than Yes you will get this:
How to refer previous task and stop the build in azure devops if there is no new data to publish an artifact
Since we need to execute different task based on the different results of Getstatus.exe running, we need set the condition based on the result of Getstatus.exe running.
To resolve this, just like the Krzysztof Madej said, we could set variable(s) based on the return value of Getstatus.exe in the inline powershell task:
$dataAvailable= $(The value of the `Getstatus.exe`)
if ($dataAvailable -eq "True")
{
Write-Host ("##vso[task.setvariable variable=Status]Yes")
}
elseif ($dataAvailable -eq "False")
{
Write-Host ("##vso[task.setvariable variable=Status]No")
}
Then set the different condition for next task:
You could check the document Specify conditions for some more details.

Azure data factory Lookup

I have a lookup activity which reads data from a SQL table and the output of this is passed onto multiple Execute Pipeline tasks as parameters. The flow is as follows
Lookup -> Exct Pipeline 1 -> Exct Pipeline 2 -> Exct Pipeline 3
.This works fine for the first pipeline however the second execute Pipeline fails with the following error.
> "The template validation failed: 'The inputs of template action 'Exct
> Pipeline 2' at line '1 and column '178987' cannot reference action
> 'Lookup'. Action 'Lookup' must either be in 'runAfter' path or within
> a scope action on the 'runAfter' path of action 'Execute Pipeline 3',
> or be a Trigger"
Another point to be noted is that the Pipeline runs fine when triggered.It only fails when in debug.
Has anyone else seen this issue?

Stop the pipeline when stage is unstable

I have a Jenkins build pipeline created using workflow plugin. At the beginning the pipeline runs a gulp build inside of the docker container and then archives test results using the following code
step([$class: 'JUnitResultArchiver', testResults: 'build/test-results/*.xml'])
In the following steps I package up the artifacts and ship them to the binary repository.
When unit tests are not passing Jenkins understands the build is unstable and marks it yellow. However it still continues with subsequent steps in the pipeline. Is there any way make the pipeline stop when unit tests are failing?
the JUnitResultArchiver will cause this condition to be true when the build is unstable:
currentBuild.result != null.
If I remember correctly it sets it to UNSTABLE, but it is enough to check that is different than null.
So you could do something like
step([$class: 'JUnitResultArchiver', testResults: 'build/test-results/*.xml'])
if (currentBuild.result == null) {
//contintue with your pipeline
} else {
//notify that the build is unstable. //or just do nothing
}
There is nothing to do at Jenkins side but at Gulp side. The call to gulp CLI needs to return a proper error value to have the sh step failing correctly.
Jenkins just interprets what the shell is returning, so you juts need to make Gulp to return a fail when tests fail (see this blog post, it seems to achieve exactly that).