For my application I wrote a custom integration test that starts the application, performs important functionality and tests many calculations. The test is part of the application's source control.
The details of every exception are written to the Trace or as an exception.
I would like to be able - on my Azure DevOps build server - to start this custom test after the build has finished.
I will adapt my test application then to mail the results to some developers.
For me this suits my needs better then test frameworks I have seen.
Currently I start Test in Debug Mode in Visual Studio (not integrated yet). It is a Console application, executed by 'Test.exe' on the command prompt. I will change that to Windows application (unattended execution, without console).
Question: how do I start my custom test on the Azure build server (who calls it Pipeline) after the build has finished? Which build step do I have to use?
Note that the integration test is also a build artifact of the same build.
It all depends what external dependencies you have if you will be able to reach them from Azure DevOps agent but since you are on self hosted you have all cards in your hands. If this is Console app you can simply call it. Let's assume you copied it to Build.ArtifactStagingDirectory you can run it as:
- pwsh: Test.exe
workingDirectory: '$(Build.ArtifactStagingDirectory)'
if you want to do as part of the same stage when you produced that file. If you want to make this in another stage then you need to first download pipeline/build artifact and call it as above (you will get this probably in a different folder, so that needs to be changed).
Related
I am trying to create a release pipeline in Azure DevOps. We already have a functioning build pipeline that works well, it is able to package the build with VSBuild and publish it as an artifact. Then in the release pipeline I am using an IIS Deployment job (which includes IIS Manage and IIS Deploy tasks) and it gets that artifact to deploy.
The problem is that we already have a publish profile (.pubxml) that should take care of pretty much everything the IIS Deployment is doing (at least as far I as I understand it). So to me it seems I have two options that don't require me to refactor the project configuration itself.
I can try to mimic the settings on the IIS Deployment job to match our .pubxml as closely as possible and manually applying any changes that aren't doable through the task settings. Obviously this is not ideal as that would require us to update both when ever we make changes and it introduces a large chance of the pipeline breaking down over time.
I can scrap the idea of using IIS Deployment and just use a VSBuild task that uses arguments /p:DeployOnBuild=true /p:PublishProfile=Staging. This doesn't seem like best practices because it means my release pipeline isn't passing a build package to deploy, it is just creating a new one at each stage.
So is there a better option that would allow me to utilize the package I created with VSBuild and the .pubxml configuration together in a deploy? If that isn't possible then are either of my options the "correct" way to handle my situation or am I just missing another method of deployment I could use?
Thank you for any help or insight you can provide. Please let me know if there is any more information I can give that would be useful.
You can try using publish settings file (*.publishsettings) for your IIS deployment.
A publish settings file (.publishsettings) is different than a publishing profile (.pubxml) created in Visual Studio. A publish settings file is created by IIS or Azure App Service, or it can be manually created, and then it can be imported into Visual Studio.
To view more details, you can see:
Publish an application to IIS by importing publish settings in Visual Studio
Deploy your app to a folder, IIS, Azure, or another destination
So unfortunately there doesn't seem to be a way I can achieve everything I wanted in this. The publish profiles are required for when we build the project so without making changes to how we configure those I need to build the project whenever I want to deploy. Ultimately I went with option #2. I essentially just copied most of the build tasks used in the testing pipeline and placed those in the release pipeline with a few modified commands to actually deploy the build once finished. It all seems to work just fine but still doesn't feel like best practices. If I am missing something please let me know and I will make updates as appropriate.
When using Azure DevOps I notice that occasionally my pull request builds will fail. After some tracking down I noticed that this is only happening when another build is already running.
It seems that the reason is that the files in the output for the build (exe, dll, note_modules, etc.) will be locked so when another build is started the new build will fail until the currently running one is finished, then I will have to manually re-queue the build again.
I am not very familiar with Azure DevOps pipelines since we recently migrated to this platform and also not sure of the best way to fix this issue. The sln's being built include .NET Framework, .NET Core, TypeScript, and Node.js if that helps at all.
I would love to post the logs and current configuration, but due to company policy I'm not allowed to... :(
Azure DevOps build pipelines fail when another build is already running
You could try to use/add a Capability, like Agent.Name to that two specific build agents then in the build definition you put that capability as a Demands.
As stated here:
How to send TFS build to a specific agent or server
The Capabilities of the agent:
Project Settings->Agent pools->Your agent pool-> Agents->Agent->Capabilities
The Demands of the build pipeline:
Options-> Demands:
In this case, when a pipeline is running in this particular agent, another new build will be in pending state until the current build is completed.
I've created CI&CD on Azure repo, and it is doing code deployment on App service.
Now created Automation that opens the browsers and do actions and generates reports for test cases. (C#+Selenium+report package).
Now I need to the setup Test plan, I know, needs to run headless testing, my the questions are
need to do without buying a Test plan package, then how we should do? (I have a subscription but not with test plans)
How to do using Azure Test plans (any recent blog pls, and no MS docs links).
any help appreciated
As far as I know, you can execute Selenium tests directly in the pipeline(Without test plan).
Here are the steps:
Add Visual Studio Test Task to run the Selenium tests and the test result will be output to the Test tab(Build Pipeline or Release Pipeline).
You also could use Publish Test Results task to publish test result.
Here is a blog about the detailed steps: Automating Selenium Tests in Azure Pipelines.
For headless testing,
You could refer to UI testing considerations.
By the way, since you have already created CI/CD, you can also consider integrating UI testing steps into CI/CD.
If you still want to configure test plan, you could refer to this doc.
I'm using Team Foundation Version Control as a source control for my .NET Core 2.1 project.
AzureDevOps is configured in continuous integration to checkout the code and build it.
We have 3 environments (Staging, PreProd, Prod). The Staging is not isometric with Prod so it is untrustworthy and we have to execute our integration tests on each environment with environmental data.
My build is generated by an agent in AzureDevOps on an OnPremise server which can only reach Prod environment.
I'd like to automate my XUnit integration tests in an AzureDevOps pipeline, however, I don't know where and how to do it. Am I supposed to execute the integration test step after building? or after releasing?
It looks like I need to deploy my binaries first on my environments, then execute the integration tests, and, if they go wrong, rollback the release.
Weird?!?
How can unblock this situation?
Regards,
If you want to run integration tests you need to first deploy your binaries to environment. You can do it as a separate:
step,
stage
pipeline
after deploying code.
Here it is up to you how you will do it. (To achieve last option you need to use pipeline triggers)
If you follow approach shift left, it means you detect issues as quickly as possible, you should don't worry about breaking them. If it happens on staging I would rather encourage you to fix the issue instead of roll backing code. Especially if it involves data model change.
And on production you can run only smoke tests, which are kind of integration tests which doesn't impact on state. They are like GET in REST - smoke tests should be idempotent, so you can run them without worrying bout changing state.
Since you use TFVC version, you could define a build pipeline to build and test your code, and then to publish artifacts. You also define a release pipeline to consume and deploy those artifacts to deployment targets.
As you have to execute integration tests on each environment with environmental data, you can run your XUnit integration tests in Release pipeline via VSTest task.
I am trying an automated build process in Azure DevOps for Salesforce. whenever a change is pushed to the repository, my build is triggered and it is working fine and pushing the changes to the related sandbox. Here is the proof for the same
Success Build Process.
The configuration of the build is Build configuration.
The build is working fine as expected. I now want to create a release which will push this change to a different environment, and I don't want this to be automated, hence the option of creating the release. The build path to the ant file in my release is exactly as it should be but I am getting this error. Release Error.
The release configuration is Release configuration
My Repository folder structure is: Folder structure. and my build.xml is within the deploy folder.
I don't know what I am doing wrong but the release is always failing and giving me the error which says:
Error: Not found antBuildFile: D:\a\r1\a\deploy\build.xml
Not found antBuildFile: D:\a\r1\a\deploy\build.xml
Based on the first image (Success Build Process), seems that you already have deployed your changes on that sandbox. Working with metadata deployment in Salesforce is different from java and .net, keep in mind that you already have the "executables", all those XML are already the code that you will change on the environment.
The second point is that on release you are in another agent, Buil and Release pipelines runs have their own lifecycle, so the code existing at the Build pipeline is not available until you send it on "drop" artifact, see Publish Build Artifacts task documentation. So that use copy task to put build.xml on publish folder, then you'll be able to use it on Release pipeline.
When you are executing ant go the /deploy folder and execute your command or check for your ant version using ant -version command.