I have an extension, initially created using the standard yo code template, and successfully uploaded to the market place. I have created a test suite, which works correctly when running locally (i.e. pressing F5), and I now wanted to add CI testing to the Github repo.
I followed the instructions on Continuous Integration and created a config file. The extension now builds successfully, however it appears that no tests are discovered.
For example, in this build I intentionally introduced a failing test, but it still passes.
Is there a step I'm missing or a good way to debug the problem?
See the Issue I opened for the answer. Currently, the tests fail silently if you do not have the required dependencies listed.
Related
We are using Azure Devops 2019 and Yaml Pipeline in order to execute out GUI Testcases. The tests get executed on Microsoft runagent. Recently we have updated our Framework to Microsoft.NET.Test.Sdk style and since then we have a strange problem as below.
Problem :- When we run the tests, tests are actualyy executed on run agent, in Logs one sees that Results were published successfully (done through TCM Test result publisher), But the Tests remain in status "In Progress" no matter whether it was actually passed or failed.
Also we noticed that the Tests which do not have any Data Source are executed rightly and their Result is also shown correctly. ([TestMethod, TestProperty("TestCaseId", "27089"), TestCategory("Smoke"), TestCategory("Selenium"), TestCategory("Appium")])
But unfortunately no Result is shown for the Tests that have Data source.
[TestMethod, TestProperty("TestCaseId", "27080"), TestCategory("Smoke"), TestCategory("Selenium")]
[DataSource("TC27080")]
Our Project has following Dependecies (after the update that we did):
Any idea what could go wrong with these new update. (We could confirm that reverting this commit solves the problem but we would like to know what is exactly wrong with the the update that we did)
Following was the problem :- MSTest nuget Package version 2.2.4 onwards has this problem with Data driven Tests. https://github.com/microsoft/testfx/issues/1023
We downgraded the MSTest package version to 2.2.3 and now the Test results are shown correctly.
I'm currently trying to integrate automatic test execution on the deployment process using Azure Devops but it's causing me an issue where some tests don't run properly.
The application I'm testing has a log-in page and I've set up a couple of tests to see if it opened properly. These ones run smoothly, but the ones that require going through the login process do not.
I've set up some snapshots amidst the process and the result was that I was getting an error on the last step of logging in. I've checked the application logs and the result I got was that no credentials were sent through the request which I found odd since they were displayed on the snapshots.
All the data being inserted was correct, I've checked the endpoints and settings on the agent that was running them and everything seemed fine.
I tried then to run the automated tests through powershell command directly on the same agent and all of them passed with no issue whatsoever.
I've tried changing my task to a powershell command but it still gives me an error.
I have no idea where my issue could be or where I should be even able to start looking for the problem, as clearly it seems that the problem is on the azure devops task but everything seems fine with it, I've tinkered around with the settings a bit but none of them seemed to have any impact on the results I was getting.
I'm using version 2 of the vstest task and am targetting the test assembly that was deployed to the agent. Any idea on what I could be missing or pointers on how to find the solution?
These ones run smoothly, but the ones that require going through the
login process do not
Based on this description, it seems that it will open a new window automatically by the task.
If I'm right, and your test will open a window to login with your private-hosted agent. At this time, the mode of your private-hosted agent must be in interactive which it can allow auto-logon enabled.
Just check this official document for more detail information.
Following this Microsoft tutorial (Run unit tests with your builds), I was expecting to be able to run my unit-tests automatically when a build is triggered, such as a Pull request.
However, when I look in the Pipeline / Builds tab and try to edit my pipeline, there is nothing that allows me to add a new task (see screenshot below).
However, there doesn't appear to be any way of adding a task. I can't even switch to the YMAL without navigating to the source via the Repo. I was hoping to use the GUI though as my YMAL is non-existent.
I have created a test solution with the following structure, which is held in the repo:
Core Solution
|_ Class Library Project (.NET Core)
|_ MSTest Test Project (.NET Core)
I was hoping to have a build step followed by a Unit Test step using the tests in my MSTest Test Project (.NET Core) project once they were built.
There appears to be a Tasks option in the Releases tab by the way, but I was expecting to be able to add tasks for builds as well, especially Unit Tests.
Being new at this, perhaps I have missed or misunderstood something. I would be grateful for any help and to be pointed in the right direction.
It seems that there is an obscure link that takes you through to the correct process, found it quite by accident, see screenshot below:
It turns out that there is a little link titled Use the visual designer that I'd missed. Seems a little odd that most of the tutorials discuss this process and yet it's partially obscured. I guess that this shows that the platform is a work in process and still being added to and improved.
Another few caveats for those descending this little rabbit hole, after selecting the Use the visual designer link, be sure to select the Empty pipeline template, or whatever is appropriate to your project/solution type, and not the YAML option at the top of the list, otherwise you'll be back where you started.
Finally, it seems that automated Unit Tests don't work on Agents other than Hosted VS2017 Agents (that said, I haven't tried the Hosted option). This is set in the very next screen by selecting the Pipeline and selecting the Agent pool from the drop-down.
One last thing... The pipeline won't run automatically unless you check the Enable continuous integration checkbox on the Triggers tab.
Once this is all done, I simply chose the tasks needed for the build by clicking the plus symbol on the Agent job 1 item.
Good luck
Kaine
You have created a new YAML style of build configuration. Currently Azure DevOps does not support of having YAML & GUI editor at the same time (this is upcoming feature in Q1/2019).
To get a GUI editor, create new build pipeline.
In this selection click "Use the visual designer".
Then at template phase, don't select YAML. Any other template will do. You can easily delete all the build steps after selecting template, so don't be afraid to choose any.
I'm working on DevSecOps. Related to that, I want to secure my application with the help of Roslyn Analyzer. For that I created an MVC Web Application along with custom ruleset in Visual Studio and it's working fine with outputs clearly.
But, when I tried to do same thing with the help of VSTS Build task in my Build Definition in VSTS account. By referring the link, I known that with the help of MSBuild task we can get the Roslyn Analyser results by default. But I didn’t get the results of Roslyn Analyzer results after running my Build Definition. Can you suggest me to “How to overcome this situation”?
If that works locally when run msbuild in command line, then you can try below items to narrow down the issue:
Add a command line task to run the msbuild command line script
directly.
Change another hosted agent.
Deploy an private agent on your dev machine, then check if that works
with the private agent.
You can also reference below articles to troubleshoot the issue:
Using, configuring and distributing Roslyn analysers in teams
Running Code Analyzers on Build Server
I have an automation script that uses maven POM.xml to import all the dependencies needed from selenium and junit. The main test uses selenium to open a browser, verify some information, close the browser and the test ends.
When run as Junit it works fine: run as Junit test
When run as Maven Test it works fine as well: run as maven test
In both scenarios, the program opens the browser and navigates through the website as it should do for an automated test.
Now I need to integrate it to VSTS so I can visualize the overall pass/fail test on the VSTS dashboard but I'm not familiarized with this tool too much yet.
So far this is what I have managed to do:
Deploy an agent on my WindowsPC (I want to execute and deploy the project on an Azure VM or another azure instance later on) NOTE: this is the same pc I'm successfully running the program using eclipse as shown in the screenshoots above. https://learn.microsoft.com/en-us/vsts/build-release/actions/agents/v2-windows?view=vsts
Create a build definition on VSTS but when I queue the definition the build fails: build definition and the build fail.
I don't know why it can't find mt config.txt file since it is located on the same hosted agent in that same directory. I'll appreciate if someone is capable of guiding me through this process so I can run the program from the VSTS and visualize the overall tests that fail and pass on the VSTS dashboard.
UPDATE: I moved the config.txt file to the public directory and the build was successful(I still need to fix this issue because I do not want my work in a public folder).
Now the problem I have is that even though the build is successful and it looks like it is running my "3 tests", When I look at my pc, nothing is happening. it should open chrome and take a screenshot, then open Firefox and take another screenshot and finally open internet explorer and take another screenshot and save each test on different folders but it is only generating folders for chrome and internet explorer (but still those folders does not have the screenshot I'm asking, maybe because the browser is not being open on the computer.)
Here is the log: https://drive.google.com/open?id=1S_MhAUmzj8i9phPQiqS06s0_1cCRrbF0
test output report generated on my computer
test output on vsts
Look at the error message. The error message tells you precisely what the problem is: java.io.FileNotFoundException: Y:\Automation Team\CopaQA\Architecture\local\config.txt (The system cannot find the path specified)
You need to not rely on hard-coded paths.
You say you registered a build agent against your VSTS account... but did you change the agent queue for your build? If the agent queue is "Hosted", you're using Microsoft's hosted agent.
I don't know why it can't find mt config.txt file since it is located on the same hosted agent in that same directory.
It turns out that Java.IO. can't read files located on a shared network drive, I solved this by using the UNC path to that file (//"computername"/"directory"/"file.txt")
Now the problem I have is that even though the build is successful and
it looks like it is running my "3 tests", When I look at my pc,
nothing is happening.
It took me a little reading to realize that to perform UI tests my agent needs to be set up in INTERACTIVE MODE. it can be done following this guide: https://learn.microsoft.com/en-us/vsts/build-release/actions/agents/v2-windows?view=vsts