I am doing some initial one off setup using [BeforeTestRun] hook for my specflow tests. This does check on some users to make sure if they exist and creates them with specific roles and permissions if they are not so the automated tests can use them. The function to do this prints a lot of useful information on the Console.Writeline.
When I run the test on my local system I can see the output from this hook function on the main feature file and the output of each scenario under each of them. But when I run the tests via Azure DevOps pipleine, I am not sure where to find the output for the [BeforeTestRun] because it is not bound a particular test scenario. The console of Run Tests Tasks has no information about this.
Can someone please help me to show this output somewhere so I can act accordingly.
I tried to use System.Diagnostics.Debug.Print, System.Diagnostics.Debug.Print, System.Diagnostics.Debug.WriteLine and System.Diagnostics.Trace.WriteLine, but nothing seems to work on pipeline console.
[BeforeTestRun]
public static void BeforeRun()
{
Console.WriteLine(
"Before Test run analyzing the users and their needed properties for performing automation run");
}
I want my output to be visible somewhere so I can act based on that information if needed to.
It's not possible for the console logs.
The product currently does not support printing console logs for passing tests and we do not currently have plans to support this in the near future.
(Source: https://developercommunity.visualstudio.com/content/problem/631082/printing-the-console-output-in-the-azure-devops-te.html)
However, there's another way:
Your build will have an attachment with the file extension .trx. This is a xml file and contains an Output element for each test (see also https://stackoverflow.com/a/55452011):
<TestRun id="[omitted]" name="[omitted] 2020-01-10 17:59:35" runUser="[omitted]" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
<Times creation="2020-01-10T17:59:35.8919298+01:00" queuing="2020-01-10T17:59:35.8919298+01:00" start="2020-01-10T17:59:26.5626373+01:00" finish="2020-01-10T17:59:35.9209479+01:00" />
<Results>
<UnitTestResult testName="TestMethod1">
<Output>
<StdOut>Test</StdOut>
</Output>
</UnitTestResult>
</Results>
</TestRun>
Related
I am running jmeter from Taurus and I need a output kpi.jtl file with url listing.
I have tried passing parameter -o modules.jmeter.properties.save.saveservice.url='true' and
-o modules.jmeter.properties="{'jmeter.save.saveservice.url':'true'}". Pipeline is running successfully but the kpi.jtl doesn't have the url. Please help
I have tried few more options like editing jmeter.properties via pipeline - which broke the pipeline and expecting input from user
user.properities- Which is ineffective.
I am expecting kpi.jtl file with all the possible logs especially url.
I believe you're using the wrong property, you should pass the next one:
modules.jmeter.csv-jtl-flags.url=true
More information: CSV file content configuration
However be informed that having a.jtl file "with all possible logs" is some form of a performance anti-pattern as it creates massive disk IO and may ruin your test. More information: 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure
This is my first question posted on stackoverflow, hence in case I did something incorrectly pleaselet me know.
Description
I am currently working on translation of freestyle projects to declarative pipelines in Jenkins (jenkinsfiles kept in Git repo). The original freestyle job was triggering PowerShell script which needed access to Global name/password pairs defined in Mask Passwords plugin section in Configure System. Solution to this problem was an additional tick in project itself (unfortunately I am not allowed adding screenshots to posts yet, hence editor uploaded screen to imgur and pasted link - please see Screenshot 1):
Screenshot 1
Therefore I started looking for possible implementation of such solution to jenkinsfile, however wothout luck.
My problem
When the script is triggered from the pipeline, it errors out stating that it cannot find relevant passwords (powershell refers to those credentials as to environment variables). This works fine when ran from freestyle project.
Which I reckon is caused by pipeline not being able to reach out to previously mentioned credentials.
What I tried
Wrapping the step into below block:
wrap([$class: 'MaskPasswordsBuildWrapper']) {
bat(batch file launching ps script)
}
Then the above block containing relevant step wrapping into
script {
wrap(...)
}
But none of them worked.
I have taken a look at other plugins like Credentials Binding Plugin or Credentials Plugin but those allow to bind/pass one credential per step, and I need to pass all credentials specified in Jenkins (I am open to move saved credentials to any other location within Jenkins).
I have looked at adding environment variable:
credentials('Credentials-ID')
But the problem is the same as with mentioned plugins.
By any chance, have anyone came across similar situation and know what can be done in order to allow pipeline to access/pass to pipeline all the credentials specified in Jenkins instead of binding/passing them one a time?
All tips are very welcome!
You can do this and the env variable will then be available throughout your job. You could define multiple env variable too.
environment {
// Use credentials() to hide the environment variable's output
MY_PERSONAL_TOKEN = credentials('Credentials-ID')
}
stages {
stage('Test Stage') {
steps {
script {
// do what you need to
}
}
}
}
If you want to configure your VS "Load Tests" to write the results to a database server, you use the following instructions.
If you want to run your "Load Tests" through powershell on a separate machine(think TFS 2018 release step), you use the following instructions.
I would like to do both, on multiple machines, in a automated manner, but there's not a great deal of documentation on this, I can run my tests like this:
.\mstest /testcontainer:"C:\XXX\ABC.loadtest"
But the results are kicked out to a "TRX" file rather than being placed into a database(there is some discussion on this). How do I put the results into a external database like when I run it locally(per instructions above)?
Note: #AdrianHHH points out that the "TRX" file is only a summary and that most of the info is stored locally(MDF/LDF file) in the user folder of current user running the load tests.
Update 1
Hmm I wonder where this is persisted:
(Curious, also click on the "?" icon in the "Manage Test Controller" box, nothing...)
It's not in the saved XML:
<RunConfigurations>
<RunConfiguration Name="Run Settings1" Description="" ResultsStoreType="Database" TimingDetailsStorage="AllIndividualDetails" SaveTestLogsOnError="true" SaveTestLogsFrequency="0" MaxErrorDetails="200" MaxErrorsPerType="1000" MaxThresholdViolations="1000" MaxRequestUrlsReported="1000" UseTestIterations="false" RunDuration="10" WarmupTime="0" CoolDownTime="0" TestIterations="100" WebTestConnectionModel="ConnectionPerUser" WebTestConnectionPoolSize="50" SampleRate="5" ValidationLevel="High" SqlTracingConnectString="" SqlTracingConnectStringDisplayValue="" SqlTracingDirectory="" SqlTracingEnabled="false" SqlTracingFileCount="2" SqlTracingRolloverEnabled="true" SqlTracingMinimumDuration="500" RunUnitTestsInAppDomain="true" CoreCount="0" ResourcesRetentionTimeInMinutes="0" AgentDiagnosticsLevel="Warning">
<CounterSetMappings>
<CounterSetMapping ComputerName="[CONTROLLER MACHINE]">
<CounterSetReferences>
<CounterSetReference CounterSetName="LoadTest" />
<CounterSetReference CounterSetName="Controller" />
</CounterSetReferences>
</CounterSetMapping>
<CounterSetMapping ComputerName="[AGENT MACHINES]">
<CounterSetReferences>
<CounterSetReference CounterSetName="Agent" />
</CounterSetReferences>
</CounterSetMapping>
</CounterSetMappings>
<LoadGeneratorLocations>
<GeoLocation Location="Default" Percentage="100" />
</LoadGeneratorLocations>
</RunConfiguration>
</RunConfigurations>
They're not persisted in my default "testsettings" file either:
<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="Local" id="02cad612-043b-447d-993e-a9b9b0547c9d"
xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
<Description>These are default test settings for a local test run.</Description>
<Deployment enabled="false" />
<Execution hostProcessPlatform="MSIL">
<TestTypeSpecific />
<AgentRule name="Execution Agents">
</AgentRule>
</Execution>
<Properties>
<Property name="TestSettingsUIType" value="UnitTest" />
</Properties>
</TestSettings>
So I need to find where ever this configuration information is being persisted, then maybe I can find a way to feed it to MSTest. Does anyone else understand how this works?
Update 2
My TRX file does contains a "connection string" but I don't think it's to my database, my database is empty, on running via powershell it completes, but all I see is the "TRX" file.
Update 3
This one is tricky, I keep trying various ways to determine where this "Manage Test Configuration" data/credentials is being stored. One of the ways I did this was to use Microsoft's Process Monitor. You can actually see where it initially is being populated from:
It's from a Application Hive, of course that's begs the question where did the "Application Hive" get populated from, that's where things get a bit murky, there's allot of different calls to many files. A common trend is that the "Temp\Local" folder is often referenced.
I deleted the entire "Temp" folder for my user account(in the process losing all my VS configuration) and upon reopening my solution it appears as though this had an effect. When I pull up my "LoadTest" file, the "Load test results store" line is now empty. In fact the entire "Manage Test Controller" window has been restored back to default(empty).
I know believe that the configuration for this "Manage Test Controller" window is persisted in the temp folder. However, I've yet to locate where it is and/or how to change/automatically populate that information with a powershell script.
Finally figured this out. Basically I used several tools to check what files were being modified when I changed the connection string, the results made it obvious:
privateregistry.bin
Once I found this it was pretty obvious that VS was maintaining it's own little registry hive. It's clearly stated in this post, so I opened it in the way described in the article and found the connection string:
This indicated that:
"The SQL Connection String is NOT stored in the loadtest files. The
setting seems to be PC specific so I had to change it on the build
server - in one loadtest file (address.loadtest) as shown, then all
the other loadtests adopt the same connection string."
So that's basically what I did, I logged into each build server and configured them so that they write all there results to my database rather than locally.
Load tests are clearly not designed to make this process easy, I don't think many people have attempted to do what I've done. All the articles just tell you to use their cloud service. I'm pretty sure that only covers web tests. If your using load testing to test unit tests you pretty much out of luck(without this work around). I really hope this gets official support in the future, it would be really nice to both run/view all types of load tests from TFS. For now though I'm going to have to keep using this work around.
In NetSuite, have a Restlet script that calls a deployed map/reduce script but the map stage shows as Failed when looking at details of status page (the getInputData stage does run and shows as Complete).
However, if I do a "Save and Execute" from the deployment of the map/reduce script, it works fine (Map stage does run).
Note that:
There is no error on execution log of either the restlet or the map/reduce scripts.
Have 'N/task' in define section of restlet as well as task in function parameters.
The map/reduce script has the Map checkbox checked. The map/reduce script deployment is not scheduled and has default values for other fields.
Using NetSuite help "See the quick brown fox jump" sample map/reduce script
Using Sandbox account
Using a custom role to post to restlet
Below is call to create.task code snippet from my Restlet. Don't know what is wrong, any help is appreciated.
var mrTask = task.create({
taskType: task.TaskType.MAP_REDUCE,
scriptId: 'customscript_test',
deploymentId: 'customdeploy_test'
});
var mrTaskId = mrTask.submit();
var taskStatus = task.checkStatus(mrTaskId);
log.debug('taskStatus', taskStatus);
You also need Documents and Files -View permission along with SuiteScript - View and SuiteScript Scheduling - Full permissions to access the Map/Reduce script.
The help for MapReduceScriptTask.submit() does not mention this but the help for ScheduledScriptTask.submit() does:
Only administrators can run scheduled scripts. If a user event script calls ScheduledScriptTask.submit(), the user event script has to be deployed with admin permissions.
I did a test of posting to my restlet using Administrator role credentials and it works fine as opposed to using my other custom role. Maybe just like ScheduledScriptTask, the MapReduceScriptTask can only be called by Administrator role ? My custom role does have SuiteScript - View and SuiteScript Scheduling - Full permissions. Thought that would do the trick but apparently not.
Anyone have any further insight on this ?
I have a feature that has prod tests to be run on prod server. Its tagged #prod. I have other non-tagged tests that can run in test envs.
In env.rb I have a Before do .. end that opens a new browser and sets up test to run in test env. I also have and a Before('#prod') block that sets up tests helpers to run in prod account.
When I run just the prod test, using cucumber --tags #prod, it runs prod tests. I expect only the Before('#prod') to run. However I see that both the BEfore blocks in env.rb file are run, creating 2 new browser instances and prod test running in second browser window. When my suite of 7 scenarios are done I have 7 blank open browser windows which my test util setup from non-tagged Before.
Given any valid test merchant account # features/step_definitions/ConsumerPortal.rb:1
We are in plain Before
We are in #prod Before
When login to pos portal as 'pos' user
I see the same happening for After tag to logout and close browser as well. Tagged one is run first and then the plain After tag. If I can read what tag a test has in my env.rb Before, I can setup accordingly. Is tehre a way to do this?
Use negation tags with your before tags.
#Before("~#prod")
#After("~#prod")
This will prevent the Before and After tags from running for all #prod tags.
Or else in general, all #Before and #After methods run for any tags.