I must be missing something obvious. I've got a couple of jobrunr jobs where i'm using the lambda enqueue format version 5.1.6. Like this:
JobId jobId = BackgroundJob.<MyService>enqueue(x -> x.doWork());
I would like to validate the plumbing and work in the jobs is executing via some integration tests with Spring, but don't see the options to run now, eager mode, etc? Thanks
You can't, I'm afraid.
You can mock the JobScheduler and capture the args. JobRunr itself is also tested very well so if you pass a job, you can rest assured it will be enqueued.
You could also put the pollIntervalInSeconds to 5 and use awaitility then to verify your job executed. There are many examples of this in the JobRunr repo.
Related
I am doing some initial one off setup using [BeforeTestRun] hook for my specflow tests. This does check on some users to make sure if they exist and creates them with specific roles and permissions if they are not so the automated tests can use them. The function to do this prints a lot of useful information on the Console.Writeline.
When I run the test on my local system I can see the output from this hook function on the main feature file and the output of each scenario under each of them. But when I run the tests via Azure DevOps pipleine, I am not sure where to find the output for the [BeforeTestRun] because it is not bound a particular test scenario. The console of Run Tests Tasks has no information about this.
Can someone please help me to show this output somewhere so I can act accordingly.
I tried to use System.Diagnostics.Debug.Print, System.Diagnostics.Debug.Print, System.Diagnostics.Debug.WriteLine and System.Diagnostics.Trace.WriteLine, but nothing seems to work on pipeline console.
[BeforeTestRun]
public static void BeforeRun()
{
Console.WriteLine(
"Before Test run analyzing the users and their needed properties for performing automation run");
}
I want my output to be visible somewhere so I can act based on that information if needed to.
It's not possible for the console logs.
The product currently does not support printing console logs for passing tests and we do not currently have plans to support this in the near future.
(Source: https://developercommunity.visualstudio.com/content/problem/631082/printing-the-console-output-in-the-azure-devops-te.html)
However, there's another way:
Your build will have an attachment with the file extension .trx. This is a xml file and contains an Output element for each test (see also https://stackoverflow.com/a/55452011):
<TestRun id="[omitted]" name="[omitted] 2020-01-10 17:59:35" runUser="[omitted]" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
<Times creation="2020-01-10T17:59:35.8919298+01:00" queuing="2020-01-10T17:59:35.8919298+01:00" start="2020-01-10T17:59:26.5626373+01:00" finish="2020-01-10T17:59:35.9209479+01:00" />
<Results>
<UnitTestResult testName="TestMethod1">
<Output>
<StdOut>Test</StdOut>
</Output>
</UnitTestResult>
</Results>
</TestRun>
I am trying to run a single set of a single set of tests within the e2e Kubernetes tests. I am quite confused as to how the tests are organized, is there a comprehensive list of all the tests?
Thanks!
Assuming, the tests are placed at ./tests/e2e path in the repository.
If the tests are written in go, they are mostly written using a standard testing library or ginkgo framework.
For running the tests written using standard testing package
Add the tags to a specific test at the start of your test file like,
// +build <my-test>
Run the tests by specifying the tag name, go test -v ./tests/e2e -tags <my-test>
For running the tests written using ginkgo
go test -ginkgo.dryRun ./tests/e2e/... to list all the tests in the package.
go test -ginkgo.focus "<regex>" ./tests/e2e/... to run specific tests mentioned in the focus regex field.
go test -ginkgo.skip "<regex>" ./tests/e2e/... to skip the specific tests mentioned in the regex field
If you have e2e.test binary, you can list all available test by setting following flag: ./e2e.test --ginkgo.DryRun. Then if you want a single test, type: ./e2e.test --ginkgo.Focus="<name of your test>", pay attentention that all special characters in the test name must be escaped. For example, if you want run only conformance tests: --ginkgo.Focus="\[Conformnce\]".
Just in case, the right way of running particularly focused e2e tests is described officially here: https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md
it would be something like this:
go run hack/e2e.go -- --test --test_args="--ginkgo.focus=${matching regex}"
I am trying to run Karma via node (gulp, specifically) persistently in the background but to manually re-run the tests i.e. I have autoWatch set to false. I start the server with something like:
karma_server = new karma.Server(config.karma)
karma_server.start()
Then elsewhere I would like to trigger the tests running when files are updated outside Karma. The API method that one would expect might work is server.refreshFiles(), but it does not do this.
Internally it seems like executor.schedule() might do the trick, but it seems to be undocumented, private, and inaccessible.
So when autoWatch is turned off, how does one trigger Karma testing with an existing server? I'm sure I must be missing something obvious as otherwise the autoWatch option would always need to be true for the server to be useful.
If you have a server already running you can use the karma runner to communicate with it:
var runner = require('karma').runner,
karmaConfig = {/* The karma config here */};
runner.run(karmaConfig, callback);
The grunt-karma plugin works like this, you can check it out for more info:
https://github.com/karma-runner/grunt-karma/blob/master/tasks/grunt-karma.js
I have been of late trying out apache spark. My question is more specific to trigger spark jobs. Here I had posted question on understanding spark jobs. After getting dirty on jobs I moved on to my requirement.
I have a REST end point where I expose API to trigger Jobs, I have used Spring4.0 for Rest Implementation. Now going ahead I thought of implementing Jobs as Service in Spring where I would submit Job programmatically, meaning when the endpoint is triggered, with given parameters I would trigger the job.
I have now few design options.
Similar to the below written job, I need to maintain several Jobs called by a Abstract Class may be JobScheduler .
/*Can this Code be abstracted from the application and written as
as a seperate job. Because my understanding is that the
Application code itself has to have the addJars embedded
which internally sparkContext takes care.*/
SparkConf sparkConf = new SparkConf().setAppName("MyApp").setJars(
new String[] { "/path/to/jar/submit/cluster" })
.setMaster("/url/of/master/node");
sparkConf.setSparkHome("/path/to/spark/");
sparkConf.set("spark.scheduler.mode", "FAIR");
JavaSparkContext sc = new JavaSparkContext(sparkConf);
sc.setLocalProperty("spark.scheduler.pool", "test");
// Application with Algorithm , transformations
extending above point have multiple versions of jobs handled by service.
Or else use an Spark Job Server to do this.
Firstly, I would like to know what is the best solution in this case, execution wise and also scaling wise.
Note : I am using a standalone cluster from spark.
kindly help.
It turns out Spark has a hidden REST API to submit a job, check status and kill.
Check out full example here: http://arturmkrtchyan.com/apache-spark-hidden-rest-api
Just use the Spark JobServer
https://github.com/spark-jobserver/spark-jobserver
There are a lot of things to consider with making a service, and the Spark JobServer has most of them covered already. If you find things that aren't good enough, it should be easy to make a request and add code to their system rather than reinventing it from scratch
Livy is an open source REST interface for interacting with Apache Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in Apache Hadoop YARN.
Here is a good client that you might find helpful: https://github.com/ywilkof/spark-jobs-rest-client
Edit: this answer was given in 2015. There are options like Livy available now.
Even I had this requirement I could do it using Livy Server, as one of the contributor Josemy mentioned. Following are the steps I took, hope it helps somebody:
Download livy zip from https://livy.apache.org/download/
Follow instructions: https://livy.apache.org/get-started/
Upload the zip to a client.
Unzip the file
Check for the following two parameters if doesn't exists, create with right path
export SPARK_HOME=/opt/spark
export HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop
Enable 8998 port on the client
Update $LIVY_HOME/conf/livy.conf with master details any other stuff needed
Note: Template are there in $LIVY_HOME/conf
Eg. livy.file.local-dir-whitelist = /home/folder-where-the-jar-will-be-kept/
Run the server
$LIVY_HOME/bin/livy-server start
Stop the server
$LIVY_HOME/bin/livy-server stop
UI: <client-ip>:8998/ui/
Submitting job:POST : http://<your client ip goes here>:8998/batches
{
"className" : "<ur class name will come here with package name>",
"file" : "your jar location",
"args" : ["arg1", "arg2", "arg3" ]
}
I want to run spring xd with Oracle(11g) which i already have in my environment. Currently my first concern is the jobs UI (my database has existing data of job executions that were performed by spring-batch and i simply want to display the details of those executions).
i'm using spring-xd-1.0.0.M5. I followed the instructions in the reference guide and i changed application.yml to have the following:
spring:
datasource:
url: jdbc:oracle:oci:MY_USERNAME/MYPWD#//orarmydomain.com:1521/myservice
username: MY_USERNAME
password: MYPWD
driverClassName: oracle.jdbc.OracleDriver
profiles:
active: default,oracle
i modified also batch-jdbc.properties to have the database configuration similar to the above.
Yet, when i start xd-singlnode.bat (or either xd-admin.bat) it seems like it ignores my oracle configuration and still uses the default hsqldb.
what am i doing wrong?
Thanks
The likely reason is that we did not upgrade the windows .bat scripts to take advantage of the property overriding via xd-config.yml. If you go into the unix script for xd-singlenode you will see that when java is invoked there there is an option
-Dspring.config.location=$XD_CONFIG
you can for now hardcode your location of that file, use file: as the prefix.
Also, The UI right now is very primitive, you will not be able to see many details about the job execution. There are however many job related commands you can execute in the shell and there is only one gap regarding step execution information as compared to what is available via spring-batch-admin.
The issue to watch for this is https://jira.springsource.org/browse/XD-1209 and it is schedule for the next milestone release.
Let me know how it goes, thanks!
Cheers,
Mark