Is it possible to instruct BuildBot to execute build steps in parallel?
I've been looking through documentation and it only seems to be possible by actually generating multiple builds / build factories.
I'm not entirely sure about Builders and Workers: I have seen that adding workers will allow me to run multiple build requests simultaneously (multiple programmers submitting PRs), but using multiple builders doesn't seem to be intended for anything like this.
So, is it possible?
You can have multiple builders executing simultaneously, for example if they listen to incoming commits on the same repository; a single commit will start all listening builders. In this scenario you can control parallelism using BuilderConfig's canStartBuild argument. And beware to have the builders work on separate resources!
Alternatively if you trigger multiple builders from a single builder and
specify waitForFinish=False, the triggered builders will run simultaneoulsy.
I believe you cannot execute build steps in parallel within a single builder. Regarding the workers, I can't tell you.
Related
I am looking into Dagger, the CICD kit.
I understand that in the Dagger pipeline, when multiple actions are executed in succession and have dependencies on each other's actions, they can be executed in succession by setting the actions that depend on the inputs and outputs in the actions.
This can be understood from the samples and explanations on the official website.
https://docs.dagger.io/1221/action/#composite-actions
So, if I want to execute an action that does not depend on an action continuously, how can I set it up?
Thanks.
I’m migrating a pipeline from Circle CI to Github Actions and am finding it a bit weird that I can only run jobs once instead of creating a job, then calling it from the workflow section, making it possible to call a job multiple times without duplicating the commands/scripts in that job.
My pipeline pushes out code to three environments, then runs a lighthouse scan for each of them. In circle ci I have 1 job to push the code to my envs and 1 job to run lighthouse. Then from my workflow section, I just call the jobs 3 times, passing the env as a parameter. Am I missing something or is there no way to do this in github actions? Do I just have to write out my commands 3 times in each job?
There are 3 main approaches for code reusing in GitHub Actions:
Reusing workflows
The obvious option is using the "Reusable workflows" feature that allows you to extract some steps into a separate "reusable" workflow and call this workflow as a job in other workflows.
Takeaways:
Reusable workflows can't call other reusable workflows.
The strategy property is not supported in any job that calls a reusable workflow.
Env variables and secrets are not inherited.
It's not convenient if you need to extract and reuse several steps inside one job.
Since it runs as a separate job, you have to use build artifacts to share files between a reusable workflow and your main workflow.
You can call a reusable workflow in synchronous or asynchronous manner (managing it by jobs ordering using needs keys).
A reusable workflow can define outputs that extract outputs/outcomes from executed steps. They can be easily used to pass data to the "main" workflow.
Dispatched workflows
Another possibility that GitHub gives us is workflow_dispatch event that can trigger a workflow run. Simply put, you can trigger a workflow manually or through GitHub API and provide its inputs.
There are actions available on the Marketplace which allow you to trigger a "dispatched" workflow as a step of "main" workflow.
Some of them also allow doing it in a synchronous manner (wait until dispatched workflow is finished). It is worth to say that this feature is implemented by polling statuses of repo workflows which is not very reliable, especially in a concurrent environment. Also, it is bounded by GitHub API usage limits and therefore has a delay in finding out a status of dispatched workflow.
Takeaways
You can have multiple nested calls, triggering a workflow from another triggered workflow. If done careless, can lead to an infinite loop.
You need a special token with "workflows" permission; your usual secrets.GITHUB_TOKEN doesn't allow you to dispatch a workflow.
You can trigger multiple dispatched workflows inside one job.
There is no easy way to get some data back from dispatched workflows to the main one.
Works better in "fire and forget" scenario. Waiting for a finish of dispatched workflow has some limitations.
You can observe dispatched workflows runs and cancel them manually.
Composite Actions
In this approach we extract steps to a distinct composite action, that can be located in the same or separate repository.
From your "main" workflow it looks like a usual action (a single step), but internally it consists of multiple steps each of which can call own actions.
Takeaways:
Supports nesting: each step of a composite action can use another composite action.
Bad visualisation of internal steps run: in the "main" workflow it's displayed as a usual step run. In raw logs you can find details of internal steps execution, but it doesn't look very friendly.
Shares environment variables with a parent job, but doesn't share secrets, which should be passed explicitly via inputs.
Supports inputs and outputs. Outputs are prepared from outputs/outcomes of internal steps and can be easily used to pass data from composite action to the "main" workflow.
A composite action runs inside the job of the "main" workflow. Since they share a common file system, there is no need to use build artifacts to transfer files from the composite action to the "main" workflow.
You can't use continue-on-error option inside a composite action.
Source: my "DRY: reusing code in GitHub Actions" article
I'm currently in the exact same boat and just found an answer. You're looking for a Composite Action, as suggested in this answer.
Reusable workflows can't call other reusable workflows.
Actually, they can, since Aug. 2022:
GitHub Actions: Improvements to reusable workflows
Reusable workflows can now be called from a matrix and other reusable workflows.
You can now nest up to 4 levels of reusable workflows giving you greater flexibility and better code reuse.
Calling a reusable workflow from a matrix allows you to create richer parameterized builds and deployments.
Learn more about nesting reusable workflows.
Learn more about using reusable workflows with the matrix strategy.
Is there any way to write a Protractor test in such a way that would allow simulation of many users at once? E.g. simulate 100 users all making a checkout at the same time, or 100 users all logging in at the same time. The purpose being to detect any possible race conditions or locking issues.
Protractor appears to be designed so that everything runs in sync, even across multiple browser instances (i.e. forks). Is there any way to accomplish what I'm doing in Protractor, or am I out of luck?
Note that I'm not referring to running tests concurrently (be that running tests under both Firefox and Chrome or spreading describes over multiple instances to speed it up), rather the ability to spawn new "threads" inside one test case to execute commands in parallel.
I have looked at the documentation on both, but am not sure what's the best choice for a given application. I have looked closer at celery, so the example will be given in those terms.
My use case is similar to this question, with each worker loading a large file remotely (one file per machine), however I also need workers to contain persistent objects. So, if a worker completes a task and returns a result, then is called again, I need to use a previously created variable for the new task.
Repeating the object creation at each task call is far too wasteful. I haven't seen a celery example to lead me to believe this is possible, I was hoping to use the worker_init signal to accomplish this.
Finally, I need a central hub to keep track of what all the workers are doing. This seems to imply a client-server architecture rather than the one provided by Celery, is this correct? If so, would IPython Parallel be a good choice given the requirements?
I'm currently evaluating Celery vs IPython parallel as well. Regarding a central hub to keep track of what the workers are doing, have you checked out the Celery Flower project here? It provides a webpage that allows you to view the status of all tasks in the queue.
Well, StackOverFlow is such a good site, most of my google search results direct to here, really.
I've seen too many posts about selenium grid2 inside and outside this place, they all explained that, grid2 has such capability to run tests in parallel, how to set up grid hub and nodes. But, no one told me how to run tests through selenium grid2, all I got was "set up hub and nodes, then run tests, then all things become parallel". But how to trigger the running through selenium grid2?
Then, I got answers myself, that is, to trigger the running with another runner, e.g., NUnit. However, NUnit can only run tests serially, not parallelly. I've also tried other runners but they can't function so well along with grid2.
So I started to doubt, whether selenium grid2 really has such capability to run tests in parallel on its own? If so, how to? What is the whole workflow?
If not, then a third-party tool is needed to trigger the running, what's more, the third-party tool must be able to trigger multiple tests at one time(multi-thread, something like that?), so that grid2 can deliver those tests to its nodes to run them at the same time. In this way, can we call it a "parallel running".
What third-party tool would be a good choice? NAnt? Jenkins?
I have a long story coping with grid2 these days, these statements above are just part of it. If you can come up with anything, please tell me, that would be really appreciated.
我对自己的英文还是有信心的，在此多谢各位的帮忙了！谢谢！
Selenium Grid 2 is capable of executing tests in parallel "provided you pass multiple commands simultaneously to the hub". You need to use a different framework like NUnit, testNG to run multiple test cases simultaneously. I use testNG for triggering multiple tests in parallel. It works absolutely fine without any issues. You can find some help on getting started here
WebDriver driver = new RemoteWebDriver(new URL("http://localhost:4444/wd/hub"), capability);
as descripted here:
http://code.google.com/p/selenium/wiki/Grid2
Tests are passed to a node which executes.