I've setup a Cadence cron workflow to execute on a specific day of each month. After registering the workflow with:
cadence --address localhost:7933 \
--domain domain-name workflow start --tl TaskListName \
--wt WorkflowTypeName --dt 60 --et 60 \
--wid Workflow.Cron --cron '0 23 23 * *'
I can see it appearing in the Cadence web UI with the Open status.
I also see the workflow info when I use the Cadence CLI with:
cadence --address localhost:8933 --domain domain-name list --workflow_id Workflow.Cron --open
However, after about 24 hours, it disappears from the Workflow list view of Cadence UI. I can still access the workflow's execution details page by using its run_id. On that page, I can see the workflow is still active and waiting for the moment defined by the cron expression to be reached.
Also, at this point, if I use the Cadence CLI with the command I mentioned earlier, it doesn't list the workflow anymore.
If I use the show command instead:
cadence --address localhost:8933 --domain domain-name show --workflow_id Workflow.Cron
I get the following:
1 WorkflowExecutionStarted {WorkflowType:{Name:WorkflowTypeName},
TaskList:{Name:TaskListName},
Input:[], ExecutionStartToCloseTimeoutSeconds:60,
TaskStartToCloseTimeoutSeconds:60,
ContinuedFailureDetails:[], LastCompletionResult:[],
OriginalExecutionRunID:9bdc7e76-1b93-47eb-b566-81d4f115a950,
Identity:cadence-cli#stenix,
FirstExecutionRunID:9bdc7e76-1b93-47eb-b566-81d4f115a950,
Attempt:0, CronSchedule:0 23 23 * *,
FirstDecisionTaskBackoffSeconds:95387}
History Source: Default Storage
The cron is set to execute a few hours later today, so I've yet to know whether the workflow will continue with its activities as expected or not. But I am wondering why the workflow is disappearing from the Cadence UI and from the CLI list command after ~24 hours?
I've also tried providing an --earliest_time to the CLI list command.
EDIT: The workflow carried on with its activities at the scheduled time. Also, it reappeared in the workflow list after/during execution. However, new cron workflows are still disappearing from the list ~24h after their creation.
Related
What could be the reason for the decision tasks to not get picked for execution in cadence cluster. They remains at pending state and finally times out. I dont see any error logs. How do I debug this ?
It’s very likely that there is no worker available and actively polling tasks for the tasklist.
Best way to confirm is to click on the tasklist naming in the webUI and see what are the workers behind the tasklist. Since it’s decision task, you should check the decision handler for the tasklist.
You can also use CLI to describe the tasklist to give the same information:
cadence tasklist desc —-tl <tasklist name>
In some extremely rare cases(I personally never seen but heard that happened in Uber with large scale cluster) that cadence server lost the task. In that case you can use CLI to either regenerate the task, or reset the workflow to unblock the workflow:
To regenerate task:
cadence adm wf refresh-tasks -w <wf id>
To reset:
cadence wf reset —-reset_type LastDecisionCompleted -w <wf id>
I am currently running Snakemake on my department's cluster (SGE). For this I have used a template given by a workshop to run and submit jobs to the scheduler and run my scripts within the different rules. The template for the profile is taken from this snakemake-gridengine repository.
However, I am running into an issue where Snakemake is not submitting the max number of jobs it should be able to the cluster at once.
snakemake --snakefile pop_split_imputation_pipeline.smk \
-j 1000 --max-status-checks-per-second 0.01 \
--profile ~/snakemake/profile -f --rerun-incomplete --use-conda
For instance, above is an example of a command used to submit a .smk pipeline to be run, which in theory should generate 1000 jobs per rule. However, within my cluster, only 10-50 jobs at any one time are being submitted. Within my config.yaml I already have set max-jobs-per-second: 1000, so clearly it should be able to submit all these jobs at once yet it doesn't.
Can anyone point to something to improve the submission speed of these jobs?
I already have written a PowerShell Script, which starts an Azure Virtual Machine over a POST request to Azure Automation. For cost reasons, those Machines should automatically stop after two hours.
Is there a Way/Function to do this easily?
For your requirement, you need to calculate the duration that the VM running yourself. You can get the event time that the last start time when the VM is in the running time. It's the UTC time. Then calculate the duration up to now yourself. Here is the Azure CLI command to get the event time:
az vm get-instance-view -g yourResourceGroup -n yourVM --query instanceView.statuses
The screenshot of the result here:
Or you can filter the activity log to get the last event "Start Virtual Machine" time. Below is the Azure CLI command:
az monitor activity-log list --resource-id yourVM_resourceId --query "[?operationName.localizedValue == 'Start Virtual Machine'].eventTimestamp" --max-events 1
I think it should be an interval query task to last for two hours as you need. In my own opinion, it should run in the script that starts the VM.
We are trying to queue from code a build but that should not run instantly but in the evening as our build pipeline is quite free in the evening and this job does not need to be run right away.
We are queuing around 20 or those builds on a daily basis and right now it is unfortunately blocking other builds. I know that we can use build priorities but it is not good enough as the build we want to "postpone" takes quite a long time and would block other builds if it would be started before the high importance build.
We also saw that it is possible to create a schedule but this sounds more like a build that should reoccur where we need the build to run only once.
There is a work-around to achieve running a build once at an appointed time using Azure CLI and CMD scheduled task. You can try to follow below steps.
1, you need to install Azure CLI. You can follow the steps in this blogs to get started with Azure CLI. [blog]:https://devblogs.microsoft.com/devops/using-azure-devops-from-the-command-line/
2, Create a CMD script like below and save it to your local disk, For more information about az pipelines commands go to https://learn.microsoft.com/en-us/cli/azure/ext/azure-devops/pipelines/build?view=azure-cli-latest#ext-azure-devops-az-pipelines-build-queue
az pipelines build queue --definition-name your-build-definition-name -o table
3,create a scheduled CMD task script using schtask.exe like below example, for more information visit https://www.windowscentral.com/how-create-task-using-task-scheduler-command-prompt
schtasks /create /tn "give-your-task-a-name" /tr "the-location-of-the-scripts-file-you-created-in-previous-step" /sc ONCE /st specify-the-time-to-run-your-build
You can save this script to your local disk too, Next time you can just run this scripts when you want to schedule your build to run in the evening.
Hope above steps can help you, This workaround seems tedious and need a little effort. But it is an once and for all work.
Azure Devops: Queue a build to run in the evening
Trigger build only once is not available for now. As you saw, there only as working days, time and time zone for schedule.
There has an user voice Scheduled builds - More flexible timing configuration which suggest more flexible time configuration including. You can vote and follow up for this user voice.
As the comment on that thread, we could Use cron syntax to specify schedules in a YAML file. As test, we can get a more detailed timing configuration, but we still could not schedule the build to run only once.
As workaround, we could schedule the build on a certain day of the week, after schedule build completed, Then we could disable the schedule manually or using the tool Azure DevOps CLI.
Hope this helps.
I am eagerly awaiting the release of Kubernetes v1.3 in mid to late June, so that I can access cron scheduling for jobs. In the meantime, what I plan to do is the following:
Deploy a job on my Kubernetes cluster
Use jenkins as a cron tool to trigger the job in defined intervals (e.g. 1 hour).
I have two questions:
How do I update a job? For replication controllers, I would simply do a rolling update, but in the jobs API spec (http://kubernetes.io/docs/user-guide/jobs/) there are no details about how to do this. For example, lets say that I want to use my jenkins deploy system to update the job whenever I do a git commit.
Is it possible to use the kubernetes API to trigger jobs? For example, I have a job that runs and then the pod is terminated on completion. Then, 1 hour later, I want to use jenkins to trigger the job again.
Thanks so much!
I am not sure if there is any fancy way to trigger a completed job, but one way to do it can be to delete and recreate the job.
Re: rolling-update: that is required for long running pods, which is what RCs control.
For jobs: You can update the podTemplateSpec in jobSpec and that will ensure that any new pod created by the job after the update will have the updated podTemplateSpec (note: already running pods will not be affected).
Hope this helps!