I have an existing, functional Dataproc workflow template, and I am attempting to create a Cloud Scheduler job to run it on a schedule. The workflow template runs without error when run via the "RUN" button on the Dataproc Workflow Template console page.
I created a Scheduler job to run this workflow template with the following command (names redacted here):
gcloud scheduler jobs create http <job-name>
--location=us-central1
--schedule="0 1 * * *"
--http-method=POST
--uri=https://dataproc.googleapis.com/v1/projects/<project-name>/regions/us-central1/workflowTemplates/<template-name>:instantiate?alt=json
--oauth-service-account-email=<service-account-name>#<project-name>.iam.gserviceaccount.com
--oauth-token-scope=https://www.googleapis.com/auth/cloud-platform
The job is successfully created, but upon attempting to run it, it returns INVALID_ARGUMENT. Here is the response:
{"#type":"type.googleapis.com/google.cloud.scheduler.logging.AttemptFinished", "jobName":"projects/<project-name>/locations/us-central1/jobs/<job-name>",
"status":"INVALID_ARGUMENT", "targetType":"HTTP", "url":"https://dataproc.googleapis.com/v1/projects/<project-name>/regions/us-central1/workflowTemplates/<workflow-name>:instantiate?alt=json"}
I get no more descriptive response than that. There is no sign the workflow began, nor any failures listed on the Dataproc Workflow console page.
I also tried a similar Scheduler job with the same service account, scope, location, and schedule, but I cut the URL down to the base workflow location: https://dataproc.googleapis.com/v1/projects/<project-name>/regions/us-central1/workflowTemplates/<template-name> and tried --http-method=GET, and that was successful, but, of course, does not instantiate the workflow.
Is there either something I'm missing, or is there at least a better way to diagnose the issue?
I have now been able to successfully kick off this workflow via Scheduler once the Service Account User role has been added to the service account that is used in creating the Scheduler job. That is, the service account itself also needs to be a service account user.
Related
I am new to creating jobs in Rundeck (community). I'd like to create a job under a project that accepts 2 parameters from the user (1. external/internal 2. IP CIDR) and then return if the IP CIDR already exists in WAF.
The current process is that user passes these parameters and the script has aws-vault command for the user to authenticate with the AWS account.
I have a shell script to do so but wondering how to do this using Rundeck jobs. Also, is there a way to allow the entire Rundeck instance (IAM roles?) to authenticate against a certain AWS account?
Thanks in advance.
To execute a script on Rundeck:
Create a new Project, create a new job, give it a name, on the workflow tab select the "Script" step (you can pass the parameters on the "arguments" textbox) put the parameters on the ), put the script content there, and save and run the job.
Create a new Project, create a new job, give it a name, on the workflow tab select the "Script file or URL" step (you can pass the parameters on the "arguments" textbox), put the script file path there, and save and run the job.
I have a shell script to do so but wondering how to do this using
Rundeck jobs. Also, is there a way to allow the entire Rundeck
instance (IAM roles?) to authenticate against a certain AWS account?
For EC2 remote instances, S3 actions, and some specific (and exclusive) Process Automation it's possible (the credentials are part of the plugin config).
For AWS WAF you can create a script using awscli tool with the rights parameters to execute it (or design your own AWS WAF plugin).
Anyway, take a look at the basic tutorial to learn how Rundeck works.
I have an Azure DevOps pipeline with a PowerShell task, which when triggered, successfully creates and configures a Windows Scheduled Task on my Windows Server.
The configuration sets the task to run using an AD Service account, 'Run whether the user is logged on or not' & 'Run with the highest privileges'.
When I try to run this task however, I get the message "The user account does not have permission to run this task."
If I manually create a scheduled task using all the same details, it runs without issue.
I've seen other threads about going into the C:\Windows\System32\Tasks folder, and change the security to update the Owner, but I still get the same result.
Has anyone had any luck setting a Scheduled task up via PowerShell script, and/or what the resolution to the permission issue?
After much tinkering around, I found that the Azure DevOps agent was set up using the default SYSTEM credentials.
I created a new ADO Agent that runs as an admin Service account, and when the Scheduled task was deployed, it ran without issue.
I tried batch: true setting described here, but it seems to:
ignore commits that are pushed when a pipeline is running. I want the last commit to trigger a pipeline after the current run of that pipeline has finished
be ignored when you publish directly from CI by pressing build
Has someone found a way to configure a pipeline to run as I have described.
You can try adding a Invoke REST API check on the agent pool. The rest api will get the previous build and evaluate its status. So when new build is queued targeting the agent pool. The Invoke REST API will be invoked, this new build will only start when the response of the rest api is evaluated to true.
Please check below steps：
1, create a service connection to your azure devops organization.
Go Project Setting--> Service connections under Pipelines-->Click new service connection--> select Generic to create a generic service connection.
Then Edit the service connection type the information shown in below screenshot. Check here to get a Personal access token.
2, add Invoke REST API check on the agent pool**.
Go Project Setting--> Agent Pools under Pipelines-->Select the agent pool--> Click the 3dots -->Click Approvals and checks.
.
3, Click the "+"--> Choose Invoke REST API
4, Edit the Invokde Rest API
Select the generic service connection to your azure devops created in the first step.
Set the fields as below:
URL suffix and parameters: _apis/build/builds?definitions=the DefinitionId of your pipeline&$top=2&queryOrder=queueTimeDescending&api-version=5.1
Success criteria: eq(root['value'][1]['status'], 'completed')
please check here for more information about build rest api.
Note: Since the Invoke Rest api check is set on agent pool scope. It may have effects on other pipelines that target this agent pool. For example, if the desired yaml pipeline is waiting its previous run to complete. And now another pipeline targeting this same agent pool is triggered, it will have to wait for the previous run of desired yaml pipeline to complete too.
I have two yaml templates defined one for creating a docker registry service connection and second for deploying some stuff via container job. The second template uses the docker registry connection which is being deployed in first template. When I am running both the templates separately then both the stages are successful but when I run them in one azure-pipelines.yaml, it fails :
There was a resource authorization issue: "The pipeline is not valid. A service connection with name shared-stratus-acr-endpoint could not be found. The service connection does not exist or has not been authorized for use. For authorization details, refer to https://aka.ms/yamlauthz."
Is there any way like dependsOn or condition that we can provide in this situation?
It's likely that you only authorized the service connection for the individual template\pipelines when you created them. The workflow is not super friendly.
What if you try and authorize the pipeline that is failing for that service connection explicitly. See docs here
You could also just authorize the service connection for all pipelines depending on your security needs.
Using the Azure portal, I can click the Run Once button on a Scheduler job to execute it. Is that functionality available via a REST call from my app? The scenario is that periodically a job is run to check for changes in data. The first time the app starts up, I would like to execute that job once as not to have to wait for the scheduled time.
What is the call to kick off a scheduled job? Would I have to duplicate code in both the scheduler job and custom API and then call the custom API from my code?
You can execute a scheduler job by sending a POST request to /jobs/<jobName> (no request body required). But you need to pass the master key of your mobile service (in the x-zumo-master HTTP header), so you should only do that from a location that is not visible to your users - you don't want anyone getting a hold of your master key, as this would open up your service for all kinds of attacks.