I have a toml job spec that looks like:
type = "webhook"
schemaVersion = 1
observationSource = """
send_to_bridge [type=bridge name="vesper-test" requestData="{\\"data\\": {\\"quote\\":\\"USD\\"}}"]
send_to_bridge
"""
However, I'm getting the following error:
job type 'webhook' has not been registered with the job.Spawner
How do I fix?
Set FEATURE_WEBHOOK_V2=true in your .env
Related
I'm trying to deploy PostgreSQL managed service with bicep and in most cases get an error:
"code": "InvalidParameterValue",
"message": "Invalid value given for parameter databaseName. Specify a valid parameter value."
I've tried various names for the DB, even in last version of the script I add random suffix to made it unique. Anyway it finishes with error, but looks like service is working. Another unexplainable thing is that sometimes script finishes without error... It's part of my IaC scenario, i need to be able to rerun it many times...
bicep code:
param location string
#secure()
param sqlserverLoginPassword string
param rand string = uniqueString(resourceGroup().id) // Generate unique String
param sqlserverName string = toLower('invivopsql-${rand}')
param sqlserverAdminName string = 'invivoadmin'
param psqlDatabaseName string = 'postgres'
resource flexibleServer 'Microsoft.DBforPostgreSQL/flexibleServers#2021-06-01' = {
name: sqlserverName
location: location
sku: {
name: 'Standard_B1ms'
tier: 'Burstable'
}
properties: {
createMode: 'Default'
version: '13'
administratorLogin: sqlserverAdminName
administratorLoginPassword: sqlserverLoginPassword
availabilityZone: '1'
storage: {
storageSizeGB: 32
}
backup: {
backupRetentionDays: 7
geoRedundantBackup: 'Disabled'
}
}
}
Please follow this git issue here for a similar error that might help you to fix your problem.
I want to use Terragrunt to deploy this example: https://github.com/aws-ia/terraform-aws-eks-blueprints/blob/main/examples/complete-kubernetes-addons/main.tf
So far, I was able to create the VPC/EKS resource without a problem, I separated each module into a different module directory, and everything worked as expected.
When I tried to do the same for the Kubernetes-addons module, I faced an issue with the data source trying to call to the cluster and failing since the cluster wasn't created at this point.
Here's my terragrunt.hcl which I'm trying to execute for this specific module:
...
terraform {
source = "git::git#github.com:aws-ia/terraform-aws-eks-blueprints.git//modules/kubernetes-addons?ref=v4.6.1"
}
locals {
# Extract needed variables for reuse
cluster_version = "${include.envcommon.locals.cluster_version}"
name = "${include.envcommon.locals.name}"
}
dependency "eks" {
config_path = "../eks"
mock_outputs = {
eks_cluster_endpoint = "https://000000000000.gr7.eu-west-3.eks.amazonaws.com"
eks_oidc_provider = "something"
eks_cluster_id = "something"
}
}
inputs = {
eks_cluster_id = dependency.eks.outputs.cluster_id
eks_cluster_endpoint = dependency.eks.outputs.eks_cluster_endpoint
eks_oidc_provider = dependency.eks.outputs.eks_oidc_provider
eks_cluster_version = local.cluster_version
...
}
The error that I'm getting here:
`
INFO[0035]
Error: error reading EKS Cluster (something): couldn't find resource
with data.aws_eks_cluster.eks_cluster,
on data.tf line 7, in data "aws_eks_cluster" "eks_cluster":
7: data "aws_eks_cluster" "eks_cluster" {
`
The kubernetes-addons module is deploying addons into an existing Kubernetes cluster. If you don't have a cluster running (apparently you don't have one when you're mocking the cluster_id variable), then you get the error of not having the aws_eks_cluster data source.
You need to create the K8s cluster first, before you can start deploying the addons.
We are using terraform to launch ECS containers in AWS infra using custom task definition.
As we didn't require full infra to be launched every-time, a part only for launching ECS container was segregated.
The launch was happening correctly, till ECS launch code was segregated, then the ECS service launch started giving an error indicating Idempotent issue.
│ Error: error creating target service: error waiting for ECS service (sandbox) creation: InvalidParameterException: Creation of service was not idempotent.
│
│ with aws_ecs_service.ecs_service_target,
│ on aws_infra_ecs.tf line 100, in resource "aws_ecs_target" "ecs_service_target":
│ 100: resource "aws_ecs_target" "ecs_service_target" {
│
ECS service is defined somewhat like below:
resource "aws_ecs_service" "ecs_service_target" {
desired_count = 1
name = "target"
launch_type = "FARGATE"
cluster = data.aws_ecs_cluster.cluster_target.id
enable_ecs_managed_tasks = true
task_definition = aws_ecs_task_definition.target_taskdef.arn
platform_version = "1.4.0"
...
load_balancer {
...
target_group_arn = data.aws_lb_target_group.aws_target.arn
}
...
network_configuration {
...
security_groups = [ data.aws_security_group.target_sg.id ]
subnets = [ "subet-5767c3c2" ] # A dynamic subnet reference id is used here
}
depends_upon = [
var.second_service_name,
aws_ecs_task_definition.target_taskdef,
data.aws_efs_access_point.target_ap
]
...
}
I was expecting the problems to be one of following kind:
Subnet selected may be different due to variable based selection
Use of indirect data references (rather than direct resource reference) may cause issue
task definition JSON encoding issue
What might be other causes for such a problem.
When i create a dataflow template, the characteristics of Runtime parameters are not persisted in the template file.
At runtime, if i try to pass a value for this parameter, i take a 400 error
I'm using Scio 0.3.2, scala 2.11.11 with apache beam (0.6).
My parameters are the following :
trait XmlImportJobParameters extends PipelineOptions {
def getInput: ValueProvider[String]
def setInput(value: ValueProvider[String]): Unit
}
They are registred with this code
val options = PipelineOptionsFactory.fromArgs(cmdlineArgs: _*).withValidation().as[XmlImportJobParameters](classOf[XmlImportJobParameters])
PipelineOptionsFactory.register(classOf[XmlImportJobParameters])
implicit val (sc, args) = ContextAndArgs(cmdlineArgs)
To create the template i call sbt with this parameters :
run-main jobs.XmlImportJob --runner=DataflowRunner --project=MyProject --templateLocation=gs://myBucket/XmlImportTemplate --tempLocation=gs://myBucket/staging --instance=myInstance
If i pass explicitly --input, it becomes a StaticValue instead of RuntimeValue, and this time, i can see it in the template file.
The template is called from a google function watching a bucket storage (inspired from https://shinesolutions.com/2017/03/23/triggering-dataflow-pipelines-with-cloud-functions/) :
...
dataflow.projects.templates.create({
projectId: projectId,
resource: {
parameters: {
input: `gs://${file.bucket}/${file.name}`
},
jobName: jobs[job].name,
gcsPath: 'gs://MyBucket/MyTemplate'
}
}
...
The 400 error :
problem running dataflow template, error was: { Error: (109c1c52dc52fec7): The workflow could not be created. Causes: (109c1c52dc52fb8e): Found unexpected parameters: ['input' (perhaps you meant 'runner')] at Request._callback (/user_code/node_modules/googleapis/node_modules/google-auth-library/lib/transporters.js:85:15) at Request.self.callback (/user_code/node_modules/googleapis/node_modules/request/request.js:188:22) at emitTwo (events.js:106:13) at Request.emit (events.js:191:7) at Request.<anonymous(/user_code/node_modules/googleapis/node_modules/request/request.js:1171:10) at emitOne (events.js:96:13) at Request.emit (events.js:188:7) at IncomingMessage.<anonymous> (/user_code/node_modules/googleapis/node_modules/request/request.js:1091:12) at IncomingMessage.g (events.js:291:16) at emitNone (events.js:91:20) code: 400, errors: [ { message: '(109c1c52dc52fec7): The workflow could not be created. Causes: (109c1c52dc52fb8e): Found unexpected parameters: [\'input\' (perhaps you meant \'runner\')]', domain: 'global', reason: 'badRequest' } ] }
Same error when i try this :
gcloud beta dataflow jobs run xmlJobImport --gcs-location gs://MyBucket/MyTemplate --parameters input=gs://MyBucket/file.csv
=>
(gcloud.beta.dataflow.jobs.run) INVALID_ARGUMENT: (260a4f3f738a8ad9): The workflow could not be created. Causes: (260a4f3f738a8f96): Found unexpected parameters: ['input' (perhaps you meant 'runner'), 'projectid' (perhaps you meant 'project'), 'table' (perhaps you meant 'zone')]
The current settings are :
Current Settings:
appName: XmlImportJob$
autoscalingAlgorithm: THROUGHPUT_BASED
input: RuntimeValueProvider{propertyName=input, default=null, value=null}
instance: StaticValueProvider{value=staging}
jobName: xml-import-job
maxNumWorkers: 1
network: staging
numWorkers: 1
optionsId: 0
project: myProjectId
projectid: StaticValueProvider{value=myProjectId}
provenance: StaticValueProvider{value=ac3}
record: StaticValueProvider{value=BIEN}
root: StaticValueProvider{value=LISTEPA}
runner: class org.apache.beam.runners.dataflow.DataflowRunner
stableUniqueNames: WARNING
streaming: false
subnetwork: regions/europe-west1/subnetworks/net-staging
table: StaticValueProvider{value=annonce}
tempLocation: gs://-flux/staging/xmlImportJob/
templateLocation: gs://-flux-templates/XmlImportTemplate
workerMachineType: n1-standard-1
zone: europe-west1-c
Environement
Coping the answer from the issue:
Scio does not currently expose ValueProvider based APIs - we now have an issue open for this #696
A working example would be something like:
object WordCount {
def main(cmdlineArgs: Array[String]): Unit = {
val (sc, args) = ContextAndArgs(cmdlineArgs)
sc.customInput("input", TextIO.read().from(sc.optionsAs[XmlImportJobParameters].getInput))
.map(_.toUpperCase)
.saveAsTextFile(args("output"))
sc.close()
}
}
For the job above, to create template:
run-main com.example.WordCount --runner=DataflowRunner --project=<project> --templateLocation=gs://<template-bucket> --tempLocation=gs://<temp-location> --output=gs://<example-of-static-arg-output>
To submit job:
gcloud beta dataflow jobs run rav-test --gcs-location=gs://<template-bucket> --parameters=input=gs://<runtime-value>
I want to develop my own workflow named "backup" in cloudify with my own plugin, but when i ran that workflow, the below error occured
'backup' workflow execution failed: RuntimeError: Workflow failed: Task failed 'script_runner.tasks.run' -> Missing cloudify_agent runtime information. This most likely means that the Compute node never started successfully
I don't understand why, anybody can solved me this problem?
Here is my main blueprint code and plugin code
My main blueprint
tosca_definitions_version: cloudify_dsl_1_2
imports:
- plugins/backup.yaml
- types/types.yaml
node_templates:
mynode:
type: cloudify.nodes.Compute
properties:
ip: "ip"
agent_config:
install_method: none
user: "user"
key: "key_uri"
myapp:
type: cloudify.nodes.ApplicationModule
interfaces:
test_platform_backup:
backup:
implementation: scripts/backup.sh
inputs:
port: 6969
post_backup:
implementation: scripts/post_backup.sh
relationships:
- type: cloudify.relationships.contained_in
target: mynode
My plugin code:
from cloudify.decorators import workflow
from cloudify.workflows import ctx
from cloudify.workflows.tasks_graph import forkjoin
#workflow
def backup(operation, type_name, operation_kwargs, is_node_operation, **kwargs):
graph = ctx.graph_mode()
send_event_starting_tasks = {}
send_event_done_tasks = {}
for node in ctx.nodes:
if type_name in node.type_hierarchy:
for instance in node.instances:
send_event_starting_tasks[instance.id] = instance.send_event('Starting to run operation')
send_event_done_tasks[instance.id] = instance.send_event('Done running operation')
for node in ctx.nodes:
if type_name in node.type_hierarchy:
for instance in node.instances:
sequence = graph.sequence()
if is_node_operation:
operation_task = instance.execute_operation(operation, kwargs=operation_kwargs)
else:
forkjoin_tasks = []
for relationship in instance.relationships:
forkjoin_tasks.append(relationship.execute_source_operation(operation))
forkjoin_tasks.append(relationship.execute_target_operation(operation))
operation_task = forkjoin(*forkjoin_tasks)
sequence.add(
send_event_starting_tasks[instance.id],
operation_task,
send_event_done_tasks[instance.id])
for node in ctx.nodes:
for instance in node.instances:
for rel in instance.relationships:
instance_starting_task = send_event_starting_tasks.get(instance.id)
target_done_task = send_event_done_tasks.get(rel.target_id)
if instance_starting_task and target_done_task:
graph.add_dependency(instance_starting_task, target_done_task)
return graph.execute()
It seems that your VM did not start.
From your code I can't understand what you are trying to do.
You don't install and agent and you don't have a fabric connection to the VM, yet you are trying to run operations on the VM.
You should either install an agent, E.g remove the "install_method: none", or add a fabric connection to the VM and run the operations with it.