aws-ecs-patterns error: Cluster for this service needs Ec2 capacity. Call addXxxCapacity() on the cluster - amazon-ecs

Hoping someone can help me here, according to AWS CDK documentation if I declare my VPC then I shouldn't declare 'capacity', but when I run cdk synth I get the following error...
throw new Error(Validation failed with the following errors:\n ${errorList});
Error: Validation failed with the following errors:
[PrerenderInfrasctutureStack/preRenderApp/Service] Cluster for this service needs Ec2 capacity. Call addXxxCapacity() on the cluster.
here is my code...
(i hope Nathan Peck sees this)
const ec2 = require('#aws-cdk/aws-ec2');
const ecsPattern = require('#aws-cdk/aws-ecs-patterns');
const ecs = require('#aws-cdk/aws-ecs');
class PrerenderInfrasctutureStack extends cdk.Stack {
/**
*
* #param {cdk.Construct} scope
* #param {string} id
* #param {cdk.StackProps=} props
*/
constructor(scope, id, props) {
super(scope, id, props);
const myVPC = ec2.Vpc.fromLookup(this, 'publicVpc', {
vpcId:'vpc-xxx'
});
const preRenderApp = new ecsPattern.ApplicationLoadBalancedEc2Service(this, 'preRenderApp', {
vpcId: myVPC,
certificate: 'arn:aws:acm:ap-southeast-2:xxx:certificate/xxx', //becuase this is spcified, then the LB will automatically use HTTPS
domainName: 'my-dev.com.au.',
domainZone:'my-dev.com.au',
listenerPort: 443,
publicLoadBalancer: true,
memoryReservationMiB: 8,
cpu: 4096,
desiredCount: 1,
taskImageOptions:{
image: ecs.ContainerImage.fromRegistry('xxx.dkr.ecr.region.amazonaws.com/express-prerender-server'),
containerPort: 3000
},
});
}
}
module.exports = { PrerenderInfrasctutureStack }

This is because if you don't explicitly pass a cluster then it uses the default cluster that exists on your account. However the default cluster starts out with no EC2 capacity, since EC2 instances cost money when they run. You can use the empty default cluster with Fargate mode since Fargate does not require EC2 capacity, it just runs your container inside Fargate, but the default cluster won't work with EC2 mode until you add EC2 instances to the cluster.
The easy solution here is to switch to ApplicationLoadBalancedFargateService instead, because Fargate services run using Fargate capacity, so they don't require EC2 instances in the cluster. Alternatively you should define your own cluster using something like:
// Create an ECS cluster
const cluster = new ecs.Cluster(this, 'Cluster', {
vpc,
});
// Add capacity to it
cluster.addCapacity('DefaultAutoScalingGroupCapacity', {
instanceType: new ec2.InstanceType("t2.xlarge"),
desiredCapacity: 3,
});
Then pass that cluster as a property when creating the ApplicationLoadBalancedEc2Service
Hope this helps!

Related

Confluent for Kubernetes can't access kafka externally

I'm trying to create an external access about confluent-kafka from an AKS cluster. I've been able to connect with control center with an Ingress but i can't create the external access from kafka.
I added this to spec of kafka:
listeners:
external:
externalAccess:
type: loadBalancer
loadBalancer:
domain: lb.example.it
advertisedPort: 39093
and it creates two load-balancer services. Then i added the external IPs on my etc/hosts file:
20.31.10.27 kafka.lb.example.it
20.31.9.167 b0.lb.example.it
But when i ceate a nodejs producer i don't know what to put into broker:
const { Kafka } = require('kafkajs')
const kafka = new Kafka({
clientId: 'my-app',
brokers: ['kafka.lb.example.it:39093']
})
const producer = kafka.producer()
console.log("produced")
const asyncOperation = async () => {
console.log("connecting")
await producer.connect()
console.log("connected")
let i = 0
try{
while(true){
await producer.send({
topic: 'topic_prova3',
messages: [
{
key: JSON.stringify("hello"),
value: JSON.stringify({"NUM":i}),
},
]
})
await producer.send({
topic: 'topic_prova3',
messages: [
{
value: JSON.stringify({"NUMERO":i.toString()}),
},
]
})
console.log("sended")
i++
await new Promise(resolve => setTimeout(resolve, 5000));
}
}
catch(err){
console.error("error: " + err)
}
await producer.disconnect()
}
asyncOperation();
This is the log of the error:
{"level":"ERROR","timestamp":"2023-01-22T08:43:20.761Z","logger":"kafkajs","message":"[Connection] Connection error: connect ECONNREFUSED 127.0.0.2:9092","broker":"kafka-0.kafka.ckafka.svc.cluster.local:9092","clientId":"my-app","stack":"Error: connect ECONNREFUSED 127.0.0.2:9092\n at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1495:16)"}
The broker should be kafka.ckafka.svc.cluster.local:9092 , instead of kafka-0.kafka.ckafka.svc.cluster.local:9092 but it does it automatically
The load balancer appears to be working (it's not throwing unknown host exception), and it returned a cluster local broker address. See above comment that explains why this happens within Kubernetes with Kafka.
You need to modify, or use, the appropriate advertised.listeners port that corresponds to the external LoadBalancer.
from an AKS cluster.
You should use Azure services to create a public DNS route rather than specifically target one IP of your cluster for any given service in /etc/hosts; especially when you'd deploy multiple replicas of the Kafka pod.

How to deploy the kinesis-video-producer Docker image from AWS's own ECR to Fargate using CDK in TypeScript?

I'm trying to stand up a proof of concept that ingests an RTSP video stream into Kinesis Video. The provided documentation has a docker image all set up that seems to have everything I need to do this, hosted by AWS on 546150905175.dkr.ecr.us-west-2.amazonaws.com. What I am having trouble with, though, is getting that deployment (via an Amplify Custom category, in TypeScript CDK) to work.
I've tried different variations on
import * as iam from "#aws-cdk/aws-iam";
import * as ecs from "#aws-cdk/aws-ecs";
import * as ec2 from "#aws-cdk/aws-ec2";
const kinesisUserAccessKey = new iam.AccessKey(this, 'KinesisStreamUserAccessKey', {
user: kinesisStreamUser,
})
const servicePrincipal = new iam.ServicePrincipal('ecs-tasks.amazonaws.com');
const executionRole = new iam.Role(this, 'IngestVideoTaskDefExecutionRole', {
assumedBy: servicePrincipal,
managedPolicies: [
iam.ManagedPolicy.fromAwsManagedPolicyName('service-role/AmazonECSTaskExecutionRolePolicy'),
]
});
const taskDefinition = new ecs.FargateTaskDefinition(this, 'IngestVideoTaskDef', {
cpu: 512,
memoryLimitMiB: 1024,
executionRole,
})
const image = ecs.ContainerImage.fromRegistry('546150905175.dkr.ecr.us-west-2.amazonaws.com/kinesis-video-producer-sdk-cpp-amazon-linux:latest');
taskDefinition.addContainer('IngestVideoContainer', {
command: [
'gst-launch-1.0',
'rtspsrc',
`location="${locationParam.secretValue.toString()}"`,
'short-header=TRUE',
'!',
'rtph264depay',
'!',
'video/x-h264,',
'format=avc,alignment=au',
'!',
'kvssink',
`stream-name="${cfnStream.name}"`,
'storage-size=512',
`access-key="${kinesisUserAccessKey.accessKeyId}"`,
`secret-key="${kinesisUserAccessKey.secretAccessKey.toString()}"`,
`aws-region="${REGION}"`,
// `aws-region="${cdk.Aws.REGION}"`,
],
image,
logging: new ecs.AwsLogDriver({
streamPrefix: 'IngestVideoContainer',
}),
})
const service = new ecs.FargateService(this, 'IngestVideoService', {
cluster,
taskDefinition,
desiredCount: 1,
securityGroups: [
ec2.SecurityGroup.fromSecurityGroupId(this, 'DefaultSecurityGroup', SECURITY_GROUP_ID)
],
vpcSubnets: {
subnets: SUBNET_IDS.map(subnetId => ec2.Subnet.fromSubnetId(this, subnetId, subnetId)),
}
})
But it seems like regardless of what I do, an amplify push just stays in 'in progress' for like an hour until I go into the CloudFormation console and cancel the stack update, but deep in the my way to the ECS Console I managed to find an actual error message:
Resourceinitializationerror: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s): RequestError: send request failed caused by: Post "https://api.ecr.us-west-2.amazonaws.com/": dial tcp 52.94.177.118:443: i/o timeout
It seems to be some kind of networking issue, but I'm not sure how to proceed. Any assistance you can provide would be wonderful. Cheers!
Figured it out. For those stuck with similar issues, you have to give it an execution role with AmazonECSTaskExecutionRolePolicy, which I already edited above, and set assignPublicIp: true in the service.

How to modify GKE cluster master_authorized_networks_config with terraform

I am creating a kube cluster with GKE in terraform. I am creating the cluster from two modules, a cluster module and a nodepool module. I'd like to create a module for the master_authorized_networks_config so that each time a new cidr is added to it terraform doesn't destroy the original cluster. Is this possible. This is an example of the block of code that's in my cluster module that I would like to change without destroying the whole cluster. These IP addresses are what allow acess to the cluster.
master_authorized_networks_config {
cidr_blocks {
cidr_block = "123.456.789/32"
display_name = "megacorp-1-nat1"
}
cidr_blocks {
cidr_block = "34.69.69.69/32"
display_name = "megacorp-1-nat2"
}
cidr_blocks {
cidr_block = "123.456.333.333/32"
display_name = "vpn-test"
}
Adding another cidr destroys the original cluster. I don't want this.

In CDK, can I wait until a Helm-installed operator is running before applying a manifest?

I'm installing the External Secrets Operator alongside Apache Pinot into an EKS cluster using CDK. I'm running into an issue that I think is being caused by CDK attempting to create a resource defined by the ESO before the ESO has actually gotten up and running. Here's the relevant code:
// install Pinot
const pinot = cluster.addHelmChart('Pinot', {
chartAsset: new Asset(this, 'PinotChartAsset', { path: path.join(__dirname, '../pinot') }),
release: 'pinot',
namespace: 'pinot',
createNamespace: true
});
// install the External Secrets Operator
const externalSecretsOperator = cluster.addHelmChart('ExternalSecretsOperator', {
chart: 'external-secrets',
release: 'external-secrets',
repository: 'https://charts.external-secrets.io',
namespace: 'external-secrets',
createNamespace: true,
values: {
installCRDs: true,
webhook: {
port: 9443
}
}
});
// create a Fargate Profile
const fargateProfile = cluster.addFargateProfile('FargateProfile', {
fargateProfileName: 'externalsecrets',
selectors: [{ 'namespace': 'external-secrets' }]
});
// create the Service Account used by the Secret Store
const serviceAccount = cluster.addServiceAccount('ServiceAccount', {
name: 'eso-service-account',
namespace: 'external-secrets'
});
serviceAccount.addToPrincipalPolicy(new iam.PolicyStatement({
effect: iam.Effect.ALLOW,
actions: [
'secretsmanager:GetSecretValue',
'secretsmanager:DescribeSecret'
],
resources: [
'arn:aws:secretsmanager:us-east-1:<MY-ACCOUNT-ID>:secret:*'
]
}))
serviceAccount.node.addDependency(externalSecretsOperator);
// create the Secret Store, an ESO Resource
const secretStoreManifest = getSecretStoreManifest(serviceAccount);
const secretStore = cluster.addManifest('SecretStore', secretStoreManifest);
secretStore.node.addDependency(serviceAccount);
secretStore.node.addDependency(fargateProfile);
// create the External Secret, another ESO resource
const externalSecretManifest = getExternalSecretManifest(secretStoreManifest)
const externalSecret = cluster.addManifest('ExternalSecret', externalSecretManifest)
externalSecret.node.addDependency(secretStore);
externalSecret.node.addDependency(pinot);
Even though I've set the ESO as a dependency to the Secret Store, when I try to deploy this I get the following error:
Received response status [FAILED] from custom resource. Message returned: Error: b'Error from server (InternalError): error when creating "/tmp/manifest.yaml": Internal error occurred:
failed calling webhook "validate.clustersecretstore.external-secrets.io": Post "https://external-secrets-webhook.external-secrets.svc:443/validate-external-secrets-io-v1beta1-clusterse
cretstore?timeout=5s": no endpoints available for service "external-secrets-webhook"\n'
If I understand correctly, this is the error you'd get if you try to add a Secret Store before the ESO is fully installed. I'm guessing that CDK does not wait until the ESO's pods are running before attempting to apply the manifest. Furthermore, if I comment out the lines the create the Secret Store and External Secret, do a cdk deploy, uncomment those lines and then deploy again, everything works fine.
Is there any way around this? Some way I can retry applying the manifest, or to wait a period of time before attempting the apply?
The addHelmChart method has a property wait that is set to false by default - setting it to true lets CDK know to not mark the installation as complete until of its its K8s resources are in a ready state.

How to not rebuild a DockerImageAsset at every deploy using aws-cdk in TypeScript?

My app is a Python API that I package as a Docker image and use with ECS Fargate (Spot Instances). The code below works.
My issue is that it rebuilds the entire image every time I deploy this – which is very time-consuming (downloads all dependencies, makes the image, uploads, etc). I want it to reuse the exact same image uploaded to ECR by aws-cdk itself.
Is there a way (env variable or else) for me to skip this when I don't touch the app's code and just make changes to the stack?
#!/usr/bin/env node
import * as cdk from "#aws-cdk/core"
import * as ecs from "#aws-cdk/aws-ecs"
import * as ec2 from "#aws-cdk/aws-ec2"
import * as ecrassets from "#aws-cdk/aws-ecr-assets"
// See https://docs.aws.amazon.com/cdk/api/latest/docs/aws-ecs-readme.html
export class Stack extends cdk.Stack {
constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props)
/**
* Repository & Image
*/
const apiDockerImage = new ecrassets.DockerImageAsset(
this,
`my-api-image`,
{
directory: `.`,
exclude: [`cdk.out`, `cdk`, `.git`]
}
)
/**
* Cluster
*/
const myCluster = new ecs.Cluster(this, "Cluster", {})
// Add Spot Capacity to the Cluster
myCluster.addCapacity(`spot-auto-scaling-group-capacity`, {
maxCapacity: 2,
minCapacity: 1,
instanceType: new ec2.InstanceType(`r5a.large`),
spotPrice: `0.0400`,
spotInstanceDraining: true
})
// A task Definition describes what a single copy of a task should look like
const myApiFargateTaskDefinition = new ecs.FargateTaskDefinition(
this,
`api-fargate-task-definition`,
{
cpu: 2048,
memoryLimitMiB: 8192,
}
)
// Add image to task def
myApiFargateTaskDefinition.addContainer(`api-container`, {
image: ecs.ContainerImage.fromEcrRepository(
apiDockerImage.repository,
`latest`
),
})
// And the service attaching the task def to the cluster
const myApiService = new ecs.FargateService(
this,
`my-api-fargate-service`,
{
cluster: myCluster,
taskDefinition: myApiFargateTaskDefinition,
desiredCount: 1,
assignPublicIp: true,
}
)
}
}
The proper solution is to build your image outside of this deployment process and just get a reference to that image in ECR.