How can I register an ECS Service as a Network Load Balancer target on a non-default port? - aws-cloudformation

I'm trying to deploy a horizontally scaling application consisting of multiple containers with a single reverse proxy in front to AWS ECS backed by EC2. For certain reasons I cannot use an Application Load Balancer, but want to use a Network Load Balancer that forwards all traffic on ports 80 and 443 to the reverse proxy container. I use AWS CDK to define the setup.
I am running into issues when trying to route traffic on both ports to the proxy. No matter what I do, all targets in the created target group point to port 80 on the container. I.e. I get a mapping of 80->80, 443->80 when I would like 80->80, 443->443.
My CDK code looks like this:
const proxyService = new ecs.Ec2Service(this, 'ProxyService', {
serviceName: 'proxy',
cluster,
taskDefinition: proxyTaskDefinition,
minHealthyPercent: 0,
desiredCount: 1,
securityGroups: [securityGroup],
cloudMapOptions: {
name: 'proxy',
cloudMapNamespace: cluster.defaultCloudMapNamespace
}
})
const loadbalancer = new lb.NetworkLoadBalancer(this, 'NetworkLoadBalancer', {
vpc,
internetFacing: true
})
new cdk.CfnOutput(this, 'LoadBalancerDnsName', {
value: loadbalancer.loadBalancerDnsName
})
loadbalancer.addListener('HTTPListener', {
port: 80
})
.addTargets('HTTPTarget', {
port: 80,
targets: [proxyService]
})
loadbalancer.addListener('HTTPSListener', {
port: 443,
})
.addTargets('HTTPSTarget', {
port: 443,
// the proxyService seems to always register itself at port 80
// by calling its attachToNetworkTargetGroup method
targets: [proxyService]
})
}
The Cloudformation generated for the Target Groups looks like this:
NetworkLoadBalancerHTTPListener792E96F1:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
DefaultActions:
- TargetGroupArn:
Ref: NetworkLoadBalancerHTTPListenerHTTPTargetGroupCEAF8C0F
Type: forward
LoadBalancerArn:
Ref: NetworkLoadBalancer8E753273
Port: 80
Protocol: TCP
Metadata:
aws:cdk:path: SplitClusterStack/NetworkLoadBalancer/HTTPListener/Resource
NetworkLoadBalancerHTTPListenerHTTPTargetGroupCEAF8C0F:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Protocol: TCP
TargetType: ip
VpcId:
Ref: VPCB9E5F0B4
Metadata:
aws:cdk:path: SplitClusterStack/NetworkLoadBalancer/HTTPListener/HTTPTargetGroup/Resource
NetworkLoadBalancerHTTPSListenerAF8F470A:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
DefaultActions:
- TargetGroupArn:
Ref: NetworkLoadBalancerHTTPSListenerHTTPSTargetGroup4BC6FF0B
Type: forward
LoadBalancerArn:
Ref: NetworkLoadBalancer8E753273
Port: 443
Protocol: TCP
Metadata:
aws:cdk:path: SplitClusterStack/NetworkLoadBalancer/HTTPSListener/Resource
NetworkLoadBalancerHTTPSListenerHTTPSTargetGroup4BC6FF0B:
Type: AWS::ElasticLoadBalancingV2::TargetGroup
Properties:
Protocol: TCP
TargetType: ip
VpcId:
Ref: VPCB9E5F0B4
Metadata:
aws:cdk:path: SplitClusterStack/NetworkLoadBalancer/HTTPSListener/HTTPSTargetGroup/Resource
After deploying this, I can edit the created target groups in the web console to register a new target pointing to 443 on the same IP and deregister port 80 to get things working.
How can I create a Loadbalancer target that:
points to the ECS service
uses port 443
I'm happy to construct this myself of even add overrides if it helps me get this solved.

The ECS service exposes a loadBalancerTarget method that can be used for this:
loadbalancer.addListener('HTTPSListener', {
port: 443,
})
.addTargets('HTTPSTarget', {
port: 443,
targets: [proxyService.loadBalancerTarget({
containerPort: 443,
containerName: 'proxy'
})]
})

Related

Docker compose Amazon ECS

As I am quite familiar with docker compose and not so familiar with amazons cloudformation I found it extremely nice to be able to basically run your docker compose files via ecs integration and viola behind the scenes everything you need is created for you. So you get your load balancer (if not already created) and your ecs cluster with your services running and everything is connected and just works. When I started wanting to do a bit more advanced things I ran into a problem that I can't seem to find an answer to online.
I have 2 services in my docker compose, my spring boot web app and my postgres db. I wanted to implement ssl and redirect all traffic to https. After a lot of research and a lot of trial and error I finally got it to work by extending my compose file with x-aws-cloudformation and adding native cloudformation yaml. When doing all of this I was forced to choose an application load balancer over a network load balancer as it operates on layer 7 (http/https). However my problem is that now I have no way of reaching my postgres database and running queries against it via for example intellij. My spring boot app works find and can read/write to my database so that works fine. Before the whole ssl implementation I didn't specify a load balancer in my compose file and so it gave me a network load balancer every time I ran my compose file. Then I could connect to my database via intellij and run queries. I have tried adding an inbound rule on my security group that basically allows all inbound traffic to my database via 5432 but that didn't help. I may not be setting the correct host when applying my connection details in intellij but I have tried using the following:
dns name of load balancer
ip-adress of load balancer
public ip of my postgres db task (launch type: fargate)
I would just like to simply reach my database and run queries against it even though it is running inside aws ecs cluster behind an application load balancer. Is there a way of achieving what I am trying to do? Or do I have to have 2 separate load balancers (one application LB and one network LB)?
Here is my docker-compose file(I have omitted a few irrelevant env variables):
version: "3.9"
x-aws-loadbalancer: arn:my-application-load-balancer
services:
my-web-app:
build:
context: .
image: hub/my-web-app
x-aws-pull_credentials: xxxxxxxx
container_name: my-app-name
ports:
- "80:80"
networks:
- my-app-network
depends_on:
- postgres
deploy:
replicas: 1
resources:
limits:
cpus: '0.5'
memory: 2048M
environment:
- SPRING_DATASOURCE_URL=jdbc:postgresql://postgres:5432/my-db?currentSchema=my-db_schema
- SPRING_DATASOURCE_USERNAME=dbpass
- SPRING_DATASOURCE_PASSWORD=dbpass
- SPRING_DATASOURCE_DRIVER-CLASS-NAME=org.postgresql.Driver
- SPRING_JPA_DATABASE_PLATFORM=org.hibernate.dialect.PostgreSQLDialect
postgres:
build:
context: docker/database
image: hub/my-db
container_name: my-db
networks:
- my-app-network
deploy:
replicas: 1
resources:
limits:
cpus: '0.5'
memory: 2048M
environment:
- POSTGRES_USER=dbpass
- POSTGRES_PASSWORD=dbpass
- POSTGRES_DB=my-db
networks:
my-app-network:
name: my-app-network
x-aws-cloudformation:
Resources:
MyWebAppTCP80TargetGroup:
Properties:
HealthCheckPath: /actuator/health
Matcher:
HttpCode: 200-499
MyWebAppTCP80Listener:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
Protocol: HTTP
Port: 80
LoadBalancerArn: xxxxx
DefaultActions:
- Type: redirect
RedirectConfig:
Port: 443
Host: "#{host}"
Path: "/#{path}"
Query: "#{query}"
Protocol: HTTPS
StatusCode: HTTP_301
MyWebAppTCP443Listener:
Type: AWS::ElasticLoadBalancingV2::Listener
Properties:
Protocol: HTTPS
Port: 443
LoadBalancerArn: xxxxxxxxx
Certificates:
- CertificateArn: "xxxxxxxxxx"
DefaultActions:
- Type: forward
ForwardConfig:
TargetGroups:
- TargetGroupArn:
Ref: MyWebAppTCP80TargetGroup
MyWebAppTCP80RedirectRule:
Type: AWS::ElasticLoadBalancingV2::ListenerRule
Properties:
ListenerArn:
Ref: MyWebAppTCP80Listener
Priority: 1
Conditions:
- Field: host-header
HostHeaderConfig:
Values:
- "*.my-app.com"
- "www.my-app.com"
- "my-app.com"
Actions:
- Type: redirect
RedirectConfig:
Host: "#{host}"
Path: "/#{path}"
Query: "#{query}"
Port: 443
Protocol: HTTPS
StatusCode: HTTP_301

Allow IPs with TCP Listener using RBAC (Envoy)

I am trying to achieve the following with Envoy:
Allow TCP traffic to a Postgres service with RBAC rules to allow only a few IPs.
This is my listener setup.
- name: listener_postgres
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 54322
filter_chains:
filters:
- name: envoy.filters.network.rbac
config:
stat_prefix: rbac_postgres
rules:
action: ALLOW
policies:
"allow":
permissions:
- any: true
principals:
- source_ip:
address_prefix: XX.XX.XX.XX
prefix_len: 32
- source_ip:
address_prefix: XX.XX.XX.XX
prefix_len: 32
- source_ip:
address_prefix: XX.XX.XX.XX
prefix_len: 32
- name: envoy.tcp_proxy
config:
stat_prefix: tcp_postgres
cluster: database_service
I can confirm that the service is setup correctly because I can remove the RBAC rules and I can connect successfully.
When the RBAC rules are added I can not connect to the Postgres database.
But for some reason nothing seems to work, I have also tried remote_ip and direct_remote_ip in place of source_ip.
Am I doing something wrong?
Thanks
Hey I ran into the same issue and this is the configuration worked for me.
I used remote_ip attribute.
Also, check the updated filters names
- name: listener_postgres
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 54322
filter_chains:
filters:
- name: envoy_rbac
config:
stat_prefix: rbac_postgres
rules:
action: ALLOW
policies:
"allow":
permissions:
- any: true
principals:
- remote_ip:
address_prefix: XX.XX.XX.XX
prefix_len: 32
- name: envoy_tcp_proxy
config:
stat_prefix: tcp_postgres
cluster: database_service
It seems that setting the attribute to 'remote_ip' as suggested by Rahul Pratap worked.
Here is a working example:
- name: listener_postgres
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 54322
filter_chains:
filters:
- name: envoy.filters.network.rbac
config:
stat_prefix: rbac_postgres
rules:
action: ALLOW
policies:
"allow":
permissions:
- any: true
principals:
- remote_ip:
address_prefix: XX.XX.XX.XX
prefix_len: 32
- name: envoy.tcp_proxy
config:
stat_prefix: tcp_postgres
cluster: database_service

Traefik 2 does not match 2 Entrypoints

i have simple question. I want to receive a response from the backend on two different ports. According to the documentation, this should be a simple task. I did so, but for some reason one part does not work. What am I missing?
http://my-service.com -> Works.
http://my-service.com:6666 -> Does not work.
I configured traefik:
Static-conf:
entryPoints:
web:
address: :80
websecure:
address: :443
traefik:
address: :9000
web2:
address: :6666
Dynamic-conf:
http:
routers:
StartPage:
entryPoints:
- web
- web2
service: MyService
rule: "Host(`my-service.com`)"
services:
MyService:
loadBalancer:
servers:
- url: "http://ca-service-svc.app/"
Traefik dashboard does not show any errors. I see my configuration.
Thanks for the help.

fargate failing on docker pull in private subnet

I am having trouble deploying a fargate cluster, and it is failing on the docker pull image with error "CannotPullContainerError". I am creating the stack with cloudformation, which is not optional, and it creates the full stack, but fails when trying to start the task based on the above error.
I have attached the cloudformation stack file which might highlight the problem, and I have doubled checked that the subnet has a route to nat(below). I also ssh'ed into an instance in the same subnet which was able to route externally. I am wondering if i have not correctly placed the pieces required i.e the service + loadbalancer are in the private subnet, or should i not be placing the internal lb in the same subnet???
This subnet is the one that currently has the placement but all 3 in the file have the same nat settings.
subnet routable (subnet-34b92250)
* 0.0.0.0/0 -> nat-05a00385366da527a
cheers in advance.
yaml cloudformaition script:
AWSTemplateFormatVersion: 2010-09-09
Description: Cloudformation stack for the new GRPC endpoints within existing vpc/subnets and using fargate
Parameters:
StackName:
Type: String
Default: cf-core-ci-grpc
Description: The name of the parent Fargate networking stack that you created. Necessary
vpcId:
Type: String
Default: vpc-0d499a68
Description: The name of the parent Fargate networking stack that you created. Necessary
Resources:
CoreGrcpInstanceSecurityGroupOpenWeb:
Type: 'AWS::EC2::SecurityGroup'
Properties:
GroupName: sgg-core-ci-grpc-ingress
GroupDescription: Allow http to client host
VpcId: !Ref vpcId
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: '80'
ToPort: '80'
CidrIp: 0.0.0.0/0
SecurityGroupEgress:
- IpProtocol: tcp
FromPort: '80'
ToPort: '80'
CidrIp: 0.0.0.0/0
LoadBalancer:
Type: 'AWS::ElasticLoadBalancingV2::LoadBalancer'
DependsOn:
- CoreGrcpInstanceSecurityGroupOpenWeb
Properties:
Name: lb-core-ci-int-grpc
Scheme: internal
Subnets:
# # pub
# - subnet-f13995a8
# - subnet-f13995a8
# - subnet-f13995a8
# pri
- subnet-34b92250
- subnet-82d85af4
- subnet-ca379b93
LoadBalancerAttributes:
- Key: idle_timeout.timeout_seconds
Value: '50'
SecurityGroups:
- !Ref CoreGrcpInstanceSecurityGroupOpenWeb
TargetGroup:
Type: 'AWS::ElasticLoadBalancingV2::TargetGroup'
DependsOn:
- LoadBalancer
Properties:
Name: tg-core-ci-grpc
Port: 3000
TargetType: ip
Protocol: HTTP
HealthCheckIntervalSeconds: 30
HealthCheckProtocol: HTTP
HealthCheckTimeoutSeconds: 10
HealthyThresholdCount: 4
Matcher:
HttpCode: '200'
TargetGroupAttributes:
- Key: deregistration_delay.timeout_seconds
Value: '20'
UnhealthyThresholdCount: 3
VpcId: !Ref vpcId
LoadBalancerListener:
Type: 'AWS::ElasticLoadBalancingV2::Listener'
DependsOn:
- TargetGroup
Properties:
DefaultActions:
- Type: forward
TargetGroupArn: !Ref TargetGroup
LoadBalancerArn: !Ref LoadBalancer
Port: 80
Protocol: HTTP
EcsCluster:
Type: 'AWS::ECS::Cluster'
DependsOn:
- LoadBalancerListener
Properties:
ClusterName: ecs-core-ci-grpc
EcsTaskRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Statement:
- Effect: Allow
Principal:
Service:
# - ecs.amazonaws.com
- ecs-tasks.amazonaws.com
Action:
- 'sts:AssumeRole'
Path: /
Policies:
- PolicyName: iam-policy-ecs-task-core-ci-grpc
PolicyDocument:
Statement:
- Effect: Allow
Action:
- 'ecr:**'
Resource: '*'
CoreGrcpTaskDefinition:
Type: 'AWS::ECS::TaskDefinition'
DependsOn:
- EcsCluster
- EcsTaskRole
Properties:
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
ExecutionRoleArn: !Ref EcsTaskRole
Cpu: '1024'
Memory: '2048'
ContainerDefinitions:
- Name: container-core-ci-grpc
Image: 'nginx:latest'
Cpu: '256'
Memory: '1024'
PortMappings:
- ContainerPort: '80'
HostPort: '80'
Essential: 'true'
EcsService:
Type: 'AWS::ECS::Service'
DependsOn:
- CoreGrcpTaskDefinition
Properties:
Cluster: !Ref EcsCluster
LaunchType: FARGATE
DesiredCount: '1'
DeploymentConfiguration:
MaximumPercent: 150
MinimumHealthyPercent: 0
LoadBalancers:
- ContainerName: container-core-ci-grpc
ContainerPort: '80'
TargetGroupArn: !Ref TargetGroup
NetworkConfiguration:
AwsvpcConfiguration:
AssignPublicIp: DISABLED
SecurityGroups:
- !Ref CoreGrcpInstanceSecurityGroupOpenWeb
Subnets:
- subnet-34b92250
- subnet-82d85af4
- subnet-ca379b93
TaskDefinition: !Ref CoreGrcpTaskDefinition
Unfortunately AWS Fargate only supports images hosted in ECR or public repositories in Docker Hub and does not support private repositories which are hosted in Docker Hub. For more info - https://forums.aws.amazon.com/thread.jspa?threadID=268415
Even we faced the same problem using AWS Fargate couple of months back. You have only two options right now:
Migrate your images to Amazon ECR.
Use AWS Batch with custom AMI, where the custom AMI is built with Docker Hub credentials in ECS config (which we are using right now).
Edit: As mentioned by Christopher Thomas in the comment, ECS fargate now supports pulling images from DockerHub Private repositories. More info on how to set it up can be found here.
Do define this policy in your ECR registry and attach the IAM role with your task.
{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "new statement",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::99999999999:role/ecsEventsRole"
},
"Action": [
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability",
"ecr:PutImage",
"ecr:InitiateLayerUpload",
"ecr:UploadLayerPart",
"ecr:CompleteLayerUpload"
]
}
]
}

Is there a platform/service agnostic definition for service handoff?

I'm curious if there is a spec for a service handoff definition. For example, if on a PaaS/IaaS a service is provisioned, end users need a hash of details of what the service is, where the endpoint can be reached, what port(s) are published, and what authentication is used. (Think HATEOAS ref_'s) I have a couple mock-ups of what one could look like:
object storage example
name: myobjstor
family: s3
about: https://aws.amazon.com/documentation/s3
zone: public
protocol:
spec: http
host: s3.mysite.com
port: 443
tls: true
authentication:
strategy: oauth2
username: someuser
password: somepassword
definition:
type: swagger
url: 'https://mysite/swagger.json'
openstack example
name: myostack
family: openstack-keystone_v2.1
about: http://developer.openstack.org/api-ref.html
zone: public
protocol:
spec: http
host: keystone.mysite.com
port: 443
tls: true
authentication:
strategy: oauth2
username: someuser
password: somepassword
definition:
type: swagger
url: 'https://mysite/swagger.json'
redis example
name: myredis
family: redis
about: http://redis.io/documentation
zone: public
protocol:
spec: redis
host: redis.mysite.com
port: 6379
options:
db: 0
nfs example
name: mynfs
family: nfs
about: http://nfs.sourceforge.net
zone: public
protocol:
spec: nfsv4
host: nfs.mysite.com
ports:
- 111
- 2049
Is there a standard like this that already exists?
I suggest you looking into the "service discovery"pattern.
There are several tools out there that make it easy to implement, but most of them describe services using key/value pairs e.g. see etcd but it looks like Consul adds a few fields that may be of use to you see here
Example Consul Definition
{
"service": {
"name": "redis",
"tags": ["master"],
"address": "127.0.0.1",
"port": 8000,
"enableTagOverride": false,
"checks": [
{
"script": "/usr/local/bin/check_redis.py",
"interval": "10s"
}
]
}
}