I have to create an auto scaling group in two regions and the only difference between the two are the subnets. us-east-1 has 1 subnet whereas us-east-2 has two. How can I use the condition to call the subnet value from region map.
Mappings:
RegionMap:
us-east-1:
AMI: "ami-066f487d3b6819b0d"
Subnet1: "subnet-0e6f12f64042ea5b1"
us-east-2:
AMI: "ami-0aef5e0adcbc7cc0f"
Subnet1: "subnet-0e6f12f64042ea5b1"
Subnet2: "subnet-0bc661bb8d98f3f03"
Conditions:
region: !Equals [!Ref us-east-2, Subnet2]
autoscaling:
Type: AWS::AutoScaling::AutoScalingGroup
Properties:
AutoScalingGroupName: asg1
VPCZoneIdentifier:
- !FindInMap [RegionMap, !Ref "AWS::Region", Subnet1]
- If regions = us-east-2 then !FindInMap [RegionMap, !Ref "AWS::Region", Subnet2] # This is what I need to figure out
I couldn't find any examples of this. Has anyone used a region map and used conditionals with it?
Related
I have been trying to find solutions to this but no luck. all services work internally. I am able to access grafana from browser with tls enabled but I am not able to reach loki port in any way(browser/postman etc.) but I can.
I can access to loki api with curl localy if I open port on for the service. but as I understand you need to expose ports from traefik to do that.
My compose file:
version: "3"
services:
grafana:
labels:
- "traefik.http.routers.grafana.entryPoints=port80"
- "traefik.http.routers.grafana.rule=host(`${DOMAIN}`)"
- "traefik.http.middlewares.grafana-redirect.redirectScheme.scheme=https"
- "traefik.http.middlewares.grafana-redirect.redirectScheme.permanent=true"
- "traefik.http.routers.grafana.middlewares=grafana-redirect"
# SSL endpoint
- "traefik.http.routers.grafana-ssl.entryPoints=port443"
- "traefik.http.routers.grafana-ssl.rule=host(`${DOMAIN}`)"
- "traefik.http.routers.grafana-ssl.tls=true"
- "traefik.http.routers.grafana-ssl.tls.certResolver=le-ssl"
- "traefik.http.routers.grafana-ssl.service=grafana-ssl"
- "traefik.http.services.grafana-ssl.loadBalancer.server.port=3000"
image: grafana/grafana:latest # or probably any other version
volumes:
- grafana-data:/var/lib/grafana
environment:
- GF_SERVER_ROOT_URL=https://${DOMAIN}
- GF_SERVER_DOMAIN=${DOMAIN}
- GF_USERS_ALLOW_SIGN_UP=false
- GF_SECURITY_ADMIN_USER=${GRAFANAUSER}
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANAPASS}
networks:
- traefik-net
loki:
image: grafana/loki
labels:
- "traefik.http.routers.loki-ssl.entryPoints=port3100"
- "traefik.http.routers.loki-ssl.rule=host(`${DOMAIN}`)"
- "traefik.http.routers.loki-ssl.tls=true"
- "traefik.http.routers.loki-ssl.tls.certResolver=le-ssl"
- "traefik.http.routers.loki-ssl.service=loki-ssl"
- "traefik.http.services.loki-ssl.loadBalancer.server.port=3100"
command: -config.file=/etc/loki/config.yaml
volumes:
- ./loki/config.yml:/etc/loki/config.yaml
- loki:/data/loki
networks:
- traefik-net
promtail:
image: grafana/promtail:2.3.0
volumes:
- /var/log:/var/log
- ./promtail:/etc/promtail-config/
command: -config.file=/etc/promtail-config/promtail.yml
networks:
- traefik-net
influx:
image: influxdb:1.7 # or any other recent version
labels:
# SSL endpoint
- "traefik.http.routers.influx-ssl.entryPoints=port8086"
- "traefik.http.routers.influx-ssl.rule=host(`${DOMAIN}`)"
- "traefik.http.routers.influx-ssl.tls=true"
- "traefik.http.routers.influx-ssl.tls.certResolver=le-ssl"
- "traefik.http.routers.influx-ssl.service=influx-ssl"
- "traefik.http.services.influx-ssl.loadBalancer.server.port=8086"
restart: always
volumes:
- influx-data:/var/lib/influxdb
environment:
- INFLUXDB_DB=grafana # set any other to create database on initialization
- INFLUXDB_HTTP_ENABLED=true
- INFLUXDB_HTTP_AUTH_ENABLED=true
- INFLUXDB_ADMIN_USER=&{DB_USER}
- INFLUXDB_ADMIN_PASSWORD=&{DB_PASS}
networks:
- traefik-net
traefik:
image: traefik:v2.9.1
ports:
- "80:80"
- "443:443"
- "3100:3100"
# expose port below only if you need access to the Traefik API
- "8080:8080"
command:
# - "--log.level=DEBUG"
- "--api=true"
- "--api.dashboard=true"
- "--providers.docker=true"
- "--entryPoints.port443.address=:443"
- "--entryPoints.port80.address=:80"
- "--entryPoints.port8086.address=:8086"
- "--entryPoints.port3100.address=:3100"
- "--certificatesResolvers.le-ssl.acme.tlsChallenge=true"
- "--certificatesResolvers.le-ssl.acme.email=${TLS_MAIL}"
- "--certificatesResolvers.le-ssl.acme.storage=/letsencrypt/acme.json"
volumes:
- traefik-data:/letsencrypt/
- /var/run/docker.sock:/var/run/docker.sock
networks:
- traefik-net
volumes:
traefik-data:
grafana-data:
influx-data:
loki:
networks:
traefik-net:
loki conf
# (default configuration)
auth_enabled: false
server:
http_listen_port: 3100
ingester:
lifecycler:
address: 127.0.0.1
ring:
kvstore:
store: inmemory
replication_factor: 1
final_sleep: 0s
chunk_idle_period: 1h # Any chunk not receiving new logs in this time will be flushed
max_chunk_age: 1h # All chunks will be flushed when they hit this age, default is 1h
chunk_target_size: 1048576 # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
chunk_retain_period: 30s # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
max_transfer_retries: 0 # Chunk transfers disabled
wal:
enabled: true
dir: /loki/wal
common:
ring:
instance_addr: 0.0.0.0
kvstore:
store: inmemory
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
storage_config:
boltdb_shipper:
active_index_directory: /loki/boltdb-shipper-active
cache_location: /loki/boltdb-shipper-cache
cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space
shared_store: filesystem
filesystem:
directory: /loki/chunks
compactor:
working_directory: /loki/boltdb-shipper-compactor
shared_store: filesystem
limits_config:
reject_old_samples: true
reject_old_samples_max_age: 168h
ingestion_burst_size_mb: 16
ingestion_rate_mb: 16
chunk_store_config:
max_look_back_period: 0s
table_manager:
retention_deletes_enabled: false
retention_period: 0s
ruler:
storage:
type: local
local:
directory: /loki/rules
rule_path: /loki/rules-temp
alertmanager_url: localhost
ring:
kvstore:
store: inmemory
enable_api: true
is it possible in Kubernetes values to refer to value in the same file? I'm doing a for loop and most env vars are fine but this one depends on some other val and need to duplicate/refer to it somehow inside values;
image:
repository: nginx
tag: stable
someCustomVal:
- name: x
value: xx
- name: y
value: yy
- name: z
value: {{ .Values.image.tag }}
btw above config doesn't work but looking for equivalent; I could just do the z value outside of for loop in the deployment but that wouldn't look nice so looking for alternative of referecnce
This isn't Kubernetes-specific, you can do this with YAML anchors:
$ cat example.yaml
image:
repository: nginx
tag: &imagetag stable
someCustomVal:
- name: x
value: xx
- name: y
value: yy
- name: z
value: *imagetag
$ ruby -ryaml -rpp -e'pp YAML.load_file("example.yaml")'
{"image"=>{"repository"=>"nginx", "tag"=>"stable"},
"someCustomVal"=>
[{"name"=>"x", "value"=>"xx"},
{"name"=>"y", "value"=>"yy"},
{"name"=>"z", "value"=>"stable"}]}
I need to trigger an alarm when the sum of the same metric (ApproximateNumberOfMessagesVisible) on two different queues exceed the value of 100
In September '17, this answer stated that the only way to do it was with a Lambda function getting the two values and summing them up via CloudWatch API.
At writing time, Feb. '19, it is possible to use "Metric Math", so there is no need to have a lambda function or an EC2 instance. Is it possible to use Metric Math to define an Alarm directly in CloudFormation ?
It is actually possible to implement the Alarm logic directly in CloudFormation.
Assuming to have two Scaling Policies ECSScaleUp and ECSScaleDown, the alarm definition will look like:
ECSWorkerSQSCumulativeAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: !Join ['-', [!Ref 'MyService', 'SQSCumulativeAlarm']]
AlarmDescription: "Trigger ECS Service Scaling based on TWO SQS queues"
Metrics:
- Id: e1
Expression: "fq + sq"
Label: "Sum of the two Metrics"
- Id: fq
MetricStat:
Metric:
MetricName: ApproximateNumberOfMessagesVisible
Namespace: AWS/SQS
Dimensions:
- Name: QueueName
Value: !GetAtt [ FirstQueue, QueueName]
Period: 60
Stat: Average
Unit: Count
ReturnData: false
- Id: sq
MetricStat:
Metric:
MetricName: ApproximateNumberOfMessagesVisible
Namespace: AWS/SQS
Dimensions:
- Name: QueueName
Value: !GetAtt [ SecondQueue, QueueName]
Period: 60
Stat: Average
Unit: Count
ReturnData: false
EvaluationPeriods: 2
Threshold: 100
ComparisonOperator: GreaterThanThreshold
AlarmActions:
- !Ref ECSScaleUp
- !Ref ECSScaleDown
OKActions:
- !Ref ECSScaleUp
- !Ref ECSScaleDown
Is there a way to parse the EMR MasterPublicDNS in Cloudformation? I don't see a replace function in Cloudformation.
ip-100-112-10-21.aws.internal
TO
100.112.10.21
Outputs:
IPAddress:
Description: IP address of the EMR clusters
Value: !GetAtt
- EMRCluster
- MasterPublicDNS
can I reference the output value in same script?
I need to use the formatted IP to set resourcerecords- or do I have to use
Type: AWS::Route53::RecordSetGroup
ResourceRecords: !Join [".",
[
!Select [1, !Split ['-', !GetAtt EMRCluster.MasterPublicDNS]],
!Select [2, !Split ['-', !GetAtt EMRCluster.MasterPublicDNS]],
!Select [3, !Split ['-', !GetAtt EMRCluster.MasterPublicDNS]],
!Select [0,
!Split ['.', !Select [4, !Split ['-', !GetAtt EMRCluster.MasterPublicDNS]]]]
]
]
gives error - Value of property ResourceRecords must be of type List of String
or
ResourceRecords: !ref IPAddress.value
If the format is always like that, you could combine Split, Select and Join Cloudformation intrinsic functions to achieve it:
Outputs:
IPAddress:
Description: IP address of the EMR clusters
Value: !Join ['.',
[
!Select: [1, !Split: ['-', !GetAtt EMRCluster.MasterPublicDNS]],
!Select: [2, !Split: ['-', !GetAtt EMRCluster.MasterPublicDNS]],
!Select: [3, !Split: ['-', !GetAtt EMRCluster.MasterPublicDNS]],
!Select: [0,
!Split: ['.', !Select: [4, !Split: ['-', !GetAtt EMRCluster.MasterPublicDNS]]]]
]
]
I know it makes you cringe, but that's the way to go in Cloudformation.
Alternatively, you could write a Cloudformation macro to do this for you.
AWS generates the ipv6 CIDR block for VPCs so its not possible to determine ahead of time. The generated CIDR block looks something like: 2a05:d018:84c:c500::/56 and is always size 56.
When creating a subnet you have to specify a size 64 block using the full prefixed value. E.g. 2a05:d018:84c:c501::/64.
It's possible to look up the ipv6 CIDR blocks for a VPC in cloudformation, but this returns the full value, not just the prefix. To create a subnet we need to be able to append something 01::/64 to the prefix to create the 64 sized block for the subnet.
I've seen solutions that use a lambda function, but this greatly complicated the templates. I'd like to do this using just the built-in intrinsic functions available in the templates.
When deploying a VPC with ipv6 subnets in the same stack, how can you generate valid ipv6 CIDR blocks for the subnets?
Here is a one liner that does the same thing using the Fn::Cidr intrinsic function.
!Select [1, !Cidr [!Select [0, !GetAtt 'Vpc.Ipv6CidrBlocks'], 256, 64]]
For a given block 2a05:d018:84c:c500::/56 this will give you 2a05:d018:84c:c501::/64
Increment the first index to get the next block.
!Select [2, !Cidr [!Select [0, !GetAtt 'Vpc.Ipv6CidrBlocks'], 256, 64]]
will give you 2a05:d018:84c:c502::/64
Also here is a full minimal example including the crucial steps of using an AWS::EC2::VPCCidrBlock resource to attach the IPv6 block to the VPC and using the DependsOn property to make sure that the VPCCidrBlock is attached before the Subnet is created.
Resources:
Vpc:
Type: AWS::EC2::VPC
Properties:
CidrBlock: !Sub '10.255.0.0/16'
VpcCidrBlockIpv6:
Type: 'AWS::EC2::VPCCidrBlock'
Properties:
VpcId: !Ref 'Vpc'
AmazonProvidedIpv6CidrBlock: true
PrivateSubnet:
Type: AWS::EC2::Subnet
DependsOn: VpcCidrBlockIpv6 # Wait for IPv6 CIDR to be attached to VPC before creating subnet
Properties:
AvailabilityZone: !Select [ 0, !GetAZs '' ]
VpcId: !Ref 'Vpc'
AssignIpv6AddressOnCreation: true
CidrBlock: !Sub '10.255.0.0/20'
Ipv6CidrBlock: !Select [1, !Cidr [!Select [0, !GetAtt 'Vpc.Ipv6CidrBlocks'], 256, 64]]
AWS VPC Service allows us to create a VPC and associate an Amazon provided /56 IPv6 CIDR block [We cannot provide our own CIDR].
Each subnet can have an IP block of /64. Which means that there can be theoretically 2^(64-56)subnets which is 2^8 = 256 subnets.
For calculating the Subnets for the IPv6 CIDR block, Cloudformation provides an intrinsic function !Cidr for us to do just that.
Syntax of the CIDR Block goes as follows:
!Cidr [ ipBlock, count, cidrBits ]
where
count --> The number of CIDRs to generate. Valid range is between 1 and 256.
cidrBits -->The number of subnet bits for the CIDR. For example, specifying a value "8" for this parameter will create a CIDR with a mask of "/24".
So our statement Becomes:
!Select [0, !Cidr [!Select [0, !GetAtt 'Vpc.Ipv6CidrBlocks'], 256, 64]]
So a sample template to create a VPC with 2 subnets with IPv6 attached to them looks like this:
AWSTemplateFormatVersion: "2010-09-09"
Resources:
Vpc:
Type: AWS::EC2::VPC
Properties:
CidrBlock: !Sub '10.255.0.0/16'
VpcCidrBlockIpv6:
Type: 'AWS::EC2::VPCCidrBlock'
Properties:
VpcId: !Ref 'Vpc'
AmazonProvidedIpv6CidrBlock: true
Subnet1:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone: !Select [ 0, !GetAZs '' ]
VpcId: !Ref 'Vpc'
AssignIpv6AddressOnCreation: true
CidrBlock: !Sub '${PrivateSubnetCIDR1}'
Ipv6CidrBlock: !Select [0, !Cidr [!Select [0, !GetAtt 'Vpc.Ipv6CidrBlocks'], 256, 64]]
Subnet2:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref 'Vpc'
AssignIpv6AddressOnCreation: true
CidrBlock: !Sub '${PrivateSubnetCIDR2}'
Ipv6CidrBlock: !Select [1, !Cidr [!Select [0, !GetAtt 'Vpc.Ipv6CidrBlocks'], 256, 64]]
Here's a way to calculate the first subnet in YAML:
Fn::Sub:
- "${VpcPart}${SubnetPart}"
- SubnetPart: 01::/64
VpcPart: !Select [0, !Split ['00::/56', !Select [0,!GetAtt YourVpc.Ipv6CidrBlocks]]]
You can determine the prefix by using a combination of Fn::Split (on 00::/56) and Fn::Select to get the prefix. Then you can append your own value to create the subnet CIDR blocks using Fn::Join. The following example assumes you have a VPC with one or more Ipv6 CIDR blocks associated with it.
Use this value for the Ipv6CidrBlock property on the subnet.
{
"Fn::Join": [
"",
[
{
"Fn::Select": [
0,
{
"Fn::Split": [
"00::/56",
{
"Fn::Select": [
0,
{
"Fn::GetAtt": [
"Vpc",
"Ipv6CidrBlocks"
]
}
]
}
]
}
]
},
"01::/64"
]
]
}