How to determine ipv6 CIDR block prefix in AWS Cloudformation when creating subnets on a VPC - aws-cloudformation

AWS generates the ipv6 CIDR block for VPCs so its not possible to determine ahead of time. The generated CIDR block looks something like: 2a05:d018:84c:c500::/56 and is always size 56.
When creating a subnet you have to specify a size 64 block using the full prefixed value. E.g. 2a05:d018:84c:c501::/64.
It's possible to look up the ipv6 CIDR blocks for a VPC in cloudformation, but this returns the full value, not just the prefix. To create a subnet we need to be able to append something 01::/64 to the prefix to create the 64 sized block for the subnet.
I've seen solutions that use a lambda function, but this greatly complicated the templates. I'd like to do this using just the built-in intrinsic functions available in the templates.
When deploying a VPC with ipv6 subnets in the same stack, how can you generate valid ipv6 CIDR blocks for the subnets?

Here is a one liner that does the same thing using the Fn::Cidr intrinsic function.
!Select [1, !Cidr [!Select [0, !GetAtt 'Vpc.Ipv6CidrBlocks'], 256, 64]]
For a given block 2a05:d018:84c:c500::/56 this will give you 2a05:d018:84c:c501::/64
Increment the first index to get the next block.
!Select [2, !Cidr [!Select [0, !GetAtt 'Vpc.Ipv6CidrBlocks'], 256, 64]]
will give you 2a05:d018:84c:c502::/64
Also here is a full minimal example including the crucial steps of using an AWS::EC2::VPCCidrBlock resource to attach the IPv6 block to the VPC and using the DependsOn property to make sure that the VPCCidrBlock is attached before the Subnet is created.
Resources:
Vpc:
Type: AWS::EC2::VPC
Properties:
CidrBlock: !Sub '10.255.0.0/16'
VpcCidrBlockIpv6:
Type: 'AWS::EC2::VPCCidrBlock'
Properties:
VpcId: !Ref 'Vpc'
AmazonProvidedIpv6CidrBlock: true
PrivateSubnet:
Type: AWS::EC2::Subnet
DependsOn: VpcCidrBlockIpv6 # Wait for IPv6 CIDR to be attached to VPC before creating subnet
Properties:
AvailabilityZone: !Select [ 0, !GetAZs '' ]
VpcId: !Ref 'Vpc'
AssignIpv6AddressOnCreation: true
CidrBlock: !Sub '10.255.0.0/20'
Ipv6CidrBlock: !Select [1, !Cidr [!Select [0, !GetAtt 'Vpc.Ipv6CidrBlocks'], 256, 64]]

AWS VPC Service allows us to create a VPC and associate an Amazon provided /56 IPv6 CIDR block [We cannot provide our own CIDR].
Each subnet can have an IP block of /64. Which means that there can be theoretically 2^(64-56)subnets which is 2^8 = 256 subnets.
For calculating the Subnets for the IPv6 CIDR block, Cloudformation provides an intrinsic function !Cidr for us to do just that.
Syntax of the CIDR Block goes as follows:
!Cidr [ ipBlock, count, cidrBits ]
where
count --> The number of CIDRs to generate. Valid range is between 1 and 256.
cidrBits -->The number of subnet bits for the CIDR. For example, specifying a value "8" for this parameter will create a CIDR with a mask of "/24".
So our statement Becomes:
!Select [0, !Cidr [!Select [0, !GetAtt 'Vpc.Ipv6CidrBlocks'], 256, 64]]
So a sample template to create a VPC with 2 subnets with IPv6 attached to them looks like this:
AWSTemplateFormatVersion: "2010-09-09"
Resources:
Vpc:
Type: AWS::EC2::VPC
Properties:
CidrBlock: !Sub '10.255.0.0/16'
VpcCidrBlockIpv6:
Type: 'AWS::EC2::VPCCidrBlock'
Properties:
VpcId: !Ref 'Vpc'
AmazonProvidedIpv6CidrBlock: true
Subnet1:
Type: AWS::EC2::Subnet
Properties:
AvailabilityZone: !Select [ 0, !GetAZs '' ]
VpcId: !Ref 'Vpc'
AssignIpv6AddressOnCreation: true
CidrBlock: !Sub '${PrivateSubnetCIDR1}'
Ipv6CidrBlock: !Select [0, !Cidr [!Select [0, !GetAtt 'Vpc.Ipv6CidrBlocks'], 256, 64]]
Subnet2:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref 'Vpc'
AssignIpv6AddressOnCreation: true
CidrBlock: !Sub '${PrivateSubnetCIDR2}'
Ipv6CidrBlock: !Select [1, !Cidr [!Select [0, !GetAtt 'Vpc.Ipv6CidrBlocks'], 256, 64]]

Here's a way to calculate the first subnet in YAML:
Fn::Sub:
- "${VpcPart}${SubnetPart}"
- SubnetPart: 01::/64
VpcPart: !Select [0, !Split ['00::/56', !Select [0,!GetAtt YourVpc.Ipv6CidrBlocks]]]

You can determine the prefix by using a combination of Fn::Split (on 00::/56) and Fn::Select to get the prefix. Then you can append your own value to create the subnet CIDR blocks using Fn::Join. The following example assumes you have a VPC with one or more Ipv6 CIDR blocks associated with it.
Use this value for the Ipv6CidrBlock property on the subnet.
{
"Fn::Join": [
"",
[
{
"Fn::Select": [
0,
{
"Fn::Split": [
"00::/56",
{
"Fn::Select": [
0,
{
"Fn::GetAtt": [
"Vpc",
"Ipv6CidrBlocks"
]
}
]
}
]
}
]
},
"01::/64"
]
]
}

Related

Cloudformation conditionals Yaml

I have to create an auto scaling group in two regions and the only difference between the two are the subnets. us-east-1 has 1 subnet whereas us-east-2 has two. How can I use the condition to call the subnet value from region map.
Mappings:
RegionMap:
us-east-1:
AMI: "ami-066f487d3b6819b0d"
Subnet1: "subnet-0e6f12f64042ea5b1"
us-east-2:
AMI: "ami-0aef5e0adcbc7cc0f"
Subnet1: "subnet-0e6f12f64042ea5b1"
Subnet2: "subnet-0bc661bb8d98f3f03"
Conditions:
region: !Equals [!Ref us-east-2, Subnet2]
autoscaling:
Type: AWS::AutoScaling::AutoScalingGroup
Properties:
AutoScalingGroupName: asg1
VPCZoneIdentifier:
- !FindInMap [RegionMap, !Ref "AWS::Region", Subnet1]
- If regions = us-east-2 then !FindInMap [RegionMap, !Ref "AWS::Region", Subnet2] # This is what I need to figure out
I couldn't find any examples of this. Has anyone used a region map and used conditionals with it?

can slot take entity values without a action function or forms in RASA?

is it possible to pass values in the entity to slots without form or writing an action function?
nlu.yml
nlu:
- intent: place_order
examples: |
- wanna [large](size) shoes for husky
- need a [small](size) [green](color) boots for pupps
- have [blue](color) socks
- would like to place an order
- lookup: size
examples: |
- small
-medium
-large
- synonym: small
examples: |
- small
- s
- tiny
- synonym: large
examples: |
- large
- l
- big
- lookup: color
examples: |
- white
- red
- green
domain.yml
version: "2.0"
intents:
- greet
- goodbye
- affirm
- deny
- mood_great
- mood_unhappy
- bot_challenge
- place_order
entities:
- size
- color
slot:
size:
type: text
color:
type: text
responses:
utter_greet:
- text: "Hey! can I assist you ?"
utter_order_list:
- text : "your order is {size} [color} boots. right?"
stories.yml
version: "2.0"
stories:
- story: place_order
steps:
- intent: greet
- action: utter_greet
- intent: place_order
- action: utter_order_list
debug output: it recognize entity , but the value is not passed to slot
Hey! can I assist you ?
Your input -> I would like to place an order for large blue shoes for my puppy
Received user message 'I would like to place an order for large blue shoes for my puppy' with intent '{'id': -2557752933293854887, 'name': 'place_order', 'confidence': 0.9996021389961243}' and entities '[{'entity': 'size', 'start': 35, 'end': 40, 'confidence_entity': 0.9921159148216248, 'value': 'large', 'extractor': 'DIETClassifier'}, {'entity': 'color', 'start': 41, 'end': 45, 'confidence_entity': 0.9969255328178406, 'value': 'blue', 'extractor': 'DIETClassifier'}]'
Failed to replace placeholders in response 'your order is {size} [color} boots. right?'. Tried to replace 'size' but could not find a value for it. There is no slot with this name nor did you pass the value explicitly when calling the response. Return response without filling the response
"slot" is an unknown keyword. you should write "slots" instead of "slot" in the domain file and it will work.

How to define a CloudWatch Alarm on the sum of two metrics with CloudFormation?

I need to trigger an alarm when the sum of the same metric (ApproximateNumberOfMessagesVisible) on two different queues exceed the value of 100
In September '17, this answer stated that the only way to do it was with a Lambda function getting the two values and summing them up via CloudWatch API.
At writing time, Feb. '19, it is possible to use "Metric Math", so there is no need to have a lambda function or an EC2 instance. Is it possible to use Metric Math to define an Alarm directly in CloudFormation ?
It is actually possible to implement the Alarm logic directly in CloudFormation.
Assuming to have two Scaling Policies ECSScaleUp and ECSScaleDown, the alarm definition will look like:
ECSWorkerSQSCumulativeAlarm:
Type: AWS::CloudWatch::Alarm
Properties:
AlarmName: !Join ['-', [!Ref 'MyService', 'SQSCumulativeAlarm']]
AlarmDescription: "Trigger ECS Service Scaling based on TWO SQS queues"
Metrics:
- Id: e1
Expression: "fq + sq"
Label: "Sum of the two Metrics"
- Id: fq
MetricStat:
Metric:
MetricName: ApproximateNumberOfMessagesVisible
Namespace: AWS/SQS
Dimensions:
- Name: QueueName
Value: !GetAtt [ FirstQueue, QueueName]
Period: 60
Stat: Average
Unit: Count
ReturnData: false
- Id: sq
MetricStat:
Metric:
MetricName: ApproximateNumberOfMessagesVisible
Namespace: AWS/SQS
Dimensions:
- Name: QueueName
Value: !GetAtt [ SecondQueue, QueueName]
Period: 60
Stat: Average
Unit: Count
ReturnData: false
EvaluationPeriods: 2
Threshold: 100
ComparisonOperator: GreaterThanThreshold
AlarmActions:
- !Ref ECSScaleUp
- !Ref ECSScaleDown
OKActions:
- !Ref ECSScaleUp
- !Ref ECSScaleDown

Is there a way to parse the EMR MasterPublicDNS in Amazon Cloudformation?

Is there a way to parse the EMR MasterPublicDNS in Cloudformation? I don't see a replace function in Cloudformation.
ip-100-112-10-21.aws.internal
TO
100.112.10.21
Outputs:
IPAddress:
Description: IP address of the EMR clusters
Value: !GetAtt
- EMRCluster
- MasterPublicDNS
can I reference the output value in same script?
I need to use the formatted IP to set resourcerecords- or do I have to use
Type: AWS::Route53::RecordSetGroup
ResourceRecords: !Join [".",
[
!Select [1, !Split ['-', !GetAtt EMRCluster.MasterPublicDNS]],
!Select [2, !Split ['-', !GetAtt EMRCluster.MasterPublicDNS]],
!Select [3, !Split ['-', !GetAtt EMRCluster.MasterPublicDNS]],
!Select [0,
!Split ['.', !Select [4, !Split ['-', !GetAtt EMRCluster.MasterPublicDNS]]]]
]
]
gives error - Value of property ResourceRecords must be of type List of String
or
ResourceRecords: !ref IPAddress.value
If the format is always like that, you could combine Split, Select and Join Cloudformation intrinsic functions to achieve it:
Outputs:
IPAddress:
Description: IP address of the EMR clusters
Value: !Join ['.',
[
!Select: [1, !Split: ['-', !GetAtt EMRCluster.MasterPublicDNS]],
!Select: [2, !Split: ['-', !GetAtt EMRCluster.MasterPublicDNS]],
!Select: [3, !Split: ['-', !GetAtt EMRCluster.MasterPublicDNS]],
!Select: [0,
!Split: ['.', !Select: [4, !Split: ['-', !GetAtt EMRCluster.MasterPublicDNS]]]]
]
]
I know it makes you cringe, but that's the way to go in Cloudformation.
Alternatively, you could write a Cloudformation macro to do this for you.

Error : H5LTfind_dataset(file_id, dataset_name_) Failed to find HDF5 dataset label

I want to use HDF5 file to input my data and labels in my CNN.
I created the hdf5 file with matlab.
Here is my code:
h5create(['uNetDataSet.h5'],'/home/alexandra/Documents/my-u-net/warwick_dataset/Warwick_Dataset/train/image',[522 775 3 numFrames]);
h5create(['uNetDataSet.h5'],'/home/alexandra/Documents/my-u-net/warwick_dataset/Warwick_Dataset/train/anno',[522 775 3 numFrames]);
h5create(['uNetDataSet.h5'],'/home/alexandra/Documents/my-u-net/warwick_dataset/Warwick_Dataset/label',[1 numFrames]);`
h5write(['uNetDataSet.h5'],'/home/alexandra/Documents/my-u-net/warwick_dataset/Warwick_Dataset/train/image',images);
h5write(['uNetDataSet.h5'],'/home/alexandra/Documents/my-u-net/warwick_dataset/Warwick_Dataset/train/anno',anno);
h5write(['uNetDataSet.h5'],'/home/alexandra/Documents/my-u-net/warwick_dataset/Warwick_Dataset/label',label);`
Where image, anno are 4D unit8 and label is a 1x85 unit16 vector.
When I display my .h5 file I got this:
HDF5 uNetDataSet.h5
Group '/'
Group '/home'
Group '/home/alexandra'
Group '/home/alexandra/Documents'
Group '/home/alexandra/Documents/my-u-net'
Group '/home/alexandra/Documents/my-u-net/warwick_dataset'
Group '/home/alexandra/Documents/my-u-net/warwick_dataset/Warwick_Dataset'
Dataset 'label'
Size: 1x85
MaxSize: 1x85
Datatype: H5T_IEEE_F64LE (double)
ChunkSize: []
Filters: none
FillValue: 0.000000
Group '/home/alexandra/Documents/my-u-net/warwick_dataset/Warwick_Dataset/train'
Dataset 'anno'
Size: 522x775x3x85
MaxSize: 522x775x3x85
Datatype: H5T_IEEE_F64LE (double)
ChunkSize: []
Filters: none
FillValue: 0.000000
Dataset 'image'
Size: 522x775x3x85
MaxSize: 522x775x3x85
Datatype: H5T_IEEE_F64LE (double)
ChunkSize: []
Filters: none
FillValue: 0.000000`
When I read the label dataset with h5read it works.
But when I try to train my network I got this error:
I0713 09:47:18.620510 4278 layer_factory.hpp:77] Creating layer loadMydata
I0713 09:47:18.620535 4278 net.cpp:91] Creating Layer loadMydata
I0713 09:47:18.620550 4278 net.cpp:399] loadMydata -> label
I0713 09:47:18.620580 4278 net.cpp:399] loadMydata -> anno
I0713 09:47:18.620600 4278 net.cpp:399] loadMydata -> image
I0713 09:47:18.620622 4278 hdf5_data_layer.cpp:79] Loading list of HDF5 filenames from: /home/alexandra/Documents/my-u-net/my_data.txt
I0713 09:47:18.620656 4278 hdf5_data_layer.cpp:93] Number of HDF5 files: 1
F0713 09:47:18.621317 4278 hdf5.cpp:14] Check failed: H5LTfind_dataset(file_id, dataset_name_) Failed to find HDF5 dataset label
*** Check failure stack trace: ***
# 0x7f2edf557daa (unknown)
# 0x7f2edf557ce4 (unknown)
# 0x7f2edf5576e6 (unknown)
# 0x7f2edf55a687 (unknown)
# 0x7f2edf908597 caffe::hdf5_load_nd_dataset_helper<>()
# 0x7f2edf907365 caffe::hdf5_load_nd_dataset<>()
# 0x7f2edf9579fe caffe::HDF5DataLayer<>::LoadHDF5FileData()
# 0x7f2edf956818 caffe::HDF5DataLayer<>::LayerSetUp()
# 0x7f2edf94fcbc caffe::Net<>::Init()
# 0x7f2edf950b45 caffe::Net<>::Net()
# 0x7f2edf91d08a caffe::Solver<>::InitTrainNet()
# 0x7f2edf91e18c caffe::Solver<>::Init()
# 0x7f2edf91e4ba caffe::Solver<>::Solver()
# 0x7f2edf930ed3 caffe::Creator_SGDSolver<>()
# 0x40e67e caffe::SolverRegistry<>::CreateSolver()
# 0x40794b train()
# 0x40590c main
# 0x7f2ede865f45 (unknown)
# 0x406041 (unknown)
# (nil) (unknown)
Aborted (core dumped)
In my .prototxt file :
layer {
top: 'label'
top:'anno'
top: 'image'
name: 'loadMydata'
type: "HDF5Data"
hdf5_data_param { source: '/home/alexandra/Documents/my-u-net/my_data.txt' batch_size: 1 }
include: { phase: TRAIN }
}
I don't know where I did something wrong, if anyone could help me it would be great !
your hdf5 file 'uNetDataSet.h5' does not have label in it.
What you have instead is '/home/alexandra/Documents/my-u-net/warwick_dataset/Warwick_Dataset/label' - I hope you can spot the difference.
Try creating the dataset with
h5create(['uNetDataSet.h5'],'/image',[522 775 3 numFrames]);
h5create(['uNetDataSet.h5'],'/anno',[522 775 3 numFrames]);
h5create(['uNetDataSet.h5'],'/label',[1 numFrames]);
Please see this answer for more details. Also note that you might need to permute the input data before saving it to hdf5 using matlab.