I've declared a kubernetes deployment like:
const ledgerDeployment = new k8s.extensions.v1beta1.Deployment("ledger", {
spec: {
template: {
metadata: {
labels: {name: "ledger"},
name: "ledger",
// namespace: namespace,
},
spec: {
containers: [
...
],
volumes: [
{
emptyDir: {},
name: "gunicorn-socket-dir"
}
]
}
}
}
});
Later on in my index.ts I want to conditionally modify the volumes of the deployment. I think this is a quirk of pulumi I haven't wrapped my head around but here's my current attempt:
if(myCondition) {
ledgerDeployment.spec.template.spec.volumes.apply(volumes =>
volumes.push(
{
name: "certificates",
secret: {
items: [
{key: "tls.key", path: "proxykey"},
{key: "tls.crt", path: "proxycert"}],
secretName: "star.builds.qwil.co"
}
})
)
)
When I do this I get the following error: Property 'mode' is missing in type '{ key: string; path: string; }' but required in type 'KeyToPath'
I suspect I'm using apply incorrectly. When I try to directly modify ledgerDeployment.spec.template.spec.volumes.push() I get an error Property 'push' does not exist on type 'Output<Volume[]>'.
What is the pattern for modifying resources in Pulumi? How can I add a new volume to my deployment?
It's not possible to modify the resource inputs after you created the resource. Instead, you should place all the logic that defines the shape of inputs before you call the constructor.
In your example, this could be:
let volumes = [
{
emptyDir: {},
name: "gunicorn-socket-dir"
}
]
if (myCondition) {
volumes.push({...});
}
const ledgerDeployment = new k8s.extensions.v1beta1.Deployment("ledger", {
// <-- use `volumes` here
});
Related
1- I have a project core-infra that encapasses all the core infra related compoents (VPCs, Subnets, ECS Cluster...etc)
2- I have microservice projects with independant stacks each used for deployment
I want to deploy a FargateService from a microservice project stack A to the already existing ECS living within the core-infra stack
Affected area/feature
Pulumi Service
ECS
Deploy microservice
FargateService
Pulumi github issue link
Pulumi Stack References are the answer here:
https://www.pulumi.com/docs/intro/concepts/stack/#stackreferences
Your core-infra stack would output the ECS cluster ID and then stack B consumes that output so it can, for example, deploy an ECS service to the given cluster
(https://www.pulumi.com/registry/packages/aws/api-docs/ecs/service/).
I was able to deploy using aws classic.
PS: The setup is way more complex than with the awsx, the doc and resources aren't exhaustive.
Now I have few issues:
The Loadbalancer isn't reachable and keeps loading forever
I don't have any logs in the CloudWatch LogGoup
Not sure how to use the LB Listner with the ECS service / Not sure about the port mapping
Here is the complete code for reference (people who're husteling) and I'd appreciate if you could suggest any improvments/answers.
// Capture the EnvVars
const appName = process.env.APP_NAME;
const namespace = process.env.NAMESPACE;
const environment = process.env.ENVIRONMENT;
// Load the Deployment Environment config.
const configMapLoader = new ConfigMapLoader(namespace, environment);
const env = pulumi.getStack();
const infra = new pulumi.StackReference(`org/core-datainfra/${env}`);
// Fetch ECS Fargate cluster ID.
const ecsClusterId = infra.getOutput('ecsClusterId');
// Fetch DeVpc ID.
const deVpcId = infra.getOutput('deVpcId');
// Fetch DeVpc subnets IDS.
const subnets = ['subnet-aaaaaaaaaa', 'subnet-bbbbbbbbb'];
// Fetch DeVpc Security Group ID.
const securityGroupId = infra.getOutput('deSecurityGroupId');
// Define the Networking for our service.
const serviceLb = new aws.lb.LoadBalancer(`${appName}-lb`, {
internal: false,
loadBalancerType: 'application',
securityGroups: [securityGroupId],
subnets,
enableDeletionProtection: false,
tags: {
Environment: environment
}
});
const serviceTargetGroup = new aws.lb.TargetGroup(`${appName}-t-g`, {
port: configMapLoader.configMap.service.http.externalPort,
protocol: configMapLoader.configMap.service.http.protocol,
vpcId: deVpcId,
targetType: 'ip'
});
const http = new aws.lb.Listener(`${appName}-listener`, {
loadBalancerArn: serviceLb.arn,
port: configMapLoader.configMap.service.http.externalPort,
protocol: configMapLoader.configMap.service.http.protocol,
defaultActions: [
{
type: 'forward',
targetGroupArn: serviceTargetGroup.arn
}
]
});
// Create AmazonECSTaskExecutionRolePolicy
const taskExecutionPolicy = new aws.iam.Policy(
`${appName}-task-execution-policy`,
{
policy: JSON.stringify({
Version: '2012-10-17',
Statement: [
{
Effect: 'Allow',
Action: [
'ecr:GetAuthorizationToken',
'ecr:BatchCheckLayerAvailability',
'ecr:GetDownloadUrlForLayer',
'ecr:BatchGetImage',
'logs:CreateLogStream',
'logs:PutLogEvents',
'ec2:AuthorizeSecurityGroupIngress',
'ec2:Describe*',
'elasticloadbalancing:DeregisterInstancesFromLoadBalancer',
'elasticloadbalancing:DeregisterTargets',
'elasticloadbalancing:Describe*',
'elasticloadbalancing:RegisterInstancesWithLoadBalancer',
'elasticloadbalancing:RegisterTargets'
],
Resource: '*'
}
]
})
}
);
// IAM role that allows Amazon ECS to make calls to the load balancer
const taskExecutionRole = new aws.iam.Role(`${appName}-task-execution-role`, {
assumeRolePolicy: JSON.stringify({
Version: '2012-10-17',
Statement: [
{
Effect: 'Allow',
Principal: {
Service: ['ecs-tasks.amazonaws.com']
},
Action: 'sts:AssumeRole'
},
{
Action: 'sts:AssumeRole',
Principal: {
Service: 'ecs.amazonaws.com'
},
Effect: 'Allow',
Sid: ''
},
{
Action: 'sts:AssumeRole',
Principal: {
Service: 'ec2.amazonaws.com'
},
Effect: 'Allow',
Sid: ''
}
]
}),
tags: {
name: `${appName}-iam-role`
}
});
new aws.iam.RolePolicyAttachment(`${appName}-role-policy`, {
role: taskExecutionRole.name,
policyArn: taskExecutionPolicy.arn
});
// New image to be pulled
const image = `${configMapLoader.configMap.service.image.repository}:${process.env.IMAGE_TAG}`;
// Set up Log Group
const awsLogGroup = new aws.cloudwatch.LogGroup(`${appName}-awslogs-group`, {
name: `${appName}-awslogs-group`,
tags: {
Application: `${appName}`,
Environment: 'production'
}
});
const serviceTaskDefinition = new aws.ecs.TaskDefinition(
`${appName}-task-definition`,
{
family: `${appName}-task-definition`,
networkMode: 'awsvpc',
executionRoleArn: taskExecutionRole.arn,
requiresCompatibilities: ['FARGATE'],
cpu: configMapLoader.configMap.service.resources.limits.cpu,
memory: configMapLoader.configMap.service.resources.limits.memory,
containerDefinitions: JSON.stringify([
{
name: `${appName}-fargate`,
image,
cpu: parseInt(
configMapLoader.configMap.service.resources.limits.cpu
),
memory: parseInt(
configMapLoader.configMap.service.resources.limits.memory
),
essential: true,
portMappings: [
{
containerPort: 80,
hostPort: 80
}
],
environment: configMapLoader.getConfigAsEnvironment(),
logConfiguration: {
logDriver: 'awslogs',
options: {
'awslogs-group': `${appName}-awslogs-group`,
'awslogs-region': 'us-east-2',
'awslogs-stream-prefix': `${appName}`
}
}
}
])
}
);
// Create a Fargate service task that can scale out.
const fargateService = new aws.ecs.Service(`${appName}-fargate`, {
name: `${appName}-fargate`,
cluster: ecsClusterId,
taskDefinition: serviceTaskDefinition.arn,
desiredCount: 5,
loadBalancers: [
{
targetGroupArn: serviceTargetGroup.arn,
containerName: `${appName}-fargate`,
containerPort: configMapLoader.configMap.service.http.internalPort
}
],
networkConfiguration: {
subnets
}
});
// Export the Fargate Service Info.
export const fargateServiceName = fargateService.name;
export const fargateServiceUrl = serviceLb.dnsName;
export const fargateServiceId = fargateService.id;
export const fargateServiceImage = image;
I created Application Gateway & then AKS cluster using C# (Pulumi Azure Native). Resources are created successfully (code is given below). While creating AKS, I enabled AGIC add-on with AddonProfiles = { //... }
AddonProfiles = {
"ingressApplicationGateway", new ManagedClusterAddonProfileArgs { //... }
}
I am trying to achieve Fanout Ingress with AGIC add-on (apps will run in AKS and will be served by Application Gateway). The problem is ingress is not working unless I manually set Contributor role to AGIC add-on managed identity at scope 'ApplicationGatewayResourceId'.
So far, I did the followings:
Not working: created an ingress (see Kubernetes manifest demo-api.yaml below). But ingress is not working
Working but requires manual intervention:
a managed identity "ingressapplicationgateway-xxx" is created automatically (in node resource group "MC_xxx") and used by AGIC add-on
I assigned Contributor role to ingressapplicationgateway-xxx managed identity at scope 'ApplicationGatewayResourceId' (using Azure portal)
created Kubernetes ingress again and this time it works
just to be confirmed, I removed Contributor role and re-created ingress -> ingress did not work
So, ingressapplicationgateway-xxx requires Contributor role which I can easily assign using portal/cli/PowerShell, but that's not what I want because I am implementing Infrastructure-as-Code using Pulumi C#
Bring your own identity is not applicable for AGIC add-on: I thought I will create Managed Identity, assign role and then use it for AGIC add-on, but according to Microsoft Documentation, "Bring your own identity" is only supported for API server and Kubelet identity (not possible for AGIC add-on)
I tried to get AGIC add-on Managed Identity ingressapplicationgateway-xxx (that belongs to node resource group MC_xxx) but failed to do so and asked question here (no answer yet).
Kubernetes manifest for ingress: demo-api.yaml
apiVersion: v1
kind: Service
metadata:
name: demo-api
namespace: default
spec:
type: ClusterIP
ports:
3. port: 80
selector:
app: demo-api
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: demo-api
namespace: default
spec:
replicas: 3
selector:
matchLabels:
app: demo-api
template:
metadata:
labels:
app: demo-api
spec:
containers:
- name: demo-api
image: hovermind.azurecr.io/demoapi:latest
ports:
- containerPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: demo-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/backend-path-prefix: "/api/" # API has url prefix -> [Route("api")]
spec:
rules:
- host: agic-appgw-public-ip.japaneast.cloudapp.azure.com
http:
paths:
- path: /api*
pathType: Prefix
backend:
service:
name: demo-api
port:
number: 80
C# Code for Application Gateway
// ... ... ...
var hubAppGwSubnet = new Subnet($"{HubVirtualNetwork}.{ApplicationGatewaySubnet}", new AzureNative.Network.SubnetArgs {
// ... ... ...
}, new CustomResourceOptions { DependsOn = { hubVnet } });
//
// Create Public IP for App Gateway
//
var agicAppGwPublicIp = new PublicIPAddress(AgicApplicationGatewayPublicIp, new AzureNative.Network.PublicIPAddressArgs {
// ... ... ...
});
//
// Application Gateway configs from stack settings file
//
var agicAppGwArgs = config.RequireObject<JsonElement>(AgicApplicationGatewayArgs);
var agicAppGwName = agicAppGwArgs.GetName();
var agicAppGwSku = agicAppGwArgs.GetSku();
var agicAppGwMinCapacity = agicAppGwArgs.GetInt(MinCapacity);
var agicAppGwMaxCapacity = agicAppGwArgs.GetInt(MaxCapacity);
// Gateway IP Config (subnet to which Application Gateway would be deployed)
var appGwIpConfig = new ApplicationGatewayIPConfigurationArgs {
Name = AppGatewayIpConfigName,
Subnet = new SubResourceArgs {
Id = hubAppGwSubnet.Id,
}
};
//
// Create App Gateway
//
var agicAppGw = new ApplicationGateway(AgicApplicationGateway, new ApplicationGatewayArgs {
ApplicationGatewayName = agicAppGwName,
ResourceGroupName = mainResourceGroup.Name,
Sku = new ApplicationGatewaySkuArgs {
Name = agicAppGwSku,
Tier = agicAppGwSku
},
AutoscaleConfiguration = new ApplicationGatewayAutoscaleConfigurationArgs {
MinCapacity = agicAppGwMinCapacity,
MaxCapacity = agicAppGwMaxCapacity
},
GatewayIPConfigurations = { appGwIpConfig },
FrontendIPConfigurations = {
new ApplicationGatewayFrontendIPConfigurationArgs {
Name = AppGatewayFrontendIpConfigName,
PublicIPAddress = new SubResourceArgs {
Id = agicAppGwPublicIp.Id,
}
},
new ApplicationGatewayFrontendIPConfigurationArgs {
Name = $"{AppGatewayFrontendIpConfigName}_private",
PrivateIPAllocationMethod = IPAllocationMethod.Static,
PrivateIPAddress = "10.10.1.5", //hubApplicationGatewaySubnet.GetFirstUsableIp(),
Subnet = new SubResourceArgs {
Id = hubAppGwSubnet.Id
}
}
},
FrontendPorts = {
new ApplicationGatewayFrontendPortArgs {
Name = AppGatewayFrontendPort80Name,
Port = Port80
},
},
HttpListeners = {
new ApplicationGatewayHttpListenerArgs {
Name = AppGatewayHttpListenerName,
Protocol = Http,
FrontendIPConfiguration = new SubResourceArgs {
Id = $"/subscriptions/{subscriptionId}/resourceGroups/{mainResourceGroup.Name}/providers/Microsoft.Network/applicationGateways/{agicAppGwName}/frontendIPConfigurations/{AppGatewayFrontendIpConfigName}"
},
FrontendPort = new SubResourceArgs {
Id = $"/subscriptions/{subscriptionId}/resourceGroups/{mainResourceGroup.Name}/providers/Microsoft.Network/applicationGateways/{agicAppGwName}/frontendPorts/{AppGatewayFrontendPort80Name}",
},
}
},
BackendAddressPools = {
new ApplicationGatewayBackendAddressPoolArgs {
Name = AppGatewayBackendPoolName,
//BackendAddresses = { }
}
},
BackendHttpSettingsCollection = {
new ApplicationGatewayBackendHttpSettingsArgs {
Name = AppGatewayHttpSettingName,
Port = 80,
Protocol = Http,
RequestTimeout = 20,
CookieBasedAffinity = ApplicationGatewayCookieBasedAffinity.Enabled
}
},
RequestRoutingRules = {
new ApplicationGatewayRequestRoutingRuleArgs {
Name = AppGatewayRoutingName,
RuleType = ApplicationGatewayRequestRoutingRuleType.Basic,
Priority = 10,
HttpListener = new SubResourceArgs {
Id = $"/subscriptions/{subscriptionId}/resourceGroups/{mainResourceGroup.Name}/providers/Microsoft.Network/applicationGateways/{agicAppGwName}/httpListeners/{AppGatewayHttpListenerName}",
},
BackendAddressPool = new SubResourceArgs {
Id = $"/subscriptions/{subscriptionId}/resourceGroups/{mainResourceGroup.Name}/providers/Microsoft.Network/applicationGateways/{agicAppGwName}/backendAddressPools/{AppGatewayBackendPoolName}",
},
BackendHttpSettings = new SubResourceArgs {
Id = $"/subscriptions/{subscriptionId}/resourceGroups/{mainResourceGroup.Name}/providers/Microsoft.Network/applicationGateways/{agicAppGwName}/backendHttpSettingsCollection/{AppGatewayHttpSettingName}",
},
}
},
WebApplicationFirewallConfiguration = new ApplicationGatewayWebApplicationFirewallConfigurationArgs { },
}, new CustomResourceOptions { DependsOn = { hubAppGwSubnet } });
C# code for AKS Cluster
// ... ... ...
var systemNodePool = new ManagedClusterAgentPoolProfileArgs { // ... ... ...};
var userNodePool = new ManagedClusterAgentPoolProfileArgs { // ... ... ... };
var aadProfile = new ManagedClusterAADProfileArgs { // ... ... ... };
var networkProfile = new ContainerServiceNetworkProfileArgs { // ... ... ... };
//
// Create AKS Cluster
//
var aksAppClusterName = "xxx-aks-appcluster-dev-japaneast";
var aksAppClusterNodeRgName = $"{AksEntities.NodePoolResourceGroupPrefix}_{aksAppClusterName}";
var aksAppClusterDnsPrefix = "xxx-yyy";
var aksAppCluster = new ManagedCluster(AksAppplicationCluster, new ManagedClusterArgs {
ResourceName = aksAppClusterName,
ResourceGroupName = mainResourceGroup.Name,
AgentPoolProfiles = { systemNodePool, userNodePool },
DnsPrefix = aksAppClusterDnsPrefix,
EnableRBAC = true,
AadProfile = aadProfile,
NetworkProfile = networkProfile,
//ServicePrincipalProfile = spProfile,
NodeResourceGroup = aksAppClusterNodeRgName,
DisableLocalAccounts = true,
Identity = new ManagedClusterIdentityArgs { Type = AzureNative.ContainerService.ResourceIdentityType.SystemAssigned },
AddonProfiles = {
{
AksEntities.AddonProfileKeys.Agic, new ManagedClusterAddonProfileArgs {
Enabled = true,
Config = {
{
AksEntities.AddonProfileKeys.ApplicationGatewayId, agicAppGw.Id
},
}
}
},
},
}, new CustomResourceOptions { DependsOn = { spokeAksAppplicationSubnet, agicAppGw } });
Trying to do something, I don't know if it's possible, and if it is, I am asking for some help on how to do it.
I have a file "test.bicep" that has an object:
{
name: 'testRafaelRule'
priority: 1001
ruleCollectionType: 'FirewallPolicyFilterRuleCollection'
action: {
type: 'Allow'
}
rules: [
{
name: 'deleteme-1'
ipProtocols: [
'Any'
]
destinationPorts: [
'*'
]
sourceAddresses: [
'192.168.0.0/16'
]
sourceIpGroups: []
destinationIpGroups: []
destinationAddresses: [
'AzureCloud.EastUS'
]
ruleType: 'NetworkRule'
destinationFqdns: []
}
]
}
and I have another file, in which I am trying to somehow input the object in test.bicep into a specific property called "ruleCollections":
resource fwll 'Microsoft.Network/firewallPolicies/ruleCollectionGroups#2020-11-01' = {
name: 'netrules'
properties: {
priority: 200
ruleCollections: [
**ADD_OBJECT_FROM_TEST.BICEP_HERE_HOW?**
]
}
}
any suggestions or links to useful documentation would be helpful.
I have looked at outputs and parameters, but I am trying to add just an object into an existing property, I am not adding an entire resource on its own, otherwise, I would output the resouce and consume it with the "module" keyword.
It's not possible straightforward, but you can leverage variables or module's output.
var RULE = {
name: 'testRafaelRule'
priority: 1001
(...)
}
resource fwll 'Microsoft.Network/firewallPolicies/ruleCollectionGroups#2020-11-01' = {
name 'netrules'
properties: {
ruleCollections: [
RULE
]
}
}
or
rule.bicep
output rule object = {
name: 'testRafaelRule'
priority: 1001
(...)
}
main.bicep
module fwrule 'rule.bicep' = {
name: 'fwrule'
}
resource fwll 'Microsoft.Network/firewallPolicies/ruleCollectionGroups#2020-11-01' = {
name 'netrules'
properties: {
ruleCollections: [
fwrule.outputs.rule
]
}
}
I see there are 2 separate metrics ApproximateNumberOfMessagesVisible and ApproximateNumberOfMessagesNotVisible.
Using number of messages visible causes processing pods to get triggered for termination immediately after they pick up the message from queue, as they're no longer visible. If I use number of messages not visible, it will not scale up.
I'm trying to scale a kubernetes service using horizontal pod autoscaler and external metric from SQS. Here is template external metric:
apiVersion: metrics.aws/v1alpha1
kind: ExternalMetric
metadata:
name: metric-name
spec:
name: metric-name
queries:
- id: metric_name
metricStat:
metric:
namespace: "AWS/SQS"
metricName: "ApproximateNumberOfMessagesVisible"
dimensions:
- name: QueueName
value: "queue_name"
period: 60
stat: Average
unit: Count
returnData: true
Here is HPA template:
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
name: hpa-name
spec:
scaleTargetRef:
apiVersion: apps/v1beta1
kind: Deployment
name: deployment-name
minReplicas: 1
maxReplicas: 50
metrics:
- type: External
external:
metricName: metric-name
targetAverageValue: 1
The problem will be solved if I can define another custom metric that is a sum of these two metrics, how else can I solve this problem?
We used a lambda to fetch two metrics and publish a custom metric that is sum of messages in-flight and waiting, and trigger this lambda using cloudwatch events at whatever frequency you want, https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#rules:action=create
Here is lambda code for reference:
const AWS = require('aws-sdk');
const cloudwatch = new AWS.CloudWatch({region: ''}); // fill region here
const sqs = new AWS.SQS();
const SQS_URL = ''; // fill queue url here
async function getSqsMetric(queueUrl) {
var params = {
QueueUrl: queueUrl,
AttributeNames: ['All']
};
return new Promise((res, rej) => {
sqs.getQueueAttributes(params, function(err, data) {
if (err) rej(err);
else res(data);
});
})
}
function buildMetric(numMessages) {
return {
Namespace: 'yourcompany-custom-metrics',
MetricData: [{
MetricName: 'mymetric',
Dimensions: [{
Name: 'env',
Value: 'prod'
}],
Timestamp: new Date(),
Unit: 'Count',
Value: numMessages
}]
}
}
async function pushMetrics(metrics) {
await new Promise((res) => cloudwatch.putMetricData(metrics, (err, data) => {
if (err) {
console.log('err', err, err.stack); // an error occurred
res(err);
} else {
console.log('response', data); // successful response
res(data);
}
}));
}
exports.handler = async (event) => {
console.log('Started');
const sqsMetrics = await getSqsMetric(SQS_URL).catch(console.error);
var queueSize = null;
if (sqsMetrics) {
console.log('Got sqsMetrics', sqsMetrics);
if (sqsMetrics.Attributes) {
queueSize = parseInt(sqsMetrics.Attributes.ApproximateNumberOfMessages) + parseInt(sqsMetrics.Attributes.ApproximateNumberOfMessagesNotVisible);
console.log('Pushing', queueSize);
await pushMetrics(buildMetric(queueSize))
}
} else {
console.log('Failed fetching sqsMetrics');
}
const response = {
statusCode: 200,
body: JSON.stringify('Pushed ' + queueSize),
};
return response;
};
This seems to be a case of Thrashing - the number of replicas keeps fluctuating frequently due to the dynamic nature of the metrics evaluated.
IMHO, you've got a couple of options here.
You could look at adding a StabilizationWindow to your HPA and also probably limit the scale down rate. You'd have to try a few combination of metrics and see what works best for you as you'd best know the nature of metrics (ApproximateNumberOfMessagesVisible in this case) you see in your infrastructure.
When I try to set a parameter with ksonnet I get an error
ks param set --env=prow workflows name some-name
ERROR Invalid params schema -- did not expect node type: *ast.ApplyBrace
My parameters file looks like
local params = import "../../components/params.libsonnet";
params {
components+: {
// Insert component parameter overrides here. Ex:
// guestbook +: {
// name: "guestbook-dev",
// replicas: params.global.replicas,
// },
workflows +: {
name: "some-name",
},
},
}
Adding a plus to params fixed it.
local params = import "../../components/params.libsonnet";
params +{
components+: {
// Insert component parameter overrides here. Ex:
// guestbook +: {
// name: "guestbook-dev",
// replicas: params.global.replicas,
// },
workflows +: {
name: "some-name",
},
},
}