I am getting the following error while running pulumi up, I am getting a templateBody update in the preview for aws:cloudformation:Stack spot-ng-01-nodes.
aws:ec2:LaunchConfiguration (spot-ng-01-nodeLaunchConfiguration):
error: deleting urn:pulumi:staging::xx-api::eks:index:NodeGroup$aws:ec2/launchConfiguration:LaunchConfiguration::spot-ng-01-nodeLaunchConfiguration: 1 error occurred:
* error deleting Autoscaling Launch Configuration (spot-ng-01-nodeLaunchConfiguration-3a59b7e): ResourceInUse: Cannot delete launch configuration spot-ng-01-nodeLaunchConfiguration-3a59b7e because it is attached to AutoScalingGroup spot-ng-01-d1815eb6-NodeGroup-UBM7XABBGVNU
status code: 400, request id: fc55d507-0884-4c50-aeba-33831646a914
This is the resource in question, but the code was not updated.
new eks.NodeGroup("spot-ng-01", {
cluster: cluster,
spotPrice: "0.1",
instanceType: "t3.xlarge",
taints,
labels: { spot: "true" },
version: "1.21",
maxSize: 60,
minSize: 1,
nodeSubnetIds: options.vpc.privateSubnetIds,
instanceProfile: new aws.iam.InstanceProfile("spot-ng-profile-01", { role: role.name }),
nodeAssociatePublicIpAddress: false,
nodeSecurityGroup: clusterSG,
clusterIngressRule: cluster.eksClusterIngressRule,
autoScalingGroupTags: {
Name: "spot",
"k8s.io/cluster-autoscaler/enabled": "true",
[`k8s.io/cluster-autoscaler/${clusterName}`]: "true",
},
});
Even after running pulumi refresh, I still get the error.
The solution required manual intervention, it might not be the best but it solved the issue.
Another LaunchConfiguration was created by pulumi, I made this new LaunchConfiguration used by the AutoscalingGroup in question. then I ran pulumi up and it was able to delete the LaunchConfiguration that was stuck. Then ran pulumi refresh.
Related
I have a CF template (serverless Typescript file to be precise) which creates a container definition in the following way.
Type: "AWS::ECS::TaskDefinition",
Properties: {
Family: "client",
RequiresCompatibilities: ["FARGATE"],
NetworkMode: "awsvpc",
Memory: 1024,
CPU: 512,
TaskRoleArn: { Ref: ECS_TASK_ROLE },
ExecutionRoleArn: { Ref: ECS_TASK_ROLE },
ContainerDefinitions: [
{
Name: application,
Essential: true,
Image: `${ACCOUNT_ID}.dkr.ecr.us-east-1.amazonaws.com/repository:client-{{resolve:ssm:param-name}}`,
.
.
.
},
],
}
Please note the usage of AWS SSM Dynamic Reference.
When I deploy it I get the following error
Could not create Change Set "change-set-name" due to: Parameters: "[ssm:param-name:167:167710252826] cannot be found.
It used to work fine and all of a sudden started breaking. Same code still works in a different env/AWS account.
Seems like it can't find a specific version of param-name for some reason. The latest param-name verion is 400. AWS SSM only keeps last 100 params. This parameter is updated quite often and it's been a long time since I deployed CF template. From the error message it seems like it is looking for version 167 (I could be wrong because I am not sure what is the number trailing 167 i.e. 167:167710252826)
What I tried:
I hardcoded the version number I pass as below and removed all the references to resolve:ssm:param-name.
Image: `${ACCOUNT_ID}.dkr.ecr.us-east-1.amazonaws.com/repository:client-1.6.3`,
It still throws the same error. Which hints that the problem is not related to code. CF is trying to run an old change set from history to create a diff and can't find an old version of the same param.
What could be the solution to this?
I can't delete the old change set. It throws an error on deleting.
I can't go back and create a parameter in SSM with version 167 if it's a version issue.
How to deploy on K8 via Pulumi using the ArgoCD Helm Chart?
Pulumi up Diagnostics:
kubernetes:helm.sh/v3:Release (argocd):
error: failed to create chart from template: chart requires kubeVersion: >=1.22.0-0 which is incompatible with Kubernetes v1.20.0
THE CLUSTER VERSION IS: v1.23.0 verified on AWS. And NOT 1.20.0
ArgoCD install yaml used with CRD2Pulumi: https://raw.githubusercontent.com/argoproj/argo-cd/master/manifests/core-install.yaml
Source:
...
cluster = eks.Cluster("argo-example") # version="1.23"
# Cluster provider
provider = k8s.Provider(
"eks",
kubeconfig=cluster.kubeconfig.apply(lambda k: json.dumps(k))
#kubeconfig=cluster.kubeconfig
)
ns = k8s.core.v1.Namespace(
'argocd',
metadata={
"name": "argocd",
},
opts=pulumi.ResourceOptions(
provider=provider
)
)
argo = k8s.helm.v3.Release(
"argocd",
args=k8s.helm.v3.ReleaseArgs(
chart="argo-cd",
namespace=ns.metadata.name,
repository_opts=k8s.helm.v3.RepositoryOptsArgs(
repo="https://argoproj.github.io/argo-helm"
),
values={
"server": {
"service": {
"type": "LoadBalancer",
}
}
},
),
opts=pulumi.ResourceOptions(provider=provider, parent=ns),
)
Any ideas as to fixing this oddity between the version error and the actual cluster version?
I've tried:
Deleting everything and starting over.
Updating to the latest ArgoCD install yaml.
I could reproduce your issue, though I am not quite sure what causes the mismatch between versions. Better open an issue at pulumi's k8s repository.
Looking at the history of https://github.com/argoproj/argo-helm/blame/main/charts/argo-cd/Chart.yaml, you can see that the kubeversion requirement has been added after 5.9.1. So using that version successfully deploys the helm chart. E.g.
import * as k8s from "#pulumi/kubernetes";
const namespaceName = "argo";
const namespace = new k8s.core.v1.Namespace("namespace", {
metadata: {
name: namespaceName,
}
});
const argo = new k8s.helm.v3.Release("argo", {
repositoryOpts: {
repo: "https://argoproj.github.io/argo-helm"
},
chart: "argo-cd",
version: "5.9.1",
namespace: namespace.metadata.name,
})
(Not Recommended) Alternatively, you could also clone the source code of the chart, comment out the kubeVersion requirement in Chart.yaml and install the chart from your local path.
Upgrade helm. I had a similar issue where my k8s was 1.25 but helm complained it was 1.20. Tried everything else, upgrading helm worked.
i am trying to experiment with scaling one of my application pods running on my raspberry pi kubernetes cluster using HPA + custom metrics but ran into several issues which despite reading the documentations on https://github.com/DirectXMan12/k8s-prometheus-adapter and troubleshooting for the past 2 days, i am still having difficulties grasping why some problems are happening.
Firstly, i built an ARM-compatible image of k8s-prometheus-adapter and install it using helm. I can confirm its running properly by checking the pod logs.
I have also set up a script which sends raspberry pis temperature to pushgateway and i can query via this Prometheus query node_temp, which will return the following series
node_temp{job="kube4"} 42
node_temp{job="kube1"} 44
node_temp{job="kube2"} 39
node_temp{job="kube3"} 40
Now i want to be able to scale one of my application pods using the above temperature values as an experiment to understand better how it works.
Below is my k8s-prometheus-adapter helm values.yml file
image:
repository: jaanhio/k8s-prometheus-adapter-arm
tag: latest
logLevel: 7
prometheus:
url: http://10.17.0.12
rules:
default: false
custom:
- seriesQuery: 'etcd_object_counts'
resources:
template: <<.Resource>>
name:
as: "etcd_object"
metricsQuery: count(etcd_object_counts)
- seriesQuery: 'node_temp'
resources:
template: <<.Resource>>
name:
as: "node_temp"
metricsQuery: count(node_temp)
After installing via helm, i ran kubectl get apiservices and can see v1beta1.custom.metrics.k8s.io listed.
i then ran kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 | jq and got the following
{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "custom.metrics.k8s.io/v1beta1",
"resources": [
{
"name": "jobs.batch/node_temp",
"singularName": "",
"namespaced": true,
"kind": "MetricValueList",
"verbs": [
"get"
]
},
{
"name": "jobs.batch/etcd_object",
"singularName": "",
"namespaced": true,
"kind": "MetricValueList",
"verbs": [
"get"
]
},
]
i then tried to query the value of the registered node_temp metrics using kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/jobs/*/node_temp but got the following response
Error from server (InternalError): Internal error occurred: unable to list matching resources
Questions:
Why is the node_temp metrics associated with jobs.batch resource type?
Why am i not able to retrieve the value of metrics via kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/jobs/*/node_temp?
What is a definitive way of figuring the path of the query? e.g /apis/custom.metrics.k8s.io/v1beta1/jobs/*/node_temp i kinda trial and error until i got see somewhat of a response. i also see some other path with namespaces in the query e.g /apis/custom.metrics.k8s.io/v1beta1/namespaces/*/metrics/foo_metrics
Any help and advice will be greatly appreciate!
Why is the node_temp metrics associated with jobs.batch resource type?
It picks the labels attached to the prometheus metrics and tries to interpret them, in this case u have clearely "job-kube4"
Why am i not able to retrieve the value of metrics via kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/jobs/*/node_temp?
Metrics are namespaced, see the "namespaced:true" so you'll need "/apis/custom.metrics.k8s.io/v1beta1/namespaces//jobs//node_temp"
What is a definitive way of figuring the path of the query? e.g /apis/custom.metrics.k8s.io/v1beta1/jobs//node_temp i kinda trial and error until i got see somewhat of a response. i also see some other path with namespaces in the query e.g /apis/custom.metrics.k8s.io/v1beta1/namespaces//metrics/foo_metrics
Check https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/custom-metrics-api.md#api-paths
After some successful projects, I have deleted the functions inside AWS-lambda, deleted the logs in CloudWatch and the IAM roles.
Also deleted the my-service folder from my Documents.
Then I followed the steps in this tutorial in serverless.
Now when I run:
serverless deploy --aws-profile testUser_atWork
where testUser_atWork is one of my profiles to connect in AWS.
I get the follow error:
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Service files not changed. Skipping deployment...
Service Information
service: my-service
stage: dev
region: us-east-1
stack: my-service-dev
api keys:
None
endpoints:
None
functions:
hello: my-service-dev-hello
//serverless.yml
service: my-service
provider:
name: aws
runtime: nodejs6.10
functions:
hello:
handler: handler.hello
And this my handler.js
'use strict';
module.exports.hello = (event, context, callback) => {
const response = {
statusCode: 200,
body: JSON.stringify({
message: 'Go Serverless v1.0! Your function executed successfully!',
input: event,
}),
};
callback(null, response);
// Use this code if you don't use the http event with the LAMBDA-PROXY integration
// callback(null, { message: 'Go Serverless v1.0! Your function executed successfully!', event });
};
I don't understand why it is skipping deployment.
have you tried :
serverless deploy --aws-profile testUser_atWork --force to force it to update the stack?
Otherwise, try deleting the stack in cloudformation, or with the serverless remove command
I'm trying to update a service using :
kubectl update service my-service \
--patch='{ "apiVersion":"v1", "spec": { "selector": { "build":"2"} } }'
I receive the following Error :
Error from server: service "\"apiVersion\":\"v1\"," not found
I have tried the following :
moving the service name to the end
Removing the apiVersion
Maybe the kubectl update is not available for service ?
For now I was making my updates by simply stoping and restarting my service. But sometime, the corresponding forwarding-port changes. So it seems to not be the good choice ...
PS:
v0.19
api_v1
I am not sure if patch is 100% working yet, but if you are going to do this, you at least need to put apiVersion inside metadata, like so:
--patch='{ metadata:{ "apiVersion":"v1" }, "spec": { "selector": { "build":"2"} } }'