Good morning,
I have a k8s cluster where multiple ingress services share a pre generated self managed certificate in GCP.
My problem is that when the certificate expires, I need to update the yaml file with the name of the new cert and apply the modified yaml file for each of the ingress to update the certs. We do it, updating the environment variable and redeploying the application. I was thinking in a better way to do it that will not require to redeploy it, I was planning to use kubectl patch to do this, anyone has already have to done something similar?
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
ingress.gcp.kubernetes.io/pre-shared-cert: cert-abc
ingress.kubernetes.io/forwarding-rule: fwd-abc
ingress.kubernetes.io/https-forwarding-rule: https-fwd-abc
ingress.kubernetes.io/https-target-proxy: tgt-https-abc
ingress.kubernetes.io/ssl-cert: cert-abc
ingress.kubernetes.io/static-ip: ip-abc
ingress.kubernetes.io/target-proxy: tgt-http-abc
ingress.kubernetes.io/url-map: lb-abc
kubernetes.io/ingress.global-static-ip-name: sta-ip-abc
creationTimestamp: 2019-01-29T22:38:10Z
generation: 2
name: abc-ingress
namespace: abc
spec:
backend:
serviceName: abc
servicePort: 80
Thanks in advance for your help.
We have similar challenges. kubectl apply works fine here as Hernan Garcia already pointed out.
A patch can do the same trick.
Our choice in fact way using helm which is quite easy to use and which makes it quite easy to update selectively values. Furthermore you have the option to rollback if something goes wrong, which is nice for automated deployments.
Related
Hi all we have a Flink application with blue-green deployment which we get using the Flink operator.
Flinkk8soperator for Apache Flink. The operator spins up the following three K8s services after a deployment:
my-flinkapp-14hdhsr (Top level service)
my-flinkapp-green
my-flinkapp-blue
The idea is that one of the two among blue green would be active and would have pods (either blue or green).
And a selector to the the active one would be stored in the top level myflinkapp-14hdhsr service with the selector flink-application-version=blue. Or green. As follows:
Labels: flink-app=my-flinkapp
flink-app-hash=14hdhsr
flink-application-version=blue
Annotations: <none>
Selector: flink-app=my-flinkapp,flink-application-version=blue,flink-deployment-type=jobmanager,
I have an ingress defined as follows which I want to use to point to the top level service.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
nginx.ingress.kubernetes.io/configuration-snippet: |
proxy_set_header Accept-Encoding "";
sub_filter '<head>' '<head> <base href="/happy-flink-ui/">';
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri"
name: flink-secure-ingress-my-flink-app
namespace: happy-flink-flink
spec:
rules:
- host: flinkui-myapp.foo.com
http:
paths:
- path: /happy-flink-ui(/|$)(.*)
backend:
serviceName: my-flinkapp-14hdhsr // This works but....
servicePort: 8081
The issue I am facing is that the top level service keeps changing the hash at the end as the flink operator changes it at every deployment.. eg. myflinkapp-89hddew .etc.
So I cannot have a static service name in the ingress definition.
So I am wondering if an ingress can choose a service based on a selector or a regular expression of the service name which can account for the top level app service name plus the hash at the end.
The flink-app-hash (i.e the hash part of the service name - 14hdsr) is also part of the Labels in the top level service. Anyway I could leverage that?
Wondering if default backend could be applied here?
Have folks using Flink operator solved this a different way?
Unfortunately that's not possible OOB to use any kind of regex/wildcards/jsonpath/variables/references inside .backend.serviceName inside Ingres controller.
You are able to use regex only in path: Ingress Path Matching
And its not going to be implemented soon: Allow variable references in backend spec is in the hanged stage.
Would be very interesting to hear any possible solution for this. It was previously discussed on stack, however without any progress: https://stackoverflow.com/a/60810435/9929015
I'm adding an Ingress as follows:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: cheddar
spec:
rules:
- host: cheddar.213.215.191.78.nip.io
http:
paths:
- backend:
service:
name: cheddar
port:
number: 80
path: /
pathType: ImplementationSpecific
but the logs complain:
W0205 15:14:07.482439 1 warnings.go:67] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
time="2021-02-05T15:14:07Z" level=info msg="Updated ingress status" namespace=default ingress=cheddar
W0205 15:18:19.104225 1 warnings.go:67] networking.k8s.io/v1beta1 IngressClass is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 IngressClassList
Why? What's the correct yaml to use?
I'm currently on microk8s 1.20
I have analyzed you issue and came to the following conclusions:
The Ingress will work and these Warnings you see are just to inform you about the available api versioning. You don't have to worry about this. I've seen the same Warnings:
#microk8s:~$ kubectl describe ing
Warning: extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
As for the "why" this is happening even when you use apiVersion: networking.k8s.io/v1, I have found the following explanation:
This is working as expected. When you create an ingress object, it can
be read via any version (the server handles converting into the
requested version). kubectl get ingress is an ambiguous request,
since it does not indicate what version is desired to be read.
When an ambiguous request is made, kubectl searches the discovery docs
returned by the server to find the first group/version that contains
the specified resource.
For compatibility reasons, extensions/v1beta1 has historically been
preferred over all other api versions. Now that ingress is the only
resource remaining in that group, and is deprecated and has a GA
replacement, 1.20 will drop it in priority so that kubectl get ingress would read from networking.k8s.io/v1, but a 1.19 server
will still follow the historical priority.
If you want to read a specific version, you can qualify the get
request (like kubectl get ingresses.v1.networking.k8s.io ...) or can
pass in a manifest file to request the same version specified in the
file (kubectl get -f ing.yaml -o yaml)
Long story short: despite the fact of using the proper apiVersion, the deprecated one is still being seen as the the default one and thus generating the Warning you experience.
I also see that changes are still being made recently so I assume that it is still being worked on.
I had the same issue and was unable to update the k8s cluster which was subscribed to release channel.
One of the reasons for this log warning generation is the ClusterRole definition of external-dns. The external-dns keep querying the ingresses in k8s cluster as per the rules defined in the Cluster role
- apiGroups: ["extensions", "networking.k8s.io"]
resources: ["ingresses"]
verbs: ["get","watch","list"]
Found in the helm chart in here
It queries the old extensions of ingress as well which keeps on generating those logs. Please update the cert-manager.
I want to know if there is versioning for ingress config similar to what we have in deployments. Suppose there is a misconfiguration I would like to revert to the previous config.
I would like to understand about generation in ingress YAML config.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/service-match: 'new-nginx: header("foo", /^bar$/)' #Canary release rule. In this example, the request header is used.
nginx.ingress.kubernetes.io/service-weight: 'new-nginx: 50,old-nginx: 50' #The route weight.
creationTimestamp: null
generation: 1
name: nginx-ingress
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/nginx-ingress
spec:
rules: ##The Ingress rule.
- host: foo.bar.com
http:
paths:
- backend:
serviceName: new-nginx
servicePort: 80
path: /
- backend:
serviceName: old-nginx
servicePort: 80
path: /
Kubernetes does not offer this natively, and neither does a management tool like Rancher.
If you want to do this, you need an infra-as-code tool, like Terreform, ansible, etc. The config files for these can be versioned in a repo.
Even without those, you can independently export a give ingress yaml, and commit it to a repo.
Slightly different way to look at a solution - you could use gitOps. I mean that you could put all your yaml's in a git repo, install ArgoCD on your cluster and then simply let ArgoCD do the syncing for you. The moment you realised you've messed something up in a yaml file, just revert the commit in the git repo. That way you maintain history and get a graceful non-opinionated solution.
I've read documentation of kubernetes annotations.
But I couldn't find basic example about using this annotations. For Example;
I have a deployment yaml like below:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
test_value: "test"
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 1
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.13
ports:
- containerPort: 80
How can I use this annotation named test_value and where.
Best Regards...
Just as Labels, Annotations are key-value pairs which represent metadata that is attached to a Kubernetes object.
But contrary to Labels, which are internally utilized to find a collection of objects which satisfy specific conditions, the purpose of Annotations is simply to attach relevant metadata, which should not be used as a filter to identify those objects.
What if we wanted to describe whose person was responsible for generating a specific .yaml file?
We could attach such information to the Kubernetes's object, so that when we need to know who created such object, we can simply run kubectl describe ...
Another useful example, could be to add an annotation to a Deployment before a rollout, explaining what modifications occurred on the new version of the Deployment object. That information could be retrieved later while checking the history of your deployment versions.
But as you have realized with the Ingress example, with Annotations we can also perform advanced configuration on such objects. This is not limited only to Ingress, and for instance you can also provide configuration for running Prometheus on a Kubernetes cluster. You can check the details here.
As mentioned in Kubernetes Documentation, labels have a limited purpose of selecting objects and finding collections of objects that satisfy certain conditions. That put some limitation on the information you can store in labels. (Valid label values must be 63 characters or less and must be empty or begin and end with an alphanumeric character ([a-z0-9A-Z]) with dashes (-), underscores (_), dots (.), and alphanumerics between.)
However, annotations are not used to filter objects, so, you can put in annotation big/small structured/unstructured data that can contain characters, you’re not permitted to use in labels. Tools and libraries can retrieve annotations and use it to add some features to your cluster.
Here are some examples of information that could be recorded in annotations:
Fields managed by a declarative configuration layer. Attaching these fields as annotations distinguishes them from default values set by clients or servers, and from auto-generated fields and fields set by auto-sizing or auto-scaling systems.
Build, release, or image information like timestamps, release IDs, git branch, PR numbers, image hashes, and registry address.
Pointers to logging, monitoring, analytics, or audit repositories.
Client library or tool information that can be used for debugging purposes: for example, name, version, and build information.
User or tool/system provenance information, such as URLs of related objects from other ecosystem components.
Lightweight rollout tool metadata: for example, config or checkpoints.
Phone or pager numbers of persons responsible, or directory entries that specify where that information can be found, such as a team website.
Options for Ingress object, (nginx,gce)
Well, you are right Annotations is like Labels. But I saw that We could customize to config with Annotations for example:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: cafe-ingress-with-annotations
annotations:
nginx.org/proxy-connect-timeout: "30s"
nginx.org/proxy-read-timeout: "20s"
nginx.org/client-max-body-size: "4m"
spec:
rules:
- host: cafe.example.com
http:
paths:
- path: /tea
backend:
serviceName: tea-svc
servicePort: 80
- path: /coffee
backend:
serviceName: coffee-svc
servicePort: 80
Nginx config can customize according to given Annotation. So how to do this. I couldn't find a tutorial.
I'll first give some background regarding annotations.
Annotations Vs Labels
Annotations are quiet different then labels.
Labels:
You use labels to group resources that you want to refer as a whole.
For example pods with the app=run, env=staging could be exposed by a service with a label selector that matches those labels or managed by a deployment or a daemon set.
Annotations:
Annotations have a few different usages like providing description and adding support for fields that are not part of the K8S API.
While labels should be short, annotations can contain much larger sets of data and can reach up to 256KB.
Annotations use cases examples
You can see below a few examples of how annotations are being used by the various providers / tools that interacts with your cluster.
1 ) Used internally by K8S - below are the annotations that are added to the API-server pod:
kubernetes.io/config.hash: 7c3646d2bcee38ee7dfb851711571ba3
kubernetes.io/config.mirror: 7c3646d2bcee38ee7dfb851711571ba3
kubernetes.io/config.seen: "2020-10-22T01:26:12.671011852+03:00"
kubernetes.io/config.source: file
2 ) If you provision a cluster with kubeadm - this will be added to the API-server pod:
annotations:
kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 10.246.38.137:6443
3 ) If you run on amazon-eks you can see that the following annotation is added to your workloads - this is for backward compatibility - read more in here):
annotations:
kubernetes.io/psp: eks.privileged
4 ) There are cases when 3rd party tools like aws-alb-ingress-controller that requires you to pass (mandatory) configuration via annotations (because those fields are not supported by the K8S api):
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: aws-alb-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/tags: Role=Backend , Environment=prod , Name=eros-ingress-alb
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80},{"HTTPS": 443}]'
alb.ingress.kubernetes.io/security-groups : sg-0e3455g455
alb.ingress.kubernetes.io/backend-protocol : HTTP
alb.ingress.kubernetes.io/target-type: instance
alb.ingress.kubernetes.io/healthcheck-path:
alb.ingress.kubernetes.io/success-codes: "200"
alb.ingress.kubernetes.io/certificate-arn:
In your case
Ask yourself what is the reason for adding the annotations.
Then make sure you use a unique prefix for your key in order to avoid collusions.
If you're not sure how to add an annotation to a yaml you can add it manually:
$kubectl annotate pod <pod-name> unique.prefix/for-my-key="value"
And then run $kubectl get po <pod-name> -o yaml to view the annotation that you added manually and copy the yaml to your VCS.
I'm following the http load balancing tutorial for creating an Ingress resource for load balancing.
As seen in the tutorial, the Ingress config file looks like this:
apiVersion: extensions/v1beta2
kind: Ingress
metadata:
name: basic-ingress
spec:
backend:
serviceName: nginx
servicePort: 80
But once I try to create the resource, I always get back this error:
Error from server (BadRequest): error when creating "basic-ingress.yaml": Ingress in version "v1beta2" cannot be handled as a Ingress: no kind "Ingress" is registered for version "extensions/v1beta2"
In the past, I saw similar errors when a value wasn't in the right type (number instead of string), but now that doesn't seem the case.
I tried in both 1.7.8-gke.0 (default) and 1.8.1-gke.1 (latest) versions.
Any help would be really much appreciated.
Cheers
Hm, seems that changing the apiVersion to extensions/v1beta1 does the trick.
[edit] Google Cloud support said that's probably a typo in the documentation. Case solved!