Problem with creating argo rollouts CRD. When I try to apply CRD it gives me: error when creating "rollout.yaml": the server could not find the requested resource (post rollouts.argoproj.io)
This is my yaml:
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: example-rollout
spec:
selector:
matchLabels:
app: example
template:
metadata:
labels:
app: example
spec:
containers:
- image: demo/template1:v1
imagePullPolicy: Always
name: demo/template1-v1
ports:
- containerPort: 8080
strategy:
canary:
stableService: app1
steps: # describes the strategy for sending traffic to the canary
- setWeight: 30 # 30% of the traffic will go to the canary for 30 seconds
- pause: {duration: 30}
- setWeight: 60 # 60% of the traffic will go to the canary for 45 seconds
- pause: {duration: 45}
- setWeight: 100 # 100% of the traffic will go to the canary for 10 seconds
- pause: {duration: 10}
Maybe the problem is because of image or labels
Related
I have a kubernetes cluster on which I have deployed a opensearch cluster and opensearch dashboard using Helm, I am also able to deploy logstash using helm successfully but I am confused on how to integrate those, I want to feed data to my Opensearch using logstash as my OBJECTIVE as I am not able to find much documentation on it as well. Any help is appreciated....Thanks in advance!
Deployed opensearch using Helm and logstash as well but unable to integrate them
Update here!!!
Have made a few changes to simplify the deployment and more control over the function,
I am testing deployment and service files this time, I will add the files below
Opensearch deployment file
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
namespace: logging
name: opensearch
labels:
component: opensearch
spec:
selector:
matchLabels:
component: opensearch
replicas: 1
serviceName: opensearch
template:
metadata:
labels:
component: opensearch
spec:
initContainers:
- name: init-sysctl
image: busybox
imagePullPolicy: IfNotPresent
command:
- sysctl
- -w
- vm.max_map_count=262144
securityContext:
privileged: true
containers:
- name: opensearch
securityContext:
capabilities:
add:
- IPC_LOCK
image: opensearchproject/opensearch
env:
- name: KUBERNETES_CA_CERTIFICATE_FILE
value: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: "cluster.name"
value: "opensearch-cluster"
- name: "network.host"
value: "0.0.0.0"
- name: "discovery.seed_hosts"
value: "[]"
- name: discovery.type
value: single-node
- name: OPENSEARCH_JAVA_OPTS
value: -Xmx512M -Xms512M
- name: "plugins.security.disabled"
value: "false"
ports:
- containerPort: 9200
name: http
protocol: TCP
- containerPort: 9300
name: transport
protocol: TCP
volumeMounts:
- name: os-mount
mountPath: /data
volumes:
- name: os-mount
persistentVolumeClaim:
claimName: nfs-pvc-os-logging
Opensearch svc file
---
apiVersion: v1
kind: Service
metadata:
name: opensearch
namespace: logging
labels:
service: opensearch
spec:
type: ClusterIP
selector:
component: opensearch
ports:
- port: 9200
targetPort: 9200
Opensearch dashboard deployment
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: open-dash
namespace: logging
spec:
replicas: 1
selector:
matchLabels:
app: open-dash
template:
metadata:
labels:
app: open-dash
spec:
# securityContext:
# runAsUser: 0
containers:
- name: opensearch-dashboard
image: opensearchproject/opensearch-dashboards:latest
ports:
- containerPort: 80
env:
# - name: ELASTICSEARCH_URL
# value: https://opensearch.logging:9200
# - name: "SERVER_HOST"
# value: "localhost"
# - name: "opensearch.hosts"
# value: https://opensearch.logging:9200
- name: OPENSEARCH_HOSTS
value: '["https://opensearch.logging:9200"]'
Opensearch Dashboard svc
---
apiVersion: v1
kind: Service
metadata:
name: opensearch
namespace: logging
labels:
service: opensearch
spec:
type: ClusterIP
selector:
component: opensearch
ports:
- port: 9200
targetPort: 9200
with the above configuration I am able to get the Dashboard UI open but in Dashboard pod logs I can see a 400 code logs can anyone please try to reproduce this issue, Also I need to integrate the logstash with this stack.
{"type":"response","#timestamp":"2023-02-20T05:05:34Z","tags":[],"pid":1,"method":"head","statusCode":400,"req":{"url":"/app/home","method":"head","headers":{"connection":"Keep-Alive","content-type":"application/json","host":"3.108.199.0:30406","user-agent":"Manticore 0.9.1","accept-encoding":"gzip,deflate","securitytenant":"user"},"remoteAddress":"10.244.1.1","userAgent":"Manticore 0.9.1"},"res":{"statusCode":400,"responseTime":2,"contentLength":9},"message":"HEAD /app/home 400 2ms - 9.0B
When deploying a logstash pod I get an error that
[WARN ] 2023-02-20 05:13:52.212 [Ruby-0-Thread-9: /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/outputs/opensearch/http_client/pool.rb:217] opensearch - Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"http://logstash:xxxxxx#opensearch.logging:9200/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>"OpenSearch Unreachable: [http://logstash:xxxxxx#opensearch.logging:9200/][Manticore::ClientProtocolException] opensearch.logging:9200 failed to respond"}
Can somebody please try to help me with this puzzle
#Benla have made changes as per your recommendation to image now I am getting the following logs in logstash
[2023-02-20T05:18:43,028][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-02-20T05:18:43,147][INFO ][org.reflections.Reflections] Reflections took 70 ms to scan 1 urls, producing 127 keys and 444 values
[2023-02-20T05:18:43,351][INFO ][logstash.javapipeline ] Pipeline main is configured with pipeline.ecs_compatibility: v8 setting. All plugins in this pipeline will default to ecs_compatibility => v8 unless explicitly configured otherwise.
[2023-02-20T05:18:43,370][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>16, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2000, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x3bf49916#/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:131 run>"}
[2023-02-20T05:18:43,811][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.44}
[2023-02-20T05:18:43,816][INFO ][logstash.inputs.beats ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2023-02-20T05:18:43,821][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2023-02-20T05:18:43,835][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2023-02-20T05:18:43,869][INFO ][org.logstash.beats.Server][main][0710cad67e8f47667bc7612580d5b91f691dd8262a4187d9eca8cf87229d04aa] Starting server on port: 5044
I started getting these endless loop of logs
[WARN ] 2023-02-20 05:13:37.191 [Ruby-0-Thread-9: /usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-opensearch-2.0.1-java/lib/logstash/outputs/opensearch/http_client/pool.rb:217] opensearch - Attempted to resurrect connection to dead OpenSearch instance, but got an error {:url=>"http://logstash:xxxxxx#opensearch.logging:9200/", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :message=>"OpenSearch Unreachable: [http://logstash:xxxxxx#opensearch.logging:9200/][Manticore::ClientProtocolException] opensearch.logging:9200 failed to respond"}
I think there is not much difference in deploying logstash<->opensearch pipeline from deploying logstash<->elasticsearch in K8S,
The only differences I've encountered are:
The docker or other container runtime image should be opensearchproject/logstash-oss-with-opensearch-output-plugin:latest rather than logstash.
The output plugin in your logstash config file sould be
opensearch rather than elasticsearch.
Other than those you can go to every Logstash<->ElasticSearch in K8S tutorial, there are many.
Good Luck!
I have configured dockerhub to build a new image with tags latest and dev-<version> as new tag <version> is appeared in GitHub. I have no idea how to configure Tekton or any other cloud-native tool to automatically deploy new images as they become available at the registry.
Here's my k8s configuration:
apiVersion: v1
kind: List
items:
- apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: my-app-ingress
spec:
rules:
- http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: my-app-service
port:
number: 80
- apiVersion: v1
kind: Service
metadata:
name: my-app-service
spec:
ports:
- port: 80
targetPort: 8000
selector:
app: my-app
type: LoadBalancer
- apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app-local-deployment
labels:
app: my-app
type: web
spec:
replicas: 2
minReadySeconds: 15
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 25%
maxSurge: 1
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
imagePullSecrets:
- name: regcred
containers:
- name: backend
image: zuber93/my-app:dev-latest
imagePullPolicy: IfNotPresent
envFrom:
- secretRef:
name: my-app-local-secret
ports:
- containerPort: 8000
readinessProbe:
httpGet:
path: /flvby
port: 8000
initialDelaySeconds: 10
periodSeconds: 5
- name: celery
image: zuber93/my-app:dev-latest
imagePullPolicy: IfNotPresent
workingDir: /code
command: [ "/code/run/celery.sh" ]
envFrom:
- secretRef:
name: my-app-local-secret
- name: redis
image: redis:latest
imagePullPolicy: IfNotPresent
Short answer is:
Either setup a webhook from dockerhub (https://docs.docker.com/docker-hub/webhooks/) to tekton using triggers.
Or (depends on your security and if your cluster is reachable from www or not)
Poll dockerhub and trigger tekton upon new image detection.
(This can be done in many different ways, simple instant service, scheduled cronjob, etc in k8s)
So, you choose push or pull. ;)
I would ask "why not trigger from your git repo directly?"
Finally, I've explored keel.sh. It can handle appearing of new images and deploy them into the cluster.
I have a service in Kubernetes that receives Http requests to create users,
Only with 1 pod, it correctly reaches 100 requests per minute, after this, it has latencies, the point is that if you hold 100 requests with 1 pod, should you keep 500 requests per minute with 5 pods?
Because even with 10 pods, when exceeding 100 orders per minute, dont correctly distributed the load and appears latency in the services.
The default load configuration I understand is round robin, the problem is that I see that the ram increases only in one of the pods and does not distribute the load correctly.
This is my service yaml deploy and my HPA yaml.
Deploy Yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: create-user-service
labels:
app: create-user-service
spec:
replicas: 1
selector:
matchLabels:
app: create-user-service
template:
metadata:
labels:
app: create-user-service
spec:
volumes:
- name: key
secret:
secretName: my-secret-key
containers:
### [LISTPARTY CONTAINER]
- name: create-user-service
image: docker/user-create/create-user-service:0.0.1
volumeMounts:
- name: key
mountPath: /var/secrets/key
ports:
- containerPort: 8080
env:
- name: PORT
value: "8080"
resources:
limits:
cpu: "2.5"
memory: 6Gi
requests:
cpu: "1.5"
memory: 5Gi
livenessProbe: ## is healthy
failureThreshold: 3
httpGet:
path: /healthcheck/livenessprobe
port: 8080
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
name: create-user-service
spec:
ports:
- port: 8080
targetPort: 8080
protocol: TCP
name: http
selector:
app: create-user-service
type: NodePort
HPA Yaml
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: create-user-service
spec:
maxReplicas: 10
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: create-user-service
metrics:
- type: Resource
resource:
name: cpu
targetAverageUtilization: 75
- type: Resource
resource:
name: memory
targetAverageUtilization: 75
- external:
metricName: serviceruntime.googleapis.com|api|request_count
metricSelector:
matchLabels:
resource.type: api
resource.labels.service: create-user-service.endpoints.user-create.cloud.goog
targetAverageValue: "3"
type: External
What may be happend?.
Thank you all.
I am for this example running the "echoheaders" Nginx in a deployment with 2 replicas. When I delete 1 pod, I sometimes get slow responses and errors for ~40 seconds.
We are running our API-gateway in Kubernetes, and need to be able to allow the Kubernetes scheduler to handle the pods as it sees fit.
We recently wanted to introduce session affinity, and for that, we wanted to migrate to the new and shiny NEG's: Network Endpoint Groups:
https://cloud.google.com/load-balancing/docs/negs/
When using NEG we experience problems during failover. Without NEG we're fine.
deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: echoheaders
labels:
app: echoheaders
spec:
replicas: 2
selector:
matchLabels:
app: echoheaders
template:
metadata:
labels:
app: echoheaders
spec:
containers:
- image: brndnmtthws/nginx-echo-headers
imagePullPolicy: Always
name: echoheaders
readinessProbe:
httpGet:
path: /
port: 8080
lifecycle:
preStop:
exec:
# Hack: wait for kube-proxy to remove endpoint before exiting, and
# gracefully shut down
command: ["bash", "-c", "sleep 10; nginx -s quit; sleep 40"]
restartPolicy: Always
terminationGracePeriodSeconds: 60
service.yaml
apiVersion: v1
kind: Service
metadata:
name: echoheaders
labels:
app: echoheaders
annotations:
cloud.google.com/neg: '{"ingress": true}'
spec:
ports:
- port: 80
protocol: TCP
targetPort: 8080
selector:
app: echoheaders
ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.global-static-ip-name: echoheaders-staging
name: echoheaders-staging
spec:
backend:
serviceName: echoheaders
servicePort: 80
When deleting a pod I get errors as shown in this image of
$ httping -G -K 35.190.69.21
(https://i.imgur.com/u14MvHN.png)
This is new behaviour when using NEG. Disabling NEG gives the old behaviour with working failover.
Any way to use Google LB, ingress, NEG and Kubernetes without errors during pod deletion?
In GCP load balancers a GET request will only be served a 502 after two subsequent backends fail to meet the response timeout or a impactful error occurred which seems more plausible.
What is possibly happening may be an interim period, wherein a Pod was due to be terminated and had received its SIGTERM, but was still considered healthy by the load balancer and was sent a request. Since this period was so brief, it wasn't able to complete the request and closed the connection.
A graceful service stop[1] in the machine will make that after receiving a SIGTERM, your service would continue to serve in-flight requests, but refuse new connections. This may solve your issue, but keep in mind that there is no guarantee of zero-downtime.
[1] https://landing.google.com/sre/sre-book/chapters/load-balancing-datacenter/#robust_approach_lame_duck
I am unable to get circuit breaking configuration to work on my elb through egress config.
ELB
elb has success rate of 25% (75% 500 error & 25% with status 200),
the elb has 4 instances, only 1 returns a successful response, other instances are configured to returns 500 error for testing purpose.
Setup
k8s: v1.7.4
istio: 0.5.0
env: k8s on aws
Egress rule
apiVersion: config.istio.io/v1alpha2
kind: EgressRule
metadata:
name: elb-egress-rule
spec:
destination:
service: xxxx.us-east-1.elb.amazonaws.com
ports:
- port: 80
protocol: http
Destination Policy
kind: DestinationPolicy
metadata:
name: elb-circuit-breaker
spec:
destination:
service: xxxx.us-east-1.elb.amazonaws.com
loadBalancing:
name: RANDOM
circuitBreaker:
simpleCb:
maxConnections: 100
httpMaxPendingRequests: 100
sleepWindow: 3m
httpDetectionInterval: 1s
httpMaxEjectionPercent: 100
httpConsecutiveErrors: 3
httpMaxRequestsPerConnection: 10
Route rules: not set
Testing
apiVersion: v1
kind: Service
metadata:
name: sleep
labels:
app: sleep
spec:
ports:
- port: 80
name: http
selector:
app: sleep
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: sleep
spec:
replicas: 1
template:
metadata:
labels:
app: sleep
spec:
containers:
- name: sleep
image: tutum/curl
command: ["/bin/sleep","infinity"]
imagePullPolicy: IfNotPresent
.
export SOURCE_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
kubectl exec -it $SOURCE_POD -c sleep bash
Sending requests in parallel from the pod
#!/bin/sh
set -m # Enable Job Control
for i in `seq 100`; do # start 100 jobs in parallel
curl xxxx.us-east-1.elb.amazonaws.com &
done
Response
Currently, Istio considers an Egress Rule to designate a single host. This single host will not be ejected due to the load balancer's panic threshold of Envoy (the sidecar proxy implementation of Istio). The default panic threshold of Envoy is 50%. This means that at least two hosts are required for one host to be ejected, so the single host of an Egress Rule will not be ejected.
This practically means that httpConsecutiveErrors does not effect the external services. This lack of functionality should be partially resolved with External Services of Istio that will replace the Egress Rules.
See documentation of the Istio External Services backed by multiple endpoints -https://github.com/istio/api/blob/master/routing/v1alpha2/external_service.proto#L113