I'm trying to perform jenkins CI/CD on Kubernetes with dynamic slaves, my jenkins version is official image 2.60.2, while the kubernetes-plugin is 1.0. After add a cloud with kubernetes, the slave can't run up. It shows:
pending—Jenkins doesn’t have label mypod
I refer to
Kubernetes Jenkins plugin - slaves always offline
to configure the jenkins system. I find the issue is described as a defect, I don't know whether this updated to latest jenkins images. Here is the link: https://github.com/jenkinsci/kubernetes-plugin/pull/127
Next error:
Jenkins doesn’t have label mypod
Could this be because of 400d1ed? KubernetesDeclarativeAgentScript.groovy probably needs to get an update then.
Does anyone know how to fix this issue?
The keyword is (as always): look at the logs! You should see your errors when issuing
kubectl logs $JENKINS_POD_NAME
Also, you can try the command below. Here, your faulted slaves will be listed. Look at the logs for these:
kubectl get pods -a
Your issue is related to JNLP communication, slave->master
My jenkins is running in a container and I had to expose the JNLP port to the cluster-node (nodePort).
apiVersion: v1
kind: Service
metadata:
name: jenkins
labels:
app: jenkins
spec:
ports:
- name: jnlp
port: 40294
targetPort: 40294
- name: http
port: 80
targetPort: 8080
selector:
app: jenkins
tier: jenkins
type: NodePort
Also in jenkins security, look for JNLP and enable ALL protocols.
I am still playing with fixed or random ports. Not sure how I can expose random port from a k8s service. Port-range is not supported in k8s.
But I am able to fire off a slave and do some work!
Related
We are using gitlab-ee and kubernetes cluster on ubuntu 18.04 os. When the pipeline is running the port-forward command line, if a user sends a request to my service while port-forward command is active the port-forward command stops.
When I run the pipeline commands from my terminal manually , I don't get any errors.I think gitlab pipeline is broken.
Example, i am running the pipeline from my terminal manually at the same time i am making requests to my service while the pipeline is running the port-forwarding command the pipeline is working as it's supposed to be but when i try to do this with gitlab pipeline any request to the server while the port-forwaring command is executing the server goes down.
How can i solve this issue?
kind: Service
apiVersion: v1
metadata:
name: project_name
namespace: branch_name
labels:
app: project_name
spec:
clusterIP: None
ports:
- name: tcp-port_number
protocol: TCP
port: port_number
targetPort: port_number
selector:
app: project_name
kubectl port-forward -n xxx service/xxxx 80:80 --address=0.0.0.0
Hope the question is clear.
Thanks in advance.
I have started a Kubernetes cluster with Minikube. I used a simple deployment file to create a deployment that runs the Registry container:
apiVersion: apps/v1
kind: Deployment
metadata:
name: registry
labels:
app: registry
spec:
replicas: 3
selector:
matchLabels:
app: registry
template:
metadata:
labels:
app: registry
spec:
containers:
- name: registry
image: registry:latest
ports:
- containerPort: 80
After this, I expose the deployment using a service:
$ kubectl expose deployment/registry --type="LoadBalancer" --port 5000
$ minikube service registry
This exposes my registry to the host machine. I can navigate to http://172.17.174.88:31826/v2/_catalog in my browser and see there's no repositories yet. I have an image running an ASP.Net WebApi project called WeatherApp on my host machine's docker. I run these commands:
$ docker tag 0a259f7ce186 172.17.174.88:31412/weatherapp
$ docker push 172.17.174.88:31412/weatherapp
This causes the following error:
The push refers to repository [172.17.174.88:31412/weatherapp] Get
https://172.17.174.88:31412/v2/: dial tcp 172.17.174.88:31412:
connect: no route to host
I think the problem is that my docker client is trying to connect to the registry over HTTPS, which will not work. How can I force my docker client to use HTTP to push the image to my registry?
I‘m afraid that you will have no chance to fall back to just http. You are forced to use https. You need to configure insecure registries in your docker client.
This might help: https://docs.docker.com/registry/insecure/
I have configured a web application pod exposed via apache on port 80. I'm unable to configure a service + ingress for accessing from the internet. The issue is that the backend services always report as UNHEALTHY.
Pod Config:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
name: webapp
name: webapp
namespace: my-app
spec:
replicas: 1
selector:
matchLabels:
name: webapp
template:
metadata:
labels:
name: webapp
spec:
containers:
- image: asia.gcr.io/my-app/my-app:latest
name: webapp
ports:
- containerPort: 80
name: http-server
Service Config:
apiVersion: v1
kind: Service
metadata:
name: webapp-service
spec:
type: NodePort
selector:
name: webapp
ports:
- protocol: TCP
port: 50000
targetPort: 80
Ingress Config:
kind: Ingress
metadata:
name: webapp-ingress
spec:
backend:
serviceName: webapp-service
servicePort: 50000
This results in backend services reporting as UNHEALTHY.
The health check settings:
Path: /
Protocol: HTTP
Port: 32463
Proxy protocol: NONE
Additional information: I've tried a different approach of exposing the deployment as a load balancer with external IP and that works perfectly. When trying to use a NodePort + Ingress, this issue persists.
With GKE, the health check on the Load balancer is created automatically when you create the ingress. Since the HC is created automatically, so are the firewall rules.
Since you have no readinessProbe configured, the LB has a default HC created (the one you listed). To debug this properly, you need to isolate where the point of failure is.
First, make sure your pod is serving traffic properly;
kubectl exec [pod_name] -- wget localhost:80
If the application has curl built in, you can use that instead of wget.
If the application has neither wget or curl, skip to the next step.
get the following output and keep track of the output:
kubectl get po -l name=webapp -o wide
kubectl get svc webapp-service
You need to keep the service and pod clusterIPs
SSH to a node in your cluster and run sudo toolbox bash
Install curl:
apt-get install curl`
Test the pods to make sure they are serving traffic within the cluster:
curl -I [pod_clusterIP]:80
This needs to return a 200 response
Test the service:
curl -I [service_clusterIP]:80
If the pod is not returning a 200 response, the container is either not working correctly or the port is not open on the pod.
if the pod is working but the service is not, there is an issue with the routes in your iptables which is managed by kube-proxy and would be an issue with the cluster.
Finally, if both the pod and the service are working, there is an issue with the Load balancer health checks and also an issue that Google needs to investigate.
As Patrick mentioned, the checks will be created automatically by GCP.
By default, GKE will use readinessProbe.httpGet.path for the health check.
But if there is no readinessProbe configured, then it will just use the root path /, which must return an HTTP 200 (OK) response (and that's not always the case, for example, if the app redirects to another path, then the GCP health check will fail).
Typically, if I have a remote server, I could access it using ssh, and VS Code gives a beautiful extension for editing and debugging codes for the remote server. But when I create pods in Kuberneters, I can't really ssh into the container and so I cannot edit the code inside the pod or machine. And the kuberneters plugin in VSCode does not really help because the plugin is used to deploy the code. So, I was wondering whether there is a way edit codes inside a pod using VSCode.
P.S. Alternatively if there is a way to ssh into a pod in a kuberneters, that will do too.
If your requirement is "kubectl edit xxx" to use VSCode.
The solution is:
For Linux,macos: export EDITOR='code --wait'
For Windows: set EDITOR=code --wait
Kubernetes + Remote Development extensions now allow:
attaching to k8s pods
open remote folders
execute remotely
debug on remote
integrated terminal into remote
must have:
kubectl
docker (minimum = docker cli - Is it possible to install only the docker cli and not the daemon)
required vscode extentions:
Kubernetes. https://marketplace.visualstudio.com/items?itemName=ms-kubernetes-tools.vscode-kubernetes-tools
Remote Development - https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack
Well a pod is just a unit of deployment in kubernetes which means you can tune the containers inside it to receive an ssh connection.
Let's start by getting a docker image that allows ssh connections. rastasheep/ubuntu-sshd:18.04 image is quite nice for this. Create a deployment with it.
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: debugger
name: debugger
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: debugger
template:
metadata:
creationTimestamp: null
labels:
app: debugger
spec:
containers:
- name: debugger
image: rastasheep/ubuntu-sshd:18.04
imagePullPolicy: "Always"
hostname: debugger
restartPolicy: Always
Now let's create a service of type LoadBalancer such that we can access the pod remotely.
---
apiVersion: v1
kind: Service
metadata:
namespace: default
labels:
app: debugger
name: debugger
spec:
type: LoadBalancer
ports:
- name: "22"
port: 22
targetPort: 22
selector:
app: debugger
status:
loadBalancer: {}
Finally, get the external ip address by running kubectl get svc | grep debugger and use it to test the ssh connection ssh root#external_ip_address
Note the user / pass of this docker image is root / root respectively.
UPDATE
Nodeport example. I tested this and it worked running ssh -p30036#ipBUT I had to enable a firewall rule to make it work. So, the nmap command that I gave you has the answer. Obviously the machines that run kubernetes don't allow inbound traffic on weird ports. Talk to them such that they can give you an external ip address or at least a port in a node.
---
apiVersion: v1
kind: Service
metadata:
name: debugger
namespace: default
labels:
app: debugger
spec:
type: NodePort
ports:
- name: "ssh"
port: 22
nodePort: 30036
selector:
app: debugger
status:
loadBalancer: {}
As mentioned in some of the other answers, you can do this although it is fraught with danger as the cluster can/will replace pods regularly and when it does, it starts a new pod idempotently from the configuration which will not have your changes.
The command below will get you a shell session in your pod , which can sometimes be helpful for debugging if you don't have adequate monitoring/local testing facilities to recreate an issue.
kubectl --namespace=exmaple exec -it my-cool-pod-here -- /bin/bash
Note You can replace the command with any tool that is installed in your container (python3, sh, bash, etc). Also know that that some base images like alpine wont have bash/shell installed be default.
This will open a bash session in the running container on the cluster, assuming you have the correct k8s RBAC permissions.
There is an Cloud Code extension available for VS Code that will serve your purpose.
You can install it in your Visual Studio Code to interact with your Kubernetes cluster.
It allows you to create minikube cluster, Google GKE, Amazon EKS or Azure AKS and manage it from VS Code (you can access cluster information, stream/view logs from pods and open interactive terminal to the container).
You can also enable continuous deployment so it will continuously watch for changes in your files, rebuild the container and redeploy application to the cluster.
It is well explained in Documentation
Hope it will be useful for your use case.
I am using the concourse helm build provided at https://github.com/kubernetes/charts/tree/master/stable/concourse to setup concourse inside of our kubernetes cluster. I have been able to get the setup working and I am able to access it within the cluster but I am having trouble accessing it outside the cluster. The notes from the build show that I can just use kubectl port-forward to get to the webpage but I don't want to have all of the developers have to forward the port just to get to the web ui. I have tried creating a service that has a node port like this:
apiVersion: v1
kind: Service
metadata:
name: concourse
namespace: concourse-ci
spec:
ports:
- port: 8080
name: atc
nodePort: 31080
- port: 2222
name: tsa
nodePort: 31222
selector:
app: concourse-web
type: NodePort
This allows me to get to the webpage and interact with it in most ways but then when I try to look at build status it never loads the events that happened. Instead a network request for /api/v1/builds/1/events is stuck in pending and the steps of the build never load. Any ideas what I can do to be able to completely access concourse external to the cluster?
EDIT: It seems like the events network request normally responds with a text/event-stream data type and maybe the Kubernetes service isn't handling an event stream correctly. Or there is something about concourse that handles event-streams different than the norm.
After plenty of investigation I have found that the the nodePort service is actually working and it is just my antivirus (Sophos) that is silently blocking the response from the events request.
Also, you can expose your port through loadbalancer in kubernetes.
kubectl get deployments
kubectl expose deployment <web pod name> --port=80 --target-port=8080 --name=expoport --type=LoadBalancer
It will create a public IP for you, and you will be able to access concourse on port 80.
not sure since I'm also a newbie but... you can configure your chart by providing your own version of https://github.com/kubernetes/charts/blob/master/stable/concourse/values.yaml
helm install stable/concourse -f custom_values.yaml
there is a 'externalURL' param, maybe worth trying to set it to your URL
## URL used to reach any ATC from the outside world.
##
# externalURL:
In addition, ... if you are on GKE, .... you can use an internal loadbalancer, ... set it up in your values.yaml file
service:
## For minikube, set this to ClusterIP, elsewhere use LoadBalancer or NodePort
## ref: https://kubernetes.io/docs/user-guide/services/#publishing-services---service-types
##
#type: ClusterIP
type: LoadBalancer
## When using web.service.type: LoadBalancer, sets the user-specified load balancer IP
# loadBalancerIP: 172.217.1.174
## Annotations to be added to the web service.
##
annotations:
# May be used in example for internal load balancing in GCP:
cloud.google.com/load-balancer-type: Internal