I've noticed some logs in my Zabbix, telling me that some random IP, from my private subnet, is trying to log in as guest user. I know the IP is 10.190.0.1 but there are currently no pods with that IP. Does anyone have any idea how to see which pod had it?
The first thing I thought of, is looking and GCP Log Exporter, but we're not adding labels to logs with what POD it is. I'm sure I should be able to see it from the terminal level. So any suggestion would be nice.
Also, I know it won't be reserved but I took a look either way
gcloud compute addresses list | grep '10.190.0.1'
<empty line>
and
kubectl get all -o wide -A | grep 10.190.0.1
<empty line>
Hi you are doing the right way.
I mean the:
kubectl get pods,svc -o wide
will effectively show you the pods and services and their IP. If the line is empty though, it is because there is no such IP in services or pods in your cluster workoads. two things to check:
maybe the IP has changed
maybe this logs come from an IP in the master node? something from the k8s control plane?
bgess
Related
There is a documentation article here explaining on how one can reserve resources on a node for system use.
What I did not manage to figure out is how can one get these values? If I understand things correctly kubectl top nodes will return available resources, but I would like to see kube-reserved, system-reserved and eviction-threshold as well.
Is it possible?
by checking the kubelet's flag, we can get the values of kube-reserved, system-reserved and eviction-threshold.
ssh into the $NODE and ps aufx | grep kubelet will list out the running kubelet and its flag.
kube-reserved and system-reserved values are only useful for scheduling as scheduler can see the allocatable resources.
To see your eviction-threshold (evictionHard or systemReserved) after login on master node first start the kubectl proxy in the background using the following command:
kubectl proxy --port=8001 &
After that run the following command to see your desired node config (replace your node name in variable.eg VAR="worker-2")
VAR="NODE_NAME"; curl -sSL "http://localhost:8001/api/v1/nodes/$VAR/proxy/configz"
You shoul see a result look like:
"evictionHard":{"imagefs.available":"15%","memory.available":"100Mi","nodefs.available":"10%","nodefs.inodesFree":"5%"},
"systemReserved":{"cpu":"600m","memory":"0.5Gi"}
Enjoy ;)
I have
an openstack, it is Queens, it has octavia for lbaas
a small (test) k8s cluster on top of it (3 nodes, 1 master), version 9.1.2
a deployment called hello which serves a simple webpage saying 'hello world', it works when accessed from within the cluster
I want to expose my deployment as a load balanced service with a floating IP.
I did kubectl expose deployment hello --type=LoadBalancer --name=my-service
It says (kubectl describe service my-service)
Error creating load balancer (will retry): failed to ensure load balancer for service default/my-service: error getting floating ip for port 9cc6442b-2b2f-4b6a-8f91-65dbc2ff13d0: Resource not found
If I manually do: openstack floating ip --port 9cc6442b-2b2f-4b6a-8f91-65dbc2ff13d0 356c8ffa-7bc2-43a9-a8d3-29147ae01727
where:
| ID | Floating IP Address | Port | Floating Network |
| 356c8ffa-7bc2-43a9-a8d3-29147ae01727 | 172.27.81.241 | None | eb31cc74-96ba-4394-aef4-0e94bec46d85 |
and /etc/kubernetes/cloud_config has:
[LoadBalancer]
subnet-id=6a6cdc35-8dda-4982-850e-53c6ee5a5085
floating-network-id=eb31cc74-96ba-4394-aef4-0e94bec46d85
use-octavia=True
(so it is looking for floating IPs on the correct network, and that subnet is the k8s internal subnet)
It all works.
So everything except "associate an IP" has worked. Why does this step fail? Where has k8s logged what it did and how it failed? I can only find docs for pod level logging (and my pod is fine, and serving it's test webpage just great).
(I have lots of quota remaining for 'make more floating ips', and several unused ones hanging around)
I was able to find this No Ports Available when trying to associate a floating IP and this Failed to Associate Floating IP. Maybe those will point you into right direction.
I would recommend that you check this page OpenStack community and look for more answers as I'm not an expert in OpenStack.
As for your question
Where has k8s logged what it did and how it failed?
You can use kubectl describe service <service_name>
Show details of a specific resource or group of resources
Print a detailed description of the selected resources, including related resources such as events or controllers. You may select a single object by name, all objects of that type, provide a name prefix, or label selector. For example:
$ kubectl describe TYPE NAME_PREFIX
For mode debug description please check Debug Services.
I was able to get it working following NFS example in Kubernetes.
https://github.com/kubernetes/kubernetes/tree/master/examples/volumes/nfs
However, when I want to automate all the steps, I need to find the IP and update nfs-pv.yaml PV file with the hard coded IP address as mentioned in the example link page.
Replace the invalid IP in the nfs PV. (In the future, we'll be able to
tie these together using the service names, but for now, you have to
hardcode the IP.)
Now, I wonder that how can we tie these together using the services names?
Or, it is not possible at the latest version of Kubernetes (as of today, the latest stable version is v1.6.2) ?
I got it working after I add kube-dns address to the each minion|node where Kubernetes is running. After login each minion, update resolv.conf file as the following;
cat /etc/resolv.conf
# Generated by NetworkManager
search openstacklocal localdomai
nameserver 10.0.0.10 # I added this line
nameserver 159.107.164.10
nameserver 153.88.112.200
....
I am not sure is it the best way but this works.
Any better solution is welcome.
You can use do this with the help of kube-dns,
check whether it's service running or not,
kubectl get svc --namespace=kube-system
and kube-dns pod also,
kubectl get pods --namespace=kube-system
you have to add respected name-server according to kube-dns on each node in cluster,
For more troubleshooting, follow this document,
https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/
I'm setting up a 2-node Kubernetes system, following the Docker Multi-Node instructions.
My problem is that kubectl get nodes only shows the master, not the worker node as well.
The setup appears to have worked, with all the expected containers running (as far as I know)
I've confirmed that networking works via flannel.
The subnet of the work node appears in the master's subnet list.
So everything looks good, except the node isn't showing up.
My questions:
Am I right in thinking the worker node should now be visible from 'get nodes'?
Does it matter whether the MASTER_IP used to do the setup was the master node's public IP address, or the docker IP? (I've tried both..)
Where do I start with debugging this?
Any pointers gratefully accepted...
Versions:
Ubuntu Trusty 14.04 LTS on both master and worker
Kubernetes v1.1.4
hyperkube:v1.0.3
Answering my own #cloudplatform question...
It turned out to be a problem in worker.sh in Kubernetes v1.1.4.
kubectl is called with "--hostname-override=$(hostname -i)"
On this machine, that returns the IPv6 address.
The K8s code is trying to turn that into a DNS name, and fails.
So looking at the log file for the kubectl container, we see this:
I0122 15:57:33.891577 1786 kubelet.go:1942] Recording NodeReady event message for node 2001:41c9:1:41f::131
I0122 15:57:33.891599 1786 kubelet.go:790] Attempting to register node 2001:41c9:1:41f::131
I0122 15:57:33.894076 1786 kubelet.go:793] Unable to register 2001:41c9:1:41f::131 with the apiserver: Node "2001:41c9:1:41f::131" is invalid: [metadata.name: invalid value '2001:41c9:1:41f::131': must be a DNS subdomain (at most 253 characters, matching regex [a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*): e.g. "example.com", metadata.labels: invalid value '2001:41c9:1:41f::131': must have at most 63 characters, matching regex (([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?: e.g. "MyValue" or ""]
So that's my problem. Take that out and it all works well.
So in answer to my 3 questions:
Yes, the worker node should be visible immediately in 'get nodes'.
I don't think it matters for getting it to work; it may matter for security reasons.
First step after checking that the basic networking is right and the containers are running: look at the log file for the new node's kubectl container.
Update: I wrote this blog post to explain how I got it working http://blog.willmer.org/2016/11/kubernetes-bytemark/
I have new setup of Kubernetes and I created replication with 2. However what I see when I do " kubectl get pods' is that one is running another is "pending". Yet when I go to my 7 test nodes and do docker ps I see that all of them are running.
What I think is happening is that I had to change the default insecure port from 8080 to 7080 (the docker app actually runs on 8080), however I don't know how to tell if I am right, or where else to look.
Along the same vein, is there any way to setup config for kubectl where I can specify the port. Doing kubectl --server="" is a bit annoying (yes I know I can alias this).
If you changed the API port, did you also update the nodes to point them at the new port?
For the kubectl --server=... question, you can use kubectl config set-cluster to set cluster info in your ~/.kube/config file to avoid having to use --server all the time. See the following docs for details:
http://kubernetes.io/v1.0/docs/user-guide/kubectl/kubectl_config.html
http://kubernetes.io/v1.0/docs/user-guide/kubectl/kubectl_config_set-cluster.html
http://kubernetes.io/v1.0/docs/user-guide/kubectl/kubectl_config_set-context.html
http://kubernetes.io/v1.0/docs/user-guide/kubectl/kubectl_config_use-context.html