I search the previews post for this but I can't find the solution, sorry.
I install metrics server on kubeadm v1.12 and I get this error from the logs:
1 master node and 1 slave node, in private network.
Get https://ip-10-0-1-154:10250/stats/summary/: x509: a certificate signed by an unknown authority, unable to fully scrape metrics from source
I don't install any certificate.
How can I install a new certificate and where I need to change this without set up a new kubernetes cluster?
Sorry for the noob question, I tried to create a new certificate but I cannot make kubelet to change.
It's a problem with kubeadm in where it generates the kubelet certificates on the nodes under /var/lib/kubelet/pki (kubelet.crt,kubelet.key) signed by a different CA from the one used for the master(s) under /etc/kubernetes/pki (ca.crt). Some background here
.
You'll have to regenerate the certificates for your kubelets signed by the CA on the master(s) /etc/kubernetes/pki/ca.crt
You can follow something like this. For example use cfssl
Something like this:
$ mkdir ~/mycerts; cd ~/mycerts
$ cp /etc/kubernetes/pki/ca.crt ca.pem
$ cp /etc/kubernetes/pki/ca.key ca-key.pem
Create a file kubelet-csr.json with something like this:
{
"CN": "kubernetes",
"hosts": [
"127.0.0.1",
"<your-node-name>",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [{
"C": "US",
"ST": "NY",
"L": "City",
"O": "Org",
"OU": "Unit"
}]
}
Create a ca-config.json file:
{
"signing": {
"default": {
"expiry": "8760h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "8760h"
}
}
}
}
Create a config.json file:
{
"signing": {
"default": {
"expiry": "168h"
},
"profiles": {
"www": {
"expiry": "8760h",
"usages": [
"signing",
"key encipherment",
"server auth"
]
},
"client": {
"expiry": "8760h",
"usages": [
"signing",
"key encipherment",
"client auth"
]
}
}
}
}
Generate the certs:
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem \
--config=ca-config.json -profile=kubernetes \
kubelet-csr.json | cfssljson -bare kubelet
Copy the files to your nodes:
$ scp kubelet.pem <node-ip>:/var/lib/kubelet/pki/kubelet.crt
$ scp kubelet-key.pem <node-ip>:/var/lib/kubelet/pki/kubelet.key
Restart the kubelet on your node:
$ systemctl restart kubelet
PD. Opened this to track the issue.
Related
I've been working with a bunch of k8s clusters for a while, using kubectl from the command line to examine information. I don't actually call kubectl directly, I wrap it in multiple scripting layers. I also don't use contexts, as it's much easier for me to specify different clusters in a different way. The resulting kubectl command line has explicit --server, --namespace, and --token parameters (and one other flag to disable tls verify).
This all works fine. I have no trouble with this.
However, I'm now trying to use telepresence, which doesn't give me a choice (yet) of not using contexts to configure this. So, I now have to figure out how to use contexts.
I ran the following (approximate) command:
kubectl config set-context mycontext --server=https://host:port --namespace=abc-def-ghi --insecure-skip-tls-verify=true --token=mytoken
And it said: "Context "mycontext " modified."
I then ran "kubectl config view -o json" and got this:
{
"kind": "Config",
"apiVersion": "v1",
"preferences": {},
"clusters": [],
"users": [],
"contexts": [
{
"name": "mycontext",
"context": {
"cluster": "",
"user": "",
"namespace": "abc-def-ghi"
}
}
],
"current-context": "mycontext"
}
That doesn't look right to me.
I then ran something like this:
telepresence --verbose --swap-deployment mydeployment --expose 8080 --run java -jar target/my.jar -Xdebug -Xrunjdwp:transport=dt_socket,address=5000,server=y,suspend=n
And it said this:
T: Error: Namespace 'abc-def-ghi' does not exist
Update:
And I can confirm that this isn't a problem with telepresence. If I just run "kubectl get pods", it fails, saying "The connection to the server localhost:8080 was refused". That tells me it obviously can't connect to the k8s server. The key is my "set-context" command. It's obviously not working, and I don't understand what I'm missing.
You don't have any clusters or credentials defined in your configuration. First, you need to define a cluster:
$ kubectl config set-cluster development --server=https://1.2.3.4 --certificate-authority=fake-ca-file
Then something like this for the user:
$ kubectl config set-credentials developer --client-certificate=fake-cert-file --client-key=fake-key-seefile
Then you define your context based on your cluster, user and namespace:
$ kubectl config set-context dev-frontend --cluster=development --namespace=frontend --user=developer
More information here
Your config should look something like this:
$ kubectl config view -o json
{
"kind": "Config",
"apiVersion": "v1",
"preferences": {},
"clusters": [
{
"name": "development",
"cluster": {
"server": "https://1.2.3.4",
"certificate-authority-data": "DATA+OMITTED"
}
}
],
"users": [
{
"name": "developer",
"user": {
"client-certificate": "fake-cert-file",
"client-key": "fake-key-seefile"
}
}
],
"contexts": [
{
"name": "dev-frontend",
"context": {
"cluster": "development",
"user": "developer",
"namespace": "frontend"
}
}
],
"current-context": "dev-frontend"
}
I'm following this instruction to setup glusterfs on my kubernetes cluster. At heketi-client/bin/heketi-cli setup-openshift-heketi-storage part, heketi-cli tells me :
Error: volume create: heketidbstorage: failed: Host 192.168.99.25 not connected
or sometimes:
Error: volume create: heketidbstorage: failed: Staging failed on 192.168.99.26. Error: Host 192.168.99.25 not connected
heketi.json is
{
"_port_comment": "Heketi Server Port Number",
"port": "8080",
"_use_auth": "Enable JWT authorization. Please enable for deployment",
"use_auth": false,
"_jwt": "Private keys for access",
"jwt": {
"_admin": "Admin has access to all APIs",
"admin": {
"key": "7319"
},
"_user": "User only has access to /volumes endpoint",
"user": {
"key": "7319"
}
},
"_glusterfs_comment": "GlusterFS Configuration",
"glusterfs": {
"_executor_comment": "Execute plugin. Possible choices: mock, kubernetes, ssh",
"executor": "kubernetes",
"_db_comment": "Database file name",
"db": "/var/lib/heketi/heketi.db",
"kubeexec": {
"rebalance_on_expansion": true
},
"sshexec": {
"rebalance_on_expansion": true,
"keyfile": "/etc/heketi/private_key",
"fstab": "/etc/fstab",
"port": "22",
"user": "root",
"sudo": false
}
},
"_backup_db_to_kube_secret": "Backup the heketi database to a Kubernetes secret when running in Kubernetes. Default is off.",
"backup_db_to_kube_secret": false
}
topology-sample.json is
{
"clusters": [
{
"nodes": [
{
"node": {
"hostnames": {
"manage": [
"redis-test25"
],
"storage": [
"192.168.99.25"
]
},
"zone": 1
},
"devices": [
{
"name": "/dev/sda7",
"destroydata": true
}
]
},
{
"node": {
"hostnames": {
"manage": [
"redis-test26"
],
"storage": [
"192.168.99.26"
]
},
"zone": 1
},
"devices": [
{
"name": "/dev/sda7",
"destroydata": true
}
]
},
{
"node": {
"hostnames": {
"manage": [
"redis-test01"
],
"storage": [
"192.168.99.113"
]
},
"zone": 1
},
"devices": [
{
"name": "/dev/sda7",
"destroydata": true
}
]
}
]
}
]
}
The heketi-cli is v8.0.0 and kubernetes is v1.12.3
How do I fix this problem?
Update: Just found that I missed the iptables part, but now the message becomes
Error: volume create: heketidbstorage: failed: Host 192.168.99.25 is not in 'Peer in Cluster' state
seems that one of the glusterfs pod cannot connect to others, I tried kubectl exec -i glusterfs-59ftx -- gluster peer status:
Number of Peers: 2
Hostname: 192.168.99.26
Uuid: 6950db9a-3d60-4625-b642-da5882396bee
State: Peer Rejected (Disconnected)
Hostname: 192.168.99.113
Uuid: 78983466-4499-48d2-8411-2c3e8c70f89f
State: Peer Rejected (Disconnected)
while the other one said:
Number of Peers: 1
Hostname: 192.168.99.26
Uuid: 23a0114d-65b8-42d6-8067-7efa014af68d
State: Peer in Cluster (Connected)
I solved these problems by myself.
For first part, the reason is that I didn't setup iptables in every nodes according to Infrastructure Requirements.
For second part according to this article, delete all file in /var/lib/glusterd except glusterd.info and then start over from Kubernete Deploy.
I'm trying to connect to a one-node cluster created successfully but so far unable to connect from the node itself or from a separate client machine.
The setup is a one node cluster where the cluster and the server/node are the same. I'm using self-signed certificates and have (in theory) installed the primary and secondary certificates on the cluster/server and an admin and standard client certificate on a client machine.
If I install an unsecure cluster on the machine, I can successfully navigate to it via a browser, but when I create a secured cluster, I'm unable to connect.
Attempting to connect via PowerShell errors with "Failed to authenticate server" identity".
Browsing to the server with Chrome just fails.
The cluster JSON config is as follows:
{
"name": "SomeFancyCluster",
"clusterConfigurationVersion": "1.0.0",
"apiVersion": "10-2017",
"nodes": [
{
"nodeName": "vm0",
"iPAddress": "some.server.name",
"nodeTypeRef": "NodeType0",
"faultDomain": "fd:/dc1/r0",
"upgradeDomain": "UD0"
}
],
"properties": {
"diagnosticsStore": {
"metadata": "Please replace the diagnostics file share with an actual file share accessible from all cluster machines.",
"dataDeletionAgeInDays": "21",
"storeType": "FileShare",
"connectionstring": "c:\\ProgramData\\SF\\DiagnosticsStore"
},
"security": {
"metadata": "The Credential type X509 indicates this is cluster is secured using X509 Certificates. The thumbprint format is - d5 ec 42 3b 79 cb e5 07 fd 83 59 3c 56 b9 d5 31 24 25 42 64.",
"ClusterCredentialType": "Windows",
"ServerCredentialType": "X509",
"WindowsIdentities": {
"ClusterIdentity": "some.server"
},
"CertificateInformation": {
"ClusterCertificate": {
"Thumbprint": "ab123456789123456789123456789123456789cd",
"ThumbprintSecondary": "ef123456789123456789123456789123456789gh",
"X509StoreName": "My"
},
"ServerCertificate": {
"Thumbprint": "ab123456789123456789123456789123456789cd",
"ThumbprintSecondary": "ef123456789123456789123456789123456789gh",
"X509StoreName": "My"
},
"ClientCertificateThumbprints": [{
"CertificateThumbprint": "ab123456789123456789123456789123456789ef",
"IsAdmin": false
}, {
"CertificateThumbprint": "ab123456789123456789123456789123456789gh",
"IsAdmin": true
}]
}
},
"nodeTypes": [
{
"name": "NodeType0",
"clientConnectionEndpointPort": "19000",
"clusterConnectionEndpointPort": "19001",
"leaseDriverEndpointPort": "19002",
"serviceConnectionEndpointPort": "19003",
"httpGatewayEndpointPort": "19080",
"reverseProxyEndpointPort": "30000",
"applicationPorts": {
"startPort": "20001",
"endPort": "20031"
},
"ephemeralPorts": {
"startPort": "20032",
"endPort": "20287"
},
"isPrimary": true
}
],
"fabricSettings": [
{
"name": "Setup",
"parameters": [
{
"name": "FabricDataRoot",
"value": "C:\\ProgramData\\SF"
},
{
"name": "FabricLogRoot",
"value": "C:\\ProgramData\\SF\\Log"
}
]
}
]
}
}
I've then attempted to connect, as per the docs using the following PowerShell command
Connect-ServiceFabricCluster -ConnectionEndpoint some.server.name:19000 -KeepAliveIntervalInSec 10 -X509Credential -ServerCertThumbprint <Server Thumbprint> -FindType FindByThumbprint -FindValue <Client Thumbprint> -StoreLocation CurrentUser -StoreName My
I installed the client certificate to 'Certificates - Current User/Personal/Certificates'
On the server, the certificates are all in 'Certificates (Local Computer)/Personal/Certificates'
With the primary server/cluster certificate also in 'Certificates (Local Computer)/Trusted Root Certification Authorities' with a CN equal to some.server.name.
Any ideas as to what's missing/wrong above?
I have a feeling this is related to the ClusterIdentity field in the configuration. Could you tell me why did you add it?
You are authenticating by certificate so remove WindowsIdentities section entirely and change ClusterCredentialType to X509 too if you want to connect to cluster by certificate. Update cluster configuration and you should be able to connect.
Otherwise provide -WindowsCredential parameter not a certificate while you are connecting.
I need to map a volume while starting the container, I am able to do it so with yaml file.
Is there an way volume mapping can be done via command line without using yaml file? just like
-v option in docker?
without using yaml file
Technically, yes: you would need a json file, as illustrated in "Create kubernetes pod with volume using kubectl run"
See kubectl run.
kubectl run -i --rm --tty ubuntu --overrides='
{
"apiVersion": "batch/v1",
"spec": {
"template": {
"spec": {
"containers": [
{
"name": "ubuntu",
"image": "ubuntu:14.04",
"args": [
"bash"
],
"stdin": true,
"stdinOnce": true,
"tty": true,
"volumeMounts": [{
"mountPath": "/home/store",
"name": "store"
}]
}
],
"volumes": [{
"name":"store",
"emptyDir":{}
}]
}
}
}
}
' --image=ubuntu:14.04 --restart=Never -- bash
We are using hyperkube's apiserver and configuring it via a manifest file:
"containers":[
{
"name": "apiserver",
"image": "gcr.io/google_containers/hyperkube-amd64:v1.2.1",
"command": [
"/hyperkube",
"apiserver",
"--service-cluster-ip-range=192.168.0.0/23",
"--service-node-port-range=9000-9999",
"--bind-address=127.0.0.1",
"--etcd-servers=http://127.0.0.1:4001",
"--admission-control=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota",
"--client-ca-file=/srv/kubernetes/ca.crt",
"--basic-auth-file=/srv/kubernetes/basic_auth.csv",
"--min-request-timeout=300",
"--tls-cert-file=/srv/kubernetes/server.cert",
"--tls-private-key-file=/srv/kubernetes/server.key",
"--token-auth-file=/srv/kubernetes/known_tokens.csv",
"--allow-privileged=true",
"--v=4"
],
"volumeMounts": [
{
"name": "data",
"mountPath": "/srv/kubernetes"
}
]
}
I'm trying to figure out how to set up a different set of tokens than in /srv/kubernetes/known_tokens.csv to have users "superuser" and "reader", instead of admin, kubelet, and kube_proxy. How can I do this?
Your manifest is using the exposed volume path /srv/kubernetes, so should be able to map that to another persistent volume (http://kubernetes.io/docs/user-guide/volumes/) and setup the new files there.
You can do that by specifying a volume:
"volumes": [
{
"name": "data",
"hostPath": {
"path": "/foo"
}
}
]