Accessing minikube via the KubernetesPodOperator in airflow - kubernetes

I am trying to make the airflow KubernetesPodOperator work with minikube. But unfortunately, the operator does not find the kubernetes cluster.
The dag returned to me the following error:
ERROR - HTTPSConnectionPool(host='192.168.49.2', port=8443): Max retries exceeded with url: /api/v1/namespaces/default/pods?labelSelector=dag_id%... (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f9a63cdcc10>: Failed to establish a new connection: [Errno 110] Connection timed out'))
I suspect an error in my config file definition which you will find below. The server address I used is the one return by minikube ip.
apiVersion: v1
clusters:
- cluster:
certificate-authority: ./ca.crt
extensions:
- extension:
last-update: Tue, 26 Apr 2022 15:16:20 CEST
provider: minikube.sigs.k8s.io
version: v1.25.2
name: cluster_info
server: https://192.168.49.2:8443
name: minikube
contexts:
- context:
cluster: minikube
extensions:
- extension:
last-update: Tue, 26 Apr 2022 15:16:20 CEST
provider: minikube.sigs.k8s.io
version: v1.25.2
name: context_info
namespace: default
user: minikube
name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
user:
client-certificate: ./client.crt
client-key: ./client.key
Any ideas of what I could have done wrong?
Thanks!

Related

error: kubectl get pods -> couldn't get current server API group list |docker-desktop

I've been building and destroying couple of terraform projects and then after couple of hours came to a weird error saying:
kubectl get pods
E0117 14:10:23.537699 21524 memcache.go:238] couldn't get current server API group list: Get "https://192.168.59.102:8443/api?t
E0117 14:10:33.558130 21524 memcache.go:238] couldn't get current server API group list: Get "https://192.168.59.102:8443/api?t
tried to check everything I can even restore and purge data on docker-desktop but it didn't help.
my .kube/config :
kubectl config view
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://kubernetes.docker.internal:6443
name: docker-desktop
- cluster:
certificate-authority: C:\Users\dani0\.minikube\ca.crt
extensions:
- extension:
last-update: Tue, 17 Jan 2023 14:04:24 IST
provider: minikube.sigs.k8s.io
version: v1.28.0
name: cluster_info
server: https://192.168.59.102:8443
name: minikube
contexts:
- context:
cluster: docker-desktop
name: docker-desktop
- context:
extensions:
- extension:
last-update: Tue, 17 Jan 2023 14:04:24 IST
provider: minikube.sigs.k8s.io
version: v1.28.0
name: context_info
namespace: default
user: minikube
name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: docker-desktop
user:
client-certificate-data: DATA+OMITTED
client-key-data: DATA+OMITTED
- name: minikube
user:
client-certificate: C:\Users\dani0\.minikube\profiles\minikube\client.crt
client-key: C:\Users\dani0\.minikube\profiles\minikube\client.key
well I've fixed it by deleting the minikube cluster with "minikube delete" and then just "minikube start" and now it seems to work to me :)

velero with velero-plugin-for-aws backup jobs failed

k3s cluster.
I have used velero helm installation:
helm install vmware-tanzu/velero --namespace velero-minio -f helm-custom-values-minio.yaml --generate-name --create-namespace
and
helm install vmware-tanzu/velero --namespace velero-aws -f helm-custom-values-aws.yaml --generate-name --create-namespace
Custom helm values:
helm-custom-values-minio.yaml
configuration:
provider: aws
backupStorageLocation:
bucket: k3s-backup
name: minio
default: false
config:
region: minio
s3ForcePathStyle: true
s3Url: http://10.10.5.15:9009
volumeSnapshotLocation:
name: minio
config:
region: minio
credentials:
secretContents:
cloud: |
[default]
aws_access_key_id=minioadm
aws_secret_access_key=<password>
initContainers:
- name: velero-plugin-for-aws
image: velero/velero-plugin-for-aws:latest
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: /target
name: plugins
snapshotsEnabled: true
deployRestic: true
and helm-custom-values-aws.yaml
configuration:
provider: aws
backupStorageLocation:
name: aws-s3
bucket: k3s-backup-aws
default: false
provider: aws
config:
region: us-east-1
s3ForcePathStyle: false
volumeSnapshotLocation:
name: aws-s3
provider: aws
config:
region: us-east-1
credentials:
secretContents:
cloud: |
[default]
aws_access_key_id=A..............MJ
aws_secret_access_key=qZ79rA/yVUq2c................xnIA
initContainers:
- name: velero-plugin-for-aws
image: velero/velero-plugin-for-aws:latest
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: /target
name: plugins
snapshotsEnabled: true
deployRestic: true
velero backup jobs:
velero create backup k3s-mongodb-restic-minio --include-namespaces mongodb --default-volumes-to-restic=true --storage-location minio -n velero-minio
velero create backup k3s-mongodb-restic-aws --include-namespaces mongodb --default-volumes-to-restic=true --storage-location aws-s3 -n velero-aws
....
They all failed:
Restic Backups:
Failed:
mongodb/mongodb-cluster-0: agent-scripts, data-volume, healthstatus, hooks, logs-volume, mongodb-cluster-keyfile, tmp
mongodb/mongodb-cluster-1: agent-scripts, data-volume, healthstatus, hooks, logs-volume, mongodb-cluster-keyfile, tmp
time="2022-10-17T17:42:32Z" level=error msg="Error backing up item" backup=velero-minio/k3s-mongodb-restic-minio error="pod volume backup failed: running Restic backup, stderr=Fatal: unable to open config file: Stat: The Access Key Id you provided does not exist in our records.\nIs there a repository at the following location?\ns3:http://10.10.5.15:9009/k3s-backup/restic/mongodb\n: exit status 1" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/restic/backupper.go:199" error.function="github.com/vmware-tanzu/velero/pkg/restic.(*backupper).BackupPodVolumes" logSource="pkg/backup/backup.go:417" name=mongodb-cluster-0
...
velero get backup-locations -n velero-aws
NAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED ACCESS MODE DEFAULT
aws-s3 aws k3s-backup-aws Available 2022-10-17 14:12:46 -0400 EDT ReadWrite
...
velero get backup-locations -n velero-minio
NAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED ACCESS MODE DEFAULT
minio aws k3s-backup Available 2022-10-17 14:16:25 -0400 EDT ReadWrite
velero backup part works without errors but restic fails for all my jobs (mongodb is the only example).
It looks like the restic can't create snapshots for my nfs pvc.
What am I doing wrong?
It looks like velero doesn't work with multiple installations, at least the restic part fails (in my case, two instances in name spaces velero-aws and velero-minio).
So, I installed only one instance of velero to work with minio.
Removed --default-volumes-to-restic=true from the backup job configuration.
Used opt-in pod volume backup with the restic integration.
Each pod that has pvc volume needs to be annotated, like the following:
kubectl -n mongodb annotate pod/mongodb-cluster-0 backup.velero.io/backup-volumes=logs-volume,data-volume
I have not tried velero-pvc-watcher, probably it works well
Now backup works with no errors.

K0s cluster is not getting created with active firewall on Redhat 8.4

I am using k0sctl to create a cluster but it's not getting set up if the firewall is on.
Operating System - RedHat 8.4
k0s version - v1.24.2+k0s.0
k0sctl version - v0.13.0
k0sctl file -
apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
name: k0s-cluster
spec:
hosts:
- ssh:
address: 10.210.24.11
user: root
port: 22
keyPath: /root/.ssh/id_rsa
role: controller
- ssh:
address: 10.210.24.12
user: root
port: 22
keyPath: /root/.ssh/id_rsa
role: worker
k0s:
version: 1.24.2+k0s.0
dynamicConfig: false
Error that I am getting is - failed to connect from worker to kubernetes api at https://X.X.X.X:6443 - check networking

Fabric v2.0 in kubernetes (minikube) - error Peer channel join - TLS issue because of pod's names

I am trying to set up the Fabric v2.0 test-network (https://hyperledger-fabric.readthedocs.io/en/release-2.0/test_network.html) on kubernetes (locally on minikube). I have an error whith peer channel join.
I created kubernetes files based on the docker-compose-test-net.yaml of the test-network. I successfully deploy the following pods:
an orderer (raft)
2 peers (peer0-org1-example-com and peer0-org2-example-com)
a fabric-tools pod.
I successfully generate the crypto material with cryptogen and configtxgen.
I successfully create the channel:
when I am in the fabric-tools pod:
bash-5.0# peer channel create -o orderer-example-com:7050 -c $CHANNEL_NAME --ordererTLSHostnameOverride orderer.example.com -f /fabric/${CHANNEL_NAME}.tx --tls --cafile $ORDERER_CA
2020-02-11 08:10:14.057 CET [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2020-02-11 08:10:14.080 CET [cli.common] readBlock -> INFO 002 Expect block, but got status: &{NOT_FOUND}
...
2020-02-11 08:10:15.105 CET [cli.common] readBlock -> INFO 00c Received block: 0
But when I try for the first peer to join the channel, I have an error. I have been spending days on this, and I cannot find a solution. Your help would be much appreciated!!
in the fabric-tools pod:
bash-5.0# peer channel join -b $CHANNEL_NAME.block
Error: error getting endorser client for channel: endorser client failed to connect to peer0-org1-example-com:7051: failed to create new connection: context deadline exceeded
what I see in the peer0-org1-example-com pod logs:
[31m2020-02-11 08:11:29.945 CET [core.comm] ServerHandshake -> ERRO 1b9[0m TLS handshake failed with error remote error: tls: bad certificate server=PeerServer remoteaddress=172.17.0.6:43270
[36m2020-02-11 08:11:29.945 CET [grpc] handleRawConn -> DEBU 1ba[0m grpc: Server.Serve failed to complete security handshake from "172.17.0.6:43270": remote error: tls: bad certificate
Thank you!!
UPDATE:
If I run peer channel join directly on the peer0-org1-example-com pod, I can see that there is a certificate issue:
addrConn.createTransport failed to connect to {peer0-org1-example-com:7051 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for peer0.org1.example.com, peer0, localhost, peer0.org1.example.com, peer0, localhost, peer0.org1.example.com, peer0, localhost, not peer0-org1-example-com". Reconnecting.
It seems that it would accept the connection for peer0.org1.example.com but not for peer0-org1-example-com. But in Kubernetes, it does not allow me to put dots in the names of the services and the deployments, that is why I put dashes. Do you know how to solve this?
I tried to make the cryptogen tool generate certificates for peer0-org1-example-com, but it messed things up. The better would be, I think, to make kubernetes names with dots, but I can't seem to make it.
The names in peer deployments files:
apiVersion: apps/v1
kind: Deployment
metadata:
name: peer0-org1-example-com
spec:
selector:
matchLabels:
name: peer0-org1-example-com
replicas: 1
template:
metadata:
labels:
name: peer0-org1-example-com
The names in peer services files:
apiVersion: v1
kind: Service
metadata:
name: peer0-org1-example-com
labels:
run: peer0-org1-example-com
spec:
type: ClusterIP
selector:
name: peer0-org1-example-com
ports:
- protocol: TCP
port: 7051
name: grpc
We had a similar dot/dash certificate issue with OpenShift and solved it by setting a CommonName with dashes for each Host in our crypto-config file. Maybe this will work for you too.
Something like this:
PeerOrgs:
- Name: Org1
Domain: org1-example-com
EnableNodeOUs: true
Specs:
- Hostname: peer0
CommonName: "peer0-org1-example-com"
- Hostname: peer1
CommonName: "peer1-org1-example-com"
CA:
Hostname: ca
CommonName: "ca-org1-example-com"
PeerOrgs:
- Name: Org2
Domain: org2-example-com
EnableNodeOUs: true
Specs:
- Hostname: peer0
CommonName: "peer0-org2-example-com"
- Hostname: peer1
CommonName: "peer1-org2-example-com"
CA:
Hostname: ca
CommonName: "ca-org2-example-com"
OrdererOrgs:
- Name: Orderer
Domain: example.com
EnableNodeOUs: true
Specs:
- Hostname: orderer
CommonName: "orderer-example-com"
UPDATE:
We also changed all dot addresses in the configtx.yaml like this:
Orderer: &OrdererDefaults
...
EtcdRaft:
Consenters:
- Host: orderer-example-com
...
Addresses:
- orderer-example-com:7050
UPDATE 2:
probably you have to change the csr part in the fabric-ca-server-config.yaml of each org too:
csr:
cn: ca-example-com
names:
- C: US
ST: "New York"
L: "New York"
O: example-com
OU:
hosts:
- localhost
- example-com
ca:
expiry: 131400h
pathlength: 1
csr:
cn: ca-org1-example-com
names:
- C: US
ST: "North Carolina"
L: "Durham"
O: org1-example-com
OU:
hosts:
- localhost
- org1-example-com
ca:
expiry: 131400h
pathlength: 1
csr:
cn: ca-org2-example-com
names:
- C: UK
ST: "Hampshire"
L: "Hursley"
O: org2-example-com
OU:
hosts:
- localhost
- org2-example-com
ca:
expiry: 131400h
pathlength: 1

Passing certificates/keys in kubeconfig throws an JSON error

Using Kubectl client (1.7.0) on Windows to connect to remote cluster.
The config file in Windows ( located in .kube) directory is configured as follows:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: C:\Users\DK05478\.kube\ca.crt
server: https://10.99.70.153:6443
name: devo
contexts:
- context:
cluster: devo
user: admindevo
name: devo
current-context: devo
kind: Config
preferences: {}
users:
- name: admindevo
user:
client-certificate-data: C:\Users\DK05478\.kube\apiserver.crt
client-key-data: C:\Users\DK05478\.kube\apiserver.key
These certificate files I have downloaded from the remote system to my localhost. But this does not work. Throws the following error->
C:\Windows\kubernetes>kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean", BuildDate:"2017-06-29T23:15:59Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"windows/amd64"}
error: Error loading config file "C:\Users\DK05478/.kube/config": [pos 107]: json: error decoding base64 binary 'C:\Users\DK05478\.kube\ca.crt': illegal base64 data at input byte 1
How can I fix this issue? whats that I am doing wrong ?
Remove the -data suffix from certificate-authority-data, client-certificate-data and client-key-data. Like what #sfgroups said, the xxx-data param is meant for a base64-encoded cert/key.
Once you do, your kubeconfig should look like this:
apiVersion: v1
clusters:
- cluster:
certificate-authority: C:\Users\DK05478\.kube\ca.crt
server: https://10.99.70.153:6443
name: devo
contexts:
- context:
cluster: devo
user: admindevo
name: devo
current-context: devo
kind: Config
preferences: {}
users:
- name: admindevo
user:
client-certificate: C:\Users\DK05478\.kube\apiserver.crt
client-key: C:\Users\DK05478\.kube\apiserver.key
certificate-authority-data: , client-certificate-data:, client-key-data: reference to the file. I think you need base64 encoded keys value here. you can look .kube/config file from your cluster master.
look at this page for base64 usage example https://kubernetes.io/docs/concepts/configuration/secret/