I have to take the pg_basebackup of my postgres cluster. So how I can take the backup of this cluster.
I use the Bitnami Helm chart for PostgreSQL installation. Following Link for reference
https://phoenixnap.com/kb/postgresql-kubernetes
from that 2nd part. Deploy PostgreSQL by Creating Configuration from Scratch.
After successful installation
I tried to take pg_basebackup from localhost
from Localhost execution-->
pg_basebackup -h localhost -U postgres -D /root/base_bkp -v
pg_basebackup: error: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
2023-01-03 07:16:42.889 GMT [1005845] FATAL: no pg_hba.conf entry for replication connection from host "10.XX.XXX.X", user "postgres", SSL off
2023-01-03 07:16:42.889 GMT [1005845] DETAIL: Client IP address resolved to "10-XX-XXX-X.postgres-postgresql-ha-pgpool.postgres.svc.cluster.local", forward lookup not checked.
host "10.XX.XXX.X" --> This is the pgpool node ip
root#hostname~a# kubectl get pods -n postgres -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
postgres-postgresql-ha-pgpool-6XXXXXX-fXXXX 1/1 Running 0 1d7h 10.XX.XXX.X <none> <none>
postgres-postgresql-ha-postgresql-0 1/1 Running 0 1d7h 10.XX.YYY.Y <none> <none>
postgres-postgresql-ha-postgresql-1 1/1 Running 0 1d7h 10.XX.ZZZ.Z <none> <none>
postgres-postgresql-ha-postgresql-2 1/1 Running 0 1d7h 10.XX.PPP.P <none> <none>
Can anyone help me that from where I need to initiate the backup and how?
Related
I have deployed a mongodb replicaset (v6.0.2) using this chart: bitnami/mongodb 13.3.1
I have enabled external access with SVC type LoadBalancer.
I have my DNS records pointing to each external SVC IP.
mongodb-0.dev -> 10.246.50.1
mongodb-1.dev -> 10.246.50.2
mongodb-2.dev -> 10.246.50.3
kubectl get pod,svc -owide
NAME READY STATUS
pod/mongodb-0 2/2 Running
pod/mongodb-1 2/2 Running
pod/mongodb-2 2/2 Running
pod/mongodb-arbiter-0 2/2 Running
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/mongodb-0-external LoadBalancer 10.43.182.101 10.246.50.1 27017:30727/TCP 6m6s app.kubernetes.io/component=mongodb,app.kubernetes.io/instance=mongodb,app.kubernetes.io/name=mongodb,statefulset.kubernetes.io/pod-name=mongodb-0
service/mongodb-1-external LoadBalancer 10.43.130.10 10.246.50.2 27017:30137/TCP 6m6s app.kubernetes.io/component=mongodb,app.kubernetes.io/instance=mongodb,app.kubernetes.io/name=mongodb,statefulset.kubernetes.io/pod-name=mongodb-1
service/mongodb-2-external LoadBalancer 10.43.86.149 10.246.50.3 27017:32246/TCP 6m6s app.kubernetes.io/component=mongodb,app.kubernetes.io/instance=mongodb,app.kubernetes.io/name=mongodb,statefulset.kubernetes.io/pod-name=mongodb-2
service/mongodb-arbiter-headless ClusterIP None <none> 27017/TCP 8m51s app.kubernetes.io/component=arbiter,app.kubernetes.io/instance=mongodb,app.kubernetes.io/name=mongodb
service/mongodb-headless ClusterIP None <none> 27017/TCP 8m51s app.kubernetes.io/component=mongodb,app.kubernetes.io/instance=mongodb,app.kubernetes.io/name=mongodb
I run a docker from OUTSIDE to test the client:
docker run --rm -it -v docker.io/bitnami/mongodb:6.0.2-debian-11-r1 bash
I CAN connect from OUTSIDE using this command:
mongosh mongodb://mongodb-0.dev:27017 --authenticationDatabase admin -u root -p root1234
Connecting to: mongodb://<credentials>#mongodb-0.dev:27017/?directConnection=true&authSource=admin
rs0 [direct: primary] test>
I CAN'T connect using this one:
mongosh mongodb://mongodb-0.dev:27017,mongodb-1.dev:27017,mongodb-2.dev:27017?replicaSet=rs0 --authenticationDatabase admin -u root -p root1234
Connecting to: mongodb://<credentials>#mongodb-0.dev:27017,mongodb-1.dev:27017,mongodb-2.dev:27017/?replicaSet=rs0&authSource=admin&appName=mongosh+1.6.0
MongoNetworkError: getaddrinfo ENOTFOUND mongodb-0.mongodb-headless.mongodb.svc.cluster.local
Of course, If I add the DNS records it works (pay attention at the prompt)::
mongodb-0.mongodb-headless.mongodb.svc.cluster.local -> 10.246.50.1
mongodb-1.mongodb-headless.mongodb.svc.cluster.local -> 10.246.50.2
mongodb-2.mongodb-headless.mongodb.svc.cluster.local -> 10.246.50.3
Connecting to: mongodb://<credentials>#mongodb-0.dev:27017,mongodb-1.dev:27017,mongodb-2.dev:27017/?replicaSet=rs0&authSource=admin&appName=mongosh+1.6.0
rs0 [primary] test>
But I don't want to do that workaround at DNS level, of course it's wrong. I also don't want to harcode IPs on /etc/hosts
Extra TIP:
From INSIDE the k8s cluster this works:
kubectl run mongo-client-6 --rm -ti --image=docker.io/bitnami/mongodb:6.0.2-debian-11-r1 -- bash
mongosh --host mongodb-headless --authenticationDatabase admin -u root -p root1234
Summary: I want to connect using the SECOND method (replica set from OUTSIDE) Any help?
I have the same issue I believe the core reason is that the internal host entries for replicas are cluster-based not external URLs so when you access the server from outside the internal logic of replicas still using the internal headless service.
I am still trying to solve this issue from my side but still.
I have a error when i want to access to my pod:
error: unable to upgrade connection: pod does not exist
it's a cluster with 3 nodes, below some details. Thanks in advance
root#kubm:~/deploy/nginx# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
kubm Ready master 37h v1.17.0 10.0.2.15 <none> Ubuntu 16.04.6 LTS 4.4.0-150-generic docker://19.3.5
kubnode Ready <none> 37h v1.17.0 10.0.2.15 <none> Ubuntu 16.04.6 LTS 4.4.0-150-generic docker://19.3.5
kubnode2 Ready <none> 37h v1.17.0 10.0.2.15 <none> Ubuntu 16.04.6 LTS 4.4.0-150-generic docker://19.3.5
root#kubm:~/deploy/nginx# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-59c9f8dff-v7dvg 1/1 Running 0 16h 10.244.2.3 kubnode2 <none> <none>
root#kubm:~/deploy/nginx# kubectl exec -it nginx-59c9f8dff-v7dvg -- /bin/bash
**error: unable to upgrade connection: pod does not exist**
I had the same issue running a cluster with Vagrant and Virtualbox the first time.
Adding KUBELET_EXTRA_ARGS=--node-ip=x.x.x.x where x.x.x.x is your VM's IP in /etc/default/kubelet (this can be part of the provisioning script for example) and then restarting kubelet (systemctl restart kubelet) fixes the issues.
This is the recommended way to add extra runtime arguments to kubelet as you can see in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf. Alternatively you can also edit the kubelet config file under /etc/kubernetes/kubelet.conf
The 10.0.2.15 IP address is the default for virtualbox NAT
If you deploy a VM using a vagrantfile, your eth0 adapter will use the 10.0.2.15 IP address and the eth1 adapter will be assigned an other IP address.
K8s uses the eth0 adapter to route packets between pods.
I had the same issue and the problem was POD status as "ImagePullBackOff". Due to this, it was throwing error
error: unable to upgrade connection: container not found ("nginx")
[mayur#mayur_cloudtest ~]$ kubectl get pods
NAME READY STATUS RESTARTS AGE
nginx-598b589c46-zcr5d 0/1 ImagePullBackOff 0 116s
[mayur#mayur_cloudtest ~]$
[mayur#mayur_cloudtest ~]$
[mayur#mayur_cloudtest ~]$ kubectl exec -it nginx-598b589c46-zcr5d -- /bin/bash
error: unable to upgrade connection: container not found ("nginx")
I use below command to get into a pod.
kubectl exec -i -t <pod-name> -- /bin/bash
Note -i and -t flag have a space on the command..
If you have multi-container pod you should pass container name with -c flag or it will by default connect to first container in POD.
ubuntu#cluster-master:~$ kubectl exec -i -t nginx -- /bin/bash
root#nginx:/# whoami
root
root#nginx:/# date
Tue Jan 7 14:12:29 UTC 2020
root#nginx:/#
Refer help section of command kubectl exec --help
I have made a HA Kubernetes cluster. FIrst I added a node and joined the other node as master role.
I basically did the multi etcd set up. This worked fine for me. I did the fail over testing which also worked fine. Now the problem is once I am done working, I drained and deleted the other node and then I shut down the other machine( a VM on GCP). But then my kubectl commands dont work... Let me share the steps:
kubectl get node(when multi node is set up)
NAME STATUS ROLES AGE VERSION
instance-1 Ready <none> 17d v1.15.1
instance-3 Ready <none> 25m v1.15.1
masternode Ready master 18d v1.16.0
kubectl get node ( when I shut down my other node)
root#masternode:~# kubectl get nodes
The connection to the server k8smaster:6443 was refused - did you specify the right host or port?
Any clue?
After reboot the server you need to do some step below:
sudo -i
swapoff -a
exit
strace -eopenat kubectl version
I have installed kubeadm and created service and pod:
packet#test:~$ kubectl get pod
NAME READY STATUS RESTARTS AGE
udp-server-deployment-6f87f5c9-466ft 1/1 Running 0 5m
udp-server-deployment-6f87f5c9-5j9rt 1/1 Running 0 5m
udp-server-deployment-6f87f5c9-g9wrr 1/1 Running 0 5m
udp-server-deployment-6f87f5c9-ntbkc 1/1 Running 0 5m
udp-server-deployment-6f87f5c9-xlbjq 1/1 Running 0 5m
packet#test:~$ kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 1h
udp-server-service NodePort 10.102.67.0 <none> 10001:30001/UDP 6m
but still I am not able to access udp-server pod:
packet#test:~$ curl http://192.168.43.161:30001
curl: (7) Failed to connect to 192.168.43.161 port 30001: Connection refused
while debugging i could see kube-proxy is running but there is no entry in IPVS:
root#test:~# ps auxw | grep kube-proxy
root 4050 0.5 0.7 44340 29952 ? Ssl 14:33 0:25 /usr/local/bin/kube-proxy --config=/var/lib/kube-proxy/config.conf
root 6094 0.0 0.0 14224 968 pts/1 S+ 15:48 0:00 grep --color=auto kube-proxy
root#test:~# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
Seems to be there is no entry in ipvsadm causing connection time out.
Regards, Ranjith
From this issue (putting aside the load balancer part),
Both externalIPs and status.loadBalancer.ingress[].ip seem to be ignored by kube-proxy in IPVS mode, so external traffic is completely unrouteable.
In contrast, kube-proxy in iptables mode creates DNAT/SNAT rules for external and loadbalancer IPs.
So check if adding a network plugin (flannel, Calico, ...) would improve the situation.
Or check out cloudnativelabs/kube-router, which is also ipvs-based.
A lean yet powerful alternative to several network components used in typical Kubernetes clusters.
All this from a single DaemonSet/Binary. It doesn't get any easier.
Since curl use tcp connection, while 30001 is a udp port, they don't work together, try a udp probe tool, like nmap.
initially I have created VM(Linux VM) using virtual box(running on window),where I found this type of issue.
Now i have created VM(Linux VM) using virtual manager(running on Linux),in this set up there is no issue and every thing works fine.
It would be great if any one tell is there any restriction from virtual box?
I have got 2 VMs nodes. Both see each other either by hostname (through /etc/hosts) or by ip address. One has been provisioned with kubeadm as a master. Another as a worker node. Following the instructions (http://kubernetes.io/docs/getting-started-guides/kubeadm/) I have added weave-net. The list of pods looks like the following:
vagrant#vm-master:~$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-vm-master 1/1 Running 0 3m
kube-system kube-apiserver-vm-master 1/1 Running 0 5m
kube-system kube-controller-manager-vm-master 1/1 Running 0 4m
kube-system kube-discovery-982812725-x2j8y 1/1 Running 0 4m
kube-system kube-dns-2247936740-5pu0l 3/3 Running 0 4m
kube-system kube-proxy-amd64-ail86 1/1 Running 0 4m
kube-system kube-proxy-amd64-oxxnc 1/1 Running 0 2m
kube-system kube-scheduler-vm-master 1/1 Running 0 4m
kube-system kubernetes-dashboard-1655269645-0swts 1/1 Running 0 4m
kube-system weave-net-7euqt 2/2 Running 0 4m
kube-system weave-net-baao6 1/2 CrashLoopBackOff 2 2m
CrashLoopBackOff appears for each worker node connected. I have spent several ours playing with network interfaces, but it seems the network is fine. I have found similar question, where the answer advised to look into the logs and no follow up. So, here are the logs:
vagrant#vm-master:~$ kubectl logs weave-net-baao6 -c weave --namespace=kube-system
2016-10-05 10:48:01.350290 I | error contacting APIServer: Get https://100.64.0.1:443/api/v1/nodes: dial tcp 100.64.0.1:443: getsockopt: connection refused; trying with blank env vars
2016-10-05 10:48:01.351122 I | error contacting APIServer: Get http://localhost:8080/api: dial tcp [::1]:8080: getsockopt: connection refused
Failed to get peers
What I am doing wrong? Where to go from there?
I ran in the same issue too. It seems weaver wants to connect to the Kubernetes Cluster IP address, which is virtual. Just run this to find the cluster ip:
kubectl get svc. It should give you something like this:
$ kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 100.64.0.1 <none> 443/TCP 2d
Weaver picks up this IP and tries to connect to it, but worker nodes does not know anything about it. Simple route will solve this issue. On all your worker nodes, execute:
route add 100.64.0.1 gw <your real master IP>
this happens with a single node setup, too. I tried several things like reapplying the configuration and recreation, but the most stable way at the moment is to perform a full tear down (as described in docs) and put the cluster up again.
I use these scripts for relaunching the cluster:
down.sh
#!/bin/bash
systemctl stop kubelet;
docker rm -f -v $(docker ps -q);
find /var/lib/kubelet | xargs -n 1 findmnt -n -t tmpfs -o TARGET -T | uniq | xargs -r umount -v;
rm -r -f /etc/kubernetes /var/lib/kubelet /var/lib/etcd;
up.sh
#!/bin/bash
systemctl start kubelet
kubeadm init
# kubectl taint nodes --all dedicated- # single node!
kubectl create -f https://git.io/weave-kube
edit: I would also give other Pod networks a try, like Calico, if this is a weave related issue
The most common causes for this may be:
- presence of a firewall (e.g. firewalld on CentOS)
- network configuration (e.g. default NAT interface on VirtualBox)
Currently kubeadm is still alpha, and this is one of the issues that has already been reported by many of the alpha testers. We are looking into fixing this by documenting the most common problems, such documentation is going to be ready closer to beta version.
Right there exists a VirtualBox+Vargant+Ansible for Ubunutu and CentOS reference implementation that provides solutions for firewall, SELinux and VirtualBox NAT issues.
/usr/local/bin/weave reset
was the fix for me - Hope its useful - and yes make sure selinux is set to disabled
and firewalld is not running (on redhat / centos) releases
kube-system weave-net-2vlvj 2/2 Running 3 11d
kube-system weave-net-42k6p 1/2 Running 3 11d
kube-system weave-net-wvsk5 2/2 Running 3 11d