pvc get stuck in pending waiting for a volume to be created, either by external provisioner "rook-ceph.rbd.csi.ceph.com" or manually - ceph

I use rook to build a ceph cluster.But my pvc get stuck in pending. When I used kubectl describe pvc, I found events from persistentvolume-controller:
waiting for a volume to be created, either by external provisioner "rook-ceph.rbd.csi.ceph.com" or manually created by system administrator
All my pods are in running state:
NAME READY STATUS RESTARTS AGE
csi-cephfsplugin-ntqk6 3/3 Running 0 14d
csi-cephfsplugin-pqxdw 3/3 Running 6 14d
csi-cephfsplugin-provisioner-c68f789b8-dt4jf 6/6 Running 49 14d
csi-cephfsplugin-provisioner-c68f789b8-rn42r 6/6 Running 73 14d
csi-rbdplugin-6pgf4 3/3 Running 0 14d
csi-rbdplugin-l8fkm 3/3 Running 6 14d
csi-rbdplugin-provisioner-6c75466c49-tzqcr 6/6 Running 106 14d
csi-rbdplugin-provisioner-6c75466c49-x8675 6/6 Running 17 14d
rook-ceph-crashcollector-compute08.dc-56b86f7c4c-9mh2j 1/1 Running 2 12d
rook-ceph-crashcollector-compute09.dc-6998676d86-wpsrs 1/1 Running 0 12d
rook-ceph-crashcollector-compute10.dc-684599bcd8-7hzlc 1/1 Running 0 12d
rook-ceph-mgr-a-69fd54cccf-tjkxh 1/1 Running 200 12d
rook-ceph-mon-at-8568b88589-2bm5h 1/1 Running 0 4d3h
rook-ceph-mon-av-7b4444c8f4-2mlpc 1/1 Running 0 4d1h
rook-ceph-mon-aw-7df9f76fcd-zzmkw 1/1 Running 0 4d1h
rook-ceph-operator-7647888f87-zjgsj 1/1 Running 1 15d
rook-ceph-osd-0-6db4d57455-p4cz9 1/1 Running 2 12d
rook-ceph-osd-1-649d74dc6c-5r9dj 1/1 Running 0 12d
rook-ceph-osd-2-7c57d4498c-dh6nk 1/1 Running 0 12d
rook-ceph-osd-prepare-compute08.dc-gxt8p 0/1 Completed 0 3h9m
rook-ceph-osd-prepare-compute09.dc-wj2fp 0/1 Completed 0 3h9m
rook-ceph-osd-prepare-compute10.dc-22kth 0/1 Completed 0 3h9m
rook-ceph-tools-6b4889fdfd-d6xdg 1/1 Running 0 12d
Here is the kubectl logs -n rook-ceph csi-cephfsplugin-provisioner-c68f789b8-dt4jf csi-provisioner
I0120 11:57:13.283362 1 csi-provisioner.go:121] Version: v2.0.0
I0120 11:57:13.283493 1 csi-provisioner.go:135] Building kube configs for running in cluster...
I0120 11:57:13.294506 1 connection.go:153] Connecting to unix:///csi/csi-provisioner.sock
I0120 11:57:13.294984 1 common.go:111] Probing CSI driver for readiness
W0120 11:57:13.296379 1 metrics.go:142] metrics endpoint will not be started because `metrics-address` was not specified.
I0120 11:57:13.299629 1 leaderelection.go:243] attempting to acquire leader lease rook-ceph/rook-ceph-cephfs-csi-ceph-com...
Here is the ceph status in toolbox container:
cluster:
id: 0b71fd4c-9731-4fea-81a7-1b5194e14204
health: HEALTH_ERR
Module 'dashboard' has failed: [('x509 certificate routines', 'X509_check_private_key', 'key values mismatch')]
Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded, 1 pg undersized
1 pgs not deep-scrubbed in time
1 pgs not scrubbed in time
services:
mon: 3 daemons, quorum at,av,aw (age 4d)
mgr: a(active, since 4d)
osd: 3 osds: 3 up (since 12d), 3 in (since 12d)
data:
pools: 1 pools, 1 pgs
objects: 2 objects, 0 B
usage: 3.3 GiB used, 3.2 TiB / 3.2 TiB avail
pgs: 2/6 objects degraded (33.333%)
1 active+undersized+degraded
I think it’s because the cluster’s health is health_err, but I don’t know how to solve it...I use raw partitions to build the ceph cluster currently: one partition on a node and two partitions on another node.
I found that there are few pods restarted several times, so I checked their logs.As for the csi-rbdplugin-provisioner pod, there is the same error in csi-resizer,csi attacher and csi-snapshotter container:
E0122 08:08:37.891106 1 leaderelection.go:321] error retrieving resource lock rook-ceph/external-resizer-rook-ceph-rbd-csi-ceph-com: Get "https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/external-resizer-rook-ceph-rbd-csi-ceph-com": dial tcp 10.96.0.1:443: i/o timeout
,and a repeating error in csi-snapshotter:
E0122 08:08:48.420082 1 reflector.go:127] github.com/kubernetes-csi/external-snapshotter/client/v3/informers/externalversions/factory.go:117: Failed to watch *v1beta1.VolumeSnapshotClass: failed to list *v1beta1.VolumeSnapshotClass: the server could not find the requested resource (get volumesnapshotclasses.snapshot.storage.k8s.io)
As for the mgr pod,there is a repeating record:
debug 2021-01-29T00:47:22.155+0000 7f10fdb48700 0 log_channel(cluster) log [DBG] : pgmap v28775: 1 pgs: 1 active+undersized+degraded; 0 B data, 337 MiB used, 3.2 TiB / 3.2 TiB avail; 2/6 objects degraded (33.333%)
It's also weird that the mon pods' names are at,av and aw rather than a,b and c.Seems like the mon pods deleted and created several times,but I don't know why.
Thanks for any advice.

Related

How can I detect CNI type/version in Kubernetes cluster?

Is there a Kubectl command or config map in the cluster that can help me find what CNI is being used?
First of all checking presence of exactly one config file in /etc/cni/net.d is a good start:
$ ls /etc/cni/net.d
10-flannel.conflist
and ip a s or ifconfig helpful for checking existence of network interfaces. e.g. flannel CNI should setup flannel.1 interface:
$ ip a s flannel.1
3: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
link/ether de:cb:d1:d6:e3:e7 brd ff:ff:ff:ff:ff:ff
inet 10.244.1.0/32 scope global flannel.1
valid_lft forever preferred_lft forever
inet6 fe80::dccb:d1ff:fed6:e3e7/64 scope link
valid_lft forever preferred_lft forever
When creating a cluster, CNI installation is typically installed using:
kubectl apply -f <add-on.yaml>
thus the networking pod will be called kube-flannel*, kube-calico* etc. depending on your networking configuration.
Then crictl will help you inspect running pods and containers.
crictl pods ls
On a controller node in a healthy cluster you should have all pods in Ready state.
crictl pods ls
POD ID CREATED STATE NAME NAMESPACE ATTEMPT RUNTIME
dc90dd87e18cf 3 minutes ago Ready coredns-6d4b75cb6d-r2j9s kube-system 0 (default)
d1ab9d0aa815a 3 minutes ago Ready kubernetes-dashboard-cd4778d69-xmtkz kube-system 0 (default)
0c151fdd92e71 3 minutes ago Ready coredns-6d4b75cb6d-bn8hr kube-system 0 (default)
40f18ce56f776 4 minutes ago Ready kube-flannel-ds-d4fd7 kube-flannel 0 (default)
0e390a68380a5 4 minutes ago Ready kube-proxy-r6cq2 kube-system 0 (default)
cd93e58d3bf70 4 minutes ago Ready kube-scheduler-c01 kube-system 0 (default)
266a33aa5c241 4 minutes ago Ready kube-apiserver-c01 kube-system 0 (default)
0910a7a73f5aa 4 minutes ago Ready kube-controller-manager-c01 kube-system 0 (default)
If your cluster is properly configured you should be able to list containers using kubectl:
kubectl get pods -n kube-system
if kubectl is not working (kube-apiserver is not running) you can fallback to crictl.
On an unhealthy cluster kubectl will show pods in CrashLoopBackOff state. crictl pods ls command will give you similar picture, only displaying pods from single node. Also check documentation for common CNI errors.
$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-6d4b75cb6d-brb9d 0/1 ContainerCreating 0 25m
coredns-6d4b75cb6d-pcrcp 0/1 ContainerCreating 0 25m
kube-apiserver-cm01 1/1 Running 27 (18m ago) 26m
kube-apiserver-cm02 0/1 Running 31 (8m11s ago) 23m
kube-apiserver-cm03 0/1 CrashLoopBackOff 33 (2m22s ago) 26m
kube-controller-manager-cm01 0/1 CrashLoopBackOff 13 (50s ago) 24m
kube-controller-manager-cm02 0/1 CrashLoopBackOff 7 (15s ago) 24m
kube-controller-manager-cm03 0/1 CrashLoopBackOff 15 (3m45s ago) 26m
kube-proxy-2dvfg 0/1 CrashLoopBackOff 8 (97s ago) 25m
kube-proxy-7gnnr 0/1 CrashLoopBackOff 8 (39s ago) 25m
kube-proxy-cqmvz 0/1 CrashLoopBackOff 8 (19s ago) 25m
kube-scheduler-cm01 1/1 Running 28 (7m15s ago) 12m
kube-scheduler-cm02 0/1 CrashLoopBackOff 28 (4m45s ago) 18m
kube-scheduler-cm03 1/1 Running 36 (107s ago) 26m
kubernetes-dashboard-cd4778d69-g8jmf 0/1 ContainerCreating 0 2m27s
crictl ps will give you containers (like docker ps), watch for high number of attempts:
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD
d54c6f1e45dea 2ae1ba6417cbc 2 seconds ago Running kube-proxy 1 347fef3ae1e98 kube-proxy-7gnnr
d6048ef9e30c7 d521dd763e2e3 41 seconds ago Running kube-apiserver 27 640658b58d1ae kube-apiserver-cm03
b6b8c7a24914e 3a5aa3a515f5d 41 seconds ago Running kube-scheduler 28 c7b710a0acf30 kube-scheduler-cm03
b0a480d2c1baf 586c112956dfc 42 seconds ago Running kube-controller-manager 8 69504853ab81b kube-controller-manager-cm03
and check logs using
crictl logs d54c6f1e45dea
Last not least /opt/cni/bin/ path usually contains binaries required for networking. Another PATH might defined in add on setup or CNI config.
$ ls /opt/cni/bin/
bandwidth bridge dhcp firewall flannel host-device host-local ipvlan loopback macvlan portmap ptp sbr static tuning vlan
Finally crictl reads /etc/crictl.yaml config, you should set proper runtime and image endpoint to match you container runtime:
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10

Pods unable to resolve hostnames in Kubernetes cluster

I'm working on aws eks, and I'm having issues with networking because none of the pods can resolve hostnames.
Checking the kube-config pods, I found this:
NAME READY STATUS RESTARTS AGE
alb-ingress-controller-68b7d9dd9-4mpnc 1/1 Running 0 3d20h
alb-ingress-controller-68b7d9dd9-kqmsl 1/1 Running 8 17d
alb-ingress-controller-68b7d9dd9-q4m87 1/1 Running 11 26d
aws-node-m6ncq 1/1 Running 34 31d
aws-node-r8bs5 1/1 Running 31 28d
aws-node-vfvjn 1/1 Running 34 31d
coredns-f47955f89-4bfb5 1/1 Running 11 26d
coredns-f47955f89-7mqp7 1/1 Running 8 17d
kube-flannel-ds-chv64 0/1 CrashLoopBackOff 1814 28d
kube-flannel-ds-fct45 0/1 CrashLoopBackOff 1831 31d
kube-flannel-ds-zs8z4 0/1 CrashLoopBackOff 1814 28d
kube-proxy-6lcst 1/1 Running 18 31d
kube-proxy-9qfkg 1/1 Running 17 28d
kube-proxy-l5qvd 1/1 Running 18 31d
and kube-flannel logs are this:
I0208 21:01:23.542634 1 main.go:218] CLI flags config: {etcdEndpoints:http://127.0.0.1:4001,http://127.0.0.1:2379 etcdPrefix:/coreos.com/network etcdKeyfile: etcdCertfile: etcdCAFile: etcdUsername: etcdPassword: help:false version:false autoDetectIPv4:false autoDetectIPv6:false kubeSubnetMgr:true kubeApiUrl: kubeAnnotationPrefix:flannel.alpha.coreos.com kubeConfigFile: iface:[] ifaceRegex:[] ipMasq:true subnetFile:/run/flannel/subnet.env subnetDir: publicIP: publicIPv6: subnetLeaseRenewMargin:60 healthzIP:0.0.0.0 healthzPort:0 charonExecutablePath: charonViciUri: iptablesResyncSeconds:5 iptablesForwardRules:true netConfPath:/etc/kube-flannel/net-conf.json setNodeNetworkUnavailable:true}
W0208 21:01:23.542713 1 client_config.go:608] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0208 21:01:23.647398 1 kube.go:120] Waiting 10m0s for node controller to sync
I0208 21:01:23.647635 1 kube.go:378] Starting kube subnet manager
I0208 21:01:24.647775 1 kube.go:127] Node controller sync successful
I0208 21:01:24.647801 1 main.go:238] Created subnet manager: Kubernetes Subnet Manager - ip-10-23-21-240.us-east-2.compute.internal
I0208 21:01:24.647811 1 main.go:241] Installing signal handlers
I0208 21:01:24.647887 1 main.go:460] Found network config - Backend type: vxlan
I0208 21:01:24.647916 1 main.go:652] Determining IP address of default interface
I0208 21:01:24.648449 1 main.go:699] Using interface with name eth0 and address 10.23.21.240
I0208 21:01:24.648476 1 main.go:721] Defaulting external address to interface address (10.23.21.240)
I0208 21:01:24.648482 1 main.go:734] Defaulting external v6 address to interface address (<nil>)
I0208 21:01:24.648543 1 vxlan.go:137] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
E0208 21:01:24.648871 1 main.go:326] Error registering network: failed to acquire lease: node "ip-10-23-21-240.us-east-2.compute.internal" pod cidr not assigned
W0208 21:01:24.649006 1 reflector.go:424] github.com/flannel-io/flannel/subnet/kube/kube.go:379: watch of *v1.Node ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
I0208 21:01:24.649060 1 main.go:440] Stopping shutdownHandler...
Any idea what could be the issue or what to try?
Thanks!
Amazon VPC CNI and Flannel cannot co-exist on EKS. Note Flannel is not on the suggested alternate compatible CNI. To get an idea what does it take to use Flannel on EKS checkout this excellent blog.

How to debug GKE internal network issue?

UPDATE 1:
Some more logs from api-servers:
https://gist.github.com/nvcnvn/47df8798e798637386f6e0777d869d4f
This question is more about debugging method for current GKE but welcome for solution.
We're using GKE version 1.22.3-gke.1500 with following configuration:
We recently facing issue that commands like kubectl logs and exec doesn't work, deleting a namespace taking forever.
Checking some service inside the cluster, it seem for some reason some network operation just randomly failed. For example metric-server keep crashing with these error logs:
message: "pkg/mod/k8s.io/client-go#v0.19.10/tools/cache/reflector.go:156: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://10.97.0.1:443/api/v1/nodes?resourceVersion=387681528": net/http: TLS handshake timeout"
HTTP request timeout also:
unable to fully scrape metrics: unable to fully scrape metrics from node gke-staging-n2d-standard-8-78c35b3a-6h16: unable to fetch metrics from node gke-staging-n2d-standard-8-78c35b3a-6h16: Get "http://10.148.15.217:10255/stats/summary?only_cpu_and_memory=true": context deadline exceeded
and I also try to restart (by kubectl delete) most of the pod in this list:
kubectl get pod
NAME READY STATUS RESTARTS AGE
event-exporter-gke-5479fd58c8-snq26 2/2 Running 0 4d7h
fluentbit-gke-gbs2g 2/2 Running 0 4d7h
fluentbit-gke-knz2p 2/2 Running 0 85m
fluentbit-gke-ljw8h 2/2 Running 0 30h
gke-metadata-server-dtnvh 1/1 Running 0 4d7h
gke-metadata-server-f2bqw 1/1 Running 0 30h
gke-metadata-server-kzcv6 1/1 Running 0 85m
gke-metrics-agent-4g56c 1/1 Running 12 (3h6m ago) 4d7h
gke-metrics-agent-hnrll 1/1 Running 13 (13h ago) 30h
gke-metrics-agent-xdbrw 1/1 Running 0 85m
konnectivity-agent-87bc84bb7-g9nd6 1/1 Running 0 2m59s
konnectivity-agent-87bc84bb7-rkhhh 1/1 Running 0 3m51s
konnectivity-agent-87bc84bb7-x7pk4 1/1 Running 0 3m50s
konnectivity-agent-autoscaler-698b6d8768-297mh 1/1 Running 0 83m
kube-dns-77d9986bd5-2m8g4 4/4 Running 0 3h24m
kube-dns-77d9986bd5-z4j62 4/4 Running 0 3h24m
kube-dns-autoscaler-f4d55555-dmvpq 1/1 Running 0 83m
kube-proxy-gke-staging-n2d-standard-8-78c35b3a-8299 1/1 Running 0 11s
kube-proxy-gke-staging-n2d-standard-8-78c35b3a-fp5u 1/1 Running 0 11s
kube-proxy-gke-staging-n2d-standard-8-78c35b3a-rkdp 1/1 Running 0 11s
l7-default-backend-7db896cb4-mvptg 1/1 Running 0 83m
metrics-server-v0.4.4-fd9886cc5-tcscj 2/2 Running 82 33h
netd-5vpmc 1/1 Running 0 30h
netd-bhq64 1/1 Running 0 85m
netd-n6jmc 1/1 Running 0 4d7h
Some logs from metrics server
https://gist.github.com/nvcnvn/b77eb02705385889961aca33f0f841c7
if you cannot use kubectl to get info from your cluster, can you try to access them by using their restfull api
http://blog.madhukaraphatak.com/understanding-k8s-api-part-2/
try to delete "metric-server" pods or get logs from it using podman or curl command.

Kubernetes can't access pod in multi worker nodes

I was following a tutorial on youtube and the guy said that if you deploy your application in a multi-cluster setup and if your service is of type NodePort, you don't have to worry from where your pod gets scheduled. You can access it with different node IP address like
worker1IP:servicePort or worker2IP:servicePort or workerNIP:servicePort
But I tried just now and this is not the case, I can only access the pod on the node from where it is scheduled and deployed. Is it correct behavior?
kubectl version --short
> Client Version: v1.18.5
> Server Version: v1.18.5
kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-66bff467f8-6pt8s 0/1 Running 288 7d22h
coredns-66bff467f8-t26x4 0/1 Running 288 7d22h
etcd-redhat-master 1/1 Running 16 7d22h
kube-apiserver-redhat-master 1/1 Running 17 7d22h
kube-controller-manager-redhat-master 1/1 Running 19 7d22h
kube-flannel-ds-amd64-9mh6k 1/1 Running 16 5d22h
kube-flannel-ds-amd64-g2k5c 1/1 Running 16 5d22h
kube-flannel-ds-amd64-rnvgb 1/1 Running 14 5d22h
kube-proxy-gf8zk 1/1 Running 16 7d22h
kube-proxy-wt7cp 1/1 Running 9 7d22h
kube-proxy-zbw4b 1/1 Running 9 7d22h
kube-scheduler-redhat-master 1/1 Running 18 7d22h
weave-net-6jjd8 2/2 Running 34 7d22h
weave-net-ssqbz 1/2 CrashLoopBackOff 296 7d22h
weave-net-ts2tj 2/2 Running 34 7d22h
[root#redhat-master deployments]# kubectl logs weave-net-ssqbz -c weave -n kube-system
DEBU: 2020/07/05 07:28:04.661866 [kube-peers] Checking peer "b6:01:79:66:7d:d3" against list &{[{e6:c9:b2:5f:82:d1 redhat-master} {b2:29:9a:5b:89:e9 redhat-console-1} {e2:95:07:c8:a0:90 redhat-console-2}]}
Peer not in list; removing persisted data
INFO: 2020/07/05 07:28:04.924399 Command line options: map[conn-limit:200 datapath:datapath db-prefix:/weavedb/weave-net docker-api: expect-npc:true host-root:/host http-addr:127.0.0.1:6784 ipalloc-init:consensus=2 ipalloc-range:10.32.0.0/12 metrics-addr:0.0.0.0:6782 name:b6:01:79:66:7d:d3 nickname:redhat-master no-dns:true port:6783]
INFO: 2020/07/05 07:28:04.924448 weave 2.6.5
FATA: 2020/07/05 07:28:04.938587 Existing bridge type "bridge" is different than requested "bridged_fastdp". Please do 'weave reset' and try again
Update:
So basically the issue is because iptables is deprecated in rhel8. But After downgrading my OS to rhel7. I can access the nodeport only on the node it is deployed.

HTTPError 400 while deploying production-ready GitLab on Google Kubernetes Engine

I'm following the official Tutorial for Deploying production-ready GitLab on Google Kubernetes Engine.
The step Create the PostgreSQL instance and database: 1. Create the Cloud SQL database that GitLab will use to store most of its metadata gave me the Error:
gcloud beta sql instances create gitlab-db --network default \
--database-version=POSTGRES_9_6 --cpu 4 --memory 15 --no-assign-ip \
--storage-auto-increase --zone us-central1-a
ERROR: (gcloud.beta.sql.instances.create) HTTPError 400: Invalid request: Project {here_stands_my_correct_Project_ID} has invalid private network
name https://compute.googleapis.com/compute/v1/projects/{here_stands_my_correct_Project_ID}/global/networks/default.
Any ideas, thank you?
EDIT: I used the following command and edited manually the gilab-db to Private IP with attached Network (default) in the Console getting a 503 Error at the end of the the tutorial.
gcloud beta sql instances create gitlab-db --database-version=POSTGRES_9_6 --cpu 4 --memory 15 --storage-auto-increase --zone us-central1-a
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
gitlab-certmanager-788c6859c6-szqqm 1/1 Running 0 28m
gitlab-gitaly-0 0/1 Pending 0 28m
gitlab-gitlab-runner-6cfb858756-l8gxr 0/1 CrashLoopBackOff 6 28m
gitlab-gitlab-shell-6cc87fcd4c-2mqph 1/1 Running 0 28m
gitlab-gitlab-shell-6cc87fcd4c-jvp8n 1/1 Running 0 27m
gitlab-issuer.1-cx8tm 0/1 Completed 0 28m
gitlab-nginx-ingress-controller-5f486c5f7b-md8rj 1/1 Running 0 28m
gitlab-nginx-ingress-controller-5f486c5f7b-rps6m 1/1 Running 0 28m
gitlab-nginx-ingress-controller-5f486c5f7b-xc8fv 1/1 Running 0 28m
gitlab-nginx-ingress-default-backend-7f87d67c8-6xhhz 1/1 Running 0 28m
gitlab-nginx-ingress-default-backend-7f87d67c8-7w2s2 1/1 Running 0 28m
gitlab-registry-8dfc8f979-9hdbr 0/1 Init:0/2 0 28m
gitlab-registry-8dfc8f979-qr5nd 0/1 Init:0/2 0 27m
gitlab-sidekiq-all-in-1-88f47878-26nh8 0/1 Init:CrashLoopBackOff 7 28m
gitlab-task-runner-74fc4ccdb9-pm592 1/1 Running 0 28m
gitlab-unicorn-5b74ffdff8-4kkj4 0/2 Init:CrashLoopBackOff 7 28m
gitlab-unicorn-5b74ffdff8-nz662 0/2 Init:CrashLoopBackOff 7 27m
kube-state-metrics-57b88466db-h7xkj 1/1 Running 0 27m
node-exporter-q4bpv 1/1 Running 0 27m
node-exporter-x8mtj 1/1 Running 0 27m
node-exporter-xrdlv 1/1 Running 0 27m
prometheus-k8s-5cf4c4cf6c-hsntr 2/2 Running 1 27m
Possibly this is because it's still in beta and not all features and/or options are working correctly.
I can advice that you check if you have just one network available.
You can do that by using gcloud compute networks list.
$ gcloud compute networks list
NAME SUBNET_MODE BGP_ROUTING_MODE IPV4_RANGE GATEWAY_IPV4
default AUTO REGIONAL
If you see only default network then there is no need to worry about providing the --network flag at all.
Also form what it looks like the instance will need to use an IP either Public or Private so you can leave out the flag --no-assign-ip.
Working command might look like this:
gcloud beta sql instances create gitlab-db --database-version=POSTGRES_9_6 --cpu 4 --memory 15 --storage-auto-increase --zone us-central1-a
You can read the docs about the flags and usage on gcloud beta sql instances create