kubernetes volume mount timeout - kubernetes

Question

I am using PVC to attach the volume to one deployment for grafana, but it times out and container kept in creating stage.
Storage class
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: grafana-storagetest
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
iopsPerGB: "10"
reclaimPolicy: Retain
allowVolumeExpansion: true
mountOptions:
- debug
volumeBindingMode: Immediate
PVC
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: grafana-claimtest
spec:
accessModes:
- ReadWriteOnce
volumeMode: Block
storageClassName: grafana-storagetest
resources:
requests:
storage: 10G
Service
apiVersion: v1
kind: Service
metadata:
annotations:
getambassador.io/config: |
---
apiVersion: ambassador/v1
kind: Mapping
name: grafana-proxytest_mapping
prefix: /v1/anonymous/grafana-proxytest
rewrite: /
service: grafana-proxytest.grafana-proxytest:8080
timeout_ms: 20000
connect_timeout_ms: 20000
labels:
app: grafana-proxytest
name: grafana-proxytest
namespace: grafana-proxytest
spec:
ports:
- name: http
port: 8080
protocol: TCP
targetPort: 8080
selector:
app: grafana-proxytest
type: ClusterIP
status:
loadBalancer: {}
Deployment
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "20"
labels:
version: v1
name: grafana-proxytest-v1
namespace: grafana-proxytest
spec:
replicas: 1
selector:
matchLabels:
app: grafana-proxytest
version: v1
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: grafana-proxytest
version: v1
spec:
containers:
image: <aws_ecr>
imagePullPolicy: Always
name: grafana-proxytest
ports:
- containerPort: 3000
protocol: TCP
resources:
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- mountPath: /var/lib/grafana
name: grafanapdtest
image: grafana/grafana:latest
imagePullPolicy: Always
name: grafanatest
ports:
- containerPort: 3000
protocol: TCP
resources:
requests:
cpu: 100m
memory: 200Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumes:
- name: grafanapdtest
persistentVolumeClaim:
claimName: grafana-claimtest
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
Pod status
grafana-proxytest-v1-7cb5b6b6cf-z5zml 0/2 ContainerCreating 0 4m18s
Describe Pod
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 4m14s (x2 over 4m14s) default-scheduler pod has unbound immediate PersistentVolumeClaims (repeated 7 times)
Normal Scheduled 4m12s default-scheduler Successfully assigned grafana-proxytest/grafana-proxytest-v1-7cb5b6b6cf-z5zml to ip-10-10-107-59.ap-southeast-1.compute.internal
Normal SuccessfulAttachVolume 4m10s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-f8ad51be-74c5-11ea-8623-068411799338"
Warning FailedMount 2m9s kubelet, ip-10-10-107-59.ap-southeast-1.compute.internal Unable to mount volumes for pod "grafana-proxytest-v1-7cb5b6b6cf-z5zml_grafana-proxytest(fcf38cab-74c5-11ea-8623-068411799338)": timeout expired waiting for volumes to attach or mount for pod "grafana-proxytest"/"grafana-proxytest-v1-7cb5b6b6cf-z5zml". list of unmounted volumes=[grafanapdtest]. list of unattached volumes=[grafanapdtest default-token-pzxdk]
Expected result - The volume should be attached properly and the pod should be in running status.
I checked and the storage class, PVC, and PV are created.
Edit 1
Added the namespace into PVC as below, but still it fails:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: grafana-claimtest
namespace: grafana-proxytest
spec:
accessModes:
- ReadWriteOnce
volumeMode: Block
storageClassName: grafana-storagetest
resources:
requests:
storage: 10G
Still getting below error:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m18s default-scheduler Successfully assigned grafana-proxytest/grafana-proxytest-v1-7cb5b6b6cf-gpk4l to ip-10-10-107-59.ap-southeast-1.compute.internal
Normal SuccessfulAttachVolume 2m16s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-5bbc63a3-755f-11ea-920e-0a280935f44c"
Warning FailedMount 15s kubelet, ip-10-10-107-59.ap-southeast-1.compute.internal Unable to mount volumes for pod "grafana-proxytest-v1-7cb5b6b6cf-gpk4l_grafana-proxytest(62c50717-755f-11ea-8623-068411799338)": timeout expired waiting for volumes to attach or mount for pod "grafana-proxytest"/"grafana-proxytest-v1-7cb5b6b6cf-gpk4l". list of unmounted volumes=[grafanapdtest]. list of unattached volumes=[grafanapdtest default-token-pzxdk]
One thing as per the above logs i says that attach succeeded for volume, but then it fails for mounting grafanapdtest, which I am not able to understand.
PVC details:
get pvc -n grafana-proxytest
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
grafana-claimtest Bound pvc-5bbc63a3-755f-11ea-920e-0a280935f44c 10Gi RWO grafana-storagetest 15m