How can I see my failing jobs with Kubernetes - kubernetes

I have an issue with a Job in kubernetes. When I want to debug, I do:
kubectl describe job -n influx pamela-1578898800
Name: pamela-1578898800
Namespace: influx
Selector: controller-uid=xxx
Labels: controller-uid=xxx
job-name=pamela-1578898800
Annotations: <none>
Controlled By: CronJob/pamela
Parallelism: 1
Completions: 1
Start Time: Mon, 13 Jan 2020 08:00:04 +0100
Pods Statuses: 0 Running / 0 Succeeded / 5 Failed
Pod Template:
Labels: controller-uid=53110b24-35d2-11ea-bca1-06ecc706e86a
job-name=pamela-1578898800
Containers:
pamela:
Image: registry.gitlab.com/xxx/pamela:latest
Port: <none>
Host Port: <none>
Limits:
cpu: 800m
memory: 1000Mi
Requests:
cpu: 800m
memory: 1000Mi
Environment Variables from:
pamela-env Secret Optional: false
Environment: <none>
Mounts:
/config from pamela-keys (rw)
/log from pamela-claim (rw,path="log")
/raw from pamela-claim (rw,path="raw")
Volumes:
pamela-claim:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: pamela-claim
ReadOnly: false
pamela-keys:
Type: Secret (a volume populated by a Secret)
SecretName: pamela-keys
Optional: false
Events: <none>
Here you can see 5 failing pods, but I don't know how to see the logs of failing pods.
When I do :
kubectl get po -A
I have no pods "pamela-xxx"
How should I see the issue ?
EDIT:
Here are the scripts I use to trigger job.
# job.yaml - THIS ONE WORKS, LAUNCHING MANUALLY
apiVersion: batch/v1
kind: Job
metadata:
name: pamela-singlerun
namespace: influx
spec:
template:
spec:
containers:
- image: registry.gitlab.com/xxx/pamela:latest
envFrom:
- secretRef:
name: pamela-env
name: pamela
volumeMounts:
- mountPath: /raw
name: pamela-claim
subPath: raw
- mountPath: /log
name: pamela-claim
subPath: log
- mountPath: /config
name: pamela-keys
restartPolicy: Never
volumes:
- name: pamela-claim
persistentVolumeClaim:
claimName: pamela-claim
- name: pamela-keys
secret:
secretName: pamela-keys
items:
- key: keys.yml
path: keys.yml
nodeSelector:
kops.k8s.io/instancegroup: pamela-nodes
imagePullSecrets:
- name: gitlab-registry
And cronjob.yml, THIS ONE DOESN'T WORK
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: pamela
namespace: influx
spec:
schedule: "0 7,19 * * *"
concurrencyPolicy: Replace
jobTemplate:
spec:
template:
spec:
containers:
- image: registry.gitlab.com/xxx/pamela:latest
envFrom:
- secretRef:
name: pamela-env
name: pamela
resources:
limits:
cpu: 800m
memory: 1000Mi
requests:
cpu: 800m
memory: 1000Mi
volumeMounts:
- mountPath: /raw
name: pamela-claim
subPath: raw
- mountPath: /log
name: pamela-claim
subPath: log
- mountPath: /config
name: pamela-keys
restartPolicy: Never
volumes:
- name: pamela-claim
persistentVolumeClaim:
claimName: pamela-claim
- name: pamela-keys
secret:
secretName: pamela-keys
items:
- key: keys.yml
path: keys.yml
nodeSelector:
kops.k8s.io/instancegroup: pamela-nodes
imagePullSecrets:
- name: gitlab-registry
EDIT 2: After running cron each 10 minutes, I can see my jobs, and I get expected results ( means it works )
pamela-1578992400-ppgtx 0/1 Completed 0 21m
pamela-1578993000-kn8nd 0/1 Completed 0 11m
But when right after this, I get:
Error from server (NotFound): pods "pamela-1578992400-ppgtx" not found
when trying to get logs, means that ttl should be 10 min. When trying to increase ttl, I get a feature-gates disabled issue. checking how to fix it
It is weird, after setting cron job each 10 min, I get:
➜ kubectl get jobs -n influx
NAME COMPLETIONS DURATION AGE
pamela-1578898800 0/1 32h 32h
pamela-1579007400 1/1 99s 159m
pamela-1579011000 1/1 97s 99m
pamela-1579014600 1/1 108s 39m
I use: schedule: "10 * * * *"
Don't understand what's going on here...

Related

why does my daemonset crash when a node goes down?

I have configured this DaemonSet in my cluster that is on the official Kubernetes page and everything works fine since it repartitions the replicas of my applications between my two available work nodes. The problem comes when one node goes down, then all the replicas start running on the other node. Once the downed node recovers the pods are not automatically partitioned between my nodes, so I have to manually remove all replicas and scale them again to get the DaemonSet to work.
How can i fix this?
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluentd-elasticsearch
namespace: kube-system
labels:
k8s-app: fluentd-logging
spec:
selector:
matchLabels:
name: fluentd-elasticsearch
template:
metadata:
labels:
name: fluentd-elasticsearch
spec:
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
containers:
- name: fluentd-elasticsearch
image: gcr.io/fluentd-elasticsearch/fluentd:v2.5.1
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers

Kubernetes cluster won't send logs to Loggly

I have a Kubernetes cluster which I am trying to get to run fluentd to send logs to Loggly for viewing. I am having an issue where I know my token is correct since it was working on another cluster before.
Below is my manifet.yaml file, I have no logs to share as the pod isn't logging anything.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluentd-es-v1.20
namespace: kube-system
labels:
k8s-app: fluentd-loggly
kubernetes.io/cluster-service: "true"
version: v1.20
spec:
selector:
matchLabels:
k8s-app: fluentd-loggly
template:
metadata:
labels:
k8s-app: fluentd-loggly
kubernetes.io/cluster-service: "true"
version: v1.20
spec:
containers:
- name: fluentd-loggly
image: garland/kubernetes-fluentd-loggly:1.0
command:
- '/bin/sh'
- '-c'
- '/usr/sbin/td-agent 2>&1 >> /var/log/fluentd.log'
env:
- name: LOGGLY_URL
value: "https://logs-01.loggly.com/inputs/<token>/tag/<tag>"
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
Edit 1
Logs requested
2021-06-21 16:33:43 +0000 [warn]: pattern not match: "2021-06-21T16:33:43.548145693Z stdout F 2021-06-21 16:33:43 UTC | CORE | ERROR | (pkg/autodiscovery/config_poller.go:123 in collect) | Unable to collect configurations from provider docker: temporary failure in dockerutil, will retry later: try delay not elapsed yet"
This repeats over and over with nothing else in the logs.

Kubernetes Pod Termination doesn't happen immediately, have to wait until grace period expires

I have a helm chart that has one deployment/pod and one service. I set the deployment terminationGracePeriodSeconds to 300s.
I didn't have any pod lifecycle hook, so if I terminate the pod, the pod should terminate immediately. However, now the pod will determine until my grace period ends!
Below is the deployment template for my pod:
$ kubectl get pod hpa-poc---jcc-7dbbd66d86-xtfc5 -o yaml
apiVersion: v1
kind: Pod
metadata:
annotations:
kubernetes.io/psp: eks.privileged
creationTimestamp: "2021-02-01T18:12:34Z"
generateName: hpa-poc-jcc-7dbbd66d86-
labels:
app.kubernetes.io/instance: hpa-poc
app.kubernetes.io/name: -
pod-template-hash: 7dbbd66d86
name: hpa-poc-jcc-7dbbd66d86-xtfc5
namespace: default
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: hpa-poc-jcc-7dbbd66d86
uid: 66db29d8-9e2d-4097-94fc-b0b827466e10
resourceVersion: "127938945"
selfLink: /api/v1/namespaces/default/pods/hpa-poc-jcc-7dbbd66d86-xtfc5
uid: 82ed4134-95de-4093-843b-438e94e408dd
spec:
containers:
- env:
- name: _CONFIG_LINK
value: xxx
- name: _USERNAME
valueFrom:
secretKeyRef:
key: username
name: hpa-jcc-poc
- name: _PASSWORD
valueFrom:
secretKeyRef:
key: password
name: hpa-jcc-poc
image: xxx
imagePullPolicy: IfNotPresent
name: -
resources:
limits:
cpu: "2"
memory: 8Gi
requests:
cpu: 500m
memory: 2Gi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-hzmwh
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: xxx
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 300
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: default-token-hzmwh
secret:
defaultMode: 420
secretName: default-token-hzmwh
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2021-02-01T18:12:34Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2021-02-01T18:12:36Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2021-02-01T18:12:36Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2021-02-01T18:12:34Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://c4c969ec149f43ff4494339930c8f0640d897b461060dd810c63a5d1f17fdc47
image: xxx
imageID: xxx
lastState: {}
name: -
ready: true
restartCount: 0
state:
running:
startedAt: "2021-02-01T18:12:35Z"
hostIP: 10.0.35.137
phase: Running
podIP: 10.0.21.35
qosClass: Burstable
startTime: "2021-02-01T18:12:34Z"
When I tried to terminate the pod (I used helm delete command), you can see in the time, it terminated after 5 min which is the grace period time.
$ helm delete hpa-poc
release "hpa-poc" uninstalled
$ kubectl get pod -w | grep hpa
hpa-poc-jcc-7dbbd66d86-xtfc5 1/1 Terminating 0 3h10m
hpa-poc-jcc-7dbbd66d86-xtfc5 0/1 Terminating 0 3h15m
hpa-poc-jcc-7dbbd66d86-xtfc5 0/1 Terminating 0 3h15m
So I suspect it's something for my pod/container configuration issue. Because I have tried with the other simple Java App deployment and it can terminate immediately once I terminate the pod.
BTW, I am using AWS EKS Cluster. Not sure its AWS specific as well.
So any suggestions?
I find the issue. When I exec into the container, I noticed there is one process running, which is the tailing log process.
So, I need to kill the process and add that into the prestop hook. After that, my container can shut down immediately.

Pod is in pending stage ( Error : FailedScheduling : nodes didn't match node selector )

I have a problem with one of the pods. It says that it is in a pending state.
If I describe the pod, this is what I can see:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal NotTriggerScaleUp 1m (x58 over 11m) cluster-autoscaler pod didn't trigger scale-up (it wouldn't fit if a new node is added): 2 node(s) didn't match node selector
Warning FailedScheduling 1m (x34 over 11m) default-scheduler 0/6 nodes are available: 6 node(s) didn't match node selector.
If I check the logs, there is nothing in there (it just outputs empty value).
--- Update ---
This is my pod yaml file
apiVersion: v1
kind: Pod
metadata:
annotations:
checksum/config: XXXXXXXXXXX
checksum/dashboards-config: XXXXXXXXXXX
creationTimestamp: 2020-02-11T10:15:15Z
generateName: grafana-654667db5b-
labels:
app: grafana-grafana
component: grafana
pod-template-hash: "2102238616"
release: grafana
name: grafana-654667db5b-tnrlq
namespace: monitoring
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: grafana-654667db5b
uid: xxxx-xxxxx-xxxxxxxx-xxxxxxxx
resourceVersion: "98843547"
selfLink: /api/v1/namespaces/monitoring/pods/grafana-654667db5b-tnrlq
uid: xxxx-xxxxx-xxxxxxxx-xxxxxxxx
spec:
containers:
- env:
- name: GF_SECURITY_ADMIN_USER
valueFrom:
secretKeyRef:
key: xxxx
name: grafana
- name: GF_SECURITY_ADMIN_PASSWORD
valueFrom:
secretKeyRef:
key: xxxx
name: grafana
- name: GF_INSTALL_PLUGINS
valueFrom:
configMapKeyRef:
key: grafana-install-plugins
name: grafana-config
image: grafana/grafana:5.0.4
imagePullPolicy: Always
name: grafana
ports:
- containerPort: 3000
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /api/health
port: 3000
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 30
resources:
requests:
cpu: 200m
memory: 100Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/grafana
name: config-volume
- mountPath: /var/lib/grafana/dashboards
name: dashboard-volume
- mountPath: /var/lib/grafana
name: storage-volume
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-tqb6j
readOnly: true
dnsPolicy: ClusterFirst
initContainers:
- command:
- sh
- -c
- cp /tmp/config-volume-configmap/* /tmp/config-volume 2>/dev/null || true; cp
/tmp/dashboard-volume-configmap/* /tmp/dashboard-volume 2>/dev/null || true
image: busybox
imagePullPolicy: Always
name: copy-configs
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /tmp/config-volume-configmap
name: config-volume-configmap
- mountPath: /tmp/dashboard-volume-configmap
name: dashboard-volume-configmap
- mountPath: /tmp/config-volume
name: config-volume
- mountPath: /tmp/dashboard-volume
name: dashboard-volume
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: default-token-tqb6j
readOnly: true
nodeSelector:
nodePool: cluster
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 300
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- emptyDir: {}
name: config-volume
- emptyDir: {}
name: dashboard-volume
- configMap:
defaultMode: 420
name: grafana-config
name: config-volume-configmap
- configMap:
defaultMode: 420
name: grafana-dashs
name: dashboard-volume-configmap
- name: storage-volume
persistentVolumeClaim:
claimName: grafana
- name: default-token-tqb6j
secret:
defaultMode: 420
secretName: default-token-tqb6j
status:
conditions:
- lastProbeTime: 2020-02-11T10:45:37Z
lastTransitionTime: 2020-02-11T10:15:15Z
message: '0/6 nodes are available: 6 node(s) didn''t match node selector.'
reason: Unschedulable
status: "False"
type: PodScheduled
phase: Pending
qosClass: Burstable
Do you know how should I further debug this?
Solution : You can do one of the two things to allow scheduler to fullfil your pod creation request.
you can choose to remove these lines from your pod yaml and start your pod creation again from scratch (if you need a selector for a reason go for approach as on next step 2)
nodeSelector:
nodePool: cluster
or
You can ensure that you add this nodePool: cluster as label to all your nodes so the pod will be scheduled by using the available selector.
You can use this command to label all nodes
kubectl label nodes <your node name> nodePool=cluster
Run above command by replacing node name from your cluster details for each node or only the nodes you want to be select with this label.
Your pod probably uses a node selector which can not fulfilled by scheduler.
Check pod description for something like that
apiVersion: v1
kind: Pod
metadata:
name: nginx
labels:
env: test
spec:
...
nodeSelector:
disktype: ssd
And check whether your nodes are labeled accordingly.
The simplest option would be to use "nodeName" in the Pod yaml.
First, get the node where you want to run the Pod:
kubectl get nodes
Use the below attribute inside the Pod definition( yaml) so that the Pod is forced to run under the below mentioned node only.
nodeName: seliiuvd05714

How can I explore Kubernetes folder?

I tried to start fabric on kubernetes.
Then I get this issue CrashLoopBackOff. After search a bit, I can see from the logs that
2019-06-05 07:30:19.216 UTC [main] main -> ERRO 001 Cannot run peer because error when setting up MSP from directory /etc/hyperledger/fabric/msp: err Could not load a valid signer certificate from directory /etc/hyperledger/fabric/msp/signcerts, err stat /etc/hyperledger/fabric/msp/signcerts: no such file or directory
How can I see if I am mounting the correct folder?
I want to access my crashed container to check if my msp folder are there.
Any help is appreciated!
edit 1: kubectl pod describe for peer1 org 1
Name: peer1-org1-7b9cf7fbd4-74b7q
Namespace: org1
Priority: 0
PriorityClassName: <none>
Node: minikube/10.0.2.15
Start Time: Wed, 05 Jun 2019 17:48:21 +0900
Labels: app=hyperledger
org=org1
peer-id=peer1
pod-template-hash=7b9cf7fbd4
role=peer
Annotations: <none>
Status: Running
IP: 172.17.0.9
Controlled By: ReplicaSet/peer1-org1-7b9cf7fbd4
Containers:
couchdb:
Container ID: docker://7b5e80103491476843d365dc234316ae55a92d66f2ea009cf9162583a76907fb
Image: hyperledger/fabric-couchdb:x86_64-1.0.0
Image ID: docker-pullable://hyperledger/fabric-couchdb#sha256:e89b0f95f6ff674fd043795090dd65a11d727ec005d925545cf0b4fc48aa221d
Port: 5984/TCP
Host Port: 0/TCP
State: Running
Started: Wed, 05 Jun 2019 17:49:49 +0900
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-sjp8t (ro)
peer1-org1:
Container ID: docker://95e743dceafbd78f7e29476302ac86d7eb48f97c9a50db3d174dc6684511c97b
Image: hyperledger/fabric-peer:x86_64-1.0.0
Image ID: docker-pullable://hyperledger/fabric-peer#sha256:b7c1c2a6b356996c3dbe2b9554055cd2b63194cd7a492a83de2dbabf7f7e3c65
Ports: 7051/TCP, 7052/TCP, 7053/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
peer
Args:
node
start
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 05 Jun 2019 17:50:58 +0900
Finished: Wed, 05 Jun 2019 17:50:58 +0900
Ready: False
Restart Count: 3
Environment:
CORE_LEDGER_STATE_STATEDATABASE: CouchDB
CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS: localhost:5984
CORE_VM_ENDPOINT: unix:///host/var/run/docker.sock
CORE_LOGGING_LEVEL: DEBUG
CORE_PEER_TLS_ENABLED: false
CORE_PEER_GOSSIP_USELEADERELECTION: true
CORE_PEER_GOSSIP_ORGLEADER: false
CORE_PEER_PROFILE_ENABLED: true
CORE_PEER_TLS_CERT_FILE: /etc/hyperledger/fabric/tls/server.crt
CORE_PEER_TLS_KEY_FILE: /etc/hyperledger/fabric/tls/server.key
CORE_PEER_TLS_ROOTCERT_FILE: /etc/hyperledger/fabric/tls/ca.crt
CORE_PEER_ID: peer1.org1
CORE_PEER_ADDRESS: peer1.org1:7051
CORE_PEER_GOSSIP_EXTERNALENDPOINT: peer1.org1:7051
CORE_PEER_LOCALMSPID: Org1MSP
Mounts:
/etc/hyperledger/fabric/msp from certificate (rw,path="peers/peer1.org1/msp")
/etc/hyperledger/fabric/tls from certificate (rw,path="peers/peer1.org1/tls")
/host/var/run/ from run (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-sjp8t (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
certificate:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: org1-pv
ReadOnly: false
run:
Type: HostPath (bare host directory volume)
Path: /run
HostPathType:
default-token-sjp8t:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-sjp8t
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m58s default-scheduler Successfully assigned org1/peer1-org1-7b9cf7fbd4-74b7q to minikube
Normal Pulling 2m55s kubelet, minikube Pulling image "hyperledger/fabric-couchdb:x86_64-1.0.0"
Normal Pulled 90s kubelet, minikube Successfully pulled image "hyperledger/fabric-couchdb:x86_64-1.0.0"
Normal Created 90s kubelet, minikube Created container couchdb
Normal Started 90s kubelet, minikube Started container couchdb
Normal Pulling 90s kubelet, minikube Pulling image "hyperledger/fabric-peer:x86_64-1.0.0"
Normal Pulled 71s kubelet, minikube Successfully pulled image "hyperledger/fabric-peer:x86_64-1.0.0"
Normal Created 21s (x4 over 70s) kubelet, minikube Created container peer1-org1
Normal Started 21s (x4 over 70s) kubelet, minikube Started container peer1-org1
Normal Pulled 21s (x3 over 69s) kubelet, minikube Container image "hyperledger/fabric-peer:x86_64-1.0.0" already present on machine
Warning BackOff 5s (x6 over 68s) kubelet, minikube Back-off restarting failed container
edit 2:
Kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
org1-artifacts-pv 500Mi RWX Retain Available 39m
org1-pv 500Mi RWX Retain Available 39m
org2-artifacts-pv 500Mi RWX Retain Available 39m
org2-pv 500Mi RWX Retain Available 39m
orgorderer1-pv 500Mi RWX Retain Available 39m
pvc-aa87a86f-876e-11e9-99ef-080027f6ce3c 10Mi RWX Delete Bound orgorderer1/orgorderer1-pv standard 39m
pvc-aadb69ff-876e-11e9-99ef-080027f6ce3c 10Mi RWX Delete Bound org2/org2-pv standard 39m
pvc-ab2e4d8e-876e-11e9-99ef-080027f6ce3c 10Mi RWX Delete Bound org2/org2-artifacts-pv standard 39m
pvc-abb04335-876e-11e9-99ef-080027f6ce3c 10Mi RWX Delete Bound org1/org1-pv standard 39m
pvc-abfaaf76-876e-11e9-99ef-080027f6ce3c 10Mi RWX Delete Bound org1/org1-artifacts-pv standard 39m
Kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
org1-artifacts-pv Bound pvc-abfaaf76-876e-11e9-99ef-080027f6ce3c 10Mi RWX standard 40m
org1-pv Bound pvc-abb04335-876e-11e9-99ef-080027f6ce3c 10Mi RWX standard 40m
edit 3: org1-cli.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: org1-artifacts-pv
spec:
capacity:
storage: 500Mi
accessModes:
- ReadWriteMany
hostPath:
path: "/opt/share/channel-artifacts"
# nfs:
# path: /opt/share/channel-artifacts
# server: localhost #change to your nfs server ip here
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
namespace: org1
name: org1-artifacts-pv
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Mi
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
namespace: org1
name: cli
spec:
replicas: 1
strategy: {}
template:
metadata:
labels:
app: cli
spec:
containers:
- name: cli
image: hyperledger/fabric-tools:x86_64-1.0.0
env:
- name: CORE_PEER_TLS_ENABLED
value: "false"
#- name: CORE_PEER_TLS_CERT_FILE
# value: /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1/peers/peer0.org1/tls/server.crt
#- name: CORE_PEER_TLS_KEY_FILE
# value: /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1/peers/peer0.org1/tls/server.key
#- name: CORE_PEER_TLS_ROOTCERT_FILE
# value: /opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1/peers/peer0.org1/tls/ca.crt
- name: CORE_VM_ENDPOINT
value: unix:///host/var/run/docker.sock
- name: GOPATH
value: /opt/gopath
- name: CORE_LOGGING_LEVEL
value: DEBUG
- name: CORE_PEER_ID
value: cli
- name: CORE_PEER_ADDRESS
value: peer0.org1:7051
- name: CORE_PEER_LOCALMSPID
value: Org1MSP
- name: CORE_PEER_MSPCONFIGPATH
value: /etc/hyperledger/fabric/msp
workingDir: /opt/gopath/src/github.com/hyperledger/fabric/peer
command: [ "/bin/bash", "-c", "--" ]
args: [ "while true; do sleep 30; done;" ]
volumeMounts:
# - mountPath: /opt/gopath/src/github.com/hyperledger/fabric/peer
# name: certificate
# subPath: scripts
- mountPath: /host/var/run/
name: run
# - mountPath: /opt/gopath/src/github.com/hyperledger/fabric/examples/chaincode/go
# name: certificate
# subPath: chaincode
- mountPath: /etc/hyperledger/fabric/msp
name: certificate
subPath: users/Admin#org1/msp
- mountPath: /opt/gopath/src/github.com/hyperledger/fabric/peer/channel-artifacts
name: artifacts
volumes:
- name: certificate
persistentVolumeClaim:
claimName: org1-pv
- name: artifacts
persistentVolumeClaim:
claimName: org1-artifacts-pv
- name: run
hostPath:
path: /var/run
org1-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: org1
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: org1-pv
spec:
capacity:
storage: 500Mi
accessModes:
- ReadWriteMany
hostPath:
path: /opt/share/crypto-config/peerOrganizations/org1
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
namespace: org1
name: org1-pv
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Mi
---
edit 3: peer1-org1
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
namespace: org1
name: peer1-org1
spec:
replicas: 1
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
app: hyperledger
role: peer
peer-id: peer1
org: org1
spec:
containers:
- name: couchdb
image: hyperledger/fabric-couchdb:x86_64-1.0.0
ports:
- containerPort: 5984
- name: peer1-org1
image: hyperledger/fabric-peer:x86_64-1.0.0
env:
- name: CORE_LEDGER_STATE_STATEDATABASE
value: "CouchDB"
- name: CORE_LEDGER_STATE_COUCHDBCONFIG_COUCHDBADDRESS
value: "localhost:5984"
- name: CORE_VM_ENDPOINT
value: "unix:///host/var/run/docker.sock"
- name: CORE_LOGGING_LEVEL
value: "DEBUG"
- name: CORE_PEER_TLS_ENABLED
value: "false"
- name: CORE_PEER_GOSSIP_USELEADERELECTION
value: "true"
- name: CORE_PEER_GOSSIP_ORGLEADER
value: "false"
- name: CORE_PEER_PROFILE_ENABLED
value: "true"
- name: CORE_PEER_TLS_CERT_FILE
value: "/etc/hyperledger/fabric/tls/server.crt"
- name: CORE_PEER_TLS_KEY_FILE
value: "/etc/hyperledger/fabric/tls/server.key"
- name: CORE_PEER_TLS_ROOTCERT_FILE
value: "/etc/hyperledger/fabric/tls/ca.crt"
- name: CORE_PEER_ID
value: peer1.org1
- name: CORE_PEER_ADDRESS
value: peer1.org1:7051
- name: CORE_PEER_GOSSIP_EXTERNALENDPOINT
value: peer1.org1:7051
- name: CORE_PEER_LOCALMSPID
value: Org1MSP
workingDir: /opt/gopath/src/github.com/hyperledger/fabric/peer
ports:
- containerPort: 7051
- containerPort: 7052
- containerPort: 7053
command: ["peer"]
args: ["node","start"]
volumeMounts:
#- mountPath: /opt/gopath/src/github.com/hyperledger/fabric/peer/channel-artifacts
# name: certificate
# subPath: channel-artifacts
- mountPath: /etc/hyperledger/fabric/msp
name: certificate
#subPath: crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/msp
subPath: peers/peer1.org1/msp
- mountPath: /etc/hyperledger/fabric/tls
name: certificate
#subPath: crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/
subPath: peers/peer1.org1/tls
- mountPath: /host/var/run/
name: run
volumes:
- name: certificate
persistentVolumeClaim:
claimName: org1-pv
- name: run
hostPath:
path: /run
---
apiVersion: v1
kind: Service
metadata:
namespace: org1
name: peer1
spec:
selector:
app: hyperledger
role: peer
peer-id: peer1
org: org1
type: NodePort
ports:
- name: externale-listen-endpoint
protocol: TCP
port: 7051
targetPort: 7051
nodePort: 30003
- name: chaincode-listen
protocol: TCP
port: 7052
targetPort: 7052
nodePort: 30004
---
You can do a kubectl edit pod <podname> -n <namespace> and change the command section to sleep 1000000000 then the pod will restart and you can get in there and see whats going. Or just delete the deployment, edit your yaml to remove the peer launch command, redeploy your yaml and see how the directories are laid out.
After a bit searching, I tried to mount the volume to nginx Kubernetes PVC sample. Changing the pods claimName to my created pvc. From there I exec bash to it and explore my file. Then I can see if I did mount the correct folder or not.