We have created .yaml file to deploy osrm/osrm-backend (https://hub.docker.com/r/osrm/osrm-backend/tags) in a Kubernetes cluster.
We initially download the pbf file in the node's volume, then we create the necessary files for the service and finally the service starts.
You may find the yaml file below:
apiVersion: v1
kind: Service
metadata:
name: osrm-albania
labels:
app: osrm-albania
spec:
ports:
- port: 5000
targetPort: 5000
name: http
selector:
app: osrm-albania
type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: osrm-albania
spec:
replicas: 1
selector:
matchLabels:
app: osrm-albania
template:
metadata:
labels:
app: osrm-albania
spec:
containers:
- name: osrm-albania
image: osrm/osrm-backend:latest
command: ["/bin/sh", "-c"]
args: ["osrm-extract -p /opt/car.lua /data/albania-latest.osm.pbf && osrm-partition /data/albania-latest.osrm && osrm-customize /data/albania-latest.osrm && osrm-routed --algorithm mld /data/albania-latest.osrm"]
ports:
- containerPort: 5000
name: osrm-port
volumeMounts:
- name: albania
readOnly: false
mountPath: /data
initContainers:
- name: get-osrm-file
image: busybox
command: ['wget', 'http://download.geofabrik.de/europe/albania-latest.osm.pbf', '--directory-prefix=/data']
volumeMounts:
- name: albania
readOnly: false
mountPath: /data
volumes:
- name: albania
emptyDir: {}
The problem is that we need to update the map data used by the osrm service, regularly. Which means to be able to re-download the pbf file and recreate the necessary files to be used by the service.
This might be achieved via kubernetes cronjobs which might has to use persistent volumes instead (Cron Jobs in Kubernetes - connect to existing Pod, execute script).
Is this the only way to achieve getting new map data and refresh the data used by the osrm service?
How exactly?
Is there a better - easier way to achieve this?
This is a tricky situation, I had the same problem in my cluster and I fixed dividing the job in more pods:
1 wget in a volume mount ('volume A')
2 extract, partition, customize in 'volume A'
3 copy 'volume A' to volume mount B
4 run osrm-routed with 'volume B'
In this way a set pod 1, 2, and 3 as a cronjob and each pod would do all operation without broke the service.
This issue was due by a large amount of time for the first 3 operation (2 to 3 hours).
Related
I am using payara/server-full in Kubernetes. I want to add a persistent volume so that all configuration made to the Payara server via the Admin UI is perstisted after the pod is recreated, including uploaded .war files.
Right now my deployment looks like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name:
spec:
selector:
matchLabels:
app: myapp
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: payara/server-full
imagePullPolicy: "Always"
ports:
- name: myapp-default
containerPort: 8080
- name: myapp-admin
containerPort: 4848
How to augment that yaml file to make use of a persistent volume?
Which path(s) within payara should be synced with the persistent volume so that Payara's configuration isn't lost after redeployment ?
Which additional yaml files do I need?
So after a longer conideration of the problem I realised I need to persist everything under /opt/payara/appserver/glassfish/domains for all configuration made via the Admin UI to be persisted. However if I simply start the pod with a volumeMount pointing to that path, i.e.
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
selector:
matchLabels:
app: myapp
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
template:
metadata:
labels:
app: myapp
spec:
volumes:
- name: myapp-vol
persistentVolumeClaim:
claimName: myapp-rwo-pvc
containers:
- name: myapp
image: payara/server-full
imagePullPolicy: "Always"
ports:
- name: myapp-default
containerPort: 8080
- name: myapp-admin
containerPort: 4848
volumeMounts:
- mountPath: "/opt/payara/appserver/glassfish/domains"
and
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: myapp-rwo-pvc
labels:
app: dont-delete-autom
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
then the Payara server won't be able to start successfully, because Kubernetes will mount an empty persistent volume into that location. Payara needs however config files which are originally located within /opt/payara/appserver/glassfish/domains.
What I needed to do is to provision the volume with the data by default located in that folder. But how to do that when the only way to access the PV is to mount it into a pod?
Fist I scaled the above deployment to 0 with:
kubectl scale --replicas=0 deployment/myapp
This deletes all pods accessing the persistent volume.
Then I created a "provisioning" pod which mounts the previously created persistent volume into /tmp.
apiVersion: v1
kind: Pod
metadata:
labels:
app: myapp
name: pv-provisioner
namespace: default
spec:
containers:
- image: payara/server-full
imagePullPolicy: Always
name: pv-provisioner
ports:
- containerPort: 8080
name: myapp-default
protocol: TCP
- containerPort: 4848
name: myapp-admin
protocol: TCP
volumeMounts:
- mountPath: "/tmp"
name: myapp-vol
resources:
limits:
cpu: "2"
memory: 2Gi
requests:
cpu: 500m
memory: 128Mi
volumes:
- name: myapp-vol
persistentVolumeClaim:
claimName: myapp-rwo-pvc
Then I used the following commands to copy the necessary data first from the "provisioning" pod to a local folder /tmp and then back from /tmp to the persistent volume (previously mounted into pv-provisioner:/tmp). There is no option to copy directly from pod:/a to pod:/b
kubectl cp pv-provisioner:/opt/payara/appserver/glassfish/domains/. tmp
kubectl cp tmp/. pv-provisioner:/tmp
As a result everything stored under /opt/payara/appserver/glassfish/domains/ in the original payara container was now copied into the persistent volume identified by the persistence volume claim "myapp-rwo-pvc".
To finish it up I deleted the provisioning pod and scaled the deployment back up:
kubectl delete pod pv-provisioner
kubectl scale --replicas=3 deployment/myapp
The payara server is now starting successfully and any configuration made via the Admin UI, including .war deployments is persisted, such that the payara pods can be killed any time and after the restart everything is as before.
Thanks for reading.
I'm trying to play with single pod multi container scenario.
The problem is one of my container (directus) is a node app that run as user 'node' with uid 1000
First try, I use hostpath as storage back end. With this, I need to change the host's directory mode with chmod manualy.
Now, I'm trying using longhorn.
And basicaly I don't want to change a host directory mod/ownership each time i deploy this deployment.
Here is my manifest
apiVersion: apps/v1
kind: Deployment
metadata:
name: lh-directus
namespace: lh-directus
spec:
replicas: 1
selector:
matchLabels:
app: lh-directus
template:
metadata:
labels:
app: lh-directus
spec:
nodeSelector:
kubernetes.io/os: linux
isGeneralDeployment: "true"
volumes:
- name: lh-directus-uploads-volume
persistentVolumeClaim:
claimName: lh-directus-uploads-pvc
- name: lh-directus-dbdata-volume
persistentVolumeClaim:
claimName: lh-directus-dbdata-pvc
containers:
# Redis Cache
- name: redis
image: redis:6
# Database
- name: database
image: postgres:12
volumeMounts:
- name: lh-directus-dbdata-volume
mountPath: /var/lib/postgresql/data
# Directus
- name: directus
image: directus/directus:latest
securityContext:
fsGroup: 1000
volumeMounts:
- name: lh-directus-uploads-volume
mountPath: /directus/uploads
When I Appy the manifest, I got error
error: error validating "lh-directus.yaml": error validating data: ValidationError(Deployment.spec.template.spec.containers[2].securityContext): unknown field "fsGroup" in io.k8s.api.core.v1.SecurityContext; if you choose to ignore these errors, turn validation off with --validate=false
I reads about initContainer ....
But Kindly please tell me how to fix this problem without initContainer and without manualy set/change host's path ownership/mod.
Sincerely
-bino-
I have a kubernetes deployment with the below spec that gets installed via helm 3.
apiVersion: apps/v1
kind: Deployment
metadata:
name: gatekeeper
spec:
replicas: 1
template:
spec:
containers:
- name: gatekeeper
image: my-gatekeeper-image:some-sha
args:
- --listen=0.0.0.0:80
- --client-id=gk-client
- --discovery-url={{ .Values.discoveryUrl }}
I need to pass the discoveryUrl value as a helm value, which is the public IP address of the nginx-ingress pod that I deploy via a different helm chart. I install the above deployment like below:
helm3 install my-nginx-ingress-chart
INGRESS_IP=$(kubectl get svc -lapp=nginx-ingress -o=jsonpath='{.items[].status.loadBalancer.ingress[].ip}')
helm3 install my-gatekeeper-chart --set discovery_url=${INGRESS_IP}
This works fine, however, Now instead of these two helm3 install, I want to have a single helm3 install, where both the nginx-ingress and the gatekeeper deployment should be created.
I understand that in the initContainer of my-gatekeeper-image we can get the nginx-ingress ip address, but I am not able to understand how to set that as an environment variable or pass to the container spec.
There are some stackoverflow questions that mention that we can create a persistent volume or secret to achieve this, but I am not sure, how that would work if we have to delete them. I do not want to create any extra objects and maintain the lifecycle of them.
It is not possible to do this without mounting a persistent volume. But the creation of persistent volume can be backed by just an in-memory store, instead of a block storage device. That way, we do not have to do any extra lifecycle management. The way to achieve that is:
apiVersion: v1
kind: ConfigMap
metadata:
name: gatekeeper
data:
gatekeeper.sh: |-
#!/usr/bin/env bash
set -e
INGRESS_IP=$(kubectl get svc -lapp=nginx-ingress -o=jsonpath='{.items[].status.loadBalancer.ingress[].name}')
# Do other validations/cleanup
echo $INGRESS_IP > /opt/gkconf/discovery_url;
exit 0
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: gatekeeper
labels:
app: gatekeeper
spec:
replicas: 1
selector:
matchLabels:
app: gatekeeper
template:
metadata:
name: gatekeeper
labels:
app: gatekeeper
spec:
initContainers:
- name: gkinit
command: [ "/opt/gk-init.sh" ]
image: 'bitnami/kubectl:1.12'
volumeMounts:
- mountPath: /opt/gkconf
name: gkconf
- mountPath: /opt/gk-init.sh
name: gatekeeper
subPath: gatekeeper.sh
readOnly: false
containers:
- name: gatekeeper
image: my-gatekeeper-image:some-sha
# ENTRYPOINT of above image should read the
# file /opt/gkconf/discovery_url and then launch
# the actual gatekeeper binary
imagePullPolicy: Always
ports:
- containerPort: 80
protocol: TCP
volumeMounts:
- mountPath: /opt/gkconf
name: gkconf
volumes:
- name: gkconf
emptyDir:
medium: Memory
- name: gatekeeper
configMap:
name: gatekeeper
defaultMode: 0555
Using init containers is indeed a valid solution but you need to be aware that by doing so you are adding complexity to your deployment.
This is because you would also need to create serviceaccount with permisions to be able to read service objects from inside of init container. Then, when having the IP, you can't just set env variable for gatekeeper container without recreating a pod so you would need to save the IP e.g. to shared file and read it from it when starting gatekeeper.
Alternatively you can reserve ip address if your cloud provided supports this feature and use this static IP when deploying nginx service:
apiVersion: v1
kind: Service
[...]
type: LoadBalancer
loadBalancerIP: "YOUR.IP.ADDRESS.HERE"
Let me know if you have any questions or if something needs clarification.
I have one spring boot microservice running on docker container, below is the Dockerfile
FROM java:8-jre
MAINTAINER <>
WORKDIR deploy/
#COPY config/* /deploy/config/
COPY ./ms.console.jar /deploy/
CMD chmod +R 777 ./ms.console.jar
CMD ["java","-jar","/deploy/ms.console.jar","console"]
EXPOSE 8384
here my configuration stores in external folder, i.e /config/console-server.yml and when I started the application, internally it will load the config (spring boot functionality).
Now I want to separate this configuration using configmap, for that I simply created one configmap and storing all the configuration details.
kubectl create configmap console-configmap
--from-file=./config/console-server.yml
kubectl describe configmap console-configmap
below are the description details:
Name: console-configmap
Namespace: default
Labels: <none>
Annotations: <none>
Data
====
console-server.yml:
----
server:
http:
port: 8385
compression:
enabled: true
mime-types: application/json,application/xml,text/html,text/xml,text/plain,text/css,application/javascript
min-response-size: 2048
---
spring:
thymeleaf:
prefix: classpath:/static
application:
name: console-service
profiles:
active: native
servlet:
multipart:
max-file-size: 30MB
max-request-size: 30MB
---
host:
gateway: http://apigateway:4000
webhook: http://localhost:9000
my deployment yml is:
apiVersion: apps/v1 # for versions before 1.8.0 use apps/v1beta1
kind: Deployment
metadata:
name: consoleservice1
spec:
selector:
matchLabels:
app: consoleservice
replicas: 1 # tells deployment to run 3 pods matching the template
template: # create pods using pod definition in this template
metadata:
labels:
app: consoleservice
spec:
containers:
- name: consoleservice
image: ms-console
ports:
- containerPort: 8384
imagePullPolicy: Always
envFrom:
- configMapRef:
name: console-configmap
imagePullSecrets:
- name: regcresd
My doubt is, I commented config folder in the Dockerfile, so while running pods, it's throwing exception because of no configuration, how I will inject this console-configmap to my deployment, what I tried already shared, but getting same issues.
First of all, how are you consuming the .yml file in your application? If you consume your yml file contents as environment variables, your config should just work fine. But I suspect that you want to consume the contents from the config file inside the container. If that is the case you have to create a volume out of the configmap as follows:
apiVersion: apps/v1 # for versions before 1.8.0 use apps/v1beta1
kind: Deployment
metadata:
name: consoleservice1
spec:
selector:
matchLabels:
app: consoleservice
replicas: 1 # tells deployment to run 3 pods matching the template
template: # create pods using pod definition in this template
metadata:
labels:
app: consoleservice
spec:
containers:
- name: consoleservice
image: ms-console
ports:
- containerPort: 8384
imagePullPolicy: Always
volumeMounts:
- mountPath: /app/config
name: config
volumes:
- name: config
configMap:
name: console-configmap
imagePullSecrets:
- name: regcresd
The file will be available in the path /app/config/console-server.yml. You have to modify it as per your needs.
do you need to load key:value pairs from the config file as environment variables then below spec would work
envFrom:
- configMapRef:
name: console-configmap
if you need the config as a file inside pod then mount the configmap as volume. following link would be helpful
https://kubernetes.io/docs/tutorials/configuration/configure-redis-using-configmap/
I define a Secret:
apiVersion: v1
kind: Secret
metadata:
name: mysecret
type: Opaque
stringData:
config.yaml: |-
apiUrl: "https://my.api.com/api/v1"
username: Administrator
password: NewPasswdTest11
And then creating volume mount in Deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: k8s-webapp-test
labels:
name: k8s-webapp-test
version: 1.0.4
spec:
replicas: 2
selector:
matchLabels:
name: k8s-webapp-test
version: 1.0.4
template:
metadata:
labels:
name: k8s-webapp-test
version: 1.0.4
spec:
nodeSelector:
kubernetes.io/os: windows
volumes:
- name: secret-volume
secret:
secretName: string-data-secret
containers:
- name: k8s-webapp-test
image: dockerstore/k8s-webapp-test:1.0.4
ports:
- containerPort: 80
volumeMounts:
- name: secret-volume
mountPath: "/secrets"
readOnly: false
So, after the deployment, I have 2 pods with volume mounts in C:\secrets (I do use Windows nodes). When I try to edit config.yaml that is located in C:\secrets folder, I get following error:
Access to the path 'c:\secrets\config.yaml' is denied.
Although I marked file as readOnly false I cannot write to it. How can I modify the file?
As you can see here it is not possible by intention:
Secret, configMap, downwardAPI and projected volumes will be mounted as read-only volumes. Applications that attempt to write to these volumes will receive read-only filesystem errors. Previously, applications were allowed to make changes to these volumes, but those changes were reverted at an arbitrary interval by the system. Applications should be re-configured to write derived files to another location
You can look into using an init container which maps the secret and then copies it to the desired location where you might be able to modify it.
As an alternative to the init container you might also use a container lifecycle hook i.e. a PostStart-hook which executes immediately after a container is created.
lifecycle:
postStart:
exec:
command:
- "/bin/sh"
- "-c"
- >
cp -r /secrets ~/secrets;
You can create secrets from within a Pod but it seems you need to utilize the Kubernetes REST API to do so:
https://kubernetes.io/docs/concepts/overview/kubernetes-api/