I have a tomcat + postgres application that I test with docker-compose. I am trying to package the application in a kubernetes config file.
For now, I am running kubernetes (and kubectl) using my Docker Desktop for Windows installation. Eventually, I want to deploy to other environments.
I am currently trying to replicate some of the volume functionality in docker-compose within the following config file.
apiVersion: v1
kind: Pod
metadata:
name: pg-pod
spec:
volumes:
- name: "pgdata-vol"
#emptyDir: {}
hostPath:
path: /c/temp/vols/pgdata
containers:
- image: postgres
name: db
ports:
- containerPort: 5432
name: http
protocol: TCP
volumeMounts:
- mountPath: "/pgdata"
name: "pgdata-vol"
env:
- name: PGDATA
value: /pgdata
When postgres launches, I get see the following error.
fixing permissions on existing directory /pgdata ... ok
creating subdirectories ... ok
selecting default max_connections ... 20
selecting default shared_buffers ... 400kB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
2019-07-26 20:43:41.844 UTC [78] FATAL: data directory "/pgdata" has wrong ownership
2019-07-26 20:43:41.844 UTC [78] HINT: The server must be started by the user that owns the data directory.
child process exited with exit code 1
initdb: removing contents of data directory "/pgdata"
running bootstrap script ...
I presume that I either need to provide some additional parameters to my volume definition or I need to try a different type of volume config (local vs hostPath).
I found a partial solution to this issue.
Interestingly, if I assign a linux-style path as my host-path (on Windows), then my pgdata-vol persists until Docker Desktop is restarted.
Instead of mounting to a real windows location
volumes:
- name: "pgdata-vol"
hostPath:
path: /c/temp/vols/pgdata
I use a "linux" location as my Windows hostPath
volumes:
- name: "pgdata-vol"
hostPath:
path: /tmp/vols/pgdata
Curiously, I cannot actually find this path from Windows. I presume this /tmp is local to my Docker Desktop instance.
This solution does not offer true persistence, but it has helped me to work around a roadblock that was impacting testing.
This is a known issue with Docker image on Windows. Right now it is not possible to correctly mount Windows directories as volumes. You may however try to workaround it by using a persistent Docker volume. For example:
db:
image: postgres
environment:
- POSTGRES_USER=<user>
- POSTGRES_PASSWORD=<pass>
- POSTGRES_DB=<db_name>
ports:
- <ports>
volumes:
- pgdata:<path>
networks:
- <network>
volumes:
pgdata:
More Information:
data directory "/var/lib/postgresql/data" has wrong ownership
postgresql-data-pgdata-has-wrong-ownership
postgres-to-work-on-persistent-windows-mount
Please let me know if that helped.
Have you tried using WSL? My setup for windows is WSL + Ubuntu + Docker for windows and I can mount volumes normally.
I've followed that tutorial to configure all my environment:
https://nickjanetakis.com/blog/setting-up-docker-for-windows-and-wsl-to-work-flawlessly
Related
I have been using below to mount docker in my deployment file. This way From the inside of my code, I can build custom images using docker command
volumeMounts:
- name: dockersock
mountPath: "/var/run/docker.sock"
volumes:
- name: dockersock
hostPath:
path: /var/run/docker.sock
Now in newer kubernetes version privileged and hostPath is disabled. Due to this on new kubernetes above will not work.
I know i have few alternatives like Kaniko for building images, but I am not sure how to use it in code and what changes I need to make in my deployment file.
I'm trying to deploy Mongo onto Azure Container Instances as part of a container group. To do this, I use a Storage Account with a file share to persist the mongo data. It's impossible to mount the volume in the /data/db default location, so I mount it elsewhere and start mongod using the --db-path flag. This all works fine using the CLI, full command below.
However, when I want to translate all of these commands into my YAML config file it doesn't work. Mongo crashes out with an unknown file or directory error. If I start the container without the --db-path flag but still mount the volume, I can exec into the running container and see that the volume is there and is attached. I can even manually create folders in the share via the Azure Portal and see them appear in the container.
Documentation and examples are a little thin on the ground, especially YAML based examples. The biggest difference with the container group is having to define a named volume separate from the container which is used by the volumeMounts property. Is it just a syntax error? Are the CLI command and the YAML not equivalent in some way?
CLI Command
az container create
--resource-group rsenu-hassPilots-group
--name mongo
--image mongo
--azure-file-volume-account-name <account>
--azure-file-volume-account-key "<key>" --azure-file-volume-share-name mongodata
--azure-file-volume-mount-path "/data/mongoaz"
--ports 27017
--cpu 1
--ip-address public
--memory 1.5
--os-type Linux
--protocol TCP
--command-line "mongod --dbpath=/data/mongoaz"
YAML Config
apiVersion: 2018-10-01
location: uksouth
name: trustNewArtGroup
properties:
containers:
- name: mongo
properties:
image: mongo:4.2.3
resources:
requests:
cpu: 1
memoryInGb: 1.5
ports:
- port: 27017
volumeMounts:
- name: database
mountPath: /data/azstorage
environmentVariables:
- name: 'MONGO_INITDB_DATABASE'
value: 'trust-new-art-db'
command:
- "mongod --dbpath=/data/azstorage"
osType: Linux
ipAddress:
type: Public
dnsNameLabel: trustnewart
ports:
- protocol: tcp
port: '27017'
volumes:
- name: database
azureFile:
sharename: mongodata
storageAccountName: <account>
storageAccountKey: <key>
tags: null
type: Microsoft.ContainerInstance/containerGroups
With a bit of help from this page in the documentation, I've discovered it was a syntax issue. The correct way to override the entrypoint in a YAML config file is as follows:
command: ['mongod', '--dbpath', '/data/azstorage']
I have an python application whose docker build takes about 15-20 minutes.
Here is how my Dockerfile looks like more or less
FROM ubuntu:18.04
...
COPY . /usr/local/app
RUN pip install -r /usr/local/app/requirements.txt
...
CMD ...
Now if I use skaffold, any code change triggers a rebuild and it is going to do a reinstall of all requirements(as from the COPY step everything else is going to be rebuild) regardless of whether they were already installed. iIn docker-compose this issue would be solved using volumes. In kubernetes, if we use volumes in the following way:
apiVersion: v1
kind: Pod
metadata:
name: test
spec:
containers:
- image: test:test
name: test-container
volumeMounts:
- mountPath: /usr/local/venv # this is the directory of the
# virtualenv of python
name: test-volume
volumes:
- name: test-volume
awsElasticBlockStore:
volumeID: <volume-id>
fsType: ext4
will this extra requirements build be resolved with skaffold?
I can't speak for skaffold specifically but the container image build can be improved. If there is layer caching available then only reinstall the dependencies when your requirements.txt changes. This is documented in the "ADD or COPY" Best Practices.
FROM ubuntu:18.04
...
COPY requirements.txt /usr/local/app/
RUN pip install -r /usr/local/app/requirements.txt
COPY . /usr/local/app
...
CMD ...
You may need to trigger updates some times if the module versions are loosely defined and say you want a new patch version. I've found requirements should be specific so versions don't slide underneath your application without your knowledge/testing.
Kaniko in-cluster builds
For kaniko builds to make use of a cache in a cluster where there is no persistent storage by default, kaniko needs either a persistent volume mounted (--cache-dir) or a container image repo (--cache-repo) with the layers available.
If your goal is to speed up the dev process: Instead of triggering an entirely new deployment process every time you change a line of code, you can switch to a sync-based dev process to deploy once and then update the files within the running containers when editing code.
Skaffold supports file sync to directly update files inside the deployed containers if you change them on your local machine. However, the docs state "File sync is alpha" (https://skaffold.dev/docs/how-tos/filesync/) and I can completely agree from working with it a while ago: The sync mechanism is only one-directional (no sync from container back to local) and pretty buggy, i.e. it crashes frequently when switching git branches, installing dependencies etc. which can be pretty annoying.
If you want a more stable alternative for sync-based Kubernetes development which is very easy to get started with, take a look at DevSpace: https://github.com/devspace-cloud/devspace
I am one of the maintainers of DevSpace and started the project because Skaffold was much too slow for our team and it did not have a file sync back then.
#Matt's answer is a great best practice (+1) - skaffold in and of itself won't solve the underlying layer cache invalidation issues which results in having to re-install the requirements during every build.
For additional performance, you can cache all the python packages in a volume mounted in your pod for example:
apiVersion: v1
kind: Pod
metadata:
name: test
spec:
containers:
- image: test:test
name: test-container
volumeMounts:
- mountPath: /usr/local/venv
name: test-volume
- mountPath: /root/.cache/pip
name: pip-cache
volumes:
- name: test-volume
awsElasticBlockStore:
volumeID: <volume-id>
fsType: ext4
- name: pip-cache
awsElasticBlockStore:
volumeID: <volume-id>
fsType: ext4
That way if the build cache is ever invalidated and you have to re-install your requirements.txt you'd be saving some time by fetching them from cache.
If you're building with kaniko you can also cache base images to a persistent disk using the kaniko-warmer, for example:
...
volumeMounts:
...
- mountPath: /cache
name: kaniko-warmer
volumes:
...
- name: kaniko-warmer
awsElasticBlockStore:
volumeID: <volume-id>
fsType: ext4
Running the kaniko-warmer inside the pod: docker run --rm -it -v /cache:/cache --entrypoint /kaniko/warmer gcr.io/kaniko-project/warmer --cache-dir=/cache --image=python:3.7-slim --image=nginx:1.17.3. Your skaffold.yaml might look something like:
apiVersion: skaffold/v1beta13
kind: Config
build:
artifacts:
- image: test:test
kaniko:
buildContext:
localDir: {}
cache:
hostPath: /cache
cluster:
namespace: jx
dockerConfig:
secretName: jenkins-docker-cfg
tagPolicy:
envTemplate:
template: '{{.DOCKER_REGISTRY}}/{{.IMAGE_NAME}}'
deploy:
kubectl: {}
I'm a new Ambassador user here. I have walked thru the tutorial, in an effort to understand how use ambassador gateway. I am attempting to run this locally via Docker Compose until it's ready for deployment to K8s in production.
My use case is that all http traffic comes in on port 80, and then directed to the appropriate service. Is it considered best practice to have a docker-compose.yaml file in the working directory that refers to services in the /config directory? I ask because this doesn't appear to actually pickup my files (the postgres startup doesn't show in console). And when I run "docker ps" I only show:
CONTAINER ID IMAGE PORTS NAMES
8bc8393ac04c 05a916199684 k8s_statsd_ambassador-8564bfb874-q97l9_default_e775d686-a93c-11e8-9caa-025000000001_0
1c00f2341caf d7cf7cf837f9 k8s_ambassador_ambassador-8564bfb874-q97l9_default_e775d686-a93c-11e8-9caa-025000000001_0
fe20c4819514 05a916199684 k8s_statsd_ambassador-8564bfb874-xzvkl_default_e775ffe6-a93c-11e8-9caa-025000000001_0
ba6415b028ba d7cf7cf837f9 k8s_ambassador_ambassador-8564bfb874-xzvkl_default_e775ffe6-a93c-11e8-9caa-025000000001_0
9df07dc5083d 05a916199684 k8s_statsd_ambassador-8564bfb874-w5vsq_default_e773ed53-a93c-11e8-9caa-025000000001_0
682e1f9902a0 d7cf7cf837f9 k8s_ambassador_ambassador-8564bfb874-w5vsq_default_e773ed53-a93c-11e8-9caa-025000000001_0
bb6d2f749491 quay.io/datawire/ambassador:0.40.2 0.0.0.0:80->80/tcp apigateway_ambassador_1
I have a docker-compose.yaml:
version: '3.1'
# Define the services/containers to be run
services:
ambassador:
image: quay.io/datawire/ambassador:0.40.2
ports:
- 80:80
volumes:
# mount a volume where we can inject configuration files
- ./config:/ambassador/config
postgres:
image: my-postgresql
ports:
- '5432:5432'
and in /config/mapping-postgres.yaml:
---
apiVersion: ambassador/v0
kind: Mapping
name: postgres_mapping
rewrite: ""
service: postgres:5432
volumes:
- ../my-postgres:/docker-entrypoint-initdb.d
environment:
- POSTGRES_MULTIPLE_DATABASES=db1, db2, db3
- POSTGRES_USER=<>
- POSTGRES_PASSWORD=<>
volumes and environment are not valid configs for Ambassador Mappings. Ambassador lets you proxy to postgres but the authentication has to be handled by your application.
Having said that, it looks like your Postgres container is not starting. (Perhaps because it needs an initial config). You can check for errors with:
$ docker ps -a | grep postgres
$ docker logs <container-id-from-previous-step>
You can also check a postgres docker compose example here.
Is it considered best practice to have a docker-compose.yaml file in the working directory that refers to services in the /config directory?
It's pretty standard, but you can use any directory you like for this.
Can we use nfs volume plugin to maintain the High Availability and Disaster Recovery among the kubernetes cluster?
I am running the pod with MongoDB. Getting the error
chown: changing ownership of '/data/db': Operation not permitted .
Cloud any body, Please suggest me how to resolve the error? (or)
Is any alternative volume plugin is suggestible to achieve HA- DR in kubernetes cluster?
chown: changing ownership of '/data/db': Operation not permitted .
You'll want to either launch the mongo container as root, so that you can chown the directory, or if the image prohibits it (as some images already have a USER mongo clause that prohibits the container from escalating privileges back up to root), then one of two things: supersede the user with a securityContext stanza in containers: or use an initContainer: to preemptively change the target folder to be the mongo UID:
Approach #1:
containers:
- name: mongo
image: mongo:something
securityContext:
runAsUser: 0
(which may require altering your cluster's config to permit such a thing to appear in a PodSpec)
Approach #2 (which is the one I use with Elasticsearch images):
initContainers:
- name: chmod-er
image: busybox:latest
command:
- /bin/chown
- -R
- "1000" # or whatever the mongo UID is, use string "1000" not 1000 due to yaml
- /data/db
volumeMounts:
- name: mongo-data # or whatever
mountPath: /data/db
containers:
- name: mongo # then run your container as before
/data/db is a mountpoint, even if you don't explicitly mount a volume there. The data is persisted to an overlay specific to the pod.
Kubernetes mounts all volumes as 0755 root.root, regardless of what the permissions for the directory were intially.
Of course mongo cannot chown that.
If you mount the volume somewhere below /data/db, you will get the same error.
And if you mount the volume above at /data, the data will not be stored on the NFS because the mountpoint at /data/db will write to the overlay instead. But you won't get that error anymore.
By adding command:["mongod"] in your Deployment Manifest, it will override the default entrypoint script and will prevent executing the chown.
...
spec:
containers:
- name: mongodb
image: mongo:4.4.0-bionic
command: ["mongod"]
...
Instead of mounting /data/db, we could mount /data. Internally mongo will create /data/db. During entrypoint, mongo tries to chown this directory but if we mount a volume directory to this mount point, as a mongo container user - it will not be able to chown. That's the cause of the issue
Here is a sample of working mongo deployment yaml
...
spec:
containers:
- name: mongo
image: mongo:latest
volumeMounts:
- mountPath: /data
name: mongo-db-volume
volumes:
- hostPath:
path: /Users/name/mongo-data
type: Directory
name: mongo-db-volume
...