From my understanding, new containers should not persist data after they die.
Right now I have two Dockerfiles. One is for my application and the other is to seed a database.
This is my project structure
/
package.json
Dockerfile
docker-compose.yml
mongo-seeding/
Dockerfile
seedDatabase.mjs
This is what my seedDatabase looks like
import mongodb from "mongodb";
import data from "./dummyData";
// Connection URL
const url = "mongodb://mongo:27017/poc";
async function mongoSeeder() {
try {
// first check to see if the database exists
const client = await mongodb.MongoClient.connect(url);
const db = client.db("poc");
const questionCollection = await db.createCollection("question");
const answerCollection = await db.createCollection("answer");
await questionCollection.insertMany(data.questions);
await answerCollection.insertMany(data.answers);
} catch (e) {
console.log(e);
}
}
mongoSeeder();
This is what my the Dockerfile for seeding the database looks like
FROM node:10
WORKDIR /usr/src/app
COPY package*.json ./
COPY mongo-seeding mongo-seeding
RUN yarn add mongodb uuid faker
CMD ["yarn", "seed"]
This is what the other Dockerfile looks like
FROM node:10
WORKDIR /usr/src/app
COPY package*.json ./
RUN yarn install
RUN yarn add mongodb uuid faker
RUN yarn global add nodemon babel-cli babel-preset-env
COPY . .
EXPOSE 3000
CMD ["yarn", "dev-deploy"]
This is what my docker-compose.yml looks like
version: "3"
services:
mongo:
container_name: mongo
image: mongo
ports:
- "4040:27017"
expose:
- "4040"
app:
container_name: ingovguru
build: .
ports:
- "3000:3000"
depends_on:
- mongo
links:
- mongo
expose:
- "3000"
seed:
container_name: seed
build:
context: .
dockerfile: ./mongo-seeding/Dockerfile
links:
- mongo
depends_on:
- mongo
- app
Initially, I am seeding the database with 1000 questions and 200 answers.
Notice that I am not using any volumes, so nothing should be persisted.
I run
docker-compose build
docker-compose up
I jump into the mongo container, and I see that I have a 1000 questions and 100 answers.
I then Ctrl-C and re-run.
I now have 2000 questions and 100 answers.
Why does this happen?
Even though you are not declaring any volumes, the mongo image itself declares volumes for /data/db and /data/configdb in containers, which cover database data (see Dockerfile#L87). With Docker Compose, these anonymous volumes are re-used between invocations of docker-compose up and docker-compose down.
To remove the volumes when you bring down a stack, you need to use the -v, --volumes option. This will remove the volumes, giving you a clean database on the next up.
docker-compose down --volumes
docker-compose down documentation
-v, --volumes Remove named volumes declared in the `volumes`
section of the Compose file and anonymous volumes
attached to containers.
The problem that you have is because the mongo image has defined some volumes on its Dockerfile, you can see them doing:
docker history --no-trunc mongo | grep VOLUME
If you only stop the mongo container (you can check doing docker ps -a | grep mongo), the created volumes are kept. If you want to remove the container and its volumnes, you can remove the stopped mongo container with docker rm CONTAINER_ID or removing all the containers created by compose with docker-compose down.
In your case, if you want to build and run the services, you have to do:
docker-compose build && docker-compose down && docker-compose up -d
Related
I'm trying to run a database in docker and a python script with it to store MQTT messages. This gave me the idea to use Docker Compose since it sounded logical that both were somewhat connected. The issue I'm having is that the Docker Containers do indeed run, but they do not store anything in the database.
When I run my script locally it does store messages so my hunch is that the Compose File is not correct.
Is this the correct way to compose a python file which stores message in a DB and the database itself (with a .js file for the credentials). Any feedback would be appreciated!
version: '3'
services:
storing_script:
build:
context: .
dockerfile: Dockerfile
depends_on: [mongo]
mongo:
image: mongo:latest
environment:
MONGO_INITDB_ROOT_USERNAME: xx
MONGO_INITDB_ROOT_PASSWORD: xx
MONGO_INITDB_DATABASE: motionDB
volumes:
- ${PWD}/mongo-data:/data/db
- ./mongo-init.js:/docker-entrypoint-initdb.d/mongo-init.js:ro
ports:
- 27018:27018
restart: unless-stopped
The DockerFile im using to build:
# set base image (host OS)
FROM python:3.8-slim
# set the working directory in the container
WORKDIR /code
# copy the dependencies file to the working
directory
COPY requirements.txt .
# install dependencies
RUN pip install -r requirements.txt
# copy the content of the local src directory to
the working directory
COPY src/ .
# command to run on container start
CMD [ "python", "./main.py"]
I think this may be due to user permission.
What I did for my docker-compose for docker deployment, is I also mount the passwd file after creating a mongodb user
volumes:
/etc/passwd:/etc/passwd:ro
This worked for me as the most straight forward solution.
I am running a simple django webapp with docker-compose. I define both a web service and a db service in a docker-compose.yml file:
version: "3.8"
services:
db:
image: postgres
volumes:
- postgres_data:/var/lib/postgresql/data
environment:
- POSTGRES_DB=postgres
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
ports:
- "8000:8000"
env_file:
- ./.env.dev
depends_on:
- db
volumes:
postgres_data:
I start the service by running:
docker-compose up -d
I can load some data in there with a custom django command that I wrote for my app. Everything is running fine (with data) on localhost:8000.
However, when I run
docker-compose down
(so without -v) and then again
docker-compose up -d
the database is empty again. The volume was not persisted. From what I read in the docker-compose docs and also in several posts here at SO, persisting the volume and reusing it when you start a new container should be the default behavior (which, if I understand it correctly, you can disable by using the --renew-anon-volumes flag).
However in my case, the volume is not persisted. Or maybe it is, but my data is gone.
By doing docker volume ls I can see that my volume (I'll use the name my_volume here) still exists after the docker-compose down command. However, the CreatedAt value has been changed. This makes me think it's a different volume with the same name, and my data is already gone, but I don't know how to confirm that.
This SO answer suggests to mount the volume on /var/lib/postgresql instead of /var/lib/postgresql/data. However, I've seen other resources (like this one) where the opposite is suggested. I've tried both, but neither option works.
Thanks for any advice.
It turns out that the Dockerfile of my app was using an entrypoint in which the following command was executed: python manage.py flush which clears all data in the database. As this gets executed every time the app container starts, it clears all data. It had nothing to do with docker-compose.
I want to run mongo db in docker container. I've pulled image and run it. So it seems work ok.
But every time I start it the DB is overwritten so I loose any changes. So I want to want to map somehow internal container storage on my local host folder.
Should I write Dockerfile or/and docker-compose.yaml? I suppose this is simple question but being new in docker I can't understand what to read to get full understanding.
You do not need to write Dockerfile and make thing complex, just use offical image as mentioned in command or compose file.
You can use both options either docker run or docker-compose but the path should be correct in mapping to keep data persistent.
Here is way
Create a data directory on a suitable volume on your host system, e.g. /my/own/datadir.
Start your mongo container like this:
$ docker run --name some-mongo -v /my/own/datadir:/data/db -d mongo
The -v /my/own/datadir:/data/db part of the command mounts the
/my/own/datadir directory from the underlying host system as /data/db
inside the container, where MongoDB by default will write its data
files.
mongo docker volume
with docker-compose
version: "2"
services:
mongo:
image: mongo:latest
restart: always
ports:
- "27017:27017"
environment:
- MONGO_INITDB_DATABASE=pastime
- MONGO_INITDB_ROOT_USERNAME=root
- MONGO_INITDB_ROOT_PASSWORD=root_password
volumes:
- /my/own/datadir:/data/db
I have to create a mongo image with some default collection and data. I am able to create mongo image with this data by referring the following link :-
How to create a Mongo Docker Image with default collections and data?
so when I run the container I get the default data.
Now when I use the app and some more data is generated(by calling API's) which gets saved again in mongodb with default data.
Now for some reason if docker container is re-started, unfortunately, all the run-time created data is gone and only default data is left. Though I am saving data using volumes.
So how to persist the run time data and default data each time docker is restarted?
I am using following docker file and docker-compose file
Dockerfile :
FROM mongo
####### working isnerting data $##########
# Modify child mongo to use /data/db2 as dbpath (because /data/db wont persist the build)
RUN mkdir -p /data/db2 \
&& echo "dbpath = /data/db2" > /etc/mongodb.conf \
&& chown -R mongodb:mongodb /data/db2
COPY . /data/db2
RUN mongod --fork --logpath /var/log/mongodb.log --dbpath /data/db2 --smallfiles \
&& mongo 127.0.0.1:27017/usaa /data/db2/config-mongo.js \
&& mongod --dbpath /data/db2 --shutdown \
&& chown -R mongodb /data/db2
# Make the new dir a VOLUME to persists it
VOLUME /data/db2
CMD ["mongod", "--config", "/etc/mongodb.conf", "--smallfiles"]
and a part of docker-compose.yml
services:
mongo:
build: ./mongodb
image: "mongo:1.2"
container_name: "mongo"
ports:
- "27017:27017"
volumes:
- ${LOCAL_DIRECTORY}:/data/db2
networks:
- some-network
Reason may be, by rebuilding docker image its creating /data/db2 directory with only default data defined in .js file. But not sure.
Please correct me what I am doing wrong or suggest a new work-flow for this problem.
Thanks much!
Because docker is stateless by default. Each time you call docker run it rebuilds the container. If you want some data to persist, you have 2 general approaches:
Not to remove the container after it exits. Just give the lovely name to your container when first starting it, like docker run --name jessica mongo and then, on subsequent calls, use docker start jessica
Use volumes to store data and share it between containers. In this case you will start your container with volume arguments, like docker run -v /home/data:/data mongo. Also, you will have to reconfigure your mongodb to save data in path /data inside container. This approach is easier and can be used to share data between different containers, as well as providing default data for the first run
UPD
When using docker-compose to start the containers, if you need your data to persist between sessions, you can simply use external volumes, which you create in advance.
First create volume, lets say lovely:
docker volume create lovely
Then use it in docker-compose.yml:
version: '3'
services:
db1:
image: whatever
volumes:
- lovely:/data
db2:
image: whatever
volumes:
- lovely:/data
volumes:
lovely:
external: true
I utilize HapiJS via docker compose 2+
.env
NODE_VIEWS_PATH=../
NODE_PUBLIC_PATH=../
MONGODB_URI=mongodb://127.0.0.1:27017/mahrio
WEB_DOMAIN=http://127.0.0.1:6085
deep down somewhere I am setting the HapiJS stuff via these .env files, but for Docker I understand I need to do some changes.. no problem I made a docker specific version
docker.env
NODE_VIEWS_PATH=../
NODE_PUBLIC_PATH=../
MONGODB_URI=mongodb://mongo:27017/mahrio
WEB_DOMAIN=http://0.0.0.0:6085
I've tried 0.0.0.0 and 127.0.0.1 , neither work
I can see everything seems to work however when I goto localhost:6085 I get no response.
127.0.0.1 didn’t send any data.
Dockerfile
FROM node:carbon
# Create app directory
RUN mkdir -p /usr/src/mahrio
WORKDIR /usr/src/mahrio
COPY package*.json /usr/src/mahrio
RUN npm install
# If you are building your code for production
# RUN npm install --only=production
COPY . /usr/src/mahrio
EXPOSE 6085
CMD ["npm", "start"]
docker-compose.yml
version: "2"
services:
app:
build: .
container_name: mahrio
depends_on:
- mongo
env_file:
- docker.env
ports:
- "6085:6085"
restart: always
mongo:
container_name: mongo
image: mongo
volumes:
- ./tmp:/data/db
ports:
- "27017:27017"
any ideas ? No errors from nodejs are coming, everything looks A-OKAY at the console and I know it works outside docker just fine.
Edit: added the docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
25e7a4c3f350 mahriomedium_app "npm start" 24 hours ago Up About a minute 0.0.0.0:6085->6085/tcp mahrio
c8d691777aa0 mongo "docker-entrypoint..." 3 days ago Up About a minute 0.0.0.0:27017->27017/tcp mongo
docker logs
> mahrio-medium#0.0.1 start /usr/src/mahrio
> node server/index.js
Running Development!
MongoDB Config...
Server running at: http://127.0.0.1:6085
MongoDB connected!
db connection opened
it turned out WEB_DOMAIN was the wrong env var
the right var is NODE_URI to set to 0.0.0.0
all works now