Trying to sync between MongoDB and Elasticsearch with MongoDbJdbcDriver by following this answer. I use docker-compose for development, and this is how it looks like for logstash:
logstash:
image: logstash:7.9.1
container_name: logstash
volumes:
- ./logstash/jars/gson-2.8.6.jar:/usr/share/logstash/logstash-core/lib/jars/gson-2.8.6.jar:ro
- ./logstash/jars/mongojdbc2.3.jar:/usr/share/logstash/logstash-core/lib/jars/mongojdbc2.3.jar:ro
- ./logstash/jars/mongo-java-driver-3.12.6.jar:/usr/share/logstash/logstash-core/lib/jars/mongo-java-driver-3.12.6.jar:ro
- ./logstash/config/pipelines.yml:/usr/share/logstash/config/pipelines.yml
- ./logstash/pipeline/mongo-to-elasticsearch.conf:/usr/share/logstash/pipeline/mongo-to-elasticsearch.conf
command: logstash
depends_on:
- elasticsearch
I run docker-compose up and it gives this error:
...
logstash | Error: unable to load mongojdbc2.3.jar from :jdbc_driver_library, file not readable (please check user and group permissions for the path)
logstash | Exception: LogStash::PluginLoadingError
...
I've checked file permission of mongojdbc2.3.jar on my machine, given read and write for me and docker group. However when I check inside logstash container, owner is not root but logstash:
// ls -l /usr/share/logstash/logstash-core/lib/jars
...
-rw-r--r-- 1 logstash logstash 2315317 Jul 24 15:13 mongo-java-driver-3.12.6.jar
-rw-r--r-- 1 logstash logstash 83500 Sep 8 01:30 mongojdbc2.3.jar
-rw-rw-r-- 1 logstash root 107210 Sep 1 23:32 org.eclipse.core.commands-3.6.0.jar
...
I tried changing ownership inside container but I couldn't do it since I don't have the sudo permission.
Please help, how I can I sync between MongoDB and Elasticsearch successfully?
Stack: MongoDB (v4.4.1), Logstash (v7.9.1), Docker (v19.03.12), Docker-compose (v1.27.3)
Try to create new image with jars in it
logstash:
build: ./logstash/
container_name: logstash
volumes:
- ./logstash/config/pipelines.yml:/usr/share/logstash/config/pipelines.yml
- ./logstash/pipeline/mongo-to-elasticsearch.conf:/usr/share/logstash/pipeline/mongo-to-elasticsearch.conf
command: logstash
depends_on:
- elasticsearch
./logstash/Dockerfile
FROM docker.elastic.co/logstash/logstash:7.9.1
COPY ./jars/ /usr/share/logstash/logstash-core/lib/jars
Related
I have a SpringBoot application running in the docker and I am using PostgreSQL as a database for this project and the database also running in the docker.
Now, I want to use MongoDb along with PostgreSQL as database to my SpringBoot application.
I added MongoDb info in the docker-compose.yml file and created new Dockerfile and ran the application. After that MongoDb got installed and running in the docker successfully.
I created a api for insertion of a document into the collection. When I hit the api I am getting the error. I think, I am not able to connect to the MongoDb which is running in the docker.
error:- com.mongodb.MongoSocketOpenException: Exception opening socket
I think I have to do configuration in the MongoDb before doing any CRUD operations.
Can anyone please share a detailed configuration of MongoDb with some examples.
Or provide some information which can help me achieve my task.
Thanks.
Docker-compose.yml
mongodb:
build:
context: mongodb
args:
DOCKER_ARTIFACTORY: ${DOCKER_ARTIFACTORY}
container_name: “mongodb”
image: mongo:6.0.4
restart: always
environment:
- MONGODB_USER=${SPRING_DATASOURCE_USERNAME:-username}
- MONGODB_PASSWORD=${SPRING_DATASOURCE_PASSWORD:-password}
ports:
- “27017:27017”
volumes:
- “/mongodata:/data/mongodb”
networks:
- somenetwork
Dockerfile
ARG DOCKER_ARTIFACTORY
FROM ${DOCKER_ARTIFACTORY}mongo:6.0.4
COPY init/mongodbsetup.sh /docker-entrypoint-initdb.d/
RUN chmod +x /docker-entrypoint-initdb.d/mongodbsetup.sh
CMD [“mongod”]
version: '3'
services:
db:
image: postgres
environment:
POSTGRES_PASSWORD: password
mongodb:
image: mongo
ports:
- "27017:27017"
environment:
MONGO_INITDB_ROOT_USERNAME: root
MONGO_INITDB_ROOT_PASSWORD: password
You can find above configuration of docker file that includes both MongoDB and PostgreSQL configurations.
I resolve the issue and successfully able to connect and able to perform CRUD operations too.
Below are the changes I did.
1. docker-compose.yml
mongodb:
build:
context: mongodb
args:
DOCKER_ARTIFACTORY: ${DOCKER_ARTIFACTORY}
container_name: mongodb
hostname: mongodb
restart: always
environment:
- MONGO_INITDB_DATABASE=databaseName
- MONGO_INITDB_ROOT_USERNAME=username
- MONGO_INITDB_ROOT_PASSWORD=password
ports:
- 27017:27017
volumes:
- /mongodata:/data/mongodb
networks:
- somenetwork
created below file (filename: mongodbsetup.sh) under projectFolder/builder/mongodb/init
#!/bin/bash
mongosh <<EOF
use code
EOF
created below file (filename: Dockerfile) under projectFolder/builder/mongodb
ARG DOCKER_ARTIFACTORY
FROM ${DOCKER_ARTIFACTORY}mongo:6.0.4
COPY init/mongodbsetup.sh /docker-entrypoint-initdb.d/
RUN chmod +x /docker-entrypoint-initdb.d/mongodbsetup.sh
CMD ["mongod"]
In application.properties file added below properties
#MongoDB configurations
spring.data.mongodb.database=databasename
spring.data.mongodb.uri=mongodb://username:password#databaseurlOrIpAddress:27017
That's it. It's working.
I'm working on a Dockerized project which has an Adminer container. I need to increase the value of post_max_size found in /usr/local/etc/php/conf.d/0-upload_large_dumps.ini.
My problem is any attempt to edit the file results in permission denied responses. Usually, this would be a prolem resolved by using sudo but I also get permission denied from that as well.
The following is the output of the directory I'm trying to edit showing the target file is owned by root:
/var/www/html $ cd /usr/local/etc/php/conf.d/
/usr/local/etc/php/conf.d $ ls -l
total 24
-rw-r--r-- 1 root root 113 Nov 18 22:10 0-upload_large_dumps.ini
-rw-r--r-- 1 root root 23 Nov 18 22:11 docker-php-ext-pdo_dblib.ini
-rw-r--r-- 1 root root 23 Nov 18 22:10 docker-php-ext-pdo_mysql.ini
-rw-r--r-- 1 root root 22 Nov 18 22:11 docker-php-ext-pdo_odbc.ini
-rw-r--r-- 1 root root 23 Nov 18 22:11 docker-php-ext-pdo_pgsql.ini
-rw-r--r-- 1 root root 17 Nov 18 17:03 docker-php-ext-sodium.ini
And the adminer section of docker-compose is as follows:
adminer:
image: adminer
restart: always
labels:
- traefik.port=8080
- traefik.frontend.rule=Host:adminer
How can I edit docker-compose so I have permissions to update the files?
There is nothing to change in your docker-compose.yaml.
If you want to change it, you can just exec in the container as the root user.
So I suppose that, right now, you are doing
docker compose exec adminer ash
And then, you are trying to edit those file.
What you can do, instead, is:
docker compose exec --user root adminer ash
So you will be able to adapt those files owned by root.
This said, mind that the philosophy of Docker is that a container should be short lived, so you would be better having your own Dockerfile to edit that configuration file for good. Another way to do it would be to mount a file over the existing one to change the configurations.
Example of adaptation in a Dockerfile:
FROM adminer
COPY ./0-upload_large_dumps.ini \
/usr/local/etc/php/conf.d/0-upload_large_dumps.ini
## ^-- copies a files from the folder where you build
## in order to override the existing configuration
Then in your docker-compose.yml:
adminer:
build: .
image: your-own-namespace/adminer
restart: always
labels:
- traefik.port=8080
- traefik.frontend.rule=Host:adminer
Example of mounting a file to override the configuration file:
adminer:
image: adminer
restart: always
labels:
- traefik.port=8080
- traefik.frontend.rule=Host:adminer
volumes:
- "./0-upload_large_dumps.ini:\
/usr/local/etc/php/conf.d/0-upload_large_dumps.ini"
## a local file `0-upload_large_dumps.ini` on your host
## will override the container ini file
I have a Postgres database running on my host. The datafiles for the database is stored at /usr/local/var/postgresql#13.
To get the full system running easily I'd like to have a Docker with a Postgres service running for other Docker apps to connect to. I would however like to have the Docker Postgres service to use the existing datafiles on the host ...
How do I set up the volume correctly to point to the hosts database files?
Do have to have a user/password when running the Docker against existing datafiles?
I have the following but can get the volume to work ...
version: "3.9"
services:
web:
build: .
ports:
- 8081:3011
depends_on:
- db
environment:
- PGHOST=db
- PGDATABASE=loggingtestdb
- PGUSER=postgres
- PGPASSWORD=postgres
db:
image: postgres
ports:
- 5432:5432
volumes:
- /usr/local/var/postgresql#13 <--- Need help here.
How do I map the container pg datafile location to the hosts pg datafile location? 🙏
Update 1
This is the datafile folder for the db on the host
After comments I updated the volumes to below
volumes:
- /usr/local/var/postgresql#13:/var/lib/postgresql/data
But when running docker compose I only get
Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /usr/local/var/postgresql#13
Update 2
/use/local works fine. But as soon as I add the /var folder to the path Docker for some reason can’t find it … What am I missing here?
I'm trying to run a docker image with PostgreSQL that has a volume configured for persisting data.
docker-compose.yml
version: '3.1'
services:
db:
image: postgres
restart: always
volumes:
- ./data:/var/lib/postgresql/data
environment:
POSTGRES_PASSWORD: example
When I start the container I see the output
fixing permissions on existing directory /var/lib/postgresql/data ... ok
and the data folder is no longer readable for me.
If I elevate myself and access the data directory I can see that the files are there. Furthermore, the command ls -ld data gives me
drwx------ 19 systemd-coredump root 4096 May 17 16:22 data
I can manually set the directory permission with sudo chmod 755 data, but that only works until I restart the container.
Why does this happen, and how can I fix it?
The other answer indeed points to the root cause of the problem, however the help page it points to does not contain a solution. Here is what I came up with to make this work for me:
start the container using your normal docker-compose file, this creates the directory with the hardcoded uid:gid (999:999)
version: '3.7'
services:
db:
image: postgres
container_name: postgres
volumes:
- ./data:/var/lib/postgresql/data
environment:
POSTGRES_USER: fake_database_user
POSTGRES_PASSWORD: fake_database_PASSWORD
stop the container and manually change the ownership to uid:gid you want (I'll use 1000:1000 for this example
$ docker stop postgres
$ sudo chown -R 1000:1000 ./data
Edit your docker file to add your desired uid:gid and start it up again using docker-compose (notice the user:)
version: '3.7'
services:
db:
image: postgres
container_name: postgres
volumes:
- ./data:/var/lib/postgresql/data
user: 1000:1000
environment:
POSTGRES_USER: fake_database_user
POSTGRES_PASSWORD: fake_database_password
The reason you can't just use user: from the start is that if the image runs as a different user it fails to create the data files.
On the image documentation page, it does mention a solution to add a volume to expose the /etc/passwd file as read-only in the image when providing --user option, however, that did not work for me with the latest image, as I was getting the following error. In fact none of the three proposed solutions worked for me.
initdb: error: could not change permissions of directory "/var/lib/postgresql/data": Operation not permitted
This is because of what is written in the dockerfile of the postgres image.
From line 15 to 18, you'll see that the group 999 and the user 999 are used. I'm guessing that in your host, they map respectively to systemd-coredump and root.
You need to know that whenever you use a user/group in an image, if the uid/gid exist in your host, then it will be mapped to it.
You can read the documentation on the docker hub from the postgres image here. There is a section Arbitrary --user Notes that explain how it works in the context of this image.
An easier and permanent solution would be as follows:
Add these lines to ~/.bashrc:
export UID=$(id -u)
export GID=$(id -g)
Reload your shell:
$ source ~/.bashrc
Modify your docker-compose.yml as follows:
version: "3.7"
services:
db:
image: postgres
volumes:
- ./tmp/db:/var/lib/postgresql/data
user: "${UID}:${GID}"
...
Source
here's what i did:
services:
postgres:
image: postgres:15.1
restart: always
environment:
- POSTGRES_USER=my_user
- POSTGRES_PASSWORD=my_user
- POSTGRES_DB=my_user
user: root
ports:
- "5432:5432"
volumes:
- /home/my_user/volumes/postgres/data:/var/lib/postgresql/data
- /home/my_user/volumes/postgres/config:/etc/postgresql
postgres_setup:
image: postgres:15.1
user: root
volumes:
- /home/my_user/volumes/postgres/data:/var/lib/postgresql/data
- /home/my_user/volumes/postgres/config:/etc/postgresql
entrypoint: [ "bash", "-c", "chmod 750 -R /var/lib/postgresql/data && chmod 750 -R /etc/postgresql"]
depends_on:
- postgres
pgadmin4:
image: dpage/pgadmin4
restart: always
environment:
- PGADMIN_DEFAULT_EMAIL=my_user#admin.com
- PGADMIN_DEFAULT_PASSWORD=my_user
- PGADMIN_LISTEN_ADDRESS=0.0.0.0
user: root
ports:
- "5050:80"
volumes:
- /home/my_user/volumes/pgadmin/data:/var/lib/pgadmin
depends_on:
- postgres_setup
the postgres_setup container just changes permissions and then shuts down
I have been struggling with a similar issue and the answer hit me when trying to work around postgres (static uid per container, configured or 70 by default on alpine, 999 on standard image), and docker limitations (no uid translation of volumes).
The answer is to utilize Linux ACL without any changes to docker-compose.yml user - just keep the default internal container user id.
mkdir -p ./data
sudo setfacl -m u:$(id -u):rwx -R ./data/
docker-compose up -d
or
docker-compose up -d
sudo setfacl -m u:$(id -u):rwx -R ./data/
The order of creating data volume's directory does not matter and as long as ACL is set after it was created, you as a user will be able to access it recursively. You can of course add additional permissions.
To check who has access to data folder simply run:
getfacl ./data
My docker-compose:
version: "2"
services:
api:
build: .
ports:
- "3007:3007"
links:
- mongo
volumes:
- .:/opt/app
mongo:
image: mongo
volumes:
- /data/db:/data/db
ports:
- "27017:27017"
I get permissionerror:
mongo_1 | chown: changing ownership of '/data/db/diagnostic.data/metrics.2017-06-27T13-32-30Z-00000': Operation not permitted
mongo_1 | chown: changing ownership of '/data/db/journal/WiredTigerLog.0000000054': Operation not permitted
mongo_1 | chown: changing ownership of '/data/db/journal/WiredTigerPreplog.0000000001': Operation not permitted
mongo_1 | chown: changing ownership of '/data/db/journal/WiredTigerPreplog.0000000002': Operation not permitted
mongo_1 | chown: changing ownership of '/data/db/WiredTiger.turtle': Operation not permitted
mongo_1 | chown: changing ownership of '/data/db/WiredTigerLAS.wt': Operation not permitted
ls-la on data:
ls -la data
total 0
drwxrwxrwx 3 root wheel 102 Dec 1 2016 .
drwxr-xr-x 35 root wheel 1258 Jun 25 04:29 ..
drwxrwxrwx# 118 root wheel 4012 Jun 27 15:33 db
If I manually change the permission of /data/db, it will be changed back.
What is the problem here? There's no problem if I run mongo locally.
I had this issue in CentOS and the solution was to turn on SELinux:
setenforce 0
It's not a mongo problem. It's a docker problem actually. when docker wants to map the volume it tries to change the permission and failed do to the user/group/selinux restrictions.
UPDATE:
There's a chown command in entrypoint.sh that tries to change the permission of directories and files in the mapped volume. read more in here.
Only the root or members of the sudo group can change the ownership of a file/directory. When you run mongodb in docker and attach a volume from the host, mongo is trying to run as the mongod user. Since this user doesn't exist on your host and root owns the volume mongod/docker is trying to own the OS looks at this as a permissions problem and you will see that error. You have a few options:
Configure mongo to run as root via editing the mongo config and copying it during the docker build process. This assumes you're using a docker file to build that image. Then it will have no problem accessing the attached volume.
Create a mongod user & group on the host and change the ownership of the data directory to that user that the OS sees no difference in ownership/permissions.
Rearchitect your system so mongo can use the default container data store size for its life and completely forgo the volume mount.
instead of mount with the directory, you can change to mount with the volume.
version: "2"
services:
api:
build: .
ports:
- "3007:3007"
links:
- mongo
volumes:
- .:/opt/app
mongo:
image: mongo
volumes:
- mongodata:/data/db
ports:
- "27017:27017"
volumes:
mongodata
This is the issue in mongo: https://github.com/docker-library/mongo/issues/232#issuecomment-355423692