i have this redis service in my docker compose file
redis:
container_name: redis
image: redis:${VERSION_REDIS}
ports:
- "6379:6379"
networks:
- backend
if i run docker-compose locally, than the redis db is working as intended.
1:M 12 Jun 2022 14:47:51.212 * Running mode=standalone, port=6379.
1:M 12 Jun 2022 14:47:51.213 # Server initialized
1:M 12 Jun 2022 14:47:51.215 * Loading RDB produced by version 6.2.5
1:M 12 Jun 2022 14:47:51.216 * RDB age 728 seconds
1:M 12 Jun 2022 14:47:51.219 * RDB memory usage when created 0.79 Mb
1:M 12 Jun 2022 14:47:51.219 * DB loaded from disk: 0.004 seconds
1:M 12 Jun 2022 14:47:51.220 * Ready to accept connections
The version used is 6.2.5-buster. If i try to use docker-compose to my AWS ECS cluster, every service starts normal but the redis task has the following protocol output:
12.6.2022, 16:32:34 1:M 12 Jun 2022 14:32:34.734 # Failed opening the RDB file crontab (in server root dir /etc) for saving: Permission denied
12.6.2022, 16:32:34 1:M 12 Jun 2022 14:32:34.164 # Failed opening the RDB file zzh (in server root dir /etc) for saving: Permission denied
12.6.2022, 16:32:32 1:M 12 Jun 2022 14:32:32.836 # Failed opening the RDB file root (in server root dir /etc) for saving: Permission denied
12.6.2022, 16:32:32 1:M 12 Jun 2022 14:32:32.646 # Failed opening the RDB file root (in server root dir /etc) for saving: Permission denied
12.6.2022, 16:32:32 1:M 12 Jun 2022 14:32:32.267 # Failed opening the RDB file root (in server root dir /etc) for saving: Permission denied
12.6.2022, 16:32:30 1:M 12 Jun 2022 14:32:30.939 # Failed opening the RDB file backup.db (in server root dir /etc) for saving: Permission denied
12.6.2022, 16:40:47 1:M 12 Jun 2022 14:40:47.168 # Background saving error
i tried to search for solutions, but i am just starting out with container services on AWS. How can i configure my docker-compose file or how to change the permissions inside aws to make redis work? Are there other ways? Thank you. The cluster is running on fargate.
Related
We have installed postgres v12 on ubuntu 20.04 (with apt install -y postgresql postgresql-contrib) and wish to enable archiving to /data/db/postgres/archive by setting the following in postgresql.conf:
max_wal_senders=2
wal_keep_segments=256
wal_sender_timeout=60s
archive_mode=on
archive_command=cp %p /data/db/postgres/archive/%f
However the postgres service fails to write there:
2022-11-15 15:02:26.212 CET [392860] FATAL: archive command failed with exit code 126
2022-11-15 15:02:26.212 CET [392860] DETAIL: The failed archive command was: archive_command=cp pg_wal/000000010000000000000002 /data/db/postgres/archive/000000010000000000000002
2022-11-15 15:02:26.213 CET [392605] LOG: archiver process (PID 392860) exited with exit code 1
sh: 1: pg_wal/000000010000000000000002: Permission denied
This directory /data/db/postgres/archive/ is owned by the postgres user and when we su postgres we are able to create and delete files without a problem.
Why can the postgresql service (running as postgres) not write to a directory it owns?
Here are the permissions on all the parents of the archive directory:
drwxr-xr-x 2 postgres root 6 Nov 15 14:59 /data/db/postgres/archive
drwxr-xr-x 3 root root 21 Nov 15 14:29 /data/db/postgres
drwxr-xr-x 3 root root 22 Nov 15 14:29 /data/db
drwxr-xr-x 5 root root 44 Nov 15 14:29 /data
2022-11-15 15:02:26.212 CET [392860] DETAIL: The failed archive command was: archive_command=cp pg_wal/000000010000000000000002 /data/db/postgres/archive/000000010000000000000002
So, your archive_command is apparently set to the peculiar string archive_command=cp %p /data/db/postgres/archive/%f.
After the %variables are substituted, the result is passed to the shell. The shell does what it was told, which is to set the (unused) environment variable 'archive_command' to be 'cp', and then tries to execute the file pg_wal/000000010000000000000002, which is not allowed to because it doesn't have the execute bit set.
I don't know how you managed to get such a deformed archive_command, but it didn't come from anything you showed us.
My goal is to run a docker-compose cluster on a VM from Google Cloud Platform. I have successfully installed docker and docker-compose:
$ uname -a
Linux instance-6 4.15.0-1083-gcp #94~16.04.1-Ubuntu SMP Sat Sep 5 22:53:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ docker -v
Docker version 19.03.13, build 4484c46d9d
$ docker-compose -v
docker-compose version 1.27.3, build 4092ae5d
I am following the basic tutorial to create a docker-compose cluster using: https://docs.docker.com/compose/gettingstarted/ (Steps #1-#4).
My app.py file is:
import time
import redis
from flask import Flask
app = Flask(__name__)
cache = redis.Redis(host='redis', port=6379)
def get_hit_count():
retries = 5
while True:
try:
return cache.incr('hits')
except redis.exceptions.ConnectionError as exc:
if retries == 0:
raise exc
retries -= 1
time.sleep(0.5)
#app.route('/')
def hello():
count = get_hit_count()
return 'Hello World! I have been seen {} times.\n'.format(count)
My requirements.txt file is:
flask
redis
My Dockerfile is:
FROM python:3.7-alpine
WORKDIR /code
ENV FLASK_APP app.py
ENV FLASK_RUN_HOST 0.0.0.0
RUN apk add --no-cache gcc musl-dev linux-headers
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
EXPOSE 5000
COPY . .
CMD ["flask", "run"]
And, my docker-compose.yml is:
version: '3'
services:
web:
build: .
ports:
- "5000:5000"
redis:
image: "redis:alpine"
Running docker-compose up gives me the correct output. One of the outputs points to where the web_1 is running.
$ docker-compose up
...
web_1 | * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
...
After pressing the link http://0.0.0.0:5000/, GCP doesn't connect. It tries to go to the URL: https://ssh.cloud.google.com/devshell/proxy?authuser=2&devshellProxyPath=%2F&port=5000&environment_name&environment_id, but then it gives the error: 500. That’s an error. There was an error. Please try again later. That’s all we know.
Going to the external IP address and putting in port 5000 also doesn't return anything. (http://IPAddress:500)
I checked the ports:
$ sudo docker-compose ps
Name Command State Ports
composetest_redis_1 docker-entrypoint.sh redis ... Up 6379/tcp
composetest_web_1 flask run Up 0.0.0.0:5000->5000/tcp
I'm not sure what the reason it. I'm guessing it is the firewall configuration from GCP. Everything is just the default settings. I also allowed HTTP and HTTPS requests in the Compute Engine VM Instance settings. Would really appreciate more guidance on what to do. Thanks in advance!
See below for full output:
$ sudo docker-compose up
Starting composetest_redis_1 ... done
Starting composetest_web_1 ... done
Attaching to composetest_redis_1, composetest_web_1
redis_1 | 1:C 23 Sep 2020 21:40:27.816 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1 | 1:C 23 Sep 2020 21:40:27.816 # Redis version=6.0.8, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1 | 1:C 23 Sep 2020 21:40:27.816 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis_1 | 1:M 23 Sep 2020 21:40:27.818 * Running mode=standalone, port=6379.
redis_1 | 1:M 23 Sep 2020 21:40:27.818 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1 | 1:M 23 Sep 2020 21:40:27.818 # Server initialized
redis_1 | 1:M 23 Sep 2020 21:40:27.818 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1 | 1:M 23 Sep 2020 21:40:27.819 * Loading RDB produced by version 6.0.8
redis_1 | 1:M 23 Sep 2020 21:40:27.819 * RDB age 27 seconds
redis_1 | 1:M 23 Sep 2020 21:40:27.819 * RDB memory usage when created 0.77 Mb
redis_1 | 1:M 23 Sep 2020 21:40:27.819 * DB loaded from disk: 0.000 seconds
redis_1 | 1:M 23 Sep 2020 21:40:27.819 * Ready to accept connections
web_1 | * Serving Flask app "app.py"
web_1 | * Environment: production
web_1 | WARNING: This is a development server. Do not use it in a production deployment.
web_1 | Use a production WSGI server instead.
web_1 | * Debug mode: off
web_1 | * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
Need to properly allow traffic through that port by:
Creating a firewall rule with a tag
Including that tag in the VM's Network settings
Link here: Network Tags
OS: RHEL 8.2
I am trying to create a systemctl service for zookeeper. It fails to access the datadir.
Here is my config for zookeeper,
dataDir=/opt/zookeeper
maxClientCnxns=20
tickTime=2000
dataDir=/var/zookeeper/
initLimit=20
syncLimit=10
server.0=master:2888:3888
clientPort=2181
admin.serverPort=8082
Permission of /opt/zookeeper is set to 777.
[user1#server1 opt]$ ls -lart
total 0
dr-xr-xr-x. 17 root root 244 Jul 3 10:56 ..
drwxr-xr-x 3 root root 27 Jul 10 10:29 rh
drw-r--r-- 2 user2 user2 6 Jul 17 08:48 hsluw_data
drw-r--r-- 2 user2 user2 6 Jul 17 08:58 hsluw_config
drwxr-xr-x. 6 root root 71 Jul 17 08:58 .
drwxrwxrwx 3 user2 user2 23 Jul 17 09:40 zookeeper
If I run the command,
./bin/zookeeper-server-start.sh config/zookeeper.properties
it gives me an error message: Unable to access datadir
[2020-07-30 10:25:50,767] ERROR Invalid configuration, only one server specified (ignoring) (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-07-30 10:25:50,767] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain)
[2020-07-30 10:25:50,769] INFO zookeeper.snapshot.trust.empty : false (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2020-07-30 10:25:50,769] ERROR Unable to access datadir, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain)
org.apache.zookeeper.server.persistence.FileTxnSnapLog$DatadirException: Cannot write to data directory /var/zookeeper/version-2
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.<init>(FileTxnSnapLog.java:132)
at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:124)
at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:106)
at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:64)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:128)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:82)
Unable to access datadir, exiting abnormally
However, sudoing the above command works,
sudo ./bin/zookeeper-server-start.sh config/zookeeper.properties
Now I have created a service in /etc/systemd/system/zookeeper.service
I wrote the service in /etc/systemd/system/zookeeper.service in this way,
[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target
[Service]
Type=simple
User=user2
ExecStart=/home/user2/kafka/bin/zookeeper-server-start.sh /home/user2/kafka/config/zookeeper.properties
ExecStop=/home/user2/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal
[Install]
WantedBy=multi-user.target
The SELinux status is disabled.
user2#server1$ sestatus
SELinux status: disabled
Now if I do the following
sudo systemctl daemon-reload
sudo systemctl start zookeeper
sudo systemctl enable zookeeper
I am getting the the same Unable to access the datadir error like the following,
[user2#server1 /]$ systemctl status zookeeper
\u25cf zookeeper.service
Loaded: loaded (/etc/systemd/system/zookeeper.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2020-07-30 10:13:19 CEST; 24s ago
Main PID: 12911 (code=exited, status=3)
Jul 30 10:13:19 server1.localdomain zookeeper-server-start.sh[12911]: org.apache.zookeeper.server.persistence.FileTxnSnapLog$Data>
Jul 30 10:13:19 server1.localdomain zookeeper-server-start.sh[12911]: at org.apache.zookeeper.server.persistence.FileTxnS>
Jul 30 10:13:19 server1.localdomain zookeeper-server-start.sh[12911]: at org.apache.zookeeper.server.ZooKeeperServerMain.>
Jul 30 10:13:19 server1.localdomain zookeeper-server-start.sh[12911]: at org.apache.zookeeper.server.ZooKeeperServerMain.>
Jul 30 10:13:19 server1.localdomain zookeeper-server-start.sh[12911]: at org.apache.zookeeper.server.ZooKeeperServerMain.>
Jul 30 10:13:19 server1.localdomain zookeeper-server-start.sh[12911]: at org.apache.zookeeper.server.quorum.QuorumPeerMai>
Jul 30 10:13:19 server1.localdomain zookeeper-server-start.sh[12911]: at org.apache.zookeeper.server.quorum.QuorumPeerMai>
Jul 30 10:13:19 server1.localdomain zookeeper-server-start.sh[12911]: Unable to access datadir, exiting abnormally
Jul 30 10:13:19 server1.localdomain systemd[1]: zookeeper.service: Main process exited, code=exited, status=3/NOTIMPLEMENTED
Jul 30 10:13:19 server1.localdomain systemd[1]: zookeeper.service: Failed with result 'exit-code'.
What am I missing here?
In the configuration file, this line
dataDir=/var/zookeeper/
appears twice. Removing that line solves the issue.
i'm trying to create a docker image run Mongodb on my Raspberry Pi 3+ with Raspbian buster. But when i build image and run container, mongodb doesn't auto start, so i add command RUN service mongod start then build image. Mongodb seems to be installed, i checked by service --status-all and mongodb listed but not started. Error come when docker try to start mongodb serive: The command '/bin/sh -c service mongod start' return a non-zero code: 1.
This is myDockerfile:
FROM cretzel/rpi-mongodb
WORKDIR /usr/src/mongodb
RUN chmod +x /var/lib/mongodb
RUN service mongod start
EXPOSE 27017
CMD ["mongod"]
I tried to fix this and be told that it doesn't have permission, so i add RUN chmod +x /var/lib/mongodb to my dockerfile but it don't work.
Can someone help?
---------------- Update from #Adiii's answer -----------------------------------
My new dockerfile:
FROM cretzel/rpi-mongodb
WORKDIR /usr/src/mongodb
COPY ./entrypoint.sh .
RUN chmod +x ./entrypoint.sh
VOLUME ./database /data/db
EXPOSE 27017
CMD ["mongod"]
entrypoint.sh
#!/bin/bash
service mongod start
exec "$#"
The change solved my error but mongodb service still cannot start. This is log file:
Starting database: mongodb failed!
db level locking enabled: 1
mongod --help for help and startup options
Mon Nov 11 16:31:15
Mon Nov 11 16:31:15 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
Mon Nov 11 16:31:15
warning: some regex utf8 things will not work. pcre build doesn't have --enable-unicode-properties
Mon Nov 11 16:31:15 [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db/ 32-bit host=38271f34412b
Mon Nov 11 16:31:15 [initandlisten]
Mon Nov 11 16:31:15 [initandlisten] ** NOTE: This is a development version (2.1.1-pre-) of MongoDB.
Mon Nov 11 16:31:15 [initandlisten] ** Not recommended for production.
Mon Nov 11 16:31:15 [initandlisten]
Mon Nov 11 16:31:15 [initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
Mon Nov 11 16:31:15 [initandlisten] ** see http://blog.mongodb.org/post/137788967/32-bit-limitations
Mon Nov 11 16:31:15 [initandlisten] ** with --journal, the limit is lower
Mon Nov 11 16:31:15 [initandlisten]
Mon Nov 11 16:31:15 [initandlisten] db version v2.1.1-pre-, pdfile version 4.5
Mon Nov 11 16:31:15 [initandlisten] git version: 47fbbdceb21fc2b791d22db7f01792500647daa9
Mon Nov 11 16:31:15 [initandlisten] build info: Linux raspberrypi 3.2.27+ #102 PREEMPT Sat Sep 1 01:00:50 BST 2012 armv6l BOOST_LIB_VERSION=1_49
Mon Nov 11 16:31:15 [initandlisten] options: {}
Mon Nov 11 16:31:15 [initandlisten] waiting for connections on port 27017
Mon Nov 11 16:31:15 [websvr] admin web console waiting for connections on port 28017
Mon Nov 11 16:32:15 [clientcursormon] mem (MB) res:20 virt:83 mapped:0
I don't understand Docker so much, it don't show any error. Please help!
You should not start the process at RUN command, each run command run at a separate shell and RUN is for installation and configuration not to start the process. To start the process in the container you need to start the process at entrypoint or CMD.
also, the container needs a process to run in foreground so service will not work in case of the container. so
EXPOSE 27017
CMD ["mongod"]
This is enough to start the mongo process, if the container is up and running it's mean mongod process is running, you do not need to check the status using service --status-all. as soon as MongoDB process dies container will die automatically.
Will suggest to use offical mongo docker image.
I'be grabbed all files from /data/db dir from other sever and now trying to start mongo with it on another one. Here are the files I have:
-rw------- 1 root root 1073741824 Feb 2 11:21 j._0
drwxr-xr-x 2 mongodb nogroup 4096 May 3 12:02 journal
-rw------- 1 root root 67108864 Feb 12 06:43 metadocs-node_db.0
-rw------- 1 root root 134217728 Jun 27 2012 metadocs-node_db.1
-rw------- 1 root root 16777216 Feb 12 06:43 metadocs-node_db.ns
-rwxr-xr-x 1 mongodb nogroup 0 May 3 12:02 mongod.lock
-rw------- 1 root root 1073741824 Aug 28 2012 prealloc.1
-rw------- 1 root root 1073741824 Aug 28 2012 prealloc.2
root#mtg:/var/lib/mongodb#
While starting, I get (from logfile /var/log/mongodb/mongodb.log):
Fri May 3 12:02:01 [initandlisten] options: { bind_ip: "127.0.0.1", config:
"/etc/mongodb.conf", dbpath: "/var/lib/mongodb",$
Fri May 3 12:02:01 [initandlisten] journal dir=/var/lib/mongodb/journal
Fri May 3 12:02:01 [initandlisten] recover : no journal files present, no recovery needed
Fri May 3 12:02:01 [initandlisten] couldn't open /var/lib/mongodb/_metadocs-node_db.ns errno:13 Permission denied
Fri May 3 12:02:01 [initandlisten] error couldn't open file /var/lib/mongodb/_metadocs-node_db.ns terminating
Fri May 3 12:02:01 dbexit:
I'm running
sudo service mongodb start
So, what I'm doing wrong? What files are missing for mongodb to start? Why I'm getting "Permission denied"? Working as root, BTW. No other instances of mongo is running. Thanks!
When starting MongoDB as a service, the packaged start script will start your mongod server under a MongoDB user. Looking at a MongoDB 2.4 init.d script for mongod shows the user as "mongod". When setting up data and log directories it is important that the MongoDB user has read/write access to the directories and files contained within.