This is basically the same service file that the celery docs tells you to use as a basic beginners file.
With the below configuration, journalctl -ex displays the error "Failed at step CHDIR spawning /bin/sh: No such file or directory".
/etc/systemd/system/celery.service
[Unit]
Description=Celery Service
After=network.target
[Service]
Type=forking
User=apache
Group=apache
#Environment=PATH=/opt/python39/lib:/home/ec2-user/DjangoProjects/myproj
#Environment=PATH=/home/ec2-user/DjangoProjects/myproj
EnvironmentFile=/etc/conf.d/celery
#WorkingDirectory=/opt/python39
WorkingDirectory=/home/ec2-usuer/DjangoProjects/myproj
ExecStart=/bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi start $CELERYD_NODES \
--pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \
--loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait $CELERYD_NODES \
--pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \
--loglevel="${CELERYD_LOG_LEVEL}"'
ExecReload=/bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi restart $CELERYD_NODES \
--pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \
--loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS'
Restart=always
[Install]
WantedBy=multi-user.target
/etc/conf.d/celery
# Name of nodes to start
# here we have a single node
#CELERYD_NODES="w1"
# or we could have three nodes:
CELERYD_NODES="w1 w2 w3"
# Absolute or relative path to the 'celery' command:
#CELERY_BIN="/home/ec2-user/.local/bin/celery"
CELERY_BIN="/opt/python39/bin/celery"
#CELERY_BIN="/virtualenvs/def/bin/celery"
CELERYD_CHDIR="/home/ec2-user/DjangoProjects/myproj"
# App instance to use
# comment out this line if you don't use an app
#CELERY_APP="myproj"
CELERY_APP="myproj.celery_tasks"
#CELERY_APP="myproj.celery_tasks:myapp"
# ^^ ??? confusion ??? ^^
# or fully qualified:
#CELERY_APP="proj.tasks:app"
# How to call manage.py
CELERYD_MULTI="multi"
# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"
# - %n will be replaced with the first part of the nodename.
# - %I will be replaced with the current child process index
# and is important when using the prefork pool to avoid race conditions.
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_LOG_LEVEL="INFO"
# you may wish to add these options for Celery Beat
CELERYBEAT_PID_FILE="/var/run/celery/beat.pid"
CELERYBEAT_LOG_FILE="/var/log/celery/beat.log"
export DJANGO_SETTINGS_MODULE="myproj.settings"
If I leave out the WorkingDirectory in the service file, it throws this error: "ModuleNotFoundError: No module named 'myproj'".
I've spent the last 2 days looking at different configurations and what not, and I haven't been able to get past one of these 2 errors. What am I missing?
I was able to solve it!
I found this link that was a tutorial... which said that WorkingDirectory and CELERYD_CHDIR are the same.
I also read something on SO that suggested using a virtual environment... so I did that, too :).
The updated files:
/etc/systemd/system/celery.service
[Unit]
Description=Celery Service
After=network.target
[Service]
Type=forking
User=apache
Group=apache
#Environment=PATH=/opt/python39/lib:/home/ec2-user/DjangoProjects/myproj
#Environment=PATH=/home/ec2-user/DjangoProjects/myproj
EnvironmentFile=/etc/conf.d/celery
#WorkingDirectory=/opt/python39
WorkingDirectory=/home/ec2-user/DjangoProjects/myproj
ExecStart=/bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi start $CELERYD_NODES \
--pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \
--loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS'
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait $CELERYD_NODES \
--pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \
--loglevel="${CELERYD_LOG_LEVEL}"'
ExecReload=/bin/sh -c '${CELERY_BIN} -A $CELERY_APP multi restart $CELERYD_NODES \
--pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \
--loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS'
Restart=always
[Install]
WantedBy=multi-user.target
/etc/conf.d/celery
# Name of nodes to start
# here we have a single node
#CELERYD_NODES="w1"
# or we could have three nodes:
CELERYD_NODES="w1 w2 w3"
# Absolute or relative path to the 'celery' command:
#CELERY_BIN="/home/ec2-user/.local/bin/celery"
#CELERY_BIN="/opt/python39/bin/celery"
CELERY_BIN="/home/ec2-user/.virtualenvs/myproj_prod/bin/celery"
#CELERY_BIN="/virtualenvs/def/bin/celery"
CELERYD_CHDIR="/home/ec2-user/DjangoProjects/myproj"
# App instance to use
# comment out this line if you don't use an app
#CELERY_APP="myproj"
#CELERY_APP="myproj.celery_tasks"
CELERY_APP="myproj.celery_tasks"
#CELERY_APP="myproj.celery_tasks:myapp"
# ^^ ??? confusion ??? ^^
# or fully qualified:
#CELERY_APP="proj.tasks:app"
# How to call manage.py
CELERYD_MULTI="multi"
# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=8"
# - %n will be replaced with the first part of the nodename.
# - %I will be replaced with the current child process index
# and is important when using the prefork pool to avoid race conditions.
CELERYD_PID_FILE="/var/run/celery/%n.pid"
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_LOG_LEVEL="INFO"
# you may wish to add these options for Celery Beat
CELERYBEAT_PID_FILE="/var/run/celery/beat.pid"
CELERYBEAT_LOG_FILE="/var/log/celery/beat.log"
DJANGO_SETTINGS_MODULE="myproj.settings"
After I created /var/run/celery/ and /var/log/celery/ folders, I ran chmod and gave access to the user and group that would be running the service to those folders - apache.
I want to run the cp-schema-registry image on AWS ECS, so I am trying to get it to run on docker locally. I have a command like this:
docker run -e SCHEMA_REGISTRY_HOST_NAME=schema-registry \
-e SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS="1.kafka.address:9092,2.kafka.address:9092,3.kafka.address:9092" \
-e SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL=SASL_PLAINTEXT \
confluentinc/cp-schema-registry:5.5.3
(I have replaced the kafka urls).
My consumers/producers connect to the cluster with params:
["sasl.mechanism"] = "PLAIN"
["sasl.username"] = <username>
["sasl.password"] = <password>
Docs seem to indicate there is a file I can create with these parameters, but I don't know how to pass this into the docker run command. Can this be done?
Thanks to OneCricketeer for the help above with the SCHEMA_REGISTRY_KAFKASTORE_SASL_JAAS_CONFIG var. The command ended up like this (I added port 8081:8081 so I could test with curl):
docker run -p 8081:8081 -e SCHEMA_REGISTRY_HOST_NAME=schema-registry \
-e SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS="1.kafka.broker:9092,2.kafka.broker:9092,3.kafka.broker:9092" \
-e SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL=SASL_SSL \
-e SCHEMA_REGISTRY_KAFKASTORE_SASL_MECHANISM=PLAIN \
-e SCHEMA_REGISTRY_KAFKASTORE_SASL_JAAS_CONFIG='org.apache.kafka.common.security.plain.PlainLoginModule required username="user" password="pass";' confluentinc/cp-schema-registry:5.5.3
Then test with curl localhost:8081/subjects and get [] as a response.
I have a few proxysql (https://proxysql.com/) instances (running in Kubernetes). However, I don't want to hardcode the db credentials in the config file (proxysql.cnf). I was hoping I could use ENV variables but I wasn't able to get that to work. What is the proper way to include secrets in a proxysql instance without hard coding passwords in plain text files?
I was thinking of including the config file as one secret and mount it in Kubernetes (seem over kill or wrong) or run envsubstr via in a startup script or init container.
Thoughts?
What I ended up doing was I ran a sidecar with an init script as a configmap:
#!/bin/sh
echo "Check if mysqld is running..."
while ! nc -z 127.0.0.1 6032; do
sleep 0.1
done
echo "mysql is running!"
echo "Loading Runtime Data..."
echo "INSERT INTO mysql_users(username,password,default_hostgroup) VALUES ('$USERNAME','$PASSWORD',1);" | mysql -u $PROXYSQL_USER -p$PROXYSQL_PASSWORD -h 127.0.0.1 -P6032
echo "LOAD MYSQL USERS TO RUNTIME;" | mysql -u $PROXYSQL_USER -p$PROXYSQL_PASSWORD -h 127.0.0.1 -P6032
echo "Runtime Data loaded."
while true; do sleep 300; done;
Seem to work nicely.
I'm using the official Postgres Docker image, trying to customize its configuration. For this purpose, I use the command sed to change max_connections for example:
sed -i -e"s/^max_connections = 100.*$/max_connections = 1000/" /var/lib/postgresql/data/postgresql.conf
I tried two methods to apply this configuration:
The first is by adding the commands to a script and copying it within the init folder: /docker-entrypoint-initdb.d.
The second method is by running the commands directly within my Dockerfile with the "RUN" command (this method worked fine with a non-official PostgreSQL image with a different path to the configuration file /etc/postgres/...).
In both cases the changes fail because the configuration file is missing (I think it's not created yet).
How should I change the configuration?
Here is the Dockerfile used to create the image:
# Database (http://www.cs3c.ma/)
FROM postgres:9.4
MAINTAINER Sabbane <contact#cs3c.ma>
ENV TERM=xterm
RUN apt-get update
RUN apt-get install -y nano
ADD scripts /scripts
# ADD scripts/setup-my-schema.sh /docker-entrypoint-initdb.d/
# Allow connections from anywhere.
RUN sed -i -e"s/^#listen_addresses =.*$/listen_addresses = '*'/" /var/lib/postgresql/data/postgresql.conf
RUN echo "host all all 0.0.0.0/0 md5" >> /var/lib/postgresql/data/pg_hba.conf
# Configure logs
RUN sed -i -e"s/^#logging_collector = off.*$/logging_collector = on/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_directory = 'pg_log'.*$/log_directory = '\/var\/log\/postgresql'/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_filename = 'postgresql-\%Y-\%m-\%d_\%H\%M\%S.log'.*$/log_filename = 'postgresql_\%a.log'/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_file_mode = 0600.*$/log_file_mode = 0644/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_truncate_on_rotation = off.*$/log_truncate_on_rotation = on/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_rotation_age = 1d.*$/log_rotation_age = 1d/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_min_duration_statement = -1.*$/log_min_duration_statement = 0/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_checkpoints = off.*$/log_checkpoints = on/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_connections = off.*$/log_connections = on/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_disconnections = off.*$/log_disconnections = on/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^log_line_prefix = '\%t \[\%p-\%l\] \%q\%u#\%d '.*$/log_line_prefix = '\%t \[\%p\]: \[\%l-1\] user=\%u,db=\%d'/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_lock_waits = off.*$/log_lock_waits = on/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#log_temp_files = -1.*$/log_temp_files = 0/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#statement_timeout = 0.*$/statement_timeout = 1800000 # in milliseconds, 0 is disabled (current 30min)/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^lc_messages = 'en_US.UTF-8'.*$/lc_messages = 'C'/" /var/lib/postgresql/data/postgresql.conf
# Performance Tuning
RUN sed -i -e"s/^max_connections = 100.*$/max_connections = 1000/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^shared_buffers =.*$/shared_buffers = 16GB/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#effective_cache_size = 128MB.*$/effective_cache_size = 48GB/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#work_mem = 1MB.*$/work_mem = 16MB/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#maintenance_work_mem = 16MB.*$/maintenance_work_mem = 2GB/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#checkpoint_segments = .*$/checkpoint_segments = 32/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#checkpoint_completion_target = 0.5.*$/checkpoint_completion_target = 0.7/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#wal_buffers =.*$/wal_buffers = 16MB/" /var/lib/postgresql/data/postgresql.conf
RUN sed -i -e"s/^#default_statistics_target = 100.*$/default_statistics_target = 100/" /var/lib/postgresql/data/postgresql.conf
VOLUME ["/var/lib/postgresql/data", "/var/log/postgresql"]
CMD ["postgres"]
With this Dockerfile, the build process produces an error:
sed: can't read /var/lib/postgresql/data/postgresql.conf: No such file or directory
With Docker Compose
When working with Docker Compose, you can use command: postgres -c option=value in your docker-compose.yml to configure Postgres.
For example, this makes Postgres log to a file:
command: postgres -c logging_collector=on -c log_destination=stderr -c log_directory=/logs
Adapting Vojtech Vitek's answer, you can use
command: postgres -c config_file=/etc/postgresql.conf
to change the config file Postgres will use. You'd mount your custom config file with a volume:
volumes:
- ./customPostgresql.conf:/etc/postgresql.conf
Here's the docker-compose.yml of my application, showing how to configure Postgres:
# Start the app using docker-compose pull && docker-compose up to make sure you have the latest image
version: '2.1'
services:
myApp:
image: registry.gitlab.com/bullbytes/myApp:latest
networks:
- myApp-network
db:
image: postgres:9.6.1
# Make Postgres log to a file.
# More on logging with Postgres: https://www.postgresql.org/docs/current/static/runtime-config-logging.html
command: postgres -c logging_collector=on -c log_destination=stderr -c log_directory=/logs
environment:
# Provide the password via an environment variable. If the variable is unset or empty, use a default password
# Explanation of this shell feature: https://unix.stackexchange.com/questions/122845/using-a-b-for-variable-assignment-in-scripts/122848#122848
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-4WXUms893U6j4GE&Hvk3S*hqcqebFgo!vZi}
# If on a non-Linux OS, make sure you share the drive used here. Go to Docker's settings -> Shared Drives
volumes:
# Persist the data between container invocations
- postgresVolume:/var/lib/postgresql/data
- ./logs:/logs
networks:
myApp-network:
# Our application can communicate with the database using this hostname
aliases:
- postgresForMyApp
networks:
myApp-network:
driver: bridge
# Creates a named volume to persist our data. When on a non-Linux OS, the volume's data will be in the Docker VM
# (e.g., MobyLinuxVM) in /var/lib/docker/volumes/
volumes:
postgresVolume:
Permission to write to the log directory
Note that when on Linux, the log directory on the host must have the right permissions.
Otherwise you'll get the slightly misleading error
FATAL: could not open log file
"/logs/postgresql-2017-02-04_115222.log": Permission denied
I say misleading, since the error message suggests that the directory in the container has the wrong permission, when in reality the directory on the host doesn't permit writing.
To fix this, I set the correct permissions on the host using
chgroup ./logs docker && chmod 770 ./logs
The postgres:9.4 image you've inherited from declares a volume at /var/lib/postgresql/data. This essentially means you can't copy any files to that path in your image; the changes will be discarded.
You have a few choices:
You could just add your own configuration files as a volume at run-time with docker run -v postgresql.conf:/var/lib/postgresql/data/postgresql.conf .... However, I'm not sure exactly how that will interact with the existing volume.
You could copy the file over when the container is started. To do that, copy your file into the build at a location which isn't underneath the volume then call a script from the entrypoint or cmd which will copy the file to the correct location and start Postgres.
Clone the project behind the Postgres official image and edit the Dockerfile to add your own config file in before the VOLUME is declared (anything added before the VOLUME instruction is automatically copied in at run-time).
Pass all config changes in command option in docker-compose file
Like this:
services:
postgres:
...
command:
- "postgres"
- "-c"
- "max_connections=1000"
- "-c"
- "shared_buffers=3GB"
- "-c"
...
When you run the official entrypoint (i.e, when you launch the container), it runs initdb in $PGDATA (/var/lib/postgresql/data by default), and then it stores two files in that directory:
postgresql.conf with default manual settings.
postgresql.auto.conf with settings overriden automatically with ALTER SYSTEM commands.
The entrypoint also executes any /docker-entrypoint-initdb.d/*.{sh,sql} files.
All this means you can supply a shell/SQL script in that folder that configures the server for the next boot (which will be immediately after the DB initialization, or the next time you boot the container).
Example:
conf.sql file:
ALTER SYSTEM SET max_connections = 6;
ALTER SYSTEM RESET shared_buffers;
Dockerfile file:
FROM posgres:9.6-alpine
COPY *.sql /docker-entrypoint-initdb.d/
RUN chmod a+r /docker-entrypoint-initdb.d/*
And then you will have to execute conf.sql manually in the already existing databases. Since configuration is stored in the volume, it will survive rebuilds.
An alternative is to pass the -c option as many times as you wish:
docker container run -d postgres -c max_connections=6 -c log_lock_waits=on
This way, you don't need to build a new image, and you don't need to care about already existing or not databases; all will be affected.
Inject custom postgresql.conf into Postgres Docker container
The default postgresql.conf file lives within the PGDATA dir (/var/lib/postgresql/data), which makes things more complicated especially when running the Postgres container for the first time, since the docker-entrypoint.sh wrapper invokes the initdb step for PGDATA dir initialization.
To customize the PostgreSQL configuration in Docker consistently, I suggest using the config_file Postgres option together with Docker volumes like this:
Production database (PGDATA dir as Persistent Volume)
docker run -d \
-v $CUSTOM_CONFIG:/etc/postgresql.conf \
-v $CUSTOM_DATADIR:/var/lib/postgresql/data \
-e POSTGRES_USER=postgres \
-p 5432:5432 \
--name postgres \
postgres:9.6 postgres -c config_file=/etc/postgresql.conf
Testing database (PGDATA dir will be discarded after docker rm)
docker run -d \
-v $CUSTOM_CONFIG:/etc/postgresql.conf \
-e POSTGRES_USER=postgres \
--name postgres \
postgres:9.6 postgres -c config_file=/etc/postgresql.conf
Debugging
Remove the -d (detach option) from docker run command to see the server logs directly.
Connect to the Postgres server with the psql client and query the configuration:
docker run -it --rm --link postgres:postgres postgres:9.6 sh -c 'exec psql -h $POSTGRES_PORT_5432_TCP_ADDR -p $POSTGRES_PORT_5432_TCP_PORT -U postgres'
psql (9.6.0)
Type "help" for help.
postgres=# SHOW all;
You can put your custom postgresql.conf in a temporary file inside the container, and overwrite the default configuration at runtime.
To do that:
Copy your custom postgresql.conf inside your container
Copy the updateConfig.sh file in /docker-entrypoint-initdb.d/
Dockerfile
FROM postgres:9.6
COPY postgresql.conf /tmp/postgresql.conf
COPY updateConfig.sh /docker-entrypoint-initdb.d/_updateConfig.sh
updateConfig.sh
#!/usr/bin/env bash
cat /tmp/postgresql.conf > /var/lib/postgresql/data/postgresql.conf
At runtime, the container will execute the script inside /docker-entrypoint-initdb.d/ and overwrite the default configuration with your custom one.
I looked through all the answers and there is another option left: You can change your CMD value in the Dockerfile (it is not the best one, but still a possible way to achieve your goal).
Basically we need to:
Copy the config file into the Docker container
Override Postgres start options
Dockerfile example:
FROM postgres:9.6
USER postgres
# Copy Postgres config file into container
COPY postgresql.conf /etc/postgresql
# Override default Postgres config file
CMD ["postgres", "-c", "config_file=/etc/postgresql/postgresql.conf"]
Though I think using command: postgres -c config_file=/etc/postgresql/postgresql.conf in your docker-compose.yml file as proposed by Matthias Braun is the best option.
I was also using the official image (FROM postgres)
and I was able to change the config by executing the following commands.
The first thing is to locate the PostgreSQL config file.
This can be done by executing this command in your running database.
SHOW config_file;
I my case it returns /data/postgres/postgresql.conf.
The next step is to find out what is the hash of your running PostgreSQL docker container.
docker ps -a
This should return a list of all the running containers. In my case it looks like this.
...
0ba35e5427d9 postgres "docker-entrypoint.s…" ....
...
Now you have to switch to the bash inside your container by executing:
docker exec -it 0ba35e5427d9 /bin/bash
Inside the container check if the config is at the correct path and display it.
cat /data/postgres/postgresql.conf
I wanted to change the max connections from 100 to 1000 and the shared buffer from 128MB to 3GB.
With the sed command I can do a search and replace with the corresponding variables ins the config.
sed -i -e"s/^max_connections = 100.*$/max_connections = 1000/" /data/postgres/postgresql.conf
sed -i -e"s/^shared_buffers = 128MB.*$/shared_buffers = 3GB/" /data/postgres/postgresql.conf
The last thing we have to do is to restart the database within the container.
Find out which version you of PostGres you are using.
cd /usr/lib/postgresql/
ls
In my case its 12
So you can now restart the database by executing the following command with the correct version in place.
su - postgres -c "PGDATA=$PGDATA /usr/lib/postgresql/12/bin/pg_ctl -w restart"
A fairly low-tech solution to this problem seems to be to declare the service (I'm using swarm on AWS and a yaml file) with your database files mounted to a persisted volume (here AWS EFS as denoted by the cloudstor:aws driver specification).
version: '3.3'
services:
database:
image: postgres:latest
volumes:
- postgresql:/var/lib/postgresql
- postgresql_data:/var/lib/postgresql/data
volumes:
postgresql:
driver: "cloudstor:aws"
postgresql_data:
driver: "cloudstor:aws"
The db comes up as initialized with the image default settings.
You edit the conf settings inside the container, e.g if you want to increase the maximum number of concurrent connections that requires a restart
stop the running container (or scale the service down to zero and then back to one)
the swarm spawns a new container, which this time around picks up your persisted configuration settings and merrily applies them.
A pleasant side-effect of persisting your configuration is that it also persists your databases (or was it the other way around) ;-)
My solution is for colleagues who needs to make changes in config before launching docker-entrypoint-initdb.d
I was needed to change 'shared_preload_libraries' setting so during it's work postgres already has new library preloaded and code in docker-entrypoint-initdb.d can use it.
So I just patched postgresql.conf.sample file in Dockerfile:
RUN echo "shared_preload_libraries='citus,pg_cron'" >> /usr/share/postgresql/postgresql.conf.sample
RUN echo "cron.database_name='newbie'" >> /usr/share/postgresql/postgresql.conf.sample
And with this patch it become possible to add extension in .sql file in docker-entrypoint-initdb.d/:
CREATE EXTENSION pg_cron;
Using docker compose you can mount a volume with postgresql.auto.conf.
Example:
version: '2'
services:
db:
image: postgres:10.9-alpine
volumes:
- postgres:/var/lib/postgresql/data:z
- ./docker/postgres/postgresql.auto.conf:/var/lib/postgresql/data/postgresql.auto.conf
ports:
- 5432:5432
I have this my project folder structure
api
-- __init__.py
--jobs/
-- __init__.py
-- celery.py
-- celeyconfig.py
-- tasks.py
--api_helpers/
--views/
tasks has a task called ExamineColumns
I launch the worker using celery worker -A api.jobs --loglevel=Info
It works fine and I can run the tasks.
This is the ourput of celery examine command
$ celery inspect registered
-> ranjith-ThinkPad-T420: OK
* ExamineColumns
* celery.backend_cleanup
* celery.chain
* celery.chord
* celery.chord_unlock
* celery.chunks
* celery.group
* celery.map
* celery.starmap
But when I try the multi mode it simply does not work. I am trying to run by running
celery multi start w1 -c3 -A api.jobs --loglevel=Info
But it does not start at all.
$ celery inspect registered
Error: No nodes replied within time constraint.
I am not sure why it is not working
You can try to run as:
/usr/bin/celery multi start w1 w2 --uid=www --loglevel=INFO --pidfile=/var/run/%n.pid --logfile=/var/log/%n.log --quiet
--uid must be user/group from your server. Not recommended use root
--quiet will not output data to console
%n.log will replace itself on w1.log and w2.log
For checking you can use ps uax | grep celery Result will be as so:
www ... /usr/local/bin/python2.7 -m celery.bin.celeryd -n w1.domain.ru --loglevel=DEBUG --logfile=/var/log/w1.log --pidfile=/var/run/w1.pid
www ... /usr/local/bin/python2.7 -m celery.bin.celeryd -n w2.domain.ru --loglevel=DEBUG --logfile=/var/log/w2.log --pidfile=/var/run/w2.pid