Diesel doesnt run migrations: relation does not exist - postgresql

Im develop my first project with rust+diesel and I have a problem: diesel doesnt run migrations, caused by error "relation does not exist", although relations exist.
My code:
main.rs:
pub fn establish_connection() -> PgConnection {
dotenv().ok();
let database_url = env::var("DATABASE_URL").expect("DATABASE_URL must be set");
PgConnection::establish(&database_url)
.unwrap_or_else(|_| panic!("Error connecting to {}", database_url))
}
embed_migrations!("./migrations");
#[actix_rt::main]
async fn main() -> std::io::Result<()> {
let connect = database::establish_connection();
match embedded_migrations::run_with_output(&connect, &mut std::io::stdout()) {
Ok(()) => println!("migrations success"),
Err(e) => panic!("migrations error: {}", e)
}
}
Dockerfile:
FROM rust:1.61 as builder
WORKDIR /build/
COPY ./Cargo.toml .
COPY ./Cargo.lock .
COPY ./src ./src
COPY ./migrations ./migrations
RUN cargo build --release
FROM ubuntu:22.04 AS run
EXPOSE 8080
WORKDIR /run/
COPY --from=builder /build/target/release/to_do .
COPY ./dist ./dist
COPY ./docker-entrypoint.sh .
RUN apt-get update -y
RUN apt-get install -y libpq-dev
# docker-entrypoint.sh - wait 5432 port aviable & run to_do app
CMD ["sh", "docker-entrypoint.sh"]
docker-compose.yml:
version: "3.8"
services:
postgres_db:
image: postgres:14.3
container_name: to_do_postgres
restart: always
ports:
- ${POSTGRES_DB_PORT}:5432
environment:
- POSTGRES_USER=${POSTGRES_DB_USER}
- POSTGRES_DB=${POSTGRES_DB_NAME}
- POSTGRES_PASSWORD=${POSTGRES_DB_PASSWORD}
app:
image: to_do_build
container_name: to_do_app
restart: always
ports:
- ${APP_PORT}:8080
depends_on:
- postgres_db
environment:
- DATABASE_URL=postgres://${POSTGRES_DB_USER}:${POSTGRES_DB_PASSWORD}#postgres_db:${POSTGRES_DB_PORT}/${POSTGRES_DB_NAME}
- JWT_SECRET=${JWT_SECRET}
When im run a service by command:
docker-compose --env-file .conf up -d
.conf:
POSTGRES_DB_USER=to_do_user
POSTGRES_DB_NAME=to_do
POSTGRES_DB_PASSWORD=to_do_password
POSTGRES_DB_PORT=5432
JWT_SECRET=secret
APP_PORT=8080
Service runned, but when looking at the logs:
docker logs to_do_app
I get an error:
// DATABASE_URL = postgres://to_do_user:to_do_password#postgres_db:5432/to_do
Running migration 20220525065207
Executing migration script 20220525065207/up.sql
thread 'main' panicked at 'migrations error: Failed with: relation "to_do" does not exist', src/main.rs:28:19
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Although if we connect to postgres via psql, we see that diesel has created its own system table:
docker exec -it to_do_postgres sh
$ psql -U to_do_user to_do
psql (14.3 (Debian 14.3-1.pgdg110+1))
Type "help" for help.
to_do=# \dt
List of relations
Schema | Name | Type | Owner
--------+----------------------------+-------+------------
public | __diesel_schema_migrations | table | to_do_user
(1 row)
I don't understand what I'm doing wrong.

I was just fighting with this myself and am also using actix-web like you. It finally worked for me though after I realized that I did not have COPY ./migrations ./migrations in my Dockerfile.
Looking at yours, I see you're not copying over diesel.toml:
COPY ./diesel.toml ./diesel.toml
Would that be it?

Related

Invalid compose name even docker-compose.yaml exists

I try to upload my rasa chatbot with okteto via docker. So i has implemented a "Dockerfile", a "docker-compose.yaml" and a "okteto.yaml". The last past weeks the code works fine. Today it wont work anymore because Okteto gives the error: Invalid compose name: must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric characterexit status 1.
I really dont understand what i should change. thanks
docker-compose.yaml:
version: '3.4'
services:
rasa-server:
image: rasa-bot:latest
working_dir: /app
build: "./"
restart: always
volumes:
- ./actions:/app/actions
- ./data:/app/data
command: bash -c "rm -rf .rasa/* && rasa train && rasa run --enable-api --cors \"*\" -p 5006"
ports:
- '5006:5006'
networks:
- all
rasa-actions-server:
image: rasa-bot:latest
working_dir: /app
build: "./"
restart: always
volumes:
- ./actions:/app/actions
command: bash -c "rasa run actions"
ports:
- '5055:5055'
networks:
- all
networks:
all:
driver: bridge
driver_opts:
com.docker.network.enable_ipv6: "true"
Dockerfile:
FROM python:3.7.13 AS BASE
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["./bot.py"]
RUN pip install --no-cache-dir --upgrade pip
RUN pip install rasa==3.3.0
ADD config.yml config.yaml
ADD domain.yml domain.yaml
ADD credentials.yml credentials.yaml
ADD endpoints.yml endpoints.yaml
okteto.yml:
name: stubu4ewi
autocreate: true
image: okteto.dev/rasa-bot:latest
command: bash
volumes:
- /root/.cache/pip
sync:
- .:/app
forward:
- 5006:5006
reverse:
- 9000:9000
Error
Found okteto manifest on /okteto/src/okteto.yml
Unmarshalling manifest...
Okteto manifest unmarshalled successfully
Found okteto compose manifest on docker-compose.yaml
Unmarshalling compose...
x Invalid compose name: must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric characterexit
status 1
Dont have any clue what went wrong. It works fine till yesterday and even when nothings changed okteto gives this error.
Tried to rename the docker-compose.yaml to: docker-compose.yml, okteto-compose.yml
That error is not about the file's name itself but the name of the services defined inside your docker-compose.yaml file.
What command did you run, and what version of the okteto cli are you using? okteto version will give it to you.
If you everr face the Problem: Rename your Repo so that it consists only of lower case alphanumeric characters or '-', and starts and ends with an alphanumeric character.
Seems like Okteto uses the Repository Name to build the Images.

Run docker-compose command without exiting

I am using a postgres image and i need to start ssh service on start.
The problem is that if I run a command in docker-compose file the proccess exits with code 0.
How can I start ssh service but keep postgres serice active too?
DOCKER FILE:
FROM postgres:13
RUN apt update && apt install openssh-server sudo -y
RUN echo 'root:password' | chpasswd
RUN sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin yes/g' /etc/ssh/sshd_config
DOCKER-COMPOSE FILE:
postgres:
container_name: db_postgres
command: sh -c "service ssh start "
image: postgresc
build:
context: ../backend_apollo_server_express
dockerfile: Dockerfile.database
environment:
- "POSTGRES_USER=lims"
- "POSTGRES_PASSWORD=lims"
volumes:
- /home/javier/lims/dockerVolumes/db:/var/lib/postgresql/data
- "/etc/timezone:/etc/timezone:ro"
- "/etc/localtime:/etc/localtime:ro"
ports:
- 5434:5432
You can try to use run postgres after you command
command: sh -c "service ssh start & postgres"
Try
command: sh -c "nohup service ssh start && service postgres start &"
In order to leave the process running in the background. This way the process won't exit

Running cron job inside container with python code and postgres database

I am new into docker, and I have created a container with python + postgres, which runs a python script that collects some data and writes it down on the SQL database. Now, I need to set this job to run each day. And then the nightmare started. I did not manage to create a separate container for this job, so I tried to create a file and copy it into the container via DockerFile (see this one down). I did not manage to run cron as entry-point for the container because then my database was not mounted. So, I create the container, access it, give full permissions to /var/www/html, and create the database table. And then I run cron. No erro, but nothing happens, no log is written on /var/log/cron.log. Here my files:
Dockerfile:
FROM postgres:latest
USER root
RUN apt-get update && apt-get install -y python3 python3-pip
RUN apt-get -y install cron nano
RUN apt-get -y install postgresql-server-dev-10 gcc python3-dev musl-dev
RUN pip3 install psycopg2 \
bs4 \
requests \
pytz
COPY temp-alerts-cron /etc/cron.d/temp-alerts-cron
RUN chmod 0777 /etc/cron.d/temp-alerts-cron
RUN chmod gu+rw /var/run/
RUN chmod gu+s /usr/sbin/cron
RUN touch /var/log/cron.log
RUN chmod 0777 /var/log/cron.log
RUN crontab /etc/cron.d/temp-alerts-cron
USER postgres
EXPOSE 5432
VOLUME ["/etc/postgresql", "/var/log/postgresql", "/var/lib/postgresql"]
The temp-alers-cron:
20 13 * * * root /var/www/html/run.sh >> /var/log/cron.log 2>&1
# Don't remove the empty line at the end of this file. It is required to run the cron job
And the called script:
echo 'inside thingy' >> /var/log/cron.log 2>&1
python3 /var/www/html/nuria_main.py
In case it is needed, here the docker-compose.yml:
services:
postgres:
container_name: 'temp-postgres'
build: # build the image from Dockerfile
context: ${PWD}
volumes: # bind mount volume for Postgres data
- pg-data:/var/lib/postgresql/data
- ./python-app:/var/www/html
restart: unless-stopped
environment:
- POSTGRES_USR=xxadmin
- POSTGRES_DB=tempdb
- POSTGRES_PASSWORD=secret
expose:
- "5432"
networks:
kong:
networks:
kong:
external:
name: kong_net
volumes:
pg-data:
Hope somebody knows what I am doing wrong. I do not get any log or error, so i am lost.
Thanks!

Gitlab CI not able to use pg_prove

I'm struggling to get a Gitlab CI up and running that uses the correct version of postgres (13) and has PGTap installed.
I deploy my project locally using a Dockerfile which uses postgres:13.3-alpine and then installs PGTap too. However, I'm not sure if I can use this Dockerfile to help with my CI issues.
In my gitlab-ci.yml file, I currently have:
variables:
GIT_SUBMODULE_STRATEGY: recursive
pgtap:
only:
refs:
- merge_request
- master
changes:
- ddl/**/*
image: postgres:13.1-alpine
services:
- name: postgres:13.1-alpine
alias: db
variables:
POSTGRES_DB: postgres
POSTGRES_USER: postgres
POSTGRES_PASSWORD: ""
POSTGRES_HOST_AUTH_METHOD: trust
script:
- psql postgres://postgres#db/postgres -c 'create extension pgtap;'
- psql postgres://postgres#db/postgres -f ddl/01.sql
- cd ddl/
- psql postgres://postgres#db/postgres -f 02.sql
- psql postgres://postgres#db/postgres -f 03.sql
- pg_prove -d postgres://postgres#db/postgres --recurse test_*
The above works until it gets to the pg_prove command at the bottom as I get the below error:
pg_prove: command not found
Is there a way I can install pg_prove using the script commands? Or is there a better way to do this?
There is an old issue closed.
To summarize, either you build you own image based on postgres:13.1-alpine installing PGTap or you use a non official image where PGTap is installed 1maa/postgres:13-alpine :
docker run -it 1maa/postgres:13-alpine sh
/ # which pg_prove
/usr/local/bin/pg_prove
Since your step image is alpine based, you can try:
script:
- apk add --no-cache --update build-base make perl perl-dev git openssl-dev
- cpan TAP::Parser::SourceHandler::pgTAP
- psql.. etc
You can probably omit some of the packages...

Docker Compose and Postgres : Name does not resolve

In Docker, I am trying to configure Postgres, get it up and running in another container, and link it to my users service, set up with the following structure:
docker-compose-dev.yml
services/
users/
manage.py
Dockerfile-dev
entrypoint.sh
project/
__init__.py
config.py
db/
create.sql
Dockerfile
docker-compose-dev.yml
version: '3.7'
services:
users:
build:
context: ./services/users
dockerfile: Dockerfile-dev
volumes:
- './services/users:/usr/src/app'
ports:
- 5001:5000
environment:
- FLASK_APP=project/__init__.py
- FLASK_ENV=development
- APP_SETTINGS=project.config.DevelopmentConfig
- DATABASE_URL=postgres://postgres:postgres#users-db:5432/users_dev
- DATABASE_TEST_URL=postgres://postgres:postgres#users-db:5432/users_test
depends_on:
- users-db
users-db:
build:
context: ./services/users/project/db
dockerfile: Dockerfile
ports:
- 5435:5432
environment:
- POSTGRES_USER=postgres
- POSTGRES_PASSWORD=postgres
Dockerfile-dev
# base image
FROM python:3.7.2-alpine
# install dependencies
RUN apk update && \
apk add --virtual build-deps gcc python-dev musl-dev && \
apk add postgresql-dev && \
apk add netcat-openbsd
# set working directory
WORKDIR /usr/src/app
# add and install requirements
COPY ./requirements.txt /usr/src/app/requirements.txt
RUN pip install -r requirements.txt
# add entrypoint.sh
COPY ./entrypoint.sh /usr/src/app/entrypoint.sh
RUN chmod +x /usr/src/app/entrypoint.sh
# add app
COPY . /usr/src/app
# run server
CMD ["/usr/src/app/entrypoint.sh"]
Here I try to extend the official Postgres image by adding a SQL file to the "docker-entrypoint-initdb.d" directory in the container.
Dockerfile
# base image
FROM postgres:11.1-alpine
# run create.sql on init
ADD create.sql /docker-entrypoint-initdb.d
create.sql
CREATE DATABASE users_prod;
CREATE DATABASE users_dev;
CREATE DATABASE users_test;
Since the "users" service is dependent not only on the container being up and running but also the actual Postgres instance being up and healthy, I added an entrypoint.sh file to "users":
entrypoint.sh
#!/bin/sh
echo "Waiting for postgres..."
while ! nc -z users-db 5432; do
sleep 0.1
done
echo "PostgreSQL started"
python manage.py run -h 0.0.0.0
manage.py
from flask.cli import FlaskGroup
from project import app, db
cli = FlaskGroup(app)
#cli.command('recreate_db')
def recreate_db():
db.drop_all()
db.create_all()
db.session.commit()
if __name__ == '__main__':
cli()
config.py
import os
class BaseConfig:
"""Base configuration"""
TESTING = False
SQLALCHEMY_TRACK_MODIFICATIONS = False
class DevelopmentConfig(BaseConfig):
"""Development configuration"""
SQLALCHEMY_DATABASE_URI = os.environ.get('DATABASE_URL')
class TestingConfig(BaseConfig):
"""Testing configuration"""
TESTING = True
SQLALCHEMY_DATABASE_URI = os.environ.get('DATABASE_TEST_URL')
class ProductionConfig(BaseConfig):
"""Production configuration"""
SQLALCHEMY_DATABASE_URI = os.environ.get('DATABASE_URL')
I run and successfully build it with:
docker-compose -f docker-compose-dev.yml up -d --build
but I'm having some network issue. If I run:
docker-compose -f docker-compose-dev.yml logs
I get:
Attaching to dev2_users_1
users_1 | Waiting for postgres...
users_1 | nc: getaddrinfo: Name does not resolve
users_1 | nc: getaddrinfo: Name does not resolve
users_1 | nc: getaddrinfo: Name does not resolve
users_1 | nc: getaddrinfo: Name does not resolve
users_1 | nc: getaddrinfo: Name does not resolve
(...)
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6ca5718c9867 dev2_users "/usr/src/app/entryp…" 2 minutes ago Up 2 minutes 0.0.0.0:5001->5000/tcp dev2_users_1
What am I missing in the code above?
You need to install bind-tools in Dockerfile-dev for Name does not resolve problem:
apk add bind-tools
Test:
/usr/src/app # nslookup users-db
nslookup: can't resolve '(null)': Name does not resolve
Name: users-db
Address 1: 172.20.0.2 stc_users-db_1_f435854a8c88.stc_default
/usr/src/app # apk add bind-tools
(1/4) Installing json-c (0.13.1-r0)
(2/4) Installing libxml2 (2.9.8-r1)
(3/4) Installing bind-libs (9.12.3-r0)
(4/4) Installing bind-tools (9.12.3-r0)
Executing busybox-1.28.4-r3.trigger
OK: 188 MiB in 62 packages
/usr/src/app # nslookup users-db
Server: 127.0.0.11
Address: 127.0.0.11#53
Non-authoritative answer:
Name: users-db
Address: 172.20.0.2