I'm unfamiliar with how to use postgres and need some help. I'm currently running OSX Yosemite.
When I start postgres I get this:
pg_ctl: could not start server
Examine the log output.
There was an error executing [start] on postgres. Check /Users/work/git/proj/var/log/postgres.log for details.
createuser: could not connect to database postgres: FATAL: could not open relation mapping file "global/pg_filenode.map": No such file or directory
The log is below.
When I try to stop postgres I get this:
Postgres not running
And when I run ps -ef |grep postgres I get this:
20010 13398 1 0 Jul07 ? 00:00:00 /usr/pgsql-9.3/bin/postgres -h -k /Users/work/git/proj/var/pg
20010 13399 13398 0 Jul07 ? 00:00:09 postgres: logger process
20010 13401 13398 0 Jul07 ? 00:00:10 postgres: checkpointer process
20010 13402 13398 0 Jul07 ? 00:00:00 postgres: writer process
20010 13403 13398 0 Jul07 ? 00:00:00 postgres: wal writer process
20010 13404 13398 0 Jul07 ? 00:00:36 postgres: autovacuum launcher process
20010 13405 13398 0 Jul07 ? 00:00:02 postgres: stats collector process
20010 18112 17723 0 10:22 pts/0 00:00:00 grep postgres
What does this all mean and how could I possibly fix this?
log text
Postgres data dir doesn't exist. Creating
The files belonging to this database system will be owned by user "rose.smith".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /Users/work/git/proj/postgres ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
creating configuration files ... ok
creating template1 database in /Users/work/git/proj/postgres/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
creating system views ... ok
loading system objects' descriptions ... ok
creating collations ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
loading PL/pgSQL server-side language ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
/usr/pgsql-9.3/bin/postgres -D /Users/work/git/proj/postgres
or
/usr/pgsql-9.3/bin/pg_ctl -D /Users/work/git/proj/postgres -l logfile start
waiting for server to start....< 2015-06-04 17:24:57.966 GMT >LOG: redirecting log output to logging collector process
< 2015-06-04 17:24:57.966 GMT >HINT: Future log output will appear in directory "pg_log".
done
server started
waiting for server to shut down.... done
server stopped
waiting for server to start....< 2015-06-04 18:10:18.044 GMT >LOG: redirecting log output to logging collector process
< 2015-06-04 18:10:18.044 GMT >HINT: Future log output will appear in directory "pg_log".
done
server started
"/Users/work/git/proj/var/log/postgres.log" 413L, 20935C
after running /usr/pgsql-9.3/bin/postgres -D /Users/work/git/proj/postgres
< 2015-07-08 14:40:36.331 GMT >FATAL: lock file "postmaster.pid" already exists
< 2015-07-08 14:40:36.331 GMT >HINT: Is another postmaster (PID 18145) running in data directory "/Users/work/git/proj/postgres"?
I can't speak to why this worked after trying these commands just a few minutes ago, but it is now working. Good luck to anyone else with the same problem.
stop postgres
killall postgres
remove postgres database with rm -rf postgres
start postgres
This website was helpful. I think my problem may have been the same as his.
I had deleted ~/Library/Containers/com.heroku.postgres or ~/Application Support/Postgres/ while the Postgres.app was still running. The old version was still running since I deleted the pid file, and it didn't know how to shut it down.
Source: https://github.com/PostgresApp/PostgresApp/issues/96
I faced same issue. I solved the problem with the following commands.
If you install postgresql using HomeBrew...
rm /usr/local/var/postgres/postmaster.pid
pg_ctl -D /usr/local/var/postgres -l /usr/local/var/postgres/server.log start
Hope this helps you!
Related
Closed. This question is not reproducible or was caused by typos. It is not currently accepting answers.
This question was caused by a typo or a problem that can no longer be reproduced. While similar questions may be on-topic here, this one was resolved in a way less likely to help future readers.
Closed 4 months ago.
Improve this question
I am setting up a local Postgres database on Docker with the postgres:14-alpine image, and running database migrations on it with golang-migrate, when I got the following error message after running the migrate tool:
error: pq: role "root" does not exist
I was running the following commands:
$ docker run --name postgres14 -p 5432:5432 -e POSTGRES_USER=root -e POSTGRES_PASSWORD=pass -d postgres:14-alpine
$ docker exec -it postgres14 createdb --user=root --owner=root demodb
$ migrate -path db/migrations -database postgresql://root:pass#localhost:5432/demodb?sslmode=disable --verbose up
These commands can also be viewed in this Makefile, and the full codebase can be found in this repository.
Here are the logs from the Postgres container:
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
Success. You can now start the database server using:
pg_ctl -D /var/lib/postgresql/data -l logfile start
waiting for server to start....2022-10-15 09:56:41.209 UTC [36] LOG: starting PostgreSQL 14.5 on x86_64-pc-linux-musl, compiled by gcc (Alpine 11.2.1_git20220219) 11.2.1 20220219, 64-bit
2022-10-15 09:56:41.211 UTC [36] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-10-15 09:56:41.217 UTC [37] LOG: database system was shut down at 2022-10-15 09:56:41 UTC
2022-10-15 09:56:41.220 UTC [36] LOG: database system is ready to accept connections
done
server started
CREATE DATABASE
/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
waiting for server to shut down...2022-10-15 09:56:41.422 UTC [36] LOG: received fast shutdown request
.2022-10-15 09:56:41.423 UTC [36] LOG: aborting any active transactions
2022-10-15 09:56:41.423 UTC [36] LOG: background worker "logical replication launcher" (PID 43) exited with exit code 1
2022-10-15 09:56:41.424 UTC [38] LOG: shutting down
2022-10-15 09:56:41.434 UTC [36] LOG: database system is shut down
done
server stopped
PostgreSQL init process complete; ready for start up.
What should I do to configure the root role correctly?
The docker image docs specify that POSTGRES_USER environment variable defaults to postgres if not set, try using that instead of root or drop the container and build it again using the correct environment variable
once you are inside the psql shell you can create a user with
CREATE USER username WITH PASSWORD 'your_password';
then to grant the user access on a specific database:
GRANT ALL PRIVILEGES ON DATABASE demodb TO username;
once that is done you can use the user in the connection string in make file
Turns out the Postgres server that was installed and setup on my OS by Hombrew was using the same port, which clashed with the requests made to the containerized database under the same port number.
This issue can be solved by either using a different port number for the containerized database, or by shutting down the database on the OS.
I ran
brew postgresql-upgrade-database
and after a long series of updates the final results are:
==> Migrating and upgrading data...
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for reg* data types in user tables ok
Checking for contrib/isn with bigint-passing mismatch ok
Checking for tables WITH OIDS ok
Checking for invalid "sql_identifier" user columns ok
Creating dump of global objects ok
Creating dump of database schemas
ok
Checking for presence of required libraries fatal
Your installation references loadable libraries that are missing from the
new installation. You can add these libraries to the new installation,
or remove the functions using them from the old installation. A list of
problem libraries is in the file:
loadable_libraries.txt
Failure, exiting
Error: Upgrading postgresql data from 11 to 12 failed!
==> Removing empty postgresql initdb database...
==> Moving postgresql data back from /usr/local/var/postgres.old to /usr/local/var/postgres...
==> Successfully started `postgresql` (label: homebrew.mxcl.postgresql)
Error: Failure while executing; `/usr/local/opt/postgresql/bin/pg_upgrade -r -b /usr/local/Cellar/postgresql#11/11.9/bin -B /usr/local/opt/postgresql/bin -d /usr/local/var/postgres.old -D /usr/local/var/postgres -j 16` exited with 1.
The only notes about this I could find were about pg developers mulling whether to print out the offending databases involved: it was unclear what the fix would be. Any hints ?
Update There is no loadable_libraries.txt in the directory that this was run.
I am creating an answer based on the comment from #LaurenzAlbe. The loadable_libraries.txt was not placed in the current directory from which the command
was run . Instead let's go find where it went:
$find / -name loadable_libraries.txt
/usr/local/var/log/loadable_libraries.txt
So what's in that file?
06:22:24/cidervuong2 $cat /usr/local/var/log/loadable_libraries.txt
could not load library "$libdir/pg_background": ERROR: could not access file "$libdir/pg_background": No such file or directory
Database: bluej
I looked into that error:
ERROR: could not access file "$libdir/pg_background"
and there is no useful information. I went ahead and nuked postgresql again and this time it installed cleanly.
brew uninstall --ignore-dependencies postgres
brew install postgres
brew services start postgresql # actually already started
and pg came up cleanly this time
502 84118 1 0 6:33AM ?? 0:00.03 /usr/local/opt/postgresql/bin/postgres -D /usr/local/var/postgres
502 84120 84118 0 6:33AM ?? 0:00.00 postgres: checkpointer
502 84121 84118 0 6:33AM ?? 0:00.03 postgres: background writer
502 84122 84118 0 6:33AM ?? 0:00.01 postgres: walwriter
502 84123 84118 0 6:33AM ?? 0:00.01 postgres: autovacuum launcher
502 84124 84118 0 6:33AM ?? 0:00.01 postgres: stats collector
502 84125 84118 0 6:33AM ?? 0:00.00 postgres: logical replication launcher
I am in the process of experimenting/tinkering/learning/breaking with Docker. I am currently writing Docker code to create a snapshotted testing environment for my application.
By snapshotted I mean that my database is reset on purpose on every restart, so that I can work with old data at a certain time. What is peculiar in my case is that I want to populate a Postgresql database at build time, not at start time. Postgresql image is ready for populating the db with sql scripts at container start, but it takes hours.
My application is made by a Tomcat 8.5 server running my WAR and a Postgresql database, which is the focus of my question now. I am creating a Gist while I write for full code.
The code I have done
Full code on Gist
I have followed a tutorial on how to build a Docker image of Postgres with a full database, rather than have Postgres populate itself on boot. This because I have a million record database and only a .sql.gz dump that sysop gave me.
So the relevant parts of the Dockerfile are
WORKDIR /opt/setup/
COPY db-setup.sh /opt/setup/
COPY db-pack.sh /opt/setup/
COPY db-run.sh /opt/setup/
RUN ./db-setup.sh
RUN ./db-pack.sh
#VOLUME $PGDATA (Note it is commented out, now)
EXPOSE 5432
The db-setup.sh is run on image build, and picks files from data-scripts.d. Of course I am not allowed to share the contents of the dump, but it's a plain .sql.gz with plenties of OIDs that take a huge amount of time to restore. The db-setup.sh shown in Gist is derived from both the tutorial and the original Postgres image so that it handles correctly the compression (the tutorial only uses plain SQL)
Build succeeds, startup fails
When I build the image, it takes considerable amount of time to load the data, which is what I want
2019-08-07 07:57:04.149 UTC [49] LOG: database system was shut down at 2019-08-07 07:57:03 UTC
2019-08-07 07:57:04.231 UTC [48] LOG: database system is ready to accept connections
done
server started
./db-setup.sh: running methodinv_pcp3.sql.gz
2019-08-07 08:49:52.052 UTC [117] ERROR: canceling autovacuum task
2019-08-07 08:49:52.052 UTC [117] CONTEXT: automatic analyze of table "postgres.public.ftt_interactive_data_492"
2019-08-07 08:49:59.086 UTC [118] ERROR: canceling autovacuum task
2019-08-07 08:49:59.086 UTC [118] CONTEXT: automatic analyze of table "postgres.public.ftt_oper_492"
2019-08-07 08:50:34.086 UTC [118] ERROR: canceling autovacuum task
2019-08-07 08:50:34.086 UTC [118] CONTEXT: automatic analyze of table "postgres.public.ftt_validation_492"
2019-08-07 08:51:11.889 UTC [119] ERROR: canceling autovacuum task
2019-08-07 08:51:11.889 UTC [119] CONTEXT: automatic analyze of table "postgres.public.ftt_oper_492"
2019-08-07 08:54:21.131 UTC [123] ERROR: canceling autovacuum task
2019-08-07 08:54:21.131 UTC [123] CONTEXT: automatic analyze of table "postgres.public.ftt_oper_492"
waiting for server to shut down...2019-08-07 08:54:28.652 UTC [48] LOG: received fast shutdown request
.2019-08-07 08:54:28.797 UTC [48] LOG: aborting any active transactions
2019-08-07 08:54:28.799 UTC [48] LOG: worker process: logical replication launcher (PID 55) exited with exit code 1
2019-08-07 08:54:28.800 UTC [50] LOG: shutting down
..2019-08-07 08:54:31.407 UTC [48] LOG: database system is shut down
done
When I run the image with docker run, startup fails because it can't find Postgres configuration
D:\IdeaProjects\pcp\ftt-containers\ftt-db-method>docker run -p 5432:5432 -l ftt-db-method ftt-db-method:latest
Restoring /var/lib/postgresql/data ...
Done.
Launching command: postgres ...
postgres: could not access the server configuration file "/var/lib/postgresql/data/postgresql.conf": No such file or directory
Originally, my Dockerfile exposed a VOLUME which is now commented out. The above output occurs both when I declare a volume (which is not exactly what I want, I am new to Docker and copied&pasted on first chance) and when I comment the volume out.
Question
What is wrong with the Docker image of Postgres fully loaded with s**tloads of data I am experimenting?
How can I effectively start Postgres with an already full database that will not (necessarily) survive container restarts?
Edit 1
By bash-ing into the container I have found that the data dump created during build time is 10K, so basically empty.
This doesn't solve my problem yet, but answers why Postgres is unable to find its beloved data dir
Edit 2
I was able to bash into a temporary container, in particular between the moment the database is restored and the data lib is packed.
Basically the Dockerfile does
RUN ./db-setup.sh
Which executes the restore of the sql
echo "$0: running $f"; gunzip -c "$f" | "${psql[#]}" > /dev/null 2>&1 ; echo ;;
The output is saved to a temporary container.
Now Dockerfile does
RUN ./db-pack.sh
Which tars /var/lib/postgresql/data into /zdata. I have
2019-08-07 16:43:51.532 UTC [42] LOG: received fast shutdown request
waiting for server to shut down....2019-08-07 16:43:51.676 UTC [42] LOG: aborting any active transactions
2019-08-07 16:43:51.679 UTC [42] LOG: worker process: logical replication launcher (PID 49) exited with exit code 1
2019-08-07 16:43:51.681 UTC [44] LOG: shutting down
...2019-08-07 16:43:54.952 UTC [42] LOG: database system is shut down
done
server stopped
Removing intermediate container 8dbe2a4e776a
---> 263896b905ce
Step 15/19 : RUN ./db-pack.sh
---> Running in 56132ecb90cc
Packing data folder: /var/lib/postgresql/data
Pack & clean finished successfully.
Removing intermediate container 56132ecb90cc
---> 1a7f8d68e8df
Step 16/19 : VOLUME $PGDATA
---> Running in 10d222beed81
Removing intermediate container 10d222beed81
---> e1a9355882d1
So I tagged 263896b905ce (YHMV if you replicate on your pc) into a new image, then executed bash on it. The data dir was empty, the script would have packed nothing
docker tag 263896b905ce examine
docker run -it --entrypoint /bin/bash examine
root#ab963ace16a1:/opt/setup# ls
data-scripts.d db-pack.sh db-run.sh db-setup.sh
root#ab963ace16a1:/opt/setup# cd /zdata/
root#ab963ace16a1:/zdata# ls
root#ab963ace16a1:/zdata# cd /var/lib/postgresql/
root#ab963ace16a1:/var/lib/postgresql# ls
data
root#ab963ace16a1:/var/lib/postgresql# cd data/
root#ab963ace16a1:/var/lib/postgresql/data# ls
root#ab963ace16a1:/var/lib/postgresql/data# ls -lah
total 8.0K
drwxrwxrwx 2 postgres postgres 4.0K Jul 17 23:55 .
drwxr-xr-x 1 postgres postgres 4.0K Jul 17 23:55 ..
root#ab963ace16a1:/var/lib/postgresql/data#
root#ab963ace16a1:/var/lib/postgresql/data# ls^C
root#ab963ace16a1:/var/lib/postgresql/data# exit
exit
Fixed
According to https://stackoverflow.com/a/52762779/471213
"why doesn't VOLUME work?" When you define a VOLUME in the Dockerfile, you can only define the target, not the source of the volume. During the build, you will only get an anonymous volume from this. That anonymous volume will be mounted at every RUN command, prepopulated with the contents of the image, and then discarded at the end of the RUN command. Only changes to the container are saved, not changes to the volume.
So I had basically to run both RUNs at the same time
RUN ./db-setup.sh && ./db-pack.sh
#RUN ./db-pack.sh
I am trying to create a hot_standby server, and I receive the following error after pg_basebackup completes. Notice I use a shell script, replicator.sh, to start the replication. Can anyone give me some insight?
My specs:
Debian Wheezy 7.6
Postgresql 9.3
Database size: ~115GB
Error:
postgres#database-master:/etc/postgresql/9.3/main$ sh replicator.sh
Stopping PostgreSQL
[ ok ] Stopping PostgreSQL 9.3 database server: main.
Cleaning up old cluster directory
Starting base backup as replicator
Password:
113720266/113720266 kB (100%), 1/1 tablespace
NOTICE: WAL archiving is not enabled; you must ensure that all required WAL segments are copied through other means to complete the backup
pg_basebackup: base backup completed
Starting Postgresql
[....] Starting PostgreSQL 9.3 database server: main[....] The PostgreSQL server failed to start.
Please check the log output: 2014-09-11 17:56:33 UTC LOG: database system was interrupted; last
known up at 2014-09-11 16:54:29 UTC 2014-09-11 17:56:33 UTC LOG: creating missing WAL directory
"pg_xlog/archive_status" 2014-09-11 17:56:33 UTC LOG: incomplete startup packet 2014-09-11 17:56:33
UTC LOG: invalid checkpoint record 2014-09-11 17:56:33 UTC FATAL: could not locate required
checkpoint record 2014-09-11 17:56:33 UTC HINT: If you are not restoring from a backup, try
removing the file "/var/lib/p[FAILesql/9.3/main/backup_label". 2014-09-11 17:56:33 UTC LOG: startup
process (PID 21972) exited with exit code 1 2014-09-11 17:56:33 UTC LOG: aborting startup due to
startup process failure ... failed! failed!
Contents of replicator.sh:
#!/bin/bash
echo Stopping PostgreSQL
/etc/init.d/postgresql stop
echo Cleaning up old cluster directory
rm -rf /var/lib/postgresql/9.3/main
echo Starting base backup as replicator
pg_basebackup -h 123.456.789.123 -D /var/lib/postgresql/9.3/main -U replicator -v -P
echo Writing recovery.conf file
sudo -u postgres bash -c "cat > /var/lib/postgresql/9.3/main/recovery.conf <<- _EOF1_
standby_mode = 'on'
primary_conninfo = 'host=123.456.789.123 port=5432 user=replicator password=XXXXX sslmode=require'
trigger_file = '/tmp/postgresql.trigger'
_EOF1_
"
echo Starting Postgresql
/etc/init.d/postgresql start
Thank you,
Jake
My best guess from the above is that the pg_basebackup failed and your shell script doesn't check for error return codes or use set -e to automatically abort after errors, so it just carried on regardless.
It's also possible that you don't have WAL archiving configured, or don't have a restore_command set in the replica. In that case, the transaction logs required to start the base backup will not be available and startup will fail.
I strongly recommend that you:
Use pg_basebackup -X stream so that the required transaction logs get copied along with the backup; and
Use set -e in your shell script, or test for errors with a suitable if ! pg_basebackup .... ; then block.
I'm trying to determine if Postgres 9.3 still has a logger process. It isn't referenced anywhere in the "PostgreSQL 9.3.4 Documentation". And I can't find it in my cluster's process list (see below). Also, does anyone know of a good general overview the memory structures in 9.3?
postgres 21397 1 0 20:51 pts/1 00:00:00 /opt/PostgreSQL/9.3/bin/postgres
postgres 21399 21397 0 20:51 ? 00:00:00 postgres: checkpointer process
postgres 21400 21397 0 20:51 ? 00:00:00 postgres: writer process
postgres 21401 21397 0 20:51 ? 00:00:00 postgres: wal writer process
postgres 21402 21397 0 20:51 ? 00:00:00 postgres: autovacuum launcher process
postgres 21403 21397 0 20:51 ? 00:00:00 postgres: archiver process last was 0001000004000092
postgres 21404 21397 0 20:51 ? 00:00:00 postgres: stats collector process
Thanks
Jim
Postgres has a logging collector process which is controlled through a config parameter,
logging_collector.
So in your postgresql.conf file, you would make sure this is set:
logging_collector = on
The blurb on this param from the postgres doc:
This parameter enables the logging collector, which is a background
process that captures log messages sent to stderr and redirects them
into log files. This approach is often more useful than logging to
syslog, since some types of messages might not appear in syslog
output. (One common example is dynamic-linker failure messages;
another is error messages produced by scripts such as
archive_command.) This parameter can only be set at server start.
It will show up in the process list with the following description:
postgres: logger process
For more info: http://www.postgresql.org/docs/current/static/runtime-config-logging.html
Regarding the memory structures, I'm not sure offhand, but would recommend you post that as a separate question.