I have set archive_timeout to 300 seconds (5 min). Yet WAL files remain open for a very long time (hours). My postgresql.conf file includes:
archive_mode = on
archive_timeout = 300
archive_command = '/usr/local/internal/pgarchive.sh "%p" "%f"'
restore_command = '/usr/local/internal/pgrestore.sh "%f" "%p"'
wal_level = logical
And I don't see any errors on postgresql 13.5 startup. The pgsql 13.5 pg_wal directory contains:
drwx------ 3 postgres postgres 4096 Oct 19 15:51 .
drwx------ 19 postgres postgres 4096 Oct 19 15:31 ..
-rw------- 1 postgres postgres 348 Oct 19 15:31 000000010000000000000097.00000028.backup
-rw------- 1 postgres postgres 16777216 Oct 19 15:51 00000001000000000000009A
-rw------- 1 postgres postgres 16777216 Oct 19 15:56 00000001000000000000009B
-rw------- 1 postgres postgres 16777216 Oct 19 15:31 00000001000000000000009C
-rw------- 1 postgres postgres 16777216 Oct 19 15:36 00000001000000000000009D
-rw------- 1 postgres postgres 16777216 Oct 19 15:41 00000001000000000000009E
drwx------ 2 postgres postgres 4096 Oct 19 15:56 archive_status
And the archive directory contains:
drwxrwxrwx 2 postgres postgres 4096 Oct 19 15:56 .
drwxrwxrwx 4 root root 4096 Oct 18 20:57 ..
-rw------- 1 postgres postgres 16777216 Oct 19 15:30 000000010000000000000094
-rw------- 1 postgres postgres 16777216 Oct 19 15:30 000000010000000000000095
-rw------- 1 postgres postgres 16777216 Oct 19 15:31 000000010000000000000096
-rw------- 1 postgres postgres 16777216 Oct 19 15:31 000000010000000000000097
-rw------- 1 postgres postgres 348 Oct 19 15:31 000000010000000000000097.00000028.backup
-rw------- 1 postgres postgres 16777216 Oct 19 15:36 000000010000000000000098
-rw------- 1 postgres postgres 16777216 Oct 19 15:41 000000010000000000000099
-rw------- 1 postgres postgres 16777216 Oct 19 15:51 00000001000000000000009A
-rw------- 1 postgres postgres 16777216 Oct 19 15:56 00000001000000000000009B
As you can see the pg_wal directory contains files 9C, 9D, and 9E which have not yet been archived. (9D and 9E are probably recycled old wal's not yet used as per comment below). But 9B has been open for an hour or more. Why is it not being archived?
Why is postgresql not forcing a new logfile every 5 min?
I have wal_keep_segments set to 3000. But directory pg_xlog contains more than 6000 WAL segments. Interesting thing that there are ~ 3000 files dated after Aug 14, so files dated before Aug 14 should not be exists, I guess. Also these files have an executable bit set.
$ ls -al pg_xlog | grep -A2 -B2 00000001000034DB0000003B
-rwx------ 1 postgres postgres 16777216 Jul 19 07:58 00000001000034DB00000039
-rwx------ 1 postgres postgres 16777216 Jul 19 07:58 00000001000034DB0000003A
-rwx------ 1 postgres postgres 16777216 Jul 19 07:58 00000001000034DB0000003B
-rw------- 1 postgres postgres 16777216 Aug 14 19:17 0000000100003826000000EA
-rw------- 1 postgres postgres 16777216 Aug 14 19:17 0000000100003826000000EB
```
This cluster has no replication slots, archive_mode is enabled but archive_command is set to /bin/true. I think the new WAL segments are recycled and total amount is about 6000 but postgres does not delete the old files for some reason. Any ideas?
PostgreSQL is not in the habit of setting executable flags on WAL segments.
Besides, it looks like there is a gap in the numbering.
These files must be there by accident, you can delete them.
I have 2 mongod instances running with the following parameters
--noprealloc --smallfiles --replSet mongors1 --dbpath /data/db --nojournal
The goal of the exercise is to create a replicated environment with a minimal disk footprint for local development purposes.
At this point in time, all is good with each respective data system being around ~32M and having the following
ls -o data/db
total 32784
-rw------- 1 999 16777216 Sep 22 11:38 local.0
-rw------- 1 999 16777216 Sep 22 11:38 local.ns
-rwxr-xr-x 1 999 2 Sep 22 11:38 mongod.lock
-rw-r--r-- 1 999 69 Sep 22 11:38 storage.bson
drwxr-xr-x 2 999 4096 Sep 22 11:38 _tmp
After logging on to the first member and running rs.initiate(); an additional 1G of disk space is utilized.
ls -o data/db
total 1080856
-rw------- 1 999 16777216 Sep 22 11:39 local.0
-rw------- 1 999 536608768 Sep 22 11:39 local.1
-rw------- 1 999 536608768 Sep 22 11:39 local.2
-rw------- 1 999 16777216 Sep 22 11:39 local.ns
-rwxr-xr-x 1 999 2 Sep 22 11:38 mongod.lock
-rw-r--r-- 1 999 69 Sep 22 11:38 storage.bson
drwxr-xr-x 2 999 4096 Sep 22 11:39 _tmp
This seems excessive given the properties of the nodes being replicated and the configuration they are running.
Mongo 3.0.6 is the version in use.
Eventually this will be scaled up to replica sets with 3 members across 2+ shards. A minimal disk requirement of 6Gb to store zero data initially seems sub-optimal.
Is there a way to reduce this to something more representative of the nodes needs?
Any help is appreciated. Thanks in advance
The local database contains the oplog, and I'll leave you to research yourself as to what size this should be for a given node. To address the question at hand, from the docs:
For 64-bit Linux, Solaris, FreeBSD, and Windows systems, MongoDB
allocates 5% of the available free disk space, but will always
allocate at least 1 gigabyte and never more than 50 gigabytes.
That's where your usage is coming from - to alter that allocation you will either need to resize the oplog or, if starting from scratch, look at the oplogSizeMB option (or for the CLI equivalent see here).
In addition to what Adam said, add the
--oplogSize X
to your parameters and replace X with the amount of MB you want the oplog to be.
I'm attempting to build a box via Fabric on Openstack. Part of the install involves installing and running PostgreSQL.
This command works fine:
$ sudo service postgresql initdb
This command fails:
$ sudo service postgresql start
Log output of failure shows no issues when I run this command:
$ cat /var/lib/pgsql/pgstartup.log
This command shows the following messages:
$ cat /var/lib/pgsql/data/pg_log/postgresql-Wed.log
LOG: could not open configuration file
"/var/lib/pgsql/data/pg_hba.conf": Permission denied
FATAL: could not load pg_hba.conf
My user while executing these commands has the following groups:
vagrant, wheel
My user is in the sudoers list under /etc/sudoers with these permissions:
vagrant ALL=(ALL) NOPASSWD: ALL
perms on pgsql:
[root#integration ~]# ls -ltr /var/lib/pgsql/
total 12
drwx------. 2 postgres postgres 4096 Sep 13 2012 backups
-rw-------. 1 postgres postgres 1152 Jun 19 20:17 pgstartup.log
drwx------. 12 postgres postgres 4096 Jun 19 20:19 data
and sub dir data:
[root#integration ~]# ls -ltr /var/lib/pgsql/data/
total 76
-rw-------. 1 postgres postgres 4 Jun 19 20:17 PG_VERSION
drwx------. 2 postgres postgres 4096 Jun 19 20:17 pg_twophase
drwx------. 2 postgres postgres 4096 Jun 19 20:17 pg_tblspc
drwx------. 2 postgres postgres 4096 Jun 19 20:17 pg_stat_tmp
drwx------. 4 postgres postgres 4096 Jun 19 20:17 pg_multixact
-rw-------. 1 postgres postgres 16886 Jun 19 20:17 postgresql.conf
-rw-------. 1 postgres postgres 1631 Jun 19 20:17 pg_ident.conf
drwx------. 3 postgres postgres 4096 Jun 19 20:17 pg_xlog
drwx------. 2 postgres postgres 4096 Jun 19 20:17 pg_subtrans
drwx------. 2 postgres postgres 4096 Jun 19 20:17 pg_clog
drwx------. 5 postgres postgres 4096 Jun 19 20:17 base
drwx------. 2 postgres postgres 4096 Jun 19 20:17 global
-rw-------. 1 postgres postgres 241 Jun 19 20:17 pg_hba.conf
drwx------. 2 postgres postgres 4096 Jun 19 20:17 pg_log
-rw-------. 1 postgres postgres 57 Jun 19 20:19 postmaster.opts
This turned out to be an selinux issue.
I disabled it and was well. Full config below.
For those of you that arent up on selinux (like me until today) the config can be found in:
/etc/selinux/config
It can also be turned off temporarily like this:
echo 0 > /selinux/enforce
Full config
[root#integration selinux]# cat config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - SELinux is fully disabled.
SELINUX=enforcing
# SELINUXTYPE= type of policy in use. Possible values are:
# targeted - Only targeted network daemons are protected.
# strict - Full SELinux protection.
Where are the files for a PostgreSQL database stored?
To see where the data directory is, use this query.
show data_directory;
To see all the run-time parameters, use
show all;
You can create tablespaces to store database objects in other parts of the filesystem. To see tablespaces, which might not be in that data directory, use this query.
SELECT *, pg_tablespace_location(oid) FROM pg_tablespace;
On Windows7 all the databases are referred by a number in the file named pg_database under C:\Program Files (x86)\PostgreSQL\8.2\data\global. Then you should search for the folder name by that number under C:\Program Files (x86)\PostgreSQL\8.2\data\base. That is the content of the database.
As suggested in "PostgreSQL database default location on Linux", under Linux you can find out using the following command:
ps aux | grep postgres | grep -- -D
Open pgAdmin and go to Properties for specific database. Find OID and then open directory
<POSTGRESQL_DIRECTORY>/data/base/<OID>
There should be your DB files.
Under my Linux installation, it's here: /var/lib/postgresql/8.x/
You can change it with initdb -D "c:/mydb/"
Everyone already answered but just for the latest updates. If you want to know where all the configuration files reside then run this command in the shell
SELECT name, setting FROM pg_settings WHERE category = 'File Locations';
I'd bet you're asking this question because you've tried pg_ctl start and received the following error:
pg_ctl: no database directory specified and environment variable PGDATA unset
In other words, you're looking for the directory to put after -D in your pg_ctl start command.
In this case, the directory you're looking for contains these files.
PG_VERSION pg_dynshmem pg_multixact
pg_snapshots pg_tblspc postgresql.conf
base pg_hba.conf pg_notify
pg_stat pg_twophase postmaster.opts
global pg_ident.conf pg_replslot
pg_stat_tmp pg_xlog postmaster.pid
pg_clog pg_logical pg_serial
pg_subtrans postgresql.auto.conf server.log
You can locate it by locating any of the files and directories above using the search provided with your OS.
For example in my case (a HomeBrew install on Mac OS X), these files are located in /usr/local/var/postgres. To start the server I type:
pg_ctl -D /usr/local/var/postgres -w start
... and it works.
Postgres stores data in files in its data directory. Follow the steps below to go to a database and its files:
The database corresponding to a postgresql table file is a directory. The location of the entire data directory can be obtained by running SHOW data_directory.
in a UNIX like OS (eg: Mac) /Library/PostgreSQL/9.4/data
Go inside the base folder in the data directory which has all the database folders: /Library/PostgreSQL/9.4/data/base
Find the database folder name by running (Gives an integer. This is the database folder name):
SELECT oid from pg_database WHERE datname = <database_name>;
Find the table file name by running (Gives an integer. This is the file name):
SELECT relname, relfilenode FROM pg_class WHERE relname = <table_name>;
This is a binary file. File details such as size and creation date time can be obtained as usual. For more info read this SO thread
The location of specific tables/indexes can be adjusted by TABLESPACEs:
CREATE TABLESPACE dbspace LOCATION '/data/dbs';
CREATE TABLE something (......) TABLESPACE dbspace;
CREATE TABLE otherthing (......) TABLESPACE dbspace;
On Mac: /Library/PostgreSQL/9.0/data/base
The directory can't be entered, but you can look at the content via: sudo du -hc data
picmate's answer is right. on windows the main DB folder location is (at least on my installation)
C:\PostgreSQL\9.2\data\base\
and not in program files.
his 2 scripts, will give you the exact directory/file(s) you need:
SELECT oid from pg_database WHERE datname = <database_name>;
SELECT relname, relfilenode FROM pg_class WHERE relname = <table_name>;
mine is in datname 16393 and relfilenode 41603
On Windows, the PGDATA directory that the PostgresSQL docs describe is at somewhere like C:\Program Files\PostgreSQL\8.1\data. The data for a particular database is under (for example) C:\Program Files\PostgreSQL\8.1\data\base\100929, where I guess 100929 is the database number.
I'm running postgres (9.5) in a docker container (on CentOS, as it happens), and as Skippy le Grand Gourou mentions in a comment above, the files are located in /var/lib/postgresql/data/.
$ docker exec -it my-postgres-db-container bash
root#c7d61efe2a5d:/# cd /var/lib/postgresql/data/
root#c7d61efe2a5d:/var/lib/postgresql/data# ls -lh
total 56K
drwx------. 7 postgres postgres 71 Apr 5 2018 base
drwx------. 2 postgres postgres 4.0K Nov 2 02:42 global
drwx------. 2 postgres postgres 18 Dec 27 2017 pg_clog
drwx------. 2 postgres postgres 6 Dec 27 2017 pg_commit_ts
drwx------. 2 postgres postgres 6 Dec 27 2017 pg_dynshmem
-rw-------. 1 postgres postgres 4.4K Dec 27 2017 pg_hba.conf
-rw-------. 1 postgres postgres 1.6K Dec 27 2017 pg_ident.conf
drwx------. 4 postgres postgres 39 Dec 27 2017 pg_logical
drwx------. 4 postgres postgres 36 Dec 27 2017 pg_multixact
drwx------. 2 postgres postgres 18 Nov 2 02:42 pg_notify
drwx------. 2 postgres postgres 6 Dec 27 2017 pg_replslot
drwx------. 2 postgres postgres 6 Dec 27 2017 pg_serial
drwx------. 2 postgres postgres 6 Dec 27 2017 pg_snapshots
drwx------. 2 postgres postgres 6 Sep 16 21:15 pg_stat
drwx------. 2 postgres postgres 63 Nov 8 02:41 pg_stat_tmp
drwx------. 2 postgres postgres 18 Oct 24 2018 pg_subtrans
drwx------. 2 postgres postgres 6 Dec 27 2017 pg_tblspc
drwx------. 2 postgres postgres 6 Dec 27 2017 pg_twophase
-rw-------. 1 postgres postgres 4 Dec 27 2017 PG_VERSION
drwx------. 3 postgres postgres 92 Dec 20 2018 pg_xlog
-rw-------. 1 postgres postgres 88 Dec 27 2017 postgresql.auto.conf
-rw-------. 1 postgres postgres 21K Dec 27 2017 postgresql.conf
-rw-------. 1 postgres postgres 37 Nov 2 02:42 postmaster.opts
-rw-------. 1 postgres postgres 85 Nov 2 02:42 postmaster.pid
A single Terminal command: pg_lsclusters, (using Ubuntu)
What you need is under Data directory:
Ver Cluster Port Status Owner Data directory Log file
10 main 5432 online postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log
11 main 5433 online postgres /var/lib/postgresql/11/main /var/log/postgresql/postgresql-11-main.log