those files are present in folder /pg_commit_ts
-rw-------. 1 postgres postgres 262144 Jun 17 12:56 0000
-rw-------. 1 postgres postgres 262144 Jun 17 12:56 0001
-rw-------. 1 postgres postgres 262144 Jun 17 12:57 0002
...
Are thoses files created only if track_commit_timestamp is on?
Yes, these files are only created if track_commit_timestamp = on. You cannot get the last committed statement, but you can use pg_last_committed_xact() to get the timestamp and transaction ID of the last committed transaction (see the documentation).
I have the following situation.
Have a master/replica setup.
Somehow the database was dropped and a new database with the same name was created by django. in this case, will the WAL files of the previous database still be there?
I have not created a backup using of the previous database using a tool like pg_basebackup, but do have some WAL files in the pg_xlog.
Now, i am trying to do the following:
- Shutdown the postgres server.
- Use the recovery.conf file in PGDATA (/var/lib/postgresql/9.3/main) directory and entering the following in there:
restore_command = 'cp /var/lib/postgresql/9.3/main/pg_xlog_backup_jan072016/%f %p'
recovery_target_time = '2016-01-07 03:00:00'
- Startup the postgres server again.
What I see in the log file is:
postgres:~/9.3/main/pg_log$ tail -100f postgresql-2016-01-07_170256.log
2016-01-07 17:02:56 UTC LOG: database system was shut down at 2016-01-07 17:02:55 UTC
2016-01-07 17:02:56 UTC LOG: starting point-in-time recovery to 2016-01-06 00:00:00+00
cp: cannot stat ‘/var/lib/postgresql/9.3/main/pg_xlog_backup_jan072016/0000000A.history’: No such file or directory
cp: cannot stat ‘/var/lib/postgresql/9.3/main/pg_xlog_backup_jan072016/0000000A00000000000000CC’: No such file or directory
2016-01-07 17:02:56 UTC LOG: consistent recovery state reached at 0/CC000090
2016-01-07 17:02:56 UTC LOG: record with zero length at 0/CC000090
2016-01-07 17:02:56 UTC LOG: redo is not required
2016-01-07 17:02:56 UTC LOG: database system is ready to accept read only connections
cp: cannot stat ‘/var/lib/postgresql/9.3/main/pg_xlog_backup_jan072016/0000000A00000000000000CC’: No such file or directory
cp: cannot stat ‘/var/lib/postgresql/9.3/main/pg_xlog_backup_jan072016/0000000B.history’: No such file or directory
2016-01-07 17:02:56 UTC LOG: selected new timeline ID: 11
cp: cannot stat ‘/var/lib/postgresql/9.3/main/pg_xlog_backup_jan072016/0000000A.history’: No such file or directory
2016-01-07 17:02:57 UTC LOG: archive recovery complete
2016-01-07 17:02:57 UTC LOG: autovacuum launcher started
2016-01-07 17:02:57 UTC LOG: database system is ready to accept connections
2016-01-07 17:02:57 UTC LOG: incomplete startup packet
My pg_xlog_backup_jan072016 folder has the foll content.:
:~/9.3/main/pg_xlog_backup_jan072016$ ls -larth
total 161M
-rw------- 1 postgres postgres 257 Jan 7 15:57 00000007.history
-rw------- 1 postgres postgres 16M Jan 7 15:57 0000000700000000000000CA
-rw------- 1 postgres postgres 16M Jan 7 15:57 0000000700000000000000C9
-rw------- 1 postgres postgres 16M Jan 7 15:57 0000000700000000000000C8
-rw------- 1 postgres postgres 16M Jan 7 15:57 0000000700000000000000C7
-rw------- 1 postgres postgres 16M Jan 7 15:57 0000000700000000000000C6
-rw------- 1 postgres postgres 16M Jan 7 15:57 0000000700000000000000C5
-rw------- 1 postgres postgres 16M Jan 7 15:57 0000000700000000000000C4
-rw------- 1 postgres postgres 16M Jan 7 15:57 0000000700000000000000C3
-rw------- 1 postgres postgres 16M Jan 7 15:57 0000000700000000000000C2
-rw------- 1 postgres postgres 16M Jan 7 15:57 0000000700000000000000C1
-rw------- 1 postgres postgres 298 Jan 7 15:57 00000007000000000000005D.00000028.backup
-rw------- 1 postgres postgres 214 Jan 7 15:57 00000006.history
-rw------- 1 postgres postgres 171 Jan 7 15:57 00000005.history
drwx------ 3 postgres postgres 4.0K Jan 7 16:54 .
drwx------ 18 postgres postgres 4.0K Jan 7 17:02 ..
drwx------ 2 postgres postgres 4.0K Jan 7 17:08 archive_status
The thing I am trying to figure out is:
Is it possible to restore from WAL without have a backup from pg_basebackup command? I am just trying to restore from WAL in an existing postgres installation.
Why does the log say that it cannot find 0000000A.history and 0000000A00000000000000CC files? The folder does not contain any such files.
Can anyone help us with this?
Thanks.
Sarthak
I have 2 mongod instances running with the following parameters
--noprealloc --smallfiles --replSet mongors1 --dbpath /data/db --nojournal
The goal of the exercise is to create a replicated environment with a minimal disk footprint for local development purposes.
At this point in time, all is good with each respective data system being around ~32M and having the following
ls -o data/db
total 32784
-rw------- 1 999 16777216 Sep 22 11:38 local.0
-rw------- 1 999 16777216 Sep 22 11:38 local.ns
-rwxr-xr-x 1 999 2 Sep 22 11:38 mongod.lock
-rw-r--r-- 1 999 69 Sep 22 11:38 storage.bson
drwxr-xr-x 2 999 4096 Sep 22 11:38 _tmp
After logging on to the first member and running rs.initiate(); an additional 1G of disk space is utilized.
ls -o data/db
total 1080856
-rw------- 1 999 16777216 Sep 22 11:39 local.0
-rw------- 1 999 536608768 Sep 22 11:39 local.1
-rw------- 1 999 536608768 Sep 22 11:39 local.2
-rw------- 1 999 16777216 Sep 22 11:39 local.ns
-rwxr-xr-x 1 999 2 Sep 22 11:38 mongod.lock
-rw-r--r-- 1 999 69 Sep 22 11:38 storage.bson
drwxr-xr-x 2 999 4096 Sep 22 11:39 _tmp
This seems excessive given the properties of the nodes being replicated and the configuration they are running.
Mongo 3.0.6 is the version in use.
Eventually this will be scaled up to replica sets with 3 members across 2+ shards. A minimal disk requirement of 6Gb to store zero data initially seems sub-optimal.
Is there a way to reduce this to something more representative of the nodes needs?
Any help is appreciated. Thanks in advance
The local database contains the oplog, and I'll leave you to research yourself as to what size this should be for a given node. To address the question at hand, from the docs:
For 64-bit Linux, Solaris, FreeBSD, and Windows systems, MongoDB
allocates 5% of the available free disk space, but will always
allocate at least 1 gigabyte and never more than 50 gigabytes.
That's where your usage is coming from - to alter that allocation you will either need to resize the oplog or, if starting from scratch, look at the oplogSizeMB option (or for the CLI equivalent see here).
In addition to what Adam said, add the
--oplogSize X
to your parameters and replace X with the amount of MB you want the oplog to be.
I'm attempting to build a box via Fabric on Openstack. Part of the install involves installing and running PostgreSQL.
This command works fine:
$ sudo service postgresql initdb
This command fails:
$ sudo service postgresql start
Log output of failure shows no issues when I run this command:
$ cat /var/lib/pgsql/pgstartup.log
This command shows the following messages:
$ cat /var/lib/pgsql/data/pg_log/postgresql-Wed.log
LOG: could not open configuration file
"/var/lib/pgsql/data/pg_hba.conf": Permission denied
FATAL: could not load pg_hba.conf
My user while executing these commands has the following groups:
vagrant, wheel
My user is in the sudoers list under /etc/sudoers with these permissions:
vagrant ALL=(ALL) NOPASSWD: ALL
perms on pgsql:
[root#integration ~]# ls -ltr /var/lib/pgsql/
total 12
drwx------. 2 postgres postgres 4096 Sep 13 2012 backups
-rw-------. 1 postgres postgres 1152 Jun 19 20:17 pgstartup.log
drwx------. 12 postgres postgres 4096 Jun 19 20:19 data
and sub dir data:
[root#integration ~]# ls -ltr /var/lib/pgsql/data/
total 76
-rw-------. 1 postgres postgres 4 Jun 19 20:17 PG_VERSION
drwx------. 2 postgres postgres 4096 Jun 19 20:17 pg_twophase
drwx------. 2 postgres postgres 4096 Jun 19 20:17 pg_tblspc
drwx------. 2 postgres postgres 4096 Jun 19 20:17 pg_stat_tmp
drwx------. 4 postgres postgres 4096 Jun 19 20:17 pg_multixact
-rw-------. 1 postgres postgres 16886 Jun 19 20:17 postgresql.conf
-rw-------. 1 postgres postgres 1631 Jun 19 20:17 pg_ident.conf
drwx------. 3 postgres postgres 4096 Jun 19 20:17 pg_xlog
drwx------. 2 postgres postgres 4096 Jun 19 20:17 pg_subtrans
drwx------. 2 postgres postgres 4096 Jun 19 20:17 pg_clog
drwx------. 5 postgres postgres 4096 Jun 19 20:17 base
drwx------. 2 postgres postgres 4096 Jun 19 20:17 global
-rw-------. 1 postgres postgres 241 Jun 19 20:17 pg_hba.conf
drwx------. 2 postgres postgres 4096 Jun 19 20:17 pg_log
-rw-------. 1 postgres postgres 57 Jun 19 20:19 postmaster.opts
This turned out to be an selinux issue.
I disabled it and was well. Full config below.
For those of you that arent up on selinux (like me until today) the config can be found in:
/etc/selinux/config
It can also be turned off temporarily like this:
echo 0 > /selinux/enforce
Full config
[root#integration selinux]# cat config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - SELinux is fully disabled.
SELINUX=enforcing
# SELINUXTYPE= type of policy in use. Possible values are:
# targeted - Only targeted network daemons are protected.
# strict - Full SELinux protection.
Where are the files for a PostgreSQL database stored?
To see where the data directory is, use this query.
show data_directory;
To see all the run-time parameters, use
show all;
You can create tablespaces to store database objects in other parts of the filesystem. To see tablespaces, which might not be in that data directory, use this query.
SELECT *, pg_tablespace_location(oid) FROM pg_tablespace;
On Windows7 all the databases are referred by a number in the file named pg_database under C:\Program Files (x86)\PostgreSQL\8.2\data\global. Then you should search for the folder name by that number under C:\Program Files (x86)\PostgreSQL\8.2\data\base. That is the content of the database.
As suggested in "PostgreSQL database default location on Linux", under Linux you can find out using the following command:
ps aux | grep postgres | grep -- -D
Open pgAdmin and go to Properties for specific database. Find OID and then open directory
<POSTGRESQL_DIRECTORY>/data/base/<OID>
There should be your DB files.
Under my Linux installation, it's here: /var/lib/postgresql/8.x/
You can change it with initdb -D "c:/mydb/"
Everyone already answered but just for the latest updates. If you want to know where all the configuration files reside then run this command in the shell
SELECT name, setting FROM pg_settings WHERE category = 'File Locations';
I'd bet you're asking this question because you've tried pg_ctl start and received the following error:
pg_ctl: no database directory specified and environment variable PGDATA unset
In other words, you're looking for the directory to put after -D in your pg_ctl start command.
In this case, the directory you're looking for contains these files.
PG_VERSION pg_dynshmem pg_multixact
pg_snapshots pg_tblspc postgresql.conf
base pg_hba.conf pg_notify
pg_stat pg_twophase postmaster.opts
global pg_ident.conf pg_replslot
pg_stat_tmp pg_xlog postmaster.pid
pg_clog pg_logical pg_serial
pg_subtrans postgresql.auto.conf server.log
You can locate it by locating any of the files and directories above using the search provided with your OS.
For example in my case (a HomeBrew install on Mac OS X), these files are located in /usr/local/var/postgres. To start the server I type:
pg_ctl -D /usr/local/var/postgres -w start
... and it works.
Postgres stores data in files in its data directory. Follow the steps below to go to a database and its files:
The database corresponding to a postgresql table file is a directory. The location of the entire data directory can be obtained by running SHOW data_directory.
in a UNIX like OS (eg: Mac) /Library/PostgreSQL/9.4/data
Go inside the base folder in the data directory which has all the database folders: /Library/PostgreSQL/9.4/data/base
Find the database folder name by running (Gives an integer. This is the database folder name):
SELECT oid from pg_database WHERE datname = <database_name>;
Find the table file name by running (Gives an integer. This is the file name):
SELECT relname, relfilenode FROM pg_class WHERE relname = <table_name>;
This is a binary file. File details such as size and creation date time can be obtained as usual. For more info read this SO thread
The location of specific tables/indexes can be adjusted by TABLESPACEs:
CREATE TABLESPACE dbspace LOCATION '/data/dbs';
CREATE TABLE something (......) TABLESPACE dbspace;
CREATE TABLE otherthing (......) TABLESPACE dbspace;
On Mac: /Library/PostgreSQL/9.0/data/base
The directory can't be entered, but you can look at the content via: sudo du -hc data
picmate's answer is right. on windows the main DB folder location is (at least on my installation)
C:\PostgreSQL\9.2\data\base\
and not in program files.
his 2 scripts, will give you the exact directory/file(s) you need:
SELECT oid from pg_database WHERE datname = <database_name>;
SELECT relname, relfilenode FROM pg_class WHERE relname = <table_name>;
mine is in datname 16393 and relfilenode 41603
On Windows, the PGDATA directory that the PostgresSQL docs describe is at somewhere like C:\Program Files\PostgreSQL\8.1\data. The data for a particular database is under (for example) C:\Program Files\PostgreSQL\8.1\data\base\100929, where I guess 100929 is the database number.
I'm running postgres (9.5) in a docker container (on CentOS, as it happens), and as Skippy le Grand Gourou mentions in a comment above, the files are located in /var/lib/postgresql/data/.
$ docker exec -it my-postgres-db-container bash
root#c7d61efe2a5d:/# cd /var/lib/postgresql/data/
root#c7d61efe2a5d:/var/lib/postgresql/data# ls -lh
total 56K
drwx------. 7 postgres postgres 71 Apr 5 2018 base
drwx------. 2 postgres postgres 4.0K Nov 2 02:42 global
drwx------. 2 postgres postgres 18 Dec 27 2017 pg_clog
drwx------. 2 postgres postgres 6 Dec 27 2017 pg_commit_ts
drwx------. 2 postgres postgres 6 Dec 27 2017 pg_dynshmem
-rw-------. 1 postgres postgres 4.4K Dec 27 2017 pg_hba.conf
-rw-------. 1 postgres postgres 1.6K Dec 27 2017 pg_ident.conf
drwx------. 4 postgres postgres 39 Dec 27 2017 pg_logical
drwx------. 4 postgres postgres 36 Dec 27 2017 pg_multixact
drwx------. 2 postgres postgres 18 Nov 2 02:42 pg_notify
drwx------. 2 postgres postgres 6 Dec 27 2017 pg_replslot
drwx------. 2 postgres postgres 6 Dec 27 2017 pg_serial
drwx------. 2 postgres postgres 6 Dec 27 2017 pg_snapshots
drwx------. 2 postgres postgres 6 Sep 16 21:15 pg_stat
drwx------. 2 postgres postgres 63 Nov 8 02:41 pg_stat_tmp
drwx------. 2 postgres postgres 18 Oct 24 2018 pg_subtrans
drwx------. 2 postgres postgres 6 Dec 27 2017 pg_tblspc
drwx------. 2 postgres postgres 6 Dec 27 2017 pg_twophase
-rw-------. 1 postgres postgres 4 Dec 27 2017 PG_VERSION
drwx------. 3 postgres postgres 92 Dec 20 2018 pg_xlog
-rw-------. 1 postgres postgres 88 Dec 27 2017 postgresql.auto.conf
-rw-------. 1 postgres postgres 21K Dec 27 2017 postgresql.conf
-rw-------. 1 postgres postgres 37 Nov 2 02:42 postmaster.opts
-rw-------. 1 postgres postgres 85 Nov 2 02:42 postmaster.pid
A single Terminal command: pg_lsclusters, (using Ubuntu)
What you need is under Data directory:
Ver Cluster Port Status Owner Data directory Log file
10 main 5432 online postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log
11 main 5433 online postgres /var/lib/postgresql/11/main /var/log/postgresql/postgresql-11-main.log