I have a postgres 9.3 database in a windows installation. The complete installation is on a raid installed.
We wanted to replace that raid with new drives since the old ones started to die. We copied all data from the raid to a new drive with the file explorer (wasn't me...), then replaced all drives and created a new raid, put back on all the old data and changed the drive letter back to the old one. Thinking that everything would work as normal.
Sadly not with postgres. The copying with file explorer destroyed all symlinks in the pg_tblspc folder. They arent symlinks anymore, they are just empty folders. So I have now 147 broken symlinks...
I checked how the symlinks working and compared to a working postgres installation. I found out that every table has an OID. For each OID there are 2 symlinks in that folder. The symlinks have an increased number.
For example, the OID of a table is 17530, then I have 2 symlinks 17538 and 17541. These pointing towards the data folder with the database name. In the folder with the database name I have 2 folders again. One of these symlinks pointing to one of these folders.
The increased numbers are always so same. So its always OID +8 and +11 (on other working installation its always +4 and +7).
All the table management is done by a program (FTK). So if you do stuff in there, its creating/deleting/updating the databases and tables for you. I think that its always 2 folders in there is because FTK is doing that in that way.
My question is now: Can I just manually create these symlinks? And then everything should work? Or is there maybe a function from postgres, where I can point to the "new" folder and it recreates the symlinks? It looks like the symlinks are managed by postgres itself. But so far I couldnt find anything about a repair function
Mistake number one was not to take a backup. Mistake number two was to create 147 tablespaces. But let's no dwell on that. What can you do?
First, facts. In the directory pg_tblspc is one symbolic link per tablespace, not per table. The name of the symbolic link is the object ID of the tablespace. You can find the object IDs with
SELECT oid, spcname FROM pg_tablespace;
Then you have to figure out what directory belong to which tablespace. There i no help for that, since that is the information that got lost. Once you know the path for a tablespace:
change to the pg_tblspc directory
run
ln -s /path/to/tablespace 12345
where 12345 is the OID of the tablespace.
Needless to say, PostgreSQL must be shut down when you do that. Once you have re-created the symbolic links for all tablespaces, you should be good.
Related
We have a PostgreSQL instance running in a VM in the Google Cloud. The nature of the queries that we run involves lots of PostgreSQL temporary table space. (5 or 6 or more TB of disk I/O every day)
This I/O continues to be a major bottleneck in our database. Currently I have it all happening on an SSD persistent disk - not because we need to save any of the data in the event of a reboot, but because PostgreSQL lays out a file structure on the disk that it then uses for the temporary tables and if the file structure is missing when the database starts up, it isn't very good.
What I'd like to do is configure the temporary tablespace on the local SSD's because of their much higher I/O throughput. Unfortunately, they get wiped out on every reboot. I'd like a simple way to be able to re-layout the disk after reboot and before PostgreSQL starts back up.
I could tar up the empty file structure and then write a script that untars it after every boot. Does that make sense? Is there a better way/best practice for doing this?
What would be awesome is if there was a PostgreSQL extension out there that did this magically.
Ideas?
I dug a bit into my previous tests and here is some summary:
PostgreSQL tablespace is just a directory - no big deal. Plus - if you will use it only as temporary table space there will be no persistent file left when you shutdown database.
You can create tablespace for temp tables on any location you want and then go to this location and check directory structure to see what PG created. But you must do under OS because PG will show you only tablespace main directory - both \db+ in psql or select oid, spcname, pg_tablespace_location(oid) from pg_tablespace; work the same way.
My example:
(I used /tempspace/pgtemp as presumed mounting point) CREATE TABLESPACE p_temp OWNER xxxxxx LOCATION '/tempspace/pgtemp'; created in my case structure /tempspace/pgtemp/PG_10_201707211
I set temp_tablespaces = 'pg_temp' in postgresql.conf and reloaded configuration.
When I used create temp table .... PG added another subdirectory - /tempspace/pgtemp/PG_10_201707211/16393 = oid of schema - but this does not matter for temp tablespace because if this subdirectory will be missing PG will create it.
PG created in this subdir files for temp table.
When I closed this session files for temp table were gone.
Now I stopped PG and tested what would happened if directories will be missing:
I deleted PG_10_201707211 with its subdir
started PG and log showed message LOG: could not open tablespace directory "pg_tblspc/166827/PG_10_201707211": No such file or directory but PG started
I tried to create temp table - I got error message ERROR: could not create directory "pg_tblspc/166827/PG_10_201707211/16393": No such file or directory SQL state: 58P01
Now (with running PG) I issued these commands in OS:
sudo mkdir -p /tempspace/pgtemp/PG_10_201707211
sudo chown postgres:postgres -R /tempspace/pgtemp
sudo chmod 700 -R /tempspace/pgtemp
I tried to create temp table again and insert and select values and everything worked OK
So conclusion is - since PG tablespace is no "big magic" just directories you can simply create bash script running on linux startup which will check (and mount if necessary) local SSD and create necessary directories for PG temp tablespace.
I have a database that seems to be broken for some reason. It's a development db for rails so I don't have a backup but I do need to continue development. I tried to just drop it but that's not working.
$ dropdb "database-name"
dropdb: database removal failed: ERROR: could not open file "global/2964": No such file or directory
Thanks in advance for any help!
There's more wrong here than a "broken" database. Something is badly wrong with your PostgreSQL data directory.
global/9264 looks like it's pg_catalog.pg_db_role_setting, which stores ALTER DATABASE ... SET ... and ALTER ROLE ... SET ... settings. This is not database-specific, it's a global table.
If you have missing files in your data directory your whole PostgreSQL data directory is probably damaged. You should back up what you can, if there's anything you care about, then rename or delete the damaged data directory and initdb a new blank one.
You won't be able to DROP this database (or do much else) because PostgreSQL can't load the files for the pg_db_role_setting table, but it needs to delete entries referring to the dropped database from there.
As for how this happened:
Have you ever run with fsync = off in postgresql.conf?
Do you have SSD storage? If so, have you had any recent sudden power loss?
Have you ever done any direct modifications of any kind inside the PostgreSQL data directory?
Is the PostgreSQL data directory on external storage that might have been suddenly removed?
Have you ever deleted postmaster.pid ?
See also https://wiki.postgresql.org/wiki/Corruption
One of my servers has a virus and the Postgres service in Windows is not running a backup and I'm using Odoo8 and even the Odoo Service is not running.
Is it possible to restore a database using only a OID directory which from what I know is the database file of Postgres.
I assume you mean /data/base/<oid> directory. Unfortunately it's not enough. There are some settings stored outside database oid directory as you called it.
Ex:
/data/glboal/ - cluster users' settings (passwords, roles etc)
/data/pg_xlog/ - WAL entries - possibly with transactions changes not "transfered" to database files yet.
/data/pg_tblspc/ - tablespaces
You need whole /data directory. Read more about PHYSICAL BACKUP.
Edit:
So, if whole /data is available for you, you can restore database to other server. There's one thing you should remember: destination postrges cluster must be at the same varsion ex. 9.4.1. When the first and seccond numbers match (ex 9.2.10 and 9.2.16) this should also work most of the times. Keeping that in mind, you just need to replace /data/ directory on destination server with your source /data directory (destination server must be stopped during that operation).
I am currently moving some data around and I am running into an interesting issue.
I have a CentOS server (6.3) up and running with Postgres 9.2 on a server with limited built in disk space; however, I do have a large amount of extremely reliable external network disk space available.
I have set the tablespace to a directory on this storage devise for my database and everything seems to be working well, until...
I realized that I have a large amount of BLOB data that needs to be stored in pg_largeobject.
I have been goggling how to set the tablespace of pg_largeobject and I did find some results, but they are horribly out dated.
I did find one article that looks promising, but I'm hesitant because the thread also references that things will/should have changed.
I have two questions...
In an ideal world, I would like to move all of postgres (including pg_largeobject) onto this external storage for ease of maintenance. Is this possible?
If not, how can I get pg_largeobject to use my network storage?
As you alluded to, your best bet is to move the entirety of PostgreSQL onto the remote storage, assuming that storage uses a reliable file network block device like iSCSI, ATAoE or NBD. I wouldn't recommend running Pg on NFS, and running it on CIFS/SMBFS just won't work.
Just:
Make a backup
Take a note of the output of SHOW data_directory; in psql
Shut PostgreSQL down
Move the data directory (the folder containing pg_xlog, pg_clog, etc) to the remote storage
Adjust the permissions on the parent directories for the datadir's new location to make sure the postgres user, postgres, group or others permissions block has at least execute on each parent directory so it can traverse the tree.
Adjust your system startup scripts to set the new location as the PostgreSQL datadir or symlink the old datadir location (output by SHOW data_directory) to the new location.
Start PostgreSQL
Unfortunately, different systems and packages find the datadir different ways. Debian/Ubuntu use pg_wrapper, for example.
I am installing postgres 8.4 on an ubuntu lucid server (no, at the moment we are using the "lucid" LTS version on that server so an upgrade is not possible yet (although we are going to start testing the system on precise quite soon now))
I have set up an own partition for the /var/lib/postgresql/8.4/main directory with a ext4 file system. (Those of you who are really into postgres installs knows what is happening now...) Since ext4 puts a lost+found directory in the root of all file system, postgres will not use that directory as its data-directory since it is initially not empty...
initdb: directory "/var/lib/postgresql/8.4/main" exists but is not empty
If you want to create a new database system, either remove or empty
the directory "/var/lib/postgresql/8.4/main" or run initdb
with an argument other than "/var/lib/postgresql/8.4/main".
The easiest way to proceed would be to remove the lost+found and recreate it after initdb has done its job. - could that cause any problems? Does the lost+found have any special attributes or anything that makes it impossible to recreate, and also, it is needed at any other time than if checkdisk finds something it needs to put there?
Another way would be to unmount the .../main/ file system, init the database, temporary mount the .../main/ filesystem somewhere else, move things over there and mount it in place. Seems to be a bit more work than the "easiest way".
Or is it some way to make initdb ignore that the directory is not empty? (couldn't see any command line switches for that)
May a lost+found directory within postgres main directory cause any problems?
At the moment I am running the system on a virtual machine for testing, so it really doesn't matter if I mess up things, but before making this an official way of installing a mission-critical system, it would be nice to have some thoughts on this.
lost+found has preallocated blocks that make it easier for fsck to move data into it when the partition is short of free blocks. To create it, better use the mklost+found command rather than mkdir.
If you don't recreate it, fsck will do it anyway when it's needed.
But if it comes to the point where fsck finds corruption within PGDATA, I'd think about going for a backup rather than counting on lost+found to retrieve anything.