Postgresql COPY command giving Permissions denied error - postgresql

I am trying to COPY a file into a table in PostgreSQL. The table owner is postgres and the file owner is postgres.
The file is in /tmp.
Still I am getting the error message:
could not open file "/tmp/file" for reading: Permission denied
I don't understand what I am doing wrong as all the posts I've found say that if I have the file in /tmp and owner is postgres then the COPY command should work.

A guess: You are using Fedora, Red Hat Enterprise Linux, CentOS, Scientific Linux, or one of the other distros that enable SELinux by default.
Either and on your particular OS/version the SELinux policies for PostgreSQL do not permit the server to read files outside the PostgreSQL data directory, or the file was created by a service covered by a targeted policy so it has a label that PostgreSQL isn't allowed to read from.
You can confirm whether or not this is the problem by running, as root:
setenforce 0
then re-testing. Run:
setenforce 1
to re-enable SELinux after testing. setenforce isn't permanent; SELinux will be automatically re-enabled on reboot anyway. Disabling SELinux permanently is not usually a good solution for issues like this; if you confirm the issue is SELinux it can be explored further.
Since you have not specified the OS or version you are using, the PostgreSQL version, the exact command you're running, ls -al on the file, \d+ on the table, etc, it's hard to give any more detail, or to know if this is more than a guess. Try updating your answer to include all that and an ls --lcontext of the file too.

COPY with a file name instructs the PostgreSQL server to directly read from or write to a file. The file must be accessible by the PostgreSQL user (the user ID the server runs as) and the name must be specified from the viewpoint of the server. (source: postgresql documentation)
So the file should be readable (or writable) by the unix user under which postgresql server is running (i.e not your user!). To be absolutly sure, you can try to run sudo -u postgres head /tmp/test.csv (assuming you are allowed to used sudo and assuming the database user is postgres).
If that fails, it might be an issue related to SELinux (as mentioned by Craig Ringer). Under the most common SELinux policy (the "targeted" reference policy), used by Red Hat/Fedora/CentOS, Scientific Linux, Debian and others... the postgresql server process is confined : it can only read/write a few file types.
The denial might not be logged in auditd's log file (/var/log/audit/audit.log) due to a donaudit rule. So the usual SELinux quick test apply e.g: stop SELinux from confining any process by running getenforce;setengorce 0;getenforce, then test postgresql's COPY. Then re-activate SELinux by running setenforce 1 (this command modify the running state, not the configuration file, so SELinux will be active (Enforcing) after reboot.
The proper way to fix that is to change the SELinux context of the file to load. A quick hack is to run:
chcon -t postgresql_tmp_t /tmp/a.csv
But this file labelling will not survive if hte filesystem is relabel or if you create a new file. You will need to create a directory with an SELinux file context mapping :
which semanage || yum install policycoreutils-python
semanage fcontext -a -t postgresql_tmp_t '/srv/psql_copydir(/.*)?'
mkdir /srv/psql_copydir
chmod 750 /srv/psql_copydir
chgrp postgres /srv/psql_copydir
restorecon -Rv /srv/psql_copydir
ls -Zd /srv/psql_copydir
Any file created in that directory should have the proper file context automatically so postgresql server can read/write it.
(to check the SELinux context under which postgres is running, runps xaZ | grep "postmaste[r]" | grep -o "[a-z_]*_t", which should print postgresql_t. To list the context types to which postgresql_t can write, use sesearch -s postgresql_t -A | grep ': file.*write'. the command sesearch belong to the setools-console RPM package).

Related

Greenplum Database :psql: could not connect to server: No such file or directory

I am bashing my head against the wall. its been 4 days.but psql is not connecting.
We have a small array of Greenplum database.In that, We have the master node. when i am trying to use psql utility
Getting this error :
[gpadmin#master gpseg-1]$ psql
psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
We tried
on searching for postmaster.pid files.
We have removed it.But still, error remains.
Use a command like ( netstat -ln; ps -ef ) | egrep '(postgres)|(postmaster)|(5432)' to try to determine whether or not an instance of the postgres server is running.
If the postmaster is not running, remove the postmaster.pid file and restart the database. While I don't use the Greenplum database, I see that instructions are here: Starting and Stopping the Greenplum Database. Do not remove the postmaster.pid file without making sure the database is not running, and note that removing the postmaster.pid file without starting the database is pointless.
It may be wise to open your postgresql.conf file and see if the listen_addresses, port, unix_socket_directory, unix_socket_group, and unix_socket_permissions settings might be a source of issues.
Since the error message referenced specifically mentions the socket file, look most closely at unix_socket_directory, unix_socket_group, and unix_socket_permissions.
If unix_socket_directory is pointing somewhere other than /tmp, then various workarounds exist.
Alternatively, and presuming that the server is running, one might try to locate the socket file without looking in the postgresql.conf file, though this might make it a bit harder to address permissions, port, etc. issues. A tool like locate, find, etc., may be used in conjunction with sudo or by the root user.
$ sudo find /tmp /var -name .s.PGSQL.5432
Presuming that the location of the .s.PGSQL.5432 file issue is the root cause of your problem, specifying the socket file location on the psql command-line is probably the most straightforward workaround. In example, if the *.s.PGSQL.5432 file is in /var/pgsql_socket directory as it is on some systems, try this, but, of course, use the actual directory where .s.PGSQL.5432 is located:
$ pgsql -h /var/pgsql_socket
If the .s.PGSQL.5432 file IS in /tmp, then the problem is more likely one of permissions, and consulting the postgresql.conf file is advised, and probably the user attempting the psql command will have to be added to a group that has access to the socket file. (Remember, log out and back in after changing group membership.)
Though the page does not necessarily seem to directly relate to this issue, do consider the Accessing the Database help as needed.
What does gpstate show? If it cannot connect, make sure the GPDB master is running:
ps ax | grep 'M master'
If the master is running, it will also show the port the master is listening on.
For the gpadmin Linux account, look in the ~/gpAdminLogs directory. There should be one or more startup logs that you can check.
That error normally means that the database is stopped. You should never remove this file unless it was left behind after a bad crash, where the file was never cleaned out. You would normally detect that situation when you start the db again - it would complain the file already existed.
I tend to look for
ps -eaf|grep -i silent
to see the postmaster processes.
If the master is down, but the segments are up, you will need to start the master only
gpstart -m
then stop everything with
gpstop -M fast
Causes for failures should be in $MASTER_DATA_DIRECTORY/pg_log and possibly in the corresponding segment pg_log directories. There may also be core files if the master or segments had a panic.

Unable to start Postgres Server because of permission denied on lock file

I restarted my Postgres server but now.
I checked my "pgstartup.log" log file. This says:
creating system views ... ok
loading system objects' descriptions ... ok
creating conversions ... ok
creating dictionaries ... ok
setting privileges on built-in objects ... ok
creating information schema ... ok
vacuuming database template1 ... ok
copying template1 to template0 ... ok
copying template1 to postgres ... ok
Success. You can now start the database server using:
/usr/bin/postgres -D /var/lib/pgsql/data
/usr/bin/pg_ctl -D /var/lib/pgsql/data -l logfile start
FATAL: could not open lock file "/tmp/.s.PGSQL.5432.lock": Permission denied
FATAL: could not open lock file "/tmp/.s.PGSQL.5432.lock": Permission denied
Do you think deleting /tmp/.s.PGSQL.5432.lock would work ?
Postgres cannot write a file to /tmp because of permissions set on /tmp directory.
As root user execute in terminal:
chmod 1777 /tmp
PostgreSQL normally deletes the lock file when terminates correctly.
It is probably due to another PostgreSQL instance running with a different user that has been terminated abnormally (a kill -9 to postmaster).
So, if you are sure no Postgres processes are running, you can probably delete that file without any issue. You should also check with the ipcs command if there is any stale shared memory segment, and in that case delete it with ipcrm.
Probably the be best way to address all these things at once is rebooting the server.
P.S.: never kill -9 any PostgreSQL process.
It looks like you probably have another PostgreSQL instance running on the same port as a different user, or you previously started this PostgreSQL instance as a different user then stopped it uncleanly.
Check the ownership of /tmp/.s.PGSQL.5432.lock:
ls -l /tmp/.s.PGSQL.5432.lock
Does it match the user you're running PostgreSQL as?
It's relatively harmless to delete the lock files in /tmp/. (Never, ever delete the postmaster.pid lock file though). If the other PostgreSQL instance is still running you'll lose the ability to connect to it over a unix socket, or you might get an error about being unable to bind to port 5432 on tcp.
I agree with #mnencia that a server reboot is the best option if it's easy and practical.
If you know that no other Postgres processes are running, please delete these 2 files and try again:
$ sudo rm /tmp/.s.PGSQL.5432.lock
$ sudo rm /tmp/.s.PGSQL.5432
Then, you would be able to run the server as a background process with the command:
$ pg_ctl -D /usr/local/var/postgres start
If you are in OS X, put the alias in the.bash_profile as below:
alias pgb='pg_ctl -D /usr/local/var/postgres start'
Now, source it with the command:
$ source ~/.bash_profile
The Postgres server will run with the command:
$ pgb
To me it was permission problem for database file, It was group/world readable. And that was wrong! Database file should be 0700. Aft
Thanks for the suggestions.
First I tried changing the permission of the lock file but it didn't work
Later I deleted the lock file which resolved my issue.
Thanks
Thanks.
I was trying to install postgres on my mac.I was receiving
FATAL: could not open lock file "/tmp/.s.PGSQL.5432.lock": Permission denied
After deleting the /tmp/.s.PGSQL.5432.lock file, the server started working.
Yes, I had the same issue, and I fixed it by running this command
$ sudo rm /tmp/.s.PGSQL.5432.lock
$ sudo rm /tmp/.s.PGSQL.5432
$ pg_ctl -D /usr/local/var/postgres start
I killed the Postgress service running on port 5432 which is something you should never do, this can be done unknowingly. So running the above commands would remove the lock file, and now when you start a new Postgres server, it would create a new process for you
Not sure, but I think it's the lock file that another Postgres instance is working on.
Restart the PC, and then that instance will leave access to that lock file. And the Postgres server will start. It works for me.

How can I move postgresql data to another directory on Ubuntu over Amazon EC2?

We've been running postgresql 8.4 for quite some time. As with any database, we are slowly reaching our threshold for space. I added another 8 GB EBS drive and mounted it to our instance and configured it to work properly on a directory called /files
Within /files, I manually created
Correct me if I'm wrong, but I believe all postgresql data is stored in /var/lib/postgresql/8.4/main
I backed up the database and I ran sudo /etc/init.d/postgresql stop. This stops the postgresql server. I tried to copy and paste the contents of /var/lib/postgresql/8.4/main into the /files directory but that turned out be a HUGE MESS! due to file permissions. I had to go in and chmod the contents of that folder just so that I could copy and paste them. Some files did not copy fully because of root permissions. I modified the data_directory parameter in postgresql.conf to point to the files directory
data_directory = '/files/postgresql/main'
and I ran sudo /etc/init.d/postgresql restart and the server failed to start. Again probably due to permission issues. Amazon EC2 only allows you to access the service as ubuntu by default. You can only access root from within the terminal which makes everything a lot more complicated.
Is there a much cleaner and more efficient step by step way of doing this?
Stop the server.
Copy the datadir while retaining permissions - use cp -aRv.
Then (easiest, as it avoids the need to modify initscripts) just move the old datadir aside and symlink the old path to the new location.
Thanks for the accepted answer. Instead of the symlink you can also use a bind mount. That way it is independent from the file system. If you want to use a dedicated hard drive for the database you can also mount it normally. to the data directory.
I did the latter. Here are my steps if someone needs a reference. I ran this as a script on many AWS instances.
# stop postgres server
sudo service postgresql stop
# create new filesystem in empty hard drive
sudo mkfs.ext4 /dev/xvdb
# mount it
mkdir /tmp/pg
sudo mount /dev/xvdb /tmp/pg/
# copy the entire postgres home dir content
sudo cp -a /var/lib/postgresql/. /tmp/pg
# mount it to the correct directory
sudo umount /tmp/pg
sudo mount /dev/xvdb /var/lib/postgresql/
# see if it is mounted
mount | grep postgres
# add the mount point to fstab
echo "/dev/xvdb /var/lib/postgresql ext4 rw 0 0" | sudo tee -a /etc/fstab
# when database is in use, observe that the correct disk is being used
watch -d grep xvd /proc/diskstats
A clarification. It is the particular AMI that you used that sets ubuntu as the default user, this may not apply to other AMIs.
In essence if you are trying move data manually, you will probably need to do so as the root user, and then make sure its available to whatever user postgres is running with.
You also do have the option of snapshotting the volume and increasing the size of the a volume created from the snapshot. Then you could replace the volume on your instance with the new volume (You probably will have to resize the partition to take advantage of all the space).

How to run postgres on centos when installed via YUM repo as default daemon user

With a freshly installed version of Postgres 9.2 via yum repository on Centos 6, how do you run postgres as a different user when it is configured to run as 'postgres:postgres' (u:g) out of the box?
In addition to AndrewPK's explanation, I'd like to note that you can also start new PostgreSQL instances as any user by stopping and disabling the system Pg service, then using:
initdb -D /path/to/data/directory
pg_ctl start -D /path/to/data/directory
This won't auto-start the server on boot, though. For that you must integrate into your init system. On CentOS 6 a simple System V-style init script in /etc/init.d/ and a suitable symlink into /etc/rc3.d/ or /etc/rc3.d/ (depending on default runlevel) is sufficient.
If running more than one instance at a time they must be on different ports. Change the port directive in postgresql.conf in the datadir or set it on startup with pg_ctl -o "-p 5433" .... You may also need to override the unix_socket_directories if your user doesn't have write permission to the default socket directory.
pg_ctl
initdb
This is only for a fresh installation (as it pertained to my situation) as it involves blowing away the data dir.
The steps I took to resolve this issue while utilizing the packaged startup scripts for a fresh installation:
Remove the postgres data dir /var/lib/pgsql/9.2/data if you've already gone through the initdb process with the postgres user:group configured as default.
Modify the startup script (/etc/init.d/postgresql-9.2) to replace all instances of postgres:postgres with NEWUSER:NEWGROUP.
Modify the startup script to replace all instances of postgres in any $SU -l postgres lines with the NEWUSER.
run /etc/init.d/postgres initdb to regenerate the cluster using the new username
Make sure any logs created are owned by the new user or remove old logs if error on initdb (the configuration file in my case was found in /var/lib/pgsql/9.2/data/postgresql.conf).
Startup postgres and it should now be running under the new user/group.
I understand this might not be what other people are looking for if they have existing postgres db's and want to restart the server to run as a different user/group combo - this was not my case, and I didn't see an answer posted anywhere for a 'fresh' install utilizing the pre-packaged startup scripts.

Eliminating non-working PostgreSQL installations on Ubuntu 10.04 and starting afresh

I find I have the wreckage of two old PostgreSQL installations on Ubuntu 10.04:
$ pg_lsclustersVersion Cluster Port Status Owner Data directory Log file
Use of uninitialized value in printf at /usr/bin/pg_lsclusters line 38.
8.4 main 5432 down /var/lib/postgresql/8.4/main /var/log/postgresql/postgresql-8.4-main.log
Use of uninitialized value in printf at /usr/bin/pg_lsclusters line 38.
9.1 main 5433 down /var/lib/postgresql/9.1/main /var/log/postgresql/postgresql-9.1-main.log
$
Attempts to perform basic functions return errors, for instance:
createuser: could not connect to database postgres: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
More information comes when I try to start the database server:
$ sudo /etc/init.d/postgresql start
* Starting PostgreSQL 9.1 database server
* Error: The cluster is owned by user id 109 which does not exist any more
...fail!
$
My question: how do I completely remove both clusters and set up a new one? I've tried removing, purging, and reinstalling postgresql, following the advice here: https://stackoverflow.com/a/2748644/621762. Now pg_lsclusters shows no clusters in existence, but the No such file or directory error persists when I try to createuser, createdb or run psql. What have I failed to do?
First, that answer you linked to was pretty unsafe - hand-editing /etc/passwd ?!? dselect where an apt wildcard would do? Crazy stuff. I'm not surprised you're having issues.
As for the no such file or directory messages: You need to make sure you have a running PostgreSQL server ("cluster") before you can use admin commands like createdb, because they make a connection to the server. The No such file or directory message is telling you that the server doesn't exist or isn't running.
Here's what's happening:
Ubuntu uses pg_wrapper to manage multiple concurrent PostgreSQL instances. The issues you're having are really with pg_wrapper.
Ideally you would've just used pg_dropcluster to get rid of the unwanted clusters. Unfortunately, by following bad advice it sounds like you've got your system into a bit of a messed-up state where the PostgreSQL packages are half-installed and kind of mangled. You need to either repair the install, or totally clean it out.
I'd clean it out. I'd recommend:
Verify that pg_lsclusters lists no database clusters
apt-get --purge remove postgresql\* - this is important
Remove /etc/postgresql/
Remove /etc/postgresql-common
Remove /var/lib/postgresql
userdel -r postgres
groupdel postgres
apt-get install postgresql-common postgresql-9.1 postgresql-contrib-9.1 postgresql-doc-9.1
It's possible that the apt-get --purge step will fail because you've removed the user IDs, etc. Re-creating the postgres user ID with useradd -r -u 109 postgres should allow you to re-run the purge successfully then delete the user afterwards.
This answer is not directly about removing a postgres instance, rather, about resoliving the issue,
Error: The cluster is owned by user ...
I got this error while trying to spin up a docker container pointed to a postgres data directory that was produced via a different container (on a different host machine).
The error is directly related to directory ownership. In my case, the system was unable to find the user that certain postgres directories was owned by in the current environment. By re-owning those directories to the right user resolves the issue. Following is an example mapping (that worked for me):
chown -R postgres:postgres /var/lib/postgresql
chown -R postgres:postgres /etc/postgresql
chown -R postgres:postgres /var/log/postgresql
chown -R postgres:postgres /var/run/postgresql