We are trying to encrypt Postgres data at rest. Can't find any documentation to encrypt Postgres data folder using LUKS with dm-encrypt.
No special instructions are necessary – PostgreSQL will use the opened encrypted filesystem just like any other file system. Just point initdb to a directory in the opened file system, and it will create a PostgreSQL cluster there.
Automatic server restarts will fail, because you need to enter the passphrase.
Of all the ways to protect a database, encrypting the file system is the least useful:
Usually, attacks on a database happen via the client, normally with SQL injection. Encrypring the file system won't help.
The other common attack vector are backups. Backups done with pg_dump or pg_basebackup are not encrypted.
But I guess you know why you need it.
Related
This might be a simple question, but I would like some clarification.
Based off the docs, Heroku has an ephemeral file system. How I interpret it is that anytime you upload a file to Heroku and there is a change in the configuration or the app is restarted, the files are gone.
However, I was wondering if this is the case if you upload data to Heroku Postgres through a dumps file.
For development, I am using a local Postgres server. From there, I would create a dumps file and then upload that file using commands found here:
https://stackoverflow.com/a/71206831/3100570
Now suppose my application makes a POST request to Heroku Postgres, would that data be persisted along with the initial data from the dumps file in the event that the application is restarted or crashed?
Ingesting data into your PostgreSQL database this way doesn't touch your dyno's filesystem. You are simply connecting to PostgreSQL and running the SQL commands contained in that file:
-f, --file=file
SQL file to run
The data will be stored in PostgreSQL in exactly the same way it would if you did a bunch of INSERTs yourself. You should have no problem ingesting data this way and then continuing to interact with your application as normal.
In PostgreSQL we can just change local md5 to trust in pg_hba.conf. then we can access all data in database using psql without need of password.So anyone can change this line who can access local machine.
So, Is there way to password protect our database even someone change pg_hba.conf to trust
( I want to create offline app and need to protect client database,I need something like ms access, once we set the password it always ask for password )
As long as client has root/administrator acces on the computer you can't do much about pg_hba. You could make it read only but root can overyde anything. You could mount config file on read only file system but this is too complicated.
Solution can be only at database level(not OS or application): crypted data and triggers where you implement supplimentary security.
I don't think postresql is the answer for your requirement, maybe SQLite is the right one.
I have a need to load data from S3 to Postgres RDS (around 50-100 GB) I don't have the option to use AWS Data Pipeline and I am looking for something similar to using the COPY command to load data in S3 into Amazon Redshift.
I would appreciate any suggestions on how I can accomplish this.
Originally, this answer was trying to use the S3 to Postgres RDS Functionality. That whole enterprise failed (see below).
The way I have finally been able to do this is:
Set-up an EC2 instance with psql installed (see below near end of post)
Copy the relevant CSVs to import from S3 to the local instance
Use the psql /copy command to import the files up
This last part is really, really important. If you use the SQL COPY command the entire RDS Postgres role structure will frustrate you to no end. It has a wonky SUPERRDSADMIN role which is not very super at all. However, if you use the psql /copy commany you apparently can do anything. I have confirmed this be the case and have started my uploads succesfully. I will come back and re-edit this post (time permitting) to add relevant documentation steps for the above.
Caveat Emptor: The post below was all the original work I had done trying to get this implemented. I don't want to bury the lead despite multiple efforts (including what can only be described as pathetic tech support from AWS) I don't believe that this feature is ready for prime time. Despite a very simple test environment, easy to replicate, AWS has not provided an effective way to not get the copy statement to crap out as follows:
The actual call to aws_s3.table_import_from_s3(...) is reporting a permission problem between RDS and S3. From my research work with psql this appears to be a C library, probably installed by AWS.
NOTICE: CURL error code: 28 when attempting to validate pre-signed URL, 1 attempt(s) remaining
NOTICE: HINT: make sure your instance is able to connect with S3.
S3 to Postgres RDS Functionality Now Added
On 2019-04-24 AWS released functionality allowing a Postgres RDS to load directly from S3. You can read the announcement here, and see the documentation page here.
I am sharing with the OP because this appears to be the AWS supported way of solving the question posed.
Key summary points:
Requires Postgres 11.1 or greater
Need access to psql and the ability to connect it to the RDS instance
Need to install the aws_s3 extension which pulls in aws_commons.
You can get to the S3 bucket by specifying credentials or by assigning IAM roles to RDS
It advertises supporting all of the same data formats as the postgres COPY command
It currently only appears to support a single file at a time (ie no regex)
The instructions are fairly detailed and provide a variety of paths to configuring (AWS CLI scripts, Console instructions, etc). Additionally, the option to use your IAM keys rather than have to set-up roles is nice.
I did not find a way to download just psql, so I had to bring down a full postgres install down to my mac, but that was no big deal with brew:
brew install postgres
and since the DB service does not get activated it is the quickest way to get psql.
Update: Decided that having psql on my mac was a security hole, port forwarding, etc. I found that there is a simple Postgres install available for AMI Linux 2 under the AMI Extras rubric. The install command is fairly simple on your ami instance type.
sudo amazon-linux-extras install postgresql10
psql is fairly easy to use, however, important to keep in mind that any instructions to psql itself are escaped by a \. Documentation on psql can be found here. Recommend going through it at least once before executing the AWS recommended scripts.
To the extent you run tight security and have access to your RDS instances seriously restricted (which I do) don't forget to open up the ports from your AMI instance running Postgres to your RDS instance.
If your preference is a GUI then you can try to use PGAdmin4. It is the AWS recommended way of connecting to RDS Postgres instances according to the docs. I was unable to get any of the SSH tunneling features to work (which is why I ended up doing the localhost SSH mapping that I used for psql). I also found it to be rather buggy in other ways. Reading reviews of the product it seems that version 4 may not be the stablest of releases.
http://docs.aws.amazon.com/redshift/latest/dg/t_loading-tables-from-s3.html
Use the COPY command to load a table in parallel from data files on
Amazon S3. You can specify the files to be loaded by using an Amazon
S3 object prefix or by using a manifest file.
The syntax to specify the files to be loaded by using a prefix is as
follows:
copy <table_name> from 's3://<bucket_name>/<object_prefix>'
authorization;
update
Another option is to mount s3 and use direct path to the csv with COPY command. I'm not sure If it will hold 100GB effectively, but worth of trying. Here is some list of options on software.
Yet another option would be "parsing" s3 file part by part with something described here to a file and COPY from named pipe, described here
And the most obvious option to just download file to local storage and use COPY I don't cover at all
Also worth of mentioning would be s3_fdw (status unstable). Readme is very laconic, but I assume you could create a foreign table leading to s3 file. Which itself means you can load data to other relation...
A linux VM with postgres 9.4 was hacked into. (Two processes taking 100% cpu, weird files in /tmp, did not reoccur after kill(s) and restart.) It was decided to install the system from scratch on a new machine (with postgres 9.6). The only data needed was in one of postgres databases. A pg_dump of the database was made after the attack.
Regardless of whether the data - the tables/rows/etc. - were modified during the attack: is it safe to restore the database in the new system?
I consider using pg_restore with the -O option (ignores the user permissions)
The two dangers are:
important data could have been modified
back doors could have been installed in your database
With the first, you're on your own how to verify that your data are ok. The safest thing would be to use a backup from before the machine was compromized, but this would mean data loss.
For the second, I would run a pg_dumpall -s and spend a day reading it carefully. Compare it with a dump from a backup made before the breach. Watch out for weird object and column names and functions with SECURITY DEFINER.
I am backing postgre SQL db using pgadmin.
However, while the db has a password the backup file has no protection. The backup file can be taken and restored to any blank database without any password !!
How do I make the backup secure / password protected? Thanks.
PostgreSQL and PgAdmin have no built-in facility for backup encryption and password protection.
There are numerous ways you can encrypt backups. You can use gnupg, but I suspect you will be happier using a zip utility that supports encryption. I recommend 7zip, which supports the zip64 format and zip file encryption.
You might want to use a tool which supports streaming encryption - and avoid writing your plain text backup to disk. Also don't use pgAdminIII, but use pg_dump instead, and pipe the backup output into your encryption program, and write the result to disk.