I have successfully encrypted the communication in mongoDB but when I try to enable the data encryption I'm getting errors. I am using the enterprise edition of mongoDB with version 3.2.4. I get the following message in the console:
ERROR: child process failed, exited with error number 14
But when I look at the logs I see detailed error as follows:
Unable to retrieve key .system, error: there are existing data files, but no valid keystore could be located.
Fatal Assertion 28561
following is the snippet of my config file:
# enable authentication
security:
authorization: enabled
enableEncryption: true
encryptionKeyFile: /home/test/mongodb-keyfile
It works fine without the enableEncryption, and encryptionKeyFile parameters. Could anyone explain what i'm missing here? thanks!
MongoDB's encrypted storage engine supports two key management options:
Key Manager : Integration with third party key management appliance via the Key Management Interoperability Protocol (KMIP).
Local Key: Use of local key management via a keyfile.
Worth mentioning that using a key manager meets regulatory key management guidelines and is recommended over the local key management.
If you are using Key Manager option, please see KMIP Master Key Rotation.
Since you are using the Local Key option, if you have a Replica Set deployment you could rotate the replica set member. This would re-sync data from the un-encrypted to the encrypted mongod.
Alternatively if you only have a standalone mongod, you could:
Back Up your database files.
Stop mongod process.
Delete or move existing database files in dbpath. Exercise extra caution! - If you are deleting, make sure you have a backup data.
Restart mongod with --enableEncryption and --encryptionKeyFile.
Restore backup files to the restarted and encrypted mongod.
As an example, you could use mongodump to dump the data, and use mongorestore to restore the dump files.
Lastly, note that MongoDB Enterprise edition is a commercially supported product. I'd suggest opening a support case if you have a Commercial Support subscription.
Related
I used mongodb on my local without authentication by running mongod as server and connecting it by just mongo as a user.
Now after creating users and running the server as mongod --auth i can connect using user id and pwd assigned by me.
The problem is while connecting back mongo with mongod it again gets connected to the database and whole database is visible again,this should not happen as I have enabled the authentication.
Using mongod as a server should block the users from getting access. Same is the problem for my personal server.
Is there any solution to enable the security permanently that is enable mongod --auth and not just mongod.
If I understand correctly, your concern is that restarting a mongod process without access control enabled allows you to access data without authentication. This is the expected outcome: a user with direct access to restart services or copy MongoDB data files has greater permissions than the mongod process.
Authentication only verifies remote client access to the mongod service: it does not encrypt or protect data files if a user has direct local read access. If you have an environment allowing remote and/or multi-user access, you should take steps to appropropriately limit access to your data files and service management.
If you want to secure data files from direct access, it sounds like you may be looking for Encryption at Rest or Disk Encryption. These security measures prevent direct read access to your data files or filesystem without appropriate credentials, but you still have to limit access appropriately and protect those credentials.
For general measures see the MongoDB Security Checklist.
I have a need to load data from S3 to Postgres RDS (around 50-100 GB) I don't have the option to use AWS Data Pipeline and I am looking for something similar to using the COPY command to load data in S3 into Amazon Redshift.
I would appreciate any suggestions on how I can accomplish this.
Originally, this answer was trying to use the S3 to Postgres RDS Functionality. That whole enterprise failed (see below).
The way I have finally been able to do this is:
Set-up an EC2 instance with psql installed (see below near end of post)
Copy the relevant CSVs to import from S3 to the local instance
Use the psql /copy command to import the files up
This last part is really, really important. If you use the SQL COPY command the entire RDS Postgres role structure will frustrate you to no end. It has a wonky SUPERRDSADMIN role which is not very super at all. However, if you use the psql /copy commany you apparently can do anything. I have confirmed this be the case and have started my uploads succesfully. I will come back and re-edit this post (time permitting) to add relevant documentation steps for the above.
Caveat Emptor: The post below was all the original work I had done trying to get this implemented. I don't want to bury the lead despite multiple efforts (including what can only be described as pathetic tech support from AWS) I don't believe that this feature is ready for prime time. Despite a very simple test environment, easy to replicate, AWS has not provided an effective way to not get the copy statement to crap out as follows:
The actual call to aws_s3.table_import_from_s3(...) is reporting a permission problem between RDS and S3. From my research work with psql this appears to be a C library, probably installed by AWS.
NOTICE: CURL error code: 28 when attempting to validate pre-signed URL, 1 attempt(s) remaining
NOTICE: HINT: make sure your instance is able to connect with S3.
S3 to Postgres RDS Functionality Now Added
On 2019-04-24 AWS released functionality allowing a Postgres RDS to load directly from S3. You can read the announcement here, and see the documentation page here.
I am sharing with the OP because this appears to be the AWS supported way of solving the question posed.
Key summary points:
Requires Postgres 11.1 or greater
Need access to psql and the ability to connect it to the RDS instance
Need to install the aws_s3 extension which pulls in aws_commons.
You can get to the S3 bucket by specifying credentials or by assigning IAM roles to RDS
It advertises supporting all of the same data formats as the postgres COPY command
It currently only appears to support a single file at a time (ie no regex)
The instructions are fairly detailed and provide a variety of paths to configuring (AWS CLI scripts, Console instructions, etc). Additionally, the option to use your IAM keys rather than have to set-up roles is nice.
I did not find a way to download just psql, so I had to bring down a full postgres install down to my mac, but that was no big deal with brew:
brew install postgres
and since the DB service does not get activated it is the quickest way to get psql.
Update: Decided that having psql on my mac was a security hole, port forwarding, etc. I found that there is a simple Postgres install available for AMI Linux 2 under the AMI Extras rubric. The install command is fairly simple on your ami instance type.
sudo amazon-linux-extras install postgresql10
psql is fairly easy to use, however, important to keep in mind that any instructions to psql itself are escaped by a \. Documentation on psql can be found here. Recommend going through it at least once before executing the AWS recommended scripts.
To the extent you run tight security and have access to your RDS instances seriously restricted (which I do) don't forget to open up the ports from your AMI instance running Postgres to your RDS instance.
If your preference is a GUI then you can try to use PGAdmin4. It is the AWS recommended way of connecting to RDS Postgres instances according to the docs. I was unable to get any of the SSH tunneling features to work (which is why I ended up doing the localhost SSH mapping that I used for psql). I also found it to be rather buggy in other ways. Reading reviews of the product it seems that version 4 may not be the stablest of releases.
http://docs.aws.amazon.com/redshift/latest/dg/t_loading-tables-from-s3.html
Use the COPY command to load a table in parallel from data files on
Amazon S3. You can specify the files to be loaded by using an Amazon
S3 object prefix or by using a manifest file.
The syntax to specify the files to be loaded by using a prefix is as
follows:
copy <table_name> from 's3://<bucket_name>/<object_prefix>'
authorization;
update
Another option is to mount s3 and use direct path to the csv with COPY command. I'm not sure If it will hold 100GB effectively, but worth of trying. Here is some list of options on software.
Yet another option would be "parsing" s3 file part by part with something described here to a file and COPY from named pipe, described here
And the most obvious option to just download file to local storage and use COPY I don't cover at all
Also worth of mentioning would be s3_fdw (status unstable). Readme is very laconic, but I assume you could create a foreign table leading to s3 file. Which itself means you can load data to other relation...
I'm trying to migrate a PostgreSQL DB persisted on cloud (on DO droplet) to RDS using AWS Database Migration Service (DMS).
I've successfully configured the replication instance and endpoints.
I've created a task with Migrate existing data and replicate ongoing changes. When I start the task it shows some error ERROR: could not access file "test_decoding": No such file or directory.
I've tried to create a replication slot manually on my DB console it throws the same error.
I've followed the procedures which was suggested on the DMS documentation for Postgres
I'm using PostgreSQL 9.4.6 on my source endpoint.
I presume that the problem is the output plugin test_decoding was not accessible to do the replication.
Please assist me to resolve this. Thanks in advance!
You must install postgresql-contrib additional supplied modules on Your source endpoint.
If it is installed, make sure, directory where test_decoding module located is the same with directory where PostgreSQL expect it.
In *nix, You can check module directory by command:
pg_config --pkglibdir
If it is not the same, copy module, or make symlink, or some other solution You prefer.
We are trying to encrypt Postgres data at rest. Can't find any documentation to encrypt Postgres data folder using LUKS with dm-encrypt.
No special instructions are necessary – PostgreSQL will use the opened encrypted filesystem just like any other file system. Just point initdb to a directory in the opened file system, and it will create a PostgreSQL cluster there.
Automatic server restarts will fail, because you need to enter the passphrase.
Of all the ways to protect a database, encrypting the file system is the least useful:
Usually, attacks on a database happen via the client, normally with SQL injection. Encrypring the file system won't help.
The other common attack vector are backups. Backups done with pg_dump or pg_basebackup are not encrypted.
But I guess you know why you need it.
I am migrating from server OLD at the old hosting company to server NEW at the new hosting company.
I want to run the clone command so I clone the mongoDB from OLD to NEW.
For OLD:
The public ip address is: 44.55.66.77.
The machine login user name is: admin, and the password is password
What is the right way to do this?
So far I can't even log into the server OLD
So far I have tried the following command prompts on NEW:
mongo -u admin -p password 44.55.66.77
mongo remote-ip:44.55.77.66 -u admin -p password
That don't work
I also tried this from mongo shell:
db.CopyDatabase('OldDb', 'NewDb', '44.55.66.77', 'admin', 'password')
and I get: the "could not connect to server" error message
Aside from firewall considerations in order to copy data between MongoDB servers, db.copyDatabase() (aka the copydb command) has a number of important usage caveats including:
copydb does not produce point-in-time snapshots of the source database; writing data to the source or destination database during the copy process will result in divergent data sets
copydb does not lock the destination server during its operation, so the copy will occasionally yield to allow other operations to complete.
There is also a known issue that copydb may not work with the role-based privileges in MongoDB 2.4 if you have authentication enabled (see SERVER-8213, which was recently fixed in the 2.5.x development releases).
A much better approach to migrating your data would be to restore from a normal backup using mongodump/mongorestore or file system snapshots. The Backup & Recovery section of the MongoDB manual has tutorials covering procedures for different deployment types.