I'm working with an Azure Postgresql database and am using the Cloud Shell to run psql scripts without problems. I'm now trying to load some shp files via the shp2pgsql command. The cloud shell responds by:
bash: shp2pgsql: command not found
Is it possible at all to use shp2pgsql with the Cloud Shell or I'm missing something? I've already successfully created the postgis extension on the Postgresql server.
Unfortunately, it seems that you cannot run the shp2pgsql command in the Azure Cloud Shell. It is just an interactive, browser-accessible shell for managing Azure resources. Not integrated with too much tool in it because of its flexibility. You can get more details about the features from the Features & tools for Azure Cloud Shell.
I suggest if you want to do something complicated, you'd better run it in a specific Azure VM for yourself. Hope this will be helpful to you.
Related
Is there an official way of installing an extension on a GCP Postgress Cloud SQL instance via Terraform?
Closest I've found is this unofficial Postgres resource but it's not immediately clear how to link the two. This issue on their tracker sort of shows how, but far from a step by step guide.
if it makes any difference, I'm trying to provision a Postgres Cloud SQL instance with PostGIS.
Thanks.
Terraform is a deployment tool, to create all your infrastructure. To install an extension on Postgres, you need a installation tool, because you need to connect to the database and to run a command.
It's the same thing if you want to create a user in the database and you want to grant some privileges on it.
In summary, you can't achieve that with Terraform. I recommend you to have an installation tool, such as Ansible to perform this action.
An alternative is to create, with Terraform, a micro VM with a startup script that connect the database, run the command and destroy itself at the end.
I have a requirement to create the users in PostgreSQL DB using Jenkins pipeline. It will be greatly appreciated if someone could share reliable links. Currently I am stuck on how to start.
I am a newbie in Jenkins.
To connect Jenkins with a PSQL Database, you would first need to setup psql driver plugin in jenkins. You can add the plugin & manage the configuration from the Jenkins Setup.
Once you are done with this, the best way to interact with the db would be to write the commands you need to execute in a shell script file, like $HOME/db_scripts/add_user.sh which you can execute from the jenkins directly.
Hope this answers your question!
I have a need to load data from S3 to Postgres RDS (around 50-100 GB) I don't have the option to use AWS Data Pipeline and I am looking for something similar to using the COPY command to load data in S3 into Amazon Redshift.
I would appreciate any suggestions on how I can accomplish this.
Originally, this answer was trying to use the S3 to Postgres RDS Functionality. That whole enterprise failed (see below).
The way I have finally been able to do this is:
Set-up an EC2 instance with psql installed (see below near end of post)
Copy the relevant CSVs to import from S3 to the local instance
Use the psql /copy command to import the files up
This last part is really, really important. If you use the SQL COPY command the entire RDS Postgres role structure will frustrate you to no end. It has a wonky SUPERRDSADMIN role which is not very super at all. However, if you use the psql /copy commany you apparently can do anything. I have confirmed this be the case and have started my uploads succesfully. I will come back and re-edit this post (time permitting) to add relevant documentation steps for the above.
Caveat Emptor: The post below was all the original work I had done trying to get this implemented. I don't want to bury the lead despite multiple efforts (including what can only be described as pathetic tech support from AWS) I don't believe that this feature is ready for prime time. Despite a very simple test environment, easy to replicate, AWS has not provided an effective way to not get the copy statement to crap out as follows:
The actual call to aws_s3.table_import_from_s3(...) is reporting a permission problem between RDS and S3. From my research work with psql this appears to be a C library, probably installed by AWS.
NOTICE: CURL error code: 28 when attempting to validate pre-signed URL, 1 attempt(s) remaining
NOTICE: HINT: make sure your instance is able to connect with S3.
S3 to Postgres RDS Functionality Now Added
On 2019-04-24 AWS released functionality allowing a Postgres RDS to load directly from S3. You can read the announcement here, and see the documentation page here.
I am sharing with the OP because this appears to be the AWS supported way of solving the question posed.
Key summary points:
Requires Postgres 11.1 or greater
Need access to psql and the ability to connect it to the RDS instance
Need to install the aws_s3 extension which pulls in aws_commons.
You can get to the S3 bucket by specifying credentials or by assigning IAM roles to RDS
It advertises supporting all of the same data formats as the postgres COPY command
It currently only appears to support a single file at a time (ie no regex)
The instructions are fairly detailed and provide a variety of paths to configuring (AWS CLI scripts, Console instructions, etc). Additionally, the option to use your IAM keys rather than have to set-up roles is nice.
I did not find a way to download just psql, so I had to bring down a full postgres install down to my mac, but that was no big deal with brew:
brew install postgres
and since the DB service does not get activated it is the quickest way to get psql.
Update: Decided that having psql on my mac was a security hole, port forwarding, etc. I found that there is a simple Postgres install available for AMI Linux 2 under the AMI Extras rubric. The install command is fairly simple on your ami instance type.
sudo amazon-linux-extras install postgresql10
psql is fairly easy to use, however, important to keep in mind that any instructions to psql itself are escaped by a \. Documentation on psql can be found here. Recommend going through it at least once before executing the AWS recommended scripts.
To the extent you run tight security and have access to your RDS instances seriously restricted (which I do) don't forget to open up the ports from your AMI instance running Postgres to your RDS instance.
If your preference is a GUI then you can try to use PGAdmin4. It is the AWS recommended way of connecting to RDS Postgres instances according to the docs. I was unable to get any of the SSH tunneling features to work (which is why I ended up doing the localhost SSH mapping that I used for psql). I also found it to be rather buggy in other ways. Reading reviews of the product it seems that version 4 may not be the stablest of releases.
http://docs.aws.amazon.com/redshift/latest/dg/t_loading-tables-from-s3.html
Use the COPY command to load a table in parallel from data files on
Amazon S3. You can specify the files to be loaded by using an Amazon
S3 object prefix or by using a manifest file.
The syntax to specify the files to be loaded by using a prefix is as
follows:
copy <table_name> from 's3://<bucket_name>/<object_prefix>'
authorization;
update
Another option is to mount s3 and use direct path to the csv with COPY command. I'm not sure If it will hold 100GB effectively, but worth of trying. Here is some list of options on software.
Yet another option would be "parsing" s3 file part by part with something described here to a file and COPY from named pipe, described here
And the most obvious option to just download file to local storage and use COPY I don't cover at all
Also worth of mentioning would be s3_fdw (status unstable). Readme is very laconic, but I assume you could create a foreign table leading to s3 file. Which itself means you can load data to other relation...
I have my log files on EC2 instance and want to load it to Redshift. Two questions:
Do I have to copy this log file to S3 before proceeding or can I directly copy from my EBS Volume.
I can see I can use copy command from SQL Workbench or Data Pipeline. But can I use it from my EC2 instance itself ? Which AWS CLI I need to install?
http://docs.aws.amazon.com/cli/latest/reference/redshift/ does
not list copy command
Not really. Redshift allows you to copy from a remote host, which, in your case, would be your EC2 instance. Documentation here.
The link you've referred to provides cluster management commands. To run SQL queries on your cluster, you can use the psql tool. Documentation here.
you can copy the data directly from EC2, but my recommendation is to save it first on S3 , also for a backup
All the documentation available online was confusing me. Finally the solution was that I wrote a simple Java file with DriverManager.getConnection() and calling copy command via stmt.executeUpdate() and it worked seamlessly. Only executeUpdate() did not return me number of records Inserted.
I have a small app, and I run backups manually using heroku pgbackups:capture on my dev machine.
I'd like to use the Heroku Scheduler send these backups to my own S3 bucket.
The thing is: pg_dump is not an available on Heroku boxes, and heroku pgbackups:capture is a local CLI command, also not available.
Is there another way to achieve this using Scheduler?