Invoking AWS Lambda with postgresql - postgresql

I have a postgresql db instance on AWS, I also have a lambda function.
I want to invoke this lambda function through a trigger function in my postgresql db.
I found this information: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/PostgreSQL-Lambda.html#PostgreSQL-Lambda-errors
I have created a policy named "aws_lambda_rds_invoke_policy" with the permission lambda:invokeFunction. Resource is '*' because I want this policy to work with all (future) lambda functions.
I have created a role named "aws_lambda_rds_role" (Trusted entities: AWS Service: rds) with this policy attached.
I assume this role is automatically attached to my rds? (If not, how can I attach it?)
I have the following SQL-code:
SELECT * FROM aws_lambda.invoke(
aws_commons.create_lambda_function_arn('my-lambda-function-arn', 'eu-central-1'),
'{"hello":"world"}'::json
);
I get the following error:
ERROR: unable to access credentials stored with the database instance
HINT: Make sure that the desired Amazon Resource Name (ARN) is associated with the feature-name: "Lambda".
CONTEXT: SQL function "invoke" statement 1. 0.000 seconds. (Line 1).
When I delete the role I just created and run the SQL code again, it works?!
Don't get me wrong, I'm happy it works, but I'm wondering why, because I didn't "authorize" my db instance to run lambda functions.
Edit:
A few moments later, it stopped working. Could it be I just have to wait a few minutes before all changes (like adding a new role) are finished?

I had the exact same challenge. Below is all I did and works ticketty boo.
Create Aurora Serverless 2.0 - PostgreSQL 13.7 on aarch64-unknown-linux-gnu, compiled by aarch64-unknown-linux-gnu-gcc (GCC) 7.4.0, 64-bit
Install aws_lambda extension
CREATE EXTENSION IF NOT EXISTS aws_lambda CASCADE;
NOTICE: installing required extension "aws_commons"
CREATE EXTENSION
Create Role With Invoke Lambda Policy to Specific Function and Attach Role to above RDS
Example Policy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "lambda:InvokeFunction",
"Resource": "arn:aws:lambda:region-name-here:rds-id-here:function:function-name-here"
}
]
}
Execute query
SELECT * FROM aws_lambda.invoke('arn-of-lambda-function', '{"body":"foo"}'::json);
Works for me. Hope it helps!

Related

Error restoring db in Azure PostgreSQL Flexible server: 'extension "azure" is not allow-listed'

I'm using pgAdmin to backup/restore a database from one Azure PostgreSQL Flexible server to another. The source server was created last year, the destination server is new.
The restore process fails early with the error:
ERROR: extension "azure" is not allow-listed for "azure_pg_admin" users in Azure Database for PostgreSQL
I came across this post https://techcommunity.microsoft.com/t5/azure-database-for-postgresql/introducing-ability-to-allow-list-extensions-in-postgresql/ba-p/3219124 announcing recent changes to PostgreSQL Flexible Server. If I'm reading this correctly, my new database server is affected by this change and I need to allow specific extensions under the "azure.extensions" server parameter.
In the backup file I can see:
CREATE EXTENSION IF NOT EXISTS azure WITH SCHEMA public;
CREATE EXTENSION IF NOT EXISTS "uuid-ossp" WITH SCHEMA public;
And in Azure Portal I can see "UUID-OSSP" under the new "azure.extensions" server parameter, though there's nothing called just "azure". I enabled UUID-OSSP but the restore process still fails with the same error.
What am I missing here?
It's suggestable to install TimescaleDB that is having the package of supportive extension.
To learn abot TimescaleDB. (https://docs.timescale.com/timescaledb/latest/)
Change the Postgres's parameters for "shared_preload_libraries"
After creating extension
CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;
Follow the entire procedure from the below link.
https://learn.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-extensions

How can I remove an enum label from pg_enum table in Google Cloud SQL?

I have a Node app (Express) that I built recently, which uses Sequelize to connect to a PostgreSQL instance. I just deployed this to GCP and used Cloud SQL to set up the database. This works quite well and the app properly connects to the DB instance.
However, when running the migrations I have I get this error: permission denied for table pg_enum
This happens on a single migration that I have which tries to remove an enum value from the database:
module.exports = {
up: (queryInterface, Sequelize) =>
queryInterface.sequelize.query(
`DELETE FROM pg_enum WHERE enumlabel = 'to' AND enumtypid = (SELECT oid FROM pg_type WHERE typname = 'enum_Subscriptions_emailType')`
),
down: (queryInterface, Sequelize) =>
queryInterface.sequelize.query(`ALTER TYPE "enum_Subscriptions_emailType" ADD VALUE 'to';`)
}
I've read here that since Cloud SQL is a managed service, it doesn't provide superuser privileges to customers like me. Is there some other way that I can get this migration run?
I've also tried running a Cloud Build process, but that also fails with the following error: ERROR: connect ENOENT /cloudsql/<project-id>:<project-region>:<db-instance>/.s.PGSQL.5432. For reference, my cloudbuild.yaml file looked like this:
steps:
- name: 'gcr.io/cloud-builders/yarn'
args: ['install']
- name: 'gcr.io/cloud-builders/yarn'
args: ['migrate']
- name: 'gcr.io/cloud-builders/gcloud'
args: ['app', 'deploy']
and my package.json scripts were:
"scripts": {
"start": "node index.js",
"migrate": "npx sequelize-cli db:migrate",
"dev": "set DEBUG=app:* && nodemon index.js",
"deploy": "gcloud builds submit --config cloudbuild.yaml ."
}
What else can I do to get around this, so that the migrations I have following this one can run and my app can function?
Removing a value from an enum is not supported by PostgreSQL. You can only add new ones or rename existing ones.
While it might work somewhat reliably by modifying the system catalogue, even this is not officially supported and needs superuser permissions for a reason - so there is no way to do it without.
The supported way to do what you want to do is to recreate the type without the value.
CREATE TYPE new_enum AS ENUM('a','b','c');
ALTER TABLE table_using_old_enum
ALTER COLUMN colum_using_old_enum
SET DATA TYPE new_enum
USING colum_using_old_enum::text::new_enum;
DROP TYPE old_enum;
ALTER TYPE new_enum RENAME TO old_enum;
This obviously only works if no entry in table_using_old_type is still set to the value that you want to remove - you need to ensure this first, otherwise the typecast in the USING clause will fail for those values.
--
Alternatively, you could also just comment out that query - if you already know that a few minutes later with a later migration the type will be removed completely, it shouldn't cause an issue if that value stays in until then.

AWS RDS: Automate postgres extension installation

I have a RDS Postgres instance. I have launched this instance using CloudFormation template. I am able to add PostGIS extension on this instance.
I will be using this template to launch multiple RDS instance frequently and every time I would need PostGIS extension on the instance. Is there any way, I can automate installation of Postgres extensions on RDS.
According to the RDS documentation the only way to enable this is by running the below SQL statement.
create extension postgis;
CREATE EXTENSION
create extension fuzzystrmatch;
CREATE EXTENSION
create extension postgis_tiger_geocoder;
CREATE EXTENSION
create extension postgis_topology;
CREATE EXTENSION
For this to happen a resource needs to be able to perform the SQL statement which requires an additional resource in order to apply this statement.
CloudFormation supports custom resources which allow a Lambda function to perform custom functionality. This Lambda would need to be able to connect to your RDS to perform the SQL command.

Amazon Postgres RDS pg_stat_statements not loaded

I configured my RDS Postgres 9.6.1 instance so, that the pg_stat_activity is loaded via 'shared_preload_libraries' parameter
shared_preload_libraries=pg_stat_statements,pg_hint_plan
The I rebooted my instance and then check if the pg_stat_statements can be loaded. But I get the error:
[55000] ERROR: pg_stat_statements must be loaded via shared_preload_libraries
The command
SHOW shared_preload_libraries;
does not show the pg_stat_statements library.
And therefore the error above still remains.
Does somebody else has got the same issue?
Ran into the same problem today.
Turns out I simply had to reboot the database (via CLI or web interface).
As mentioned on the AWS docs:
When you change a static parameter and save the DB parameter group, the parameter change takes effect after you manually reboot the DB instance.
UPDATE
I tried myself. Added to Parameters->shared_preload_libraries values:pg_stat_statements,pg_hint_plan. Clicked "save changes", agreed on "Apply Immediately". Indeed no effect. When I check DB Parameter Group - it shows modified group, but pg_stat_statements unusable. So I think bad on JS and try aws cli:
aws rds describe-db-instances --db-instance-identifier p5
it reveals that group was not indeed changed!:
"OptionGroupMemberships": [
{
"Status": "in-sync",
"OptionGroupName": "default:postgres-9-5"
}
],
So I did:
aws rds modify-db-instance --db-parameter-group-name with-contrib --apply-immediately --db-instance-identifier p5
aws rds reboot-db-instance --db-instance-identifier p5
aws rds describe-db-instances --db-instance-identifier p5
And It was changed. So I try to:
mon=> create extension pg_stat_statements ;
CREATE EXTENSION
mon=> select count(*) from pg_stat_statements;
count
-------
26
(1 row)
Voila. Not sure if it was Chrome (Version 57.0.2987.133 (64-bit)) feature or JS failure, or me not getting intuitive clicking steps, but I failed to apply changes as well. Please try with aws cli to see if you success there.
PREVIOUS
String requires quotes I think.:
String: In general, enclose the value in single quotes, doubling any
single quotes within the value. Quotes can usually be omitted if the
value is a simple number or identifier, however.
So maybe setting shared_preload_libraries this way will help:
shared_preload_libraries = 'pg_stat_statements,pg_hint_plan'
https://www.postgresql.org/docs/current/static/runtime-config-client.html:
shared_preload_libraries (string)
This variable specifies one or more shared libraries to be preloaded
at server start. This parameter can only be set at server start. If a
specified library is not found, the server will fail to start.

How to pipe data from AWS Postgres RDS to S3 (then Redshift)?

I'm using AWS data pipeline service to pipe data from a RDS MySql database to s3 and then on to Redshift, which works nicely.
However, I also have data living in an RDS Postres instance which I would like to pipe the same way but I'm having a hard time setting up the jdbc-connection. If this is unsupported, is there a work-around?
"connectionString": "jdbc:postgresql://THE_RDS_INSTANCE:5432/THE_DB”
Nowadays you can define a copy-activity to extract data from a Postgres RDS instance into S3. In the Data Pipeline interface:
Create a data node of the type SqlDataNode. Specify table name and select query
Setup the database connection by specifying RDS instance ID (the instance ID is in your URL, e.g. your-instance-id.xxxxx.eu-west-1.rds.amazonaws.com) along with username, password and database name.
Create a data node of the type S3DataNode
Create a Copy activity and set the SqlDataNode as input and the S3DataNode as output
this doesn't work yet. aws hasnt built / released the functionality to connect nicely to postgres. you can do it in a shellcommandactivity though. you can write a little ruby or python code to do it and drop that in a script on s3 using scriptUri. you could also just write a psql command to dump the table to a csv and then pipe that to OUTPUT1_STAGING_DIR with "staging: true" in that activity node.
something like this:
{
"id": "DumpCommand",
"type": "ShellCommandActivity",
"runsOn": { "ref": "MyEC2Resource" },
"stage": "true",
"output": { "ref": "S3ForRedshiftDataNode" },
"command": "PGPASSWORD=password psql -h HOST -U USER -d DATABASE -p 5432 -t -A -F\",\" -c \"select blah_id from blahs\" > ${OUTPUT1_STAGING_DIR}/my_data.csv"
}
i didn't run this to verify because it's a pain to spin up a pipeline :( so double check the escaping in the command.
pros: super straightforward and requires no additional script files to upload to s3
cons: not exactly secure. your db password will be transmitted over the wire without encryption.
look into the new stuff aws just launched on parameterized templating data pipelines: http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-custom-templates.html. it looks like it will allow encryption of arbitrary parameters.
AWS now allow partners to do near real time RDS -> Redshift inserts.
https://aws.amazon.com/blogs/aws/fast-easy-free-sync-rds-to-redshift/