psql: FATAL: PAM authentication failed for user " " aws rds Postgresql - postgresql

Question

I want to connect to aws rds POSTGRESQL in dev from my own computer.
I followed all the steps on how to do it from bunch of articles:
https://aws.amazon.com/premiumsupport/knowledge-center/rds-postgresql-connect-using-iam/
https://aws.amazon.com/blogs/database/using-iam-authentication-to-connect-with-pgadmin-amazon-aurora-postgresql-or-amazon-rds-for-postgresql/.
The problem is if I create the database in aws console interface, I am able to log in ONLY once.
psql -h database.xxxxxxxx.us-west-2.rds.amazonaws.com -U user_name -d database
Other times I try to log in with the same any other command, I get
psql: FATAL: PAM authentication failed for user "user_name"
First and only time I login, I create a user
CREATE USER user_name WITH LOGIN;
GRANT rds_iam TO user_name;
All other attempts including the other steps logging with the iam token etc, I get an error:
psql: FATAL: PAM authentication failed for user "user_name"
If I delete the database from aws console interface and then create a brand new one, I am able to log in only ONCE and and then get the error no matter what I do.
nc Command gives me Connection succeeded at all times I run it:
nc -zv DB-instance-endpoint port
The commands I am using :
export RDSHOST="database.xxxxxxxx.us-west-2.rds.amazonaws.com"
export PGPASSWORD="$(aws rds generate-db-auth-token --hostname $RDSHOST --port 5432 --region us-west-2 --username user_name)"
I get the error if I use the PGPASSWORD in pgAdmin window.
Also, I am trying to connect from the the terminal either mine or ssh into ec2, I use this command:
psql "host=$RDSHOST port=5432 sslmode=verify-full sslrootcert=./rds-combined-ca-bundle.pem dbname=database user=user_name"
and I still get the same error
psql: FATAL: PAM authentication failed for user "user_name"
or
If I use another command, without the .pem certificate
psql --host=database.xxxxxxxx.us-west-2.rds.amazonaws.com --port=5432 --username=user_name --password --dbname=database
Then it asks me for a password and Then I get this error
psql: error: FATAL: PAM authentication failed for user "user_name"
FATAL: pg_hba.conf rejects connection for host "222.22.22.22", user "user_name", database "database", SSL off
"222.22.22.22" is My Ip, I changed it of course.
I attached all the required and all the RDS access Policies to my user and still getting this error.
I am just no sure what to do at this point as I went through every single article and cannot find a solution.

Answer 1

I had a similar problem and after some playing around with psql utility I found the reason for these errors. You shall export your temporary database password/token to shell of the machine/service etc where the connection will be initiated from.
So, if psql connection is initiated from Bastion, the below command should also be run on the same Bastion server.
export PGPASSWORD="$(aws rds generate-db-auth-token --hostname $RDSHOST --port 5432 --region us-west-2 --username user_name)"
or generate it elsewhere and export its value as
export PGPASSWORD="temporary_token_generated_for_user_name"
With this exported $PGPASSWORD variable, it psql should connect straight away, without promting for any additional passwords

Answer 2

I found the solution finally. So if anyone has the same issue and goes nuts about it, here is the solution:
If everything is working as I described above and the only error you get is PAM.. then:
your config file is not properly set up. It does not have the username you are trying to connect, the region, and the keys.
~/.aws/config
[profile PROFILE_NAME]
output=json
region=us-west-1
aws_access_key_id=foo
aws_secret_access_key=bar
Here is the link to the question on how to set it up:
AWS : The config profile (MyName) could not be found