I have a PostgreSQL DB sitting on my local machine (Windows) and I would like to import it into my Hortonworks Sandbox using Apache Sqoop. While something like this sounds great, the complicating factor is that my Sandbox is sitting in a Docker container, so statements such as sqoop list-tables --connect jdbc:postgresql://127.0.0.1/ambari --username ambari -P seem to run into authentication errors. I believe the issue comes from trying to connect to the local host from inside the docker container.
I looked at this post on connecting to a MySQL DB from within a container and this one to try to use PostgreSQL instead, but have so far been unsuccessful. I have tried connecting to '127.0.0.1' and '172.17.0.1' (the host's IP) in order to connect to my local host from within Docker. I have also adjusted PostgreSQL's configuration file to listen for connections on all IP addresses. However, I still get the following error messages when I run sqoop list-tables --connect jdbc:postgresql://<ip>:5432/<db_name> --username postgres -P (where <ip> is either 127.0.0.1 or 172.17.0.1, and <db_name> is the name of my database)
For connecting with 127.0.0.1:
ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: org.postgresql.util.PSQLException: FATAL: Ident authentication failed for user "postgres"
For connecting with 172.17.0.1:
Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
Any suggestions would be very helpful!
If this is just for local testing and not for production level coding, you can enable all trusted connections to your database by updating the pg_hba.conf file
Locate your pg_hba.conf file inside your postgres data directory
Vim the file and update it with the following lines:
#TYPE DATABASE USER ADDRESS METHOD
local all all trust
host all all 0.0.0.0/0 trust
host all all ::1/128 trust
Restart your postgres service
If you do this, your first use case (using 127.0.0.1) should work
Related
I am trying to install Postgres as a backend DB instead of Sqllite. I followed these steps but ran into an error. can someone help, please?
Steps followed:
activated virtual env.
installed postgreSQL
sudo apt-get install postgresql postgresql-contrib
created psql object with:
sudo -u postgres psql
4.create the user and database for the airflow
CREATE USER **** PASSWORD '****';
postgres=# CREATE DATABASE airflow;
postgres=# GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO newt;
updated postgresql.conf with listen_addresses= '*'
added pg_hba.conf with
host db-name user-name ip-address trust
updated airflow.cfg file with
sql_alchemy_conn = postgresql+psycopg2://user-name:password#localhost:5432/db-name
restarted airflow db.
airflow resetdb
After the 8th steps I got below error, what am i doing wrong here? any advise
Error message
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?
After changing the pg_hba.conf and postgresql.conf you have to tell PostreSQL that a change has been made. You can do it by either restarting the database, e.g. on Ubuntu
$ sudo service postgresql restart
... or by executing the function pg_reload_conf
SELECT pg_reload_conf():
In case client and database are in different machines, make sure that there is no firewall between them messing things up.
Unrelated note: Also keep in mind that setting a connection to trust in the pg_hba.conf means that no password is required. For testing purposes more than legit but quite dangerous if forgotten in production ;)
I am trying to connect to a VM ubuntu from my local computer. On the VM I have created a postgresql database. Then I followed all steps that I could find on several tutorial when it comes to allow access for remote connexion to the db:
add host all all 0.0.0.0/0 trust to the pg_hba.conf file
add listen_addresses = '*' to the postgresql.conf file
remove the firewall by executing sudo ufw allow 5432/tcp
restart postresql by executing sudo systemctl restart postgresql
By using the command psql and then \c gives You are now connected to database "postgres" as user "postgres".
I then create a password by executing ALTER USER postgres WITH PASSWORD 'password';
Then I use postico to connect locally. I want to connect with this same default superuser postgres for testing. I use the VM ip address as host, db name is postgres user postgres, password is password and port 5432 as mentioned above.
After trying to connect for a while with the message opening connection to server, the result is:
could not connect to server: Operation timed out
Is the server running on host "***.**.**.***" and accepting
TCP/IP connections on port 5432?
Any help please? I did all steps I could find on many tutorials but still failing to connect.
I'm trying to setup postgres cluster of two nodes (primary and standby). In order to activate automatic failover, I'm using pgpool-II.
I followed the following article: https://www.pgpool.net/docs/41/en/html/example-cluster.html
and the only thing difference I did is installing postgresql version 12 instead of version 11.
Knowing that I'm trying it useing two centos7 images on VMware. I faced the following issues:
When I run systemctl status pgpool.service on both nodes, it returned success.
Also I can access postgresql using the watchdog delegate IP.
But what testing failover, everything goes wrong.
Scenario 1:
I accessed my database using watchdog delegate Ip.
I disconnect the standby server.
Result: My session to postgresql continued to work for less than a minute and then it failed. and I'm unable to connect again, until I reconnect the standby node, and restart the pgpool service again.
Scenario 2:
I accessed my database using watchdog delegate Ip.
I disconnect the primary server.
Result: My session stopped directly. and the standby server is not promoted to be master.
I noticed something (might be related to the above described problem): when I try to run the following command
psql 192.168.220.146 -p 9999 -U postgres -c "show pool_nodes"
it fails to work and returned the following:
psql: error: could not connect to server: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.9999"
However if I ran: psql 192.168.220.160 -p 5432 -U postgres
it works fine and I can access the postgres shell.
My pool_hba file:
# "local" is for Unix domain socket connections only
local all all trust
# IPv4 local connections:
host all all 127.0.0.1/32 trust
host all all ::1/128 trust
host all pgpool 0.0.0.0/0 scram-sha-256
host all postgres 0.0.0.0/0 scram-sha-256
Any help would be appreciated.
I followed the following article: https://www.pgpool.net/docs/41/en/html/example-cluster.html and the only thing difference I did is installing postgresql version 11.
I not ping delegate_IP = '192.168.1.233'. May i help you?
Thanks you.
you are not providing -h argument to psql for specifying the IP address. So effectively psql is trying to connect to UNIX domain socket and considering the IP address in the command as the database name.
Try putting -h before the IP address
psql -h 192.168.220.146 -p 9999 -U postgres -c "show pool_nodes"
I have Amazon EC2 instance running Ubuntu. I have installed and configured PostgreSQL.
Contents of the file /etc/postgresql/9.3/main/pg_hba.conf:
local all all md5
host all all 0.0.0.0/0 md5
Also in postgresql.conf
I have set listen_addresses='*'.
The test command below is successfully starting psql console.
psql -U postgres testdb
Now I am trying to connect pgAdmin4 from MacOS.
I have created a SSH tunnel with following command:
ssh -i ~/.ssh/test.pem -fN -L 5433:localhost:5432 ubuntu#mytestdomain.com
Now I have following details in pgAdmin:
When I save, I get this output:
Unable to connect to server: server closed the connection unexpectedly. This probably means the server terminated abnormally before or while processing the request.
What am I doing wrong ?
Here is the solution,
install pgadmin 4 into your system. and configure the below-added configurations. if the below configurations do not work then please check that DB user permissions on AWS. because of the restriction on IP level so it may not be able to access.
I'm playing around pgpool2.
I'm connecting to postgresql which is running on port 5432 with command psql -U postgres -p 5432 and it's connecting normally.
When I'm connecting to pgpool2 running on 9999 port with command psql -U postgres -p 9999 - it returns back dialog like:
psql: root#master:/linux/path#
What I'm doing wrong? Is there way to connect to pgpool using psql?
I cannot connect to it from application also.
UPD I've got following error pool_do_auth: maybe protocol version mismatch (current version 3)
Solved : Problem was in pgpool auth on backends. Editing pg_hba.conf fixed problem.
This error message and behavior are because of a problem with pgpool authenticating to postgres, not with you authenticating to pgpool as you might suspect. If you check the postgres logs, it will tell you which line in your postgres pg_hba.conf is being rejected.
However, since (IIRC) pgpool cannot do peer authentication, it is probably this one, which says that all local (unix socket) connections will be authenticated by pretending the system username is the postgres username:
local all all peer
You want to change that to:
local all all trust
At least to make things work. However, you will lose some security this way. However, if all your users (that can access the unix socket) are trusted, this is okayish.