Psycopg2 - multiple hosts for a PGSQL database - postgresql

Does anyone know if it's currently possible to have multiple hosts and ports in psycopg2 connection string for PostgreSQL databases?
postgresql+psycopg2://<username>:<password>#host.domain.name/database is what I have - is it possible to specify multiple hosts if I have primary/secondary setup for failovers?
Regards,

I believe this is due to an incorrect documentation.
this link suggests that using psycopg2 I have to specify the format as
username:password#/database?host=name1:port&host=name2&host=name3
The above will always give error, and also will not fall back on alternative hosts. It will take a random host on the list. You need to provide target_session_attrs parameter as per this documentation
So the correct way (at least one option) is to use the below format
postgresql+psycopg2://username:password#/<db_name>?host=<primary_db_host>&host=<secondary_db_host>&port=<port_no>&target_session_attrs=primary
Please note (I didn't test these)
It might not work if you have primary/secondary hosts running off of different ports
Since I am using psycopg2 through SQLAlchemy, I only tried this format, but other format might work too

From your example it looks like you are using SQLAlchemy. I do not have that set up but I can show thepsycopg2 portion. In this case Postgres 14 is listening on port 5432 and Postgres 12 on 5442:
import psycopg2
con = psycopg2.connect("postgresql://postgres#localhost:5432,localhost:5442/test")
cur = con.cursor()
cur.execute("select version()")
cur.fetchone()
('PostgreSQL 14.3 on x86_64-pc-linux-gnu, compiled by gcc (SUSE Linux) 7.5.0, 64-bit',)
--sudo systemctl stop postgres14
con = psycopg2.connect("postgresql://postgres#localhost:5432,localhost:5442/test")
cur = con.cursor()
cur.execute("select version()")
cur.fetchone()
('PostgreSQL 12.11 on x86_64-pc-linux-gnu, compiled by gcc (SUSE Linux) 7.5.0, 64-bit',)
For more information see libpq connection string, in particular 34.1.1.3. Specifying Multiple Hosts and the paragraph just above it. It will not automatically fail over.

Related

How to fix "ERROR: column c.relhasoids does not exist" in Postgres?

I’m trying to CREATE TABLE command in Postgresql.
After creating a table, if I punch in TABLE table name, it works.
But I punch in \d table name, I keep getting an error below.
ERROR: column c.relhasoids does not exist
LINE 1: ...riggers, c.relrowsecurity, c.relforcerowsecurity, c.relhasoi...
I attempted DROP DATABASE table name recreated a database and recreated a table again several times. But it didn't work.
Any suggestions would be appreciated! Thank you.
I am able to reproduce your error if I am using Postgres v.12 and an older client (v.11 or earlier):
[root#def /]# psql -h 172.17.0.3
psql (11.5, server 12.0)
WARNING: psql major version 11, server major version 12.
Some psql features might not work.
Type "help" for help.
postgres=# create table mytable (id int, name text);
CREATE TABLE
postgres=# table mytable;
id | name
----+------
(0 rows)
postgres=# \d mytable;
ERROR: column c.relhasoids does not exist
LINE 1: ...riggers, c.relrowsecurity, c.relforcerowsecurity, c.relhasoi...
^
postgres=#
This is because in v. 12, table OIDs are no longer treated as special columns, and hence the relhasoids column is no longer necessary. Please make sure you're using a v. 12 psql binary so you don't encounter this error.
You may not necessarily be using psql, so the more general answer here is to make sure you’re using a compatible client.
For anyone running Postgres as a Docker container:
Instead of running psql from the host, run it from inside the container e.g.
docker exec -it postgres_container_name psql your_connection_string
The Postgres image always ships with the corresponding—and thus always updated—version of psql so you don't have to worry about having the correct version installed on the host machine.
If you're using DataGrip, there's an easy fix:
Try using "Introspect using JDBC metadata". This fixed it for me when (I think) I had a version mismatch between postgresql server and DataGrip client.
Under your connection settings -> Options tab -> check Introspect using JDBC metadata
According to https://www.jetbrains.com/help/datagrip/data-sources-and-drivers-dialog.html#optionsTab :
Switch to the JDBC-based introspector.
To retrieve information about database objects (DB metadata), DataGrip
uses the following introspectors:
A native introspector (might be unavailable for certain DBMS). The
native introspector uses DBMS-specific tables and views as a source of
metadata. It can retrieve DBMS-specific details and produce a more
precise picture of database objects.
A JDBC-based introspector (available for all the DBMS). The JDBC-based
introspector uses the metadata provided by the JDBC driver. It can
retrieve only standard information about database objects and their
properties.
Consider using the JDBC-based intorspector when the native
introspector fails or is not available.
The native introspector can fail, when your database server version is
older than the minimum version supported by DataGrip.
You can try to switch to the JDBC-based introspector to fix problems
with retrieving the database structure information from your database.
For example, when the schemas that exist in your database or database
objects below the schema level are not shown in the Database tool
window.
The issue is the client (psql) is a different version from the postgres server. I have seen this issue with psql version 11 talking to postgres version 12. To solve this issue upgrade the psql version to 12.
If you are running a docker postgres, you can exec into the container then use the psql client installed there.
# get the container id with this
docker ps
# Then exec into the container, please note the host will now be 120.0.0.1
docker exec -it c12e8c6b8eb5 /bin/bash
I had this issue because my psql was 9.2 and the server version was 12.7.
So ... clearly the psql client needs to be updated. But how?
Before you go downloading/installing anything though you may already have the right version. In my case I did.
I executed which psql which showed my version was coming from /usr/bin/psql.
I then checked /usr/pgsql-12/bin and found there was a psql in there.
So all I needed to do was ensure psql was picked up from there.
There are a number of places that could be controlling this; in my case I just added this line to my .pgsql_profile (in the postgres user's home directory):
export PATH="/usr/pgsql-12/bin:$PATH"
Logging out and back in as postgres and executing which psql showed the change had been successful:
which psql
/usr/pgsql-12/bin/psql
This answer is specific to pgcli
If you are using pgcli you may be encountering this issue. It's solved by updating the python package pgspecial.
If you installed pgcli using pip, you can simply do, depending on your python version:
pip install -U pgspecial
or
pip3 install -U pgspecial
If you are using Ubuntu and intalled pgcli using apt, you can either switch it to pip with:
sudo apt remove --purge pgcli
pip3 install pgcli
or update the distribution package python-pgspecial or python3-pgspecial from the Ubuntu packages web site. In that case you may need to update its dependencies as well.
I had this issue today, was unable to continue work due to this, strangely the application code is working fine.
Later, found this issue is only occurring if I use OmniDb client I use to connect to DB.
I have switched client to default pgAdmin 4 that comes with postgres installation & issue is not occurring anymore pgAdmin 4. Link: https://www.pgadmin.org/download/pgadmin-4-windows/
Its possible that OmniDb client might be older, but no time to troubleshoot it, using pgAdmin 4 for now.
Hope that helps.
Just update DataGrip solved this issue, Datagrip updated to version DataGrip 2019.3.3, Build #DB-193.6494.42, built on February 12, 2020, Now working :)
Just for DataGrip users!
I had the same issue today too. In my case, the problem was solved when I deleted the version 12 and installed the version 11. Seems that v12 has some features that must be create along the others columns.
I had the same problem.
But I found the solution by downloading the latest build on 14/10/2019
Follow the link:
https://postbird.paxa.kuber.host/2019_10_14.06_42-master-7a9e949
I hope it helps
To fix this, edit Postgres.php file and comment the lines from hasObjectID function as shown below.
function hasObjectID($table) {
$c_schema = $this->_schema;
$this->clean($c_schema);
$this->clean($table);
/*
$sql = "SELECT relhasoids FROM pg_catalog.pg_class WHERE relname='{$table}'
AND relnamespace = (SELECT oid FROM pg_catalog.pg_namespace WHERE nspname='{$c_schema}')";
$rs = $this->selectSet($sql);
if ($rs->recordCount() != 1) return null;
else {
$rs->fields['relhasoids'] = $this->phpBool($rs->fields['relhasoids']);
return $rs->fields['relhasoids'];
}
*/
}
I had the same issue when using PgAdmin to query the database.
Once I installed the newest version of PgAdmin the error disappeared!
You might also try restarting pgadmin.
After upgrading from postgres96 to postgres12 I had the same issue. My pgadmin was running psql v12.0 so that wasn't the issue. I restarted pgadmin for a separate issue and the relhasoids issue went away.
If anyone could explain to me why this worked that would be appreciated.
Just use version 11.
how to install version 11
https://websiteforstudents.com/how-to-install-postgresql-11-on-ubuntu-16-04-18-04-servers/
I also got same issue with my postgresql tables. I have fixed this issue by below query.
ALTER Table MyDataBase.table_name add column column_name data_type default 0 not null;
commit;

Postgresql - unrecognized configuration parameter

I exported a postgresql database from an external server, and attempted to import it into my local server but got this error:
unrecognized configuration parameter "idle_in_transaction_session_timeout"
Does this kind of error mean that the two servers are using different versions of postgresql? I looked into that, and the external server is running:
version
PostgreSQL 9.5.4 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 4.8.2-19ubuntu1) 4.8.2, 64-bit
and my server is running:
version
PostgreSQL 9.5.5 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.2) 5.4.0 20160609, 64-bit
Pretty much the same thing. Is there a site where you can see all of the valid config parameters for each version? And is there a way to sync up two databases like this, so incompatibilities like this get patched up automatically?
According to Postgresql 9.6 Release Notes the idle_in_transaction_session_timeout parameter was introduced in version 9.6.
E.2.3.1.10. Server Configuration
Allow sessions to be terminated automatically if they are in
idle-in-transaction state for too long (Vik Fearing)
This behavior is controlled by the new configuration parameter
idle_in_transaction_session_timeout. It can be useful to prevent
forgotten transactions from holding locks or preventing vacuum cleanup
for too long.
Since you are using version 9.5 on the server, the parameter is not recognized.
It's possible that you used version 9.6 of the Postgresql client to export data from the the source 9.5 server and the parameter was introduced in the dump file. If this was the case I would recommend using a 9.5 client version to export and import the data.
The accepted answer is the way to go, but if for some reason you can not upgrade version, here is a workaround.
Export using plain text. You probably want to use compression too. pg_dump -F c -Z 9 dbname > file.zip
Before import, we need to remove the offending parameter. To do that we can use zcat and grep. zcat file.zip | grep -vw "idle_in_transaction_session_timeout" | psql -d newdb
Note that there are drawbacks using psql instead of pg_import. For instance, one can not use the -j to import concurrently.

Prevent Postgres 9.2 from starting

I have upgraded from 9.2 to 9.3 successfully on ubuntu. However,
/etc/init.d/postgresql start
starts both 9.2 and 9.3
Although the above command can accept that the version number and successfully starts and stops each one, is there any method I can use to make this command start 9.3 only.
The reason is that, I am not able to reboot the system now, but I am afraid when it is rebooted both servers can start.
My short term solution is to adjust the port numbers to make my application use 9.3 database. However, I would like to learn about more permanent and robust solutions.
Thanks in advance,
Steve
Ubuntu uses pg_wrapper to manage PostgreSQL installs. See the Ubuntu PostgreSQL wiki page.
You'll want to pg_dropcluster the 9.2 cluster, if you wish to actually destroy the old data. Or un-install PostgreSQL 9.2. Or modify the config file (don't remember the name right now) in /etc/postgresql/9.2/ that controls whether Pg starts or not. It's called something like start.conf or pg_ctl.conf or something.
You may also want to reverse the configured ports so your new 9.3 runs on 5432 and your not-started-by-default 9.2 tuns on 5433. That is in postgresql.conf.
Steve Harman's response worked perfectly for me, too:
Thanks for the response. In the /etc/postgresql/9.2/main/ directory, there is start.conf. If you change the single line in that file from 'auto' to 'disabled' then, /etc/init.d/postgresql start will not start 9.2. – Steve Harman Jan 1 at 16:55
On the other hand, just fyi, the command output is that both versions of the server are starting (which is not true and is coming from the service starting scripts)
user#server:/etc/postgresql/9.3/main$ sudo service postgresql start
* Starting PostgreSQL 9.1 database server
...done.
* Starting PostgreSQL 9.3 database server
...done.

connect failed with SQL-HY001

I am using Perl 5.14.2 on Ubuntu wheezy 64 bit.
I am trying to connect to a Pervasive SQL server v9.5 that is installed on a windows 2008 machine.
I tested the connection with isql and it works properly, I tested with the following command:
isql -v <db_name>
I use the DBD:ODBC v1.39 (latest version) perl module installed from cpan.
I am using the following command to connect to sql server:
use DBI;
my $db = DBI->connect('dbi:ODBC:Moked');
Moked is the connection name that was defined in the unixodbc v2.2.14 in odbc.ini.
/etc/odbc.ini
[Moked]
Driver=PERVASIVE_ODBC
Description=Pervasive ODBC Interface: Moked
ServerName=<ip>:<port>
DBQ=MIDA
UID=
PWD=
OpenMode=0
PvTranslate=
when I try to connect with the command i showed above I get error SQL-HY001 that by googling I understood that it relates to memory allocation, usually people get these while querying and not while trying to connect/
it seems that DBI does recognize the connection because if I type a different name, for example 'Moked2', i get the error SQL-IM002 instead.
any ideas?
how can I debug this issue further?
any information regarding the issue would be greatly appreciated.
thanks!
You are getting the memory allocation error because the PSQL v11 client cannot reliably connect to a PSQL v9.5 server. If you want to use the v11 client, you need the v11 server too. You might be able to get away with the v11 client at a Btrieve level but there were significant changes between v9 and v11 on the ODBC side.

MySQL Dump and Import not preserving encoding?

I am trying to copy a table from on MySQL database on a remote machine to another MySQL database on my local machine. I noticed that after importing the dump to my local machine, there were characters like ’ instead of single quotes.
I assumed this was an encoding issue, so I went into both databases and ran show create table posts, near the end of both, I saw CHARSET=utf8. Also, I ran file -i on the dump file, both before and after scping it to my local machine, and they were both utf8.
However, when I import this file, I get this before:
attendees—policy makers,
and after:
attendees—policy makers,
I am not sure why this is happening, everything is using utf8, what am I missing?
EDIT: I am using mysql Ver 14.12 Distrib 5.0.75, for debian-linux-gnu (x86_64) remotely, and mysql Ver 14.14 Distrib 5.5.25a, for osx10.7 (i386) locally.
On both systems you must check that your connection encoding is correct:
SHOW VARIABLES LIKE 'character_set_%'
Usually seeing characters like that is the result of double-encoding. Make sure you can match up the connection and client encoding to be exactly the same. There is a number of command line options that can facilitate this, or if you're using a driver or client, something in there can tweak it.