How to use lobject function in psycopg2 - postgresql

I am trying to insert Large Binary data into postgresql using psycopg2. I understand bytea datatype is more common to use but testing BLOB for any future use cases.
Versions of postgresql and psycopg2 is below.
pip list | grep psycopg2
psycopg2 (2.5.1)
rpm -qa | grep postgres
postgresql-server-9.2.15-1.el7_2.x86_64
I use python 2.7.5
python -V
Python 2.7.5
Below is my code snippet
file = "/home/test/jefferson_love_memorial_514993.jpg"
with open(file,"r") as fd:
try:
# First connect to postgresql server
conn = psycopg2.connect("dbname='sample' user='sample' host='10.1.0.19' password='sample'")
# Initate the session with postgresql to write large object instance
lobj = conn.lobject(0,'r',0)
# Write the data to database
lobj.write(fd.read())
except (psycopg2.Warning, psycopg2.Error) as e:
print "Exception: {}".format(e)
However, after I execute the code I get no error but nothing is inserted into the table.
-bash-4.2$ psql -d sample
psql (9.2.15)
Type "help" for help.
sample=# SELECT * FROM pg_largeobject_metadata;
lomowner | lomacl
----------+--------
(0 rows)
sample=# SELECT * FROM pg_largeobject;
loid | pageno | data
------+--------+------
(0 rows)
May I ask what is lacking in my code?

I found the reason.
I have forgotten to do conn.commit() after lobj.write().
After doing commit it works perfectly.

Related

\COPY TO Postgres statement working in bash but not with python psycopg2

I am trying to use psycopg2 in my script to export data from a Postgres database to a file.
I can successfully run the following from my terminal and it works, no problem:
psql -h myhost -p myport -U myuser -d mydbname -c "\COPY ( SELECT member_id FROM member_reward_transaction LIMIT 5) TO ~/Desktop/testexport.txt (FORMAT csv, DELIMITER '|', HEADER 0)"
I could presumably call the above using subprocess, but I would like to know why the following is not working for me:
import configparser
config = configparser.ConfigParser()
config.read('config/qa_config.ini')
dbname=config['postgres-rewards']['db_name']
host=config['postgres-rewards']['host']
port=config['postgres-rewards']['port']
user=config['postgres-rewards']['user']
password=config['postgres-rewards']['password']
con = psycopg2.connect(database=dbname,user=user,password=password,host=host,port=port)
cur = con.cursor()
f = open('exports/test_export.csv')
cur.copy_to(f, 'member_reward_transaction', columns=('member_id', 'sponsor_id'), sep=",")
con.commit()
con.close()
The error when I run the script:
File "tests2.py", line 17, in <module>
cur.copy_to(f, 'member_reward_transaction', columns=('member_id', 'sponsor_id'), sep=",")
psycopg2.errors.WrongObjectType: cannot copy from partitioned table "member_reward_transaction"
HINT: Try the COPY (SELECT ...) TO variant.
using Python 3.6.5, PostgreSQL 11.5
Like the error message says, you have to use
COPY (SELECT ... FROM partitioned_table) TO STDOUT;
if you want to use a partitioned table.
Your psql command does that, but psycopg2's copy_to uses plain old
COPY partitioned_table TO STDOUT;
which doesn't work.
Use copy_expert which allows you to submit your own COPY statement.

PostGIS type "geometry" doesn't exist when pg_dump copying database to server

Similar question to many that have already been asked, but I have yet to find a solution.
I'm copying a Postgres db from my local machine to a server using:
pg_dump -C -h localhost -U localuser dbname | psql -h remotehost -U remoteuser dbname
However, whenever I try to execute this, any tables with PostGIS geography columns are skipped and not copied over, and I'm left with the following error:
ERROR: type "<mydb>.geography" does not exist
Attempted solutions:
On this server, I have successfully installed PostGIS and created the relevant extensions:
mydb=# select postgis_version();
postgis_version
---------------------------------------
2.4 USE_GEOS=1 USE_PROJ=1 USE_STATS=1
(1 row)
Additionally, geometry is a recognized data type:
mydb=# \dT *.geometry
List of data types
Schema | Name | Description
--------+----------+-----------------------------------------
public | geometry | postgis type: Planar spatial data type.
(1 row)
My search_path includes everything relevant:
mydb=# show search_path;
search_path
------------------------
mydb, public, postgis
and
mydb=# SELECT r.rolname, d.datname, rs.setconfig
FROM pg_db_role_setting rs
LEFT JOIN pg_roles r ON r.oid = rs.setrole
LEFT JOIN pg_database d ON d.oid = rs.setdatabase
WHERE r.rolname = 'mydb' OR d.datname = 'mydb';
rolname | datname | setconfig
---------+---------+----------------------
| mydb | {search_path=public}
(1 row)
My only guess as to what might be causing this headache is that the superuser on my local machine isn't the same name as the remote superuser. I.e., localuser and remoteuser don't match.
The rest of the answer assumes that the error you quote is the first error while restoring the dump. If not, it might be the consequence of an earlier error (e.g., failure to CREATE EXTENSION postgis).
If PostGIS is installed on both servers, the message suggests that it is installed in different schemas on both databases.
Examine the result of
\dx postgis
on both databases to check.
Since PostGIS is not a relocatable extension, you have to drop and re-create it to move it to a different schema.

Syntax error at or near AS integer when creating a sequence

I am getting "error at or near AS integer" in below code when I am trying to execute with psql.
CREATE SEQUENCE public.auth_group_id_seq
AS integer
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
The above sql statement is from the backup file of local machine postgres version 11 and executing in EC2 postgres version 9.3. I am new to postgres and getting no idea as the sql is generated by postgres only so it should work with psql. Thanks in advance.
Postgres 9.3 (which is no longer supported) did not support the AS data_type option. This was introduced in version 10.
You could try using pg_dump from your 9.3 installation to do the dump, but I am not sure if that works.
Some workaround that works for me. In this case, you simply have to remove AS integer from the dump file.
sed 's/ AS integer$//g' your_dump_file.out > tmp.out
mv tmp.out your_dump_file.out

could not access file "$libdir/plpgsql" while creating template_postgis

I have installed PostgreSQL and Postgis on my Mac before when it's 9.3 and 2.1 but never used.
After upgrading pg to 4.5_2 and postgis to 2.2 (the latest now), I try to build my first postgis based app and creating template_postgis like this:
createdb template_postgis
psql -d template_postgis -f /usr/local/share/postgis/postgis.sql
And got this error:
SET
BEGIN
psql:/usr/local/share/postgis/postgis.sql:77: ERROR: could not access file "$libdir/plpgsql": No such file or directory
...
So I run pg_config --libdir it says /usr/local/lib, and pg_config --pkglibdir returns /usr/local/lib/postgresql.
And I found plpgsql.so under pkglibdir not libdir,
so I create a symbol link named plpgsql.so and another plpgsql under libdir, but it still not works.
What really is the problem there? How can I fix this?
My Mac is running the newest OS X 10.11 and both of them are installed and upgraded by homebrew.
Update:
Inspired by #Stefan D. , I run psql -c "\dx" to list all the installed extensions which results:
Name | Version | Schema | Description
---------+---------+------------+---------------------------------------------------------------------
plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language
postgis | 2.1.7 | public | PostGIS geometry, geography, and raster spatial types and functions
So I run ALTER EXTENSION postgis UPDATE TO "2.2.0"; to manually update the version of postgis, but got another error: could not open extension control file "/usr/local/Cellar/postgresql/9.4.5/share/postgresql/extension/postgis.control": No such file or directory.
This is wired as my PostgreSQL is installed in /usr/local/Cellar/postgresql/9.4.5_2/. And postgis.control is not listed in /usr/local/Cellar/postgresql/9.4.5_2/share/postgresql/extension/ either.
After locating, I find all the .control files have an symbol link in /usr/local/share/postgresql/extension/.
What a coinincidence: I just had a similar problem and were searching for a solution when I found your question.
I also updated my postgresql/postgis installation a couple of days ago, and when I wanted to use my database (which I created with before the update) it today I got an error similiar to yours (for me it was that the postgis 2.1 library could not be found)
Meanwhile I could fix it. Here is how I did it:
My problem was: The database was set up with the old Postgresql/postgis version. And it still looks for the old libraries. You can check this by listing all your extensions with the following command:
psql --username=gisuser --dbname=gis -c "\dx"
For me this gave me this output:
List of installed extensions
Name | Version | Schema | Description
------------------+---------+------------+---------------------------------------------------------------------
plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language
postgis | 2.1.7 | public | PostGIS geometry, geography, and raster spatial types and functions
postgis_topology | 2.1.7 | topology | PostGIS topology spatial types and functions
(3 rows)
As you can see the postgis and postgis_topology extension versions (2.1.7) do not fit my installed versions (2.2.0)
To fix that I executed the following commands to update the extension versions:
psql --username=gisuser --dbname=gis -c "ALTER EXTENSION postgis UPDATE;"
psql --username=gisuser --dbname=gis -c "ALTER EXTENSION postgis_topology UPDATE;"
You will probably have to alter the command a bit (username, database, and the extension you want to update)
Check whether it worked again with:
psql --username=gisuser --dbname=gis -c "\dx"
Now the output is:
List of installed extensions
Name | Version | Schema | Description
------------------+---------+------------+---------------------------------------------------------------------
plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language
postgis | 2.2.0 | public | PostGIS geometry, geography, and raster spatial types and functions
postgis_topology | 2.2.0 | topology | PostGIS topology spatial types and functions
(3 rows)
(Note the changed version number for postgis and postgis_topology)
Finally, I got out what caused this issue: Although there's only one postgres installed on my Mac, but there are TWO postgres exits!
What really happened is I have upgraded postgres program without shutting down, neither brew did, so I have a postgres 9.4.5_2 installed locally, and an old one in the memory!
So the solutions is really simple: Shutting down the old one and start the new installed postgres!
Anyway, thanks for your kind help!
Probably You have installed multipe instances of Postgres on Your Mac.
When you execute pg_config --pkglibdir, make sure it's the one associated with your installation. Run:
which pg_config

Where are PostgreSQL database files in the cluster directory?

On OSX, I recently installed PostgreSQL via Homebrew:
brew install postgresql
I then created a new database cluster:
initdb /usr/local/var/postgres
I confirm that postgresql server is running with the expected database cluster:
$ ps auxwww | grep postgres
0:00.03 /usr/local/opt/postgresql/bin/postgres -D /usr/local/var/postgres
I create a new database:
createdb mynewdb
I see that it exists.
$ psql
<user>=# \l
mynewdb | <user> | UTF8 | en_US.UTF-8 | en_US.UTF-8
But, I see no obvious changes to the cluster directory (e.g., just by checking contents ls -lt).
Where is the database written / stored in the cluster directory (or sub-directories)?
as #ahorsewithnoname says they are in the base directory, (on some other platfotms data/base)
the numeric name is the OID of the database and in the directory will be files named for the OID of the table or index they relate to.
databse oids can be determined with this query. oid is like a hidden unique id column.
select oid,* from pg_catalog.pg_database;
table and index oids can be found like this,
select * from pg_catalog.pg_class;
column relifilenode for the base of the filename (thanks #jjames)