I am logged in as the postgres user and am trying to export a blob from a Postgresql 9.3 database as using this query:
select lo_export(imgs.rast, '/tmp/img.tif') from imgs where rid = 1;
but I get this error:
ERROR: function lo_export(bytea, unknown) does not exist
Do I need to install the lo_export function? If so, how?
You're trying to use a function, which is used to export a large object, which is something different than a blob (bytea column type in Postgres).
There's no library function to export bytea to a file. You just use in your client program something like (Python example):
cursor = conn.execute("select rast from img were rid=%(rid)s", {"rid":1})
result = cursor.fetchone()
with tempfile.NamedTemporaryFile(mode="wb", suffix=".tif", delete=False) as f:
f.write(result[0])
result_filename = f.name
Please remember that this reads the whole data to memory multiple times, so if the data can be fairly large you can prefer to read it in chunks using substring(rast from %(chunk_start)s for %(chunk_size)s). And ensure that the column is saved externally using set storage external.
Related
We are seeing issue in table value which are populated from DB2 (source) to Postgres (Target).
I have including here all the job details for each component.
Based on the above approach and once the data has been populated, when we run the below query in the Postgres DB.
SELECT * FROM VMRCTTA1.VMRRCUST_SUMM where cust_gssn_cd='XY03666699' ;
SELECT * FROM VMRCTTA1.VMRRCUST_SUMM where cust_cntry_cd='847' ;
There will be no records were returned however, when we run the same query with Trim as below it works.
SELECT * FROM VMRCTTA1.VMRRCUST_SUMM where trim(cust_gssn_cd)='XY03666699' ;
SELECT * FROM VMRCTTA1.VMRRCUST_SUMM where trim(cust_cntry_cd)='847' ;
Below are the ways we have tried to overcome this but no luck.
Used tmap between source and target component.
Used trim in source component under Advanced setting.
Change the datatype in Postgres DB of cust_cntry_cd from char(5) to Character varying, this will allow value without any length restriction.
Please suggest what is missing as we have this issue in almost all the table where we have character/varchar columns.
We are using TOS.
The data type is probably character(5) in DB2.
That means that the trailing spaces are part of the column and will be migrated. You have to compare with
cust_cntry_cd = '847 '
or cast the right argument to character(5):
cust_cntry_cd = CAST ('847' AS character(5))
Maybe you could delete all spaces in the advanced settings of the tDB2Input component.
Like the screen :
I have a large dataset I want to load into a SQLite in-memory database. I plan on loading the data from a file exported from Postgres. What file format and load mechanism is the fastest?
Currently I'm considering the following two options:
Importing a CSV file (copy). Reference.
Running a SQL file (pg_dump) with INSERT statements using a single transaction. Reference.
Which is faster? Is there a third faster option, maybe?
This will be done as part of a Python 3 script. Does that affect the choice?
If nobody has any experience with this, I'll make sure to post benchmarks as an answer later.
Edit: This question has gotten a downvote. From the comments it seems this is due to the lack of benchmarking. If not, please let me know how to improve this question. I definitely don't expect anybody to perform benchmarking for me. I'm simply hoping that someone has prior experience with bulk loading into SQLite.
Turns out there is no great way to do this using pg_dump and insert statements in a performant way. We end up with inserting line-by-line from the source file both when we use the CSV and the pg_dump strategies. We're going with the CSV method loading 10000 rows each batch using executemany.
import sqlite3
from datetime import datetime
import csv
conn = sqlite3.connect(":memory:")
cur = conn.cursor()
create_query = """
CREATE VIRTUAL TABLE my_table USING fts4(
id INTEGER,
my_field TEXT
);
"""
cur.execute(create_query)
csv.field_size_limit(2147483647)
from_time = datetime.now()
with open('test.csv', 'r', encoding="utf8") as file:
csv_file = csv.reader(file)
header = next(csv_file)
query_template = """
INSERT INTO my_table (id, my_field)
VALUES (?, ?);
"""
for batch in split_iterable_by_size(csv_file, 10000):
cur.executemany(query_template, batch)
conn.commit()
On our system and dataset this took 2 hours 30 minutes. We're not testing the alternative.
In my project i want to collect PostgreSQL server's performance counter. For that i want query to collect it from the database. i am new to postgreSQL. when i am searching, i got something like,
SELECT * FROM pg_stat_database
but when i am use this in java in the following manner, Here Map_PostgreSQL is a hashmap.
while(rs.next())
{
Counter_Name.add(rs.getString(1).trim());
Map_PostgreSQL.put(rs.getString(1).trim(), rs.getString(2));
}
I got output like
{12024=template0, 1=template1, 12029=postgres}
What is the actual query to collect its status variables like "SHOW GLOBAL STATUS" in MySQL.
Thanks in advance..
1st, try to launch the sql query in your PostgreSQL Shell to see exactly which data are returned and how it is organised in rows and columns.
You'll see that the hashmap keys are your datid (database ids) and the values are your databases names.
I think you assumed that statistics were structured in "rows" whereas they are structured in columns.
Don't forget : PostgreSQL is a database server which means it can handle several databases (and in fact, it has several databases because some of them are already created such as the 'postgres' database itself - which Postgres (the server) uses internally, or 'template0').
By launching :
SELECT * FROM pg_stat_database;
You're asking the server to return statistics for every databases (provided you're allowed to get them)
If you want to only have stats for your own database, do :
SELECT * FROM pg_stat_database WHERE datname='your_database_name';
Hope this helped
I am using PostgreSQL 9.0.3. I have an Excel spreadsheet with lots of data to load into couple of tables in Windows OS.
I have written the script to get the data from input file and Insert into some 15 tables. This can't be done with COPY or Import. I named the input file as DATALD.
I find out the psql command -d to point the db and -f for the script sql. But I need to know the commands how to feed the input file along with the script so that the data gets inserted into the tables..
For example this is what I have done:
begin
for emp in (select distinct w_name from DATALD where w_name <> 'w_name')
--insert in a loop
INSERT INTO tblemployer( id_employer, employer_name,date_created, created_by)
VALUES (employer_id,emp.w_name,now(),'SYSTEM1');
Can someone please help?
For an SQL script you must ..
either have the data inlined in your script (in the same file).
or you need to utilize COPY to import the data into Postgres.
I suppose you use a temporary staging table, since the format doesn't seem to fit the target tables. Code example:
How to bulk insert only new rows in PostreSQL
There are other options like pg_read_file(). But:
Use of these functions is restricted to superusers.
Intended for special purposes.
I am attempting to use COPY FROM STDIN to import data into my table. One of the columns in my table is of type geometry. My command looks something like this...
COPY "WeatherStations" ("Station_ID", "Station_Code", "Station_Name", "Station_Location") FROM stdin;
1 KAVP WILKES-BARRE ST_GeomFromText('POINT(41.338055 -75.724166)')
2 KOKV WINCHESTER ST_GeomFromText('POINT(39.143333 -78.144444)')
3 KSHD SHENANDOAH ST_GeomFromText('POINT(38.263611 -78.896388)')
...
However, I think it is attempting to insert the text "ST_GeomFromText('POINT..." and failing instead of evaluating the expression and inserting the result of the expression. Does anyone know what might be going on here and how I can get the actual geoms inserted?
I had a bad time figuring out how to bulk copy/load geometry data into PostGIS using the COPY FROM STDIN command, I couldn't find official documentation on this topic.
Altering the column during the bulk load (the ALTER TABLE / SET DATA TYPE / USING) was not an option to me because it is only supported in PostGIS 2.0+ for the Geometry type, nor was acceptable the use of a temporary table.
There is indeed a direct way to do it (at least in PostGIS 1.5.2+).
You can simply rewrite the data for your copy statement this way, using a simple WKT (Well-known text) representation for your Geometry data:
1 KAVP WILKES-BARRE POINT(41.338055 -75.724166)
2 KOKV WINCHESTER POINT(39.143333 -78.144444)
3 KSHD SHENANDOAH POINT(38.263611 -78.896388)
If you have enforced a SRID constraint on the geometry column you'll have to use the following syntax (in this example the SRID is 4326) known as EWKT (Extended Well-Known Text, which is a PostGIS specific format):
1 KAVP WILKES-BARRE SRID=4326;POINT(41.338055 -75.724166)
2 KOKV WINCHESTER SRID=4326;POINT(39.143333 -78.144444)
3 KSHD SHENANDOAH SRID=4326;POINT(38.263611 -78.896388)
Closing note: there must be no space between "POINT" and the opening parenthesis "(", or the COPY will still return error saying your geometry data has an invalid format.
You could omit the function wrapping the text, import into a temporary table with text column, and then run INSERT/SELECT into the permanent table with the function doing the conversion in that step.
INSERT INTO "WeatherStations"
("Station_ID", "Station_Code", "Station_Name", "Station_Location")
SELECT "Station_ID", "Station_Code", "Station_Name",
ST_GeomFromText("Station_Location")
FROM "TempWeatherStations";
You will keep all the values in .csv file and try like this:
CAT /path/file/demo.csv | psql -u <username> -h <localhost> -d<database>
-c "COPY "WeatherStations" ("Station_ID", "Station_Code", "Station_Name",
"Station_Location") FROM stdin;"
This will work.
Point's value looks something like this: 0101000020E6100000DA722EC555552B40CDCCCCCCCC0C4840.
I typically keep latitude and longitude columns in my tables and build spatial data with triggers.
I don't know how to copy POINTs from stdin otherwise.