Having a hard time working out what do I need to do to insert this export from MySQL:
INSERT INTO cities VALUES ('Nameoftheciy',0x0000000001010000005C21ACC6129C48407AFF1F274CA65EC0);
I tried wrapping the value in single quotes or extracting the data as POINT() but it also returns an errors in PostgreSQL.
What is the right approach here?
Related
I'm trying to insert around 15000 rows with 200 columns in one batch into a table in PostgreSQL (with TimescaleDB extension) via Python by using the psycopg2 library. Some of the values I want to insert might be larger than the table's column datatype allows. This results in:
ERROR: smallint out of range, SQL state: 22003
I haven't found a way to get more information about the location of the error to handle it.
In MySQL, the column and the row where the error occured is reported back by default and it is even possible to clip the values to the max value of its datatype (which also would be fine). Is there a way to handle this in a similar manner in PostgreSQL?
I'm using redash on a PostgreSQL db and for some of our tables, querying all columns using SELECT * returns an error akin to column X is not JSON serializable. My understanding is that it's because a column contains a datetime range (and doesn't play well with redash).
Is there a way in postgres to easily retrieve all columns, while converting a specfic one to text? Or even converting all columns to text?
So far, in order to do this, I have to list out every column name and convert the culprit to text.
My hope is finding something along the lines of:
SELECT created_at::text, *
or...
SELECT *::text
Where the wonky column (created_at) is converted to text, and somehow appended to everything else.
I've got a table with 4 rows in it in a non-production database used for development. There are 2 varchar columns that I want to convert to bytea. I don't care about the contents so I could of course drop the columns and then add them back, but I became confused when I tried to just change the type:
alter table whatever
alter column col_1 set data type bytea using null,
alter column col_2 set data type bytea using null;
When I try that, the psql client just hangs. By that I mean that it just sits there giving no feedback until I eventually hit ^C and it aborts. I've tried that with a little test table and it works fine, but for some reason it doesn't work on the real table (which, really, is also just a "little test table").
The using clause doesn't seem to make a difference one way or the other; I can leave it out or give other values, and the command does the same thing.
I don't get an error, I just don't get anything. Is that what I should expect?
I'm on 9.1 on ubuntu 14.10 if it matters.
I don't care about the contents
In that case, this works on an empty table:
ALTER TABLE tablename
ALTER COLUMN colname TYPE bytea USING colname::bytea
;
Simple:
Get the active locks from pg_locks:
select t.relname,l.locktype,page,virtualtransaction,pid,mode,granted from pg_locks l, pg_stat_all_tables t where l.relation=t.relid order by relation asc;
Copy the pid(ex: 14210) from above result and substitute in the below command.
SELECT pg_terminate_backend('14210')
I've a lot of records that are originally from MySQL. I massaged the data so it will be successfully inserted into PostgreSQL using ActiveRecord. This I can easily do with insertions on row basis i.e one row at a time. This is very slow I want to do bulk insert but this fails if any of the rows contains invalid data. Is there anyway I can achieve bulk insert and only the invalid rows failing instead of the whole bulk?
COPY
When using SQL COPY for bulk insert (or its equivalent \copy in the psql client), failure is not an option. COPY cannot skip illegal lines. You have to match your input format to the table you import to.
If data itself (not decorators) is violating your table definition, there are ways to make this a lot more tolerant though. For instance: create a temporary staging table with all columns of type text. COPY to it, then fix offending rows with SQL commands before converting to the actual data type and inserting into the actual target table.
Consider this related answer:
How to bulk insert only new rows in PostreSQL
Or this more advanced case:
"ERROR: extra data after last expected column" when using PostgreSQL COPY
If NULL values are offending, remove the NOT NULL constraint from your target table temporarily. Fix the rows after COPY, then reinstate the constraint. Or take the route with the staging table, if you cannot afford to soften your rules temporarily.
Sample code:
ALTER TABLE tbl ALTER COLUMN col DROP NOT NULL;
COPY ...
-- repair, like ..
-- UPDATE tbl SET col = 0 WHERE col IS NULL;
ALTER TABLE tbl ALTER COLUMN col SET NOT NULL;
Or you just fix the source table. COPY tells you the number of the offending line. Use an editor of your preference and fix it, then retry. I like to use vim for that.
INSERT
For an INSERT (like commented) the check for NULL values is trivial:
To skip a row with a NULL value:
INSERT INTO (col1, ...
SELECT col1, ...
WHERE col1 IS NOT NULL
To insert sth. else instead of a NULL value (empty string in my example):
INSERT INTO (col1, ...
SELECT COALESCE(col1, ''), ...
A common work-around for this is to import the data into a TEMPORARY or UNLOGGED table with no constraints and, where data in the input is sufficiently bogus, text typed columns.
You can then do INSERT INTO ... SELECT queries against the data to populate the real table with a big query that cleans up the data during import. You can use a lot of CASE statements for this. The idea is to transform the data in one pass.
You might be able to do many of the fixes in Ruby as you read the data in, then push the data to PostgreSQL using COPY ... FROM STDIN. This is possible with Ruby's Pg gem, see eg https://bitbucket.org/ged/ruby-pg/src/tip/sample/copyfrom.rb .
For more complicated cases, look at Pentaho Kettle or Talend Studio ETL tools.
I have a table in postgresql with a PostGIS geometry(point, 4326) column (location, using SRID 4326) and I have a Python application that using SQL Alchemy updates the table (the rest of the columns) without any problem.
Now, I need to update the location column and I know I can use the proper text representation of a given location to update the column using SQL Alchemy without the need to use GEOAlchemy, for instance I can update the column with the value: '0101000020E6100000AEAC7EB61F835DC0241CC418A2F74040'
which corresponds to lat:33.9346343 long:-118.0488106
The question is: is there a way to compute in Python this '0101000020E6100000AEAC7EB61F835DC0241CC418A2F74040' having this (33.9346343 ,-118.0488106) as an input without querying the database? or any way to update the column using a proper text input?
I know I can use SQLAlchemy to execute this query:
select st_setsrid(st_makepoint(-118.0488106, 33.9346343),4326)
and obtain the value to update the column, but I want to avoid that.
Thanks in advance!
The solution to this problem is rather easier than it seems. To update the field using text and the input lat-long all I needed to do was defining the SRID in the text assign:
location = 'SRID=4326;POINT(-118.0488106 33.9346343)'
This will update the geometry(point,4326) column properly and when you do a select in the table the value of the column is the expected one:
"0101000020E6100000AEAC7EB61F835DC0241CC418A2F74040"
Thanks guys!