I'm trying to insert around 15000 rows with 200 columns in one batch into a table in PostgreSQL (with TimescaleDB extension) via Python by using the psycopg2 library. Some of the values I want to insert might be larger than the table's column datatype allows. This results in:
ERROR: smallint out of range, SQL state: 22003
I haven't found a way to get more information about the location of the error to handle it.
In MySQL, the column and the row where the error occured is reported back by default and it is even possible to clip the values to the max value of its datatype (which also would be fine). Is there a way to handle this in a similar manner in PostgreSQL?
Related
I have a table of 30GB in size. I am running an ETL with an AWS Glue job that copies the table to an S3 bucket.
I try to bookmark using the combination of a couple of columns as the bookmark key. Some of the columns have rows with null values.
I get this error:
An error occurred while calling o97.getDynamicFrame. Incorrect DATETIME value: 'null'
I would like to ask if there is any way to give the column a default value.
The other alternative was moving the entire table without bookmark which I don't think is efficient.
I have to write a sql script to modify a lot of types of columns in my db2 database.
Everything goes well excpet for one specific table (script used is the same as others tables) and db2 returns always an error I don't understand.
Here is my script :
ALTER TABLE "TEST"."CLIENT"
ALTER COLUMN C_CODE
SET DATA TYPE CHAR(16 OCTETS);
and the error :
SQL Error [42997]: Function not supported (Reason code = "21")..
SQLCODE=-270, SQLSTATE=42997, DRIVER=4.26.14
I try to modify some others columns on the same table, but I always receive the same error.
Do you, by any chance, have an idea?
Thanks in advance
The error SQL0270N (sqlcode = -270) has many possible causes, and the specific cause is indicated by the "reason code".
In this case the "reason code 21" means:
A column cannot be dropped or have its length, data type, security,
nullability, or hidden attribute altered on a table that is a base
table for a materialized query table.
The documentation for this sqlcode on Db2-LUW is at:
https://www.ibm.com/docs/en/db2/11.5?topic=messages-sql0250-sql0499#sql0270n
Search for SQL0270N on that page, and notice the suggested user response:
To drop or alter a column in a table that is a base table for a materialized query table, perform the following steps:
1. Drop the dependent materialized query table.
2. Drop the column of the base table, or alter the length, data type, nullability, or hidden attribute of this column.
3. Re-create the materialized query table.
I have a postgres table with two columns (an identificator and a date) that are a composite primary key. I would like to hash the concatenation in another column, generating this value everytime a new record is inserted. For that I'm trying to alter my table in order to create a generated column:
ALTER TABLE my_table ADD COLUMN hash_id_date VARCHAR(50)
GENERATED ALWAYS AS (MD5(my_table.original_id||'-'||my_table.time))
STORED;
This raises me the following error:
ERROR: syntax error at or near "("
LINE 4: GENERATED ALWAYS AS (MD5(my_table.original_id,'-',my_table.t...
^
SQL state: 42601
Character: 178
I'm turning into madness to find where is the syntax error... I've read about STABLE and IMMUTABLE functions and generated columns should always have an IMMUTABLE function as expression. As far as I know MD5 is IMMUTABLE but the error message is not even capable to reach that level.
Any help?
Assuming the basic functionality for calculating the MD5 is common you can create a function for the calculation. Use this function wherever it's needed, including updating your current rows and invoke from a trigger on yo your table. If the particular MD5 calculation is not all that common you can just put the calculation in the trigger function and also use it in a independent update for current rows. See here for example with assumption it is common in your app.
I have a stored procedure that has started to fail for no reason. Well there must be one but I can't find it!
This is the process I have followed a number of times before with no problem.
The source server works fine!
I am doing a pg_dump of the database on source server and imported it onto another server - This is fine I can see all the data and do updates.
Then I run a stored procedure on the imported database that does the following on the database which has 2 identical schema's -
For each table in schema1
Truncate table in schema2
INSERT INTO schema2."table" SELECT * FROM schema1."table" WHERE "Status" in ('A','N');
Next
However this gives me an error now when it did not before -
The error is
*** Error ***
ERROR: column "HBA" is of type boolean but expression is of type integer
SQL state: 42804
Hint: You will need to rewrite or cast the expression.
Why am I getting this - The only difference between the last time I followed this procedure and this time is that the table in question now has an extra column added to it so the "HBA" boolean column is not the last field. But then why would it work in original database!
I have tried removing all data, dropping and rebuilding table these all fail.
However if I drop column and adding it back in if works - Is there something about Boolean fields that mean they need to be the last field!
Any help greatly apprieciated.
Using Postgres 9.1
The problem here - tables in different schemas were having different column order.
If you do not explicitly specify column list and order in INSERT INTO table(...) or use SELECT * - you are relying on the column order of the table (and now you see why it is a bad thing).
You were trying to do something like
INSERT INTO schema2.table1(id, bool_column, int_column) -- based on the order of columns in schema2.table1
select id, int_column, bool_column -- based on the order of columns in schema1.table1
from schema1.table1;
And such query caused cast error because column type missmatch.
I have a table in postgresql with a PostGIS geometry(point, 4326) column (location, using SRID 4326) and I have a Python application that using SQL Alchemy updates the table (the rest of the columns) without any problem.
Now, I need to update the location column and I know I can use the proper text representation of a given location to update the column using SQL Alchemy without the need to use GEOAlchemy, for instance I can update the column with the value: '0101000020E6100000AEAC7EB61F835DC0241CC418A2F74040'
which corresponds to lat:33.9346343 long:-118.0488106
The question is: is there a way to compute in Python this '0101000020E6100000AEAC7EB61F835DC0241CC418A2F74040' having this (33.9346343 ,-118.0488106) as an input without querying the database? or any way to update the column using a proper text input?
I know I can use SQLAlchemy to execute this query:
select st_setsrid(st_makepoint(-118.0488106, 33.9346343),4326)
and obtain the value to update the column, but I want to avoid that.
Thanks in advance!
The solution to this problem is rather easier than it seems. To update the field using text and the input lat-long all I needed to do was defining the SRID in the text assign:
location = 'SRID=4326;POINT(-118.0488106 33.9346343)'
This will update the geometry(point,4326) column properly and when you do a select in the table the value of the column is the expected one:
"0101000020E6100000AEAC7EB61F835DC0241CC418A2F74040"
Thanks guys!