SQL Server Realtime Replication pipeline is not working if table has a column of type decimal(38, 32) in the source. It keeps on running but no data is pull.
If I closely look at the Stackdriver logs, I see this error:
Numeric Field 'XXXXXXXX' has invalid precision '1' and scale '32'. Precision must be at most 38 and scale must be at most 9
The pipeline works if I deselect this column.
Bigquery is considering decimal(38,32) as a NUMERIC DECIMAL data type that supports precision 38 and scale up to 9.
I would suggest you to reduce it to the limits that the NUMERIC DECIMAL type admits or use the decimal type BIGNUMERIC DECIMAL which supports precision up to 76 and scale up to 38.
Related
I have files on S3 where two columns contain only positive integers which can be of 10^26. Unfortunately, according to AWS Docs Athena only supports values in a range up to 2^63-1 (approx 10^19). So at the moment these column are represented as a string.
When it comes to filtering it is not that big of an issue, as I can use regex. For example, if I want to get all records between 5e^21 and 6e^21 my query would look like:
SELECT *
FROM database.table
WHERE (regexp_like(col_1, '^5[\d]{21}$'))
I have approx 300M rows (approx 12GB in parquet) and it takes about 7 seconds, so performance wise it ok.
However, sometimes I would like to perform some math operation on these two big columns, e.g subtract one big column from another. Casting these records to DOUBLE wouldn't work due to approximation error. Ideally, I would want to stay within Athena. At the moment, I have about 100M rows that are greater then 2^63-1, but this number can grow in a future.
What would be the right way to approach problem of having numerical records that exceed available range? Also what are your thoughts on using regex for filtering? Is there a better/more appropriate way to do it?
You can cast numbers of the form 5e21 to an approximate 64bit double or an exact numeric 128bit decimal. First you'll need to remove the caret ^, with the replace function. Then a simple cast will work:
SELECT CAST(replace('5e^21', '^', '') as DOUBLE);
_col0
--------
5.0E21
or
SELECT CAST(replace('5e^21', '^', '') as DECIMAL);
_col0
------------------------
5000000000000000000000
If you are going to this table often, I would rewrite it the new data type to save processing time.
I was doing some tests on Postgres using the tinyint extension when I came across something surprising regarding its range. On typing select -128::tinyint it gave me an ERROR: tinyint out of range message which was not what I was expecting at all.
Assuming negative numbers should be 1 greater (or is it less) than the positive maximum (127 for single byte integers) I thought it was a bug with the extension, however on trying this with non-extended numbers I found exactly the same thing was happening.
select -32768::smallint -> out of range
select -2147483648::integer -> out of range
select -9223372036854775808::bigint -> out of range
Referring to the numeric data type documentation (https://www.postgresql.org/docs/current/datatype-numeric.html)
these numbers should all be possible - all negative numbers one less -32767, -2147483647, -9223372036854775807 work correctly so I am curious as to why this is happening, or does this even happen with other peoples copies.
I tried using both postgresql 10 and postgresql 11 on a ubuntu 16.x desktop.
I think this is because the cast operator :: has a higher precedence that the minus sign.
So -32768::smallint is executed as -1 * 32768::smallint which indeed is invalid.
Using parentheses fixes this: (-32768)::smallint or using the SQL standard cast() operator: cast(-32768 as smallint)
I'm changing a column in the database from "money" to "numeric" per some previous advice here.
Looking through the data types in postgres -- https://www.postgresql.org/docs/current/static/datatype-numeric.html -- I can't see any differences between numeric and decimal in the descriptions.
What is the difference between decimal and numeric, and is there any reason I should use numeric instead of decimal for prices in my database?
According to the manual they are the same.
The types decimal and numeric are equivalent. Both types are part of
the SQL standard.
https://www.postgresql.org/docs/current/static/datatype-numeric.html
The difference lies in the SQL standard which allows for different behaviour:
NUMERIC must be exactly as precise as it is defined — so if you define 4 decimal places, the DB must always store 4 decimal places.
DECIMAL must be at least as precise as it is defined. This means that the database can actually store more digits than specified (due to the behind-the-scenes storage having space for extra digits). This means the database might store 1.00005 instead of 1.0000, affecting future calculations.
Difference between DECIMAL and NUMERIC
I have a column of decimal (5,5). I want to put default value as "+000.00000". I tried with several options such in meta data such as setting precision=5 or using the option "Packed to no(separate)", setting sign position=leading, but all I am getting value as "+000000". How can I correct this?
am new to postgresql (redshift)
i am copying CSV files from S3 to RedShift and there's an error about trying to save 2.35555E7 number into a numeric | 18, 0 column . what is the right datatype for this datum ?
thanks
numeric (18,0) implies a scale of zero, which is a way of saying no decimals -- it's a bit like a smaller bigint.
http://www.postgresql.org/docs/current/static/datatype-numeric.html
If you want to keep it as numeric, you want to use numeric instead -- with no precision or scale.
If not, just use a real or a double precision type, depending on the number of significant digits (6 vs 15, respectively) you want to keep around.
Your example data (2.35555E7) suggests you're using real, so probably try that one first.
Note: select 2.35555E7::numeric(18,0) works fine per the comments, but I assume there's some other data in your set that is causing issues.