I am copying over 20,000 rows from a table in Postgres to Bigquery. In both Postgres and Bigquery the datatype is Numeric (13,2) or Numeric (12,8) for columns that are Numeric.
It manages to copy over the correct values from Postgres for all the columns except the ones with Numeric datatype. It would move a decimal place or won't capture the full number.
I am not sure if Numeric types are need to be redefined in bigquery or if I need to add a wrapper.
Related
I am trying to write a variety of columns to redshift from a dynamic frame using the DynamicFrameWriter.from_jdbc_conf method, but all DECIMAL fields end up as a column of NULLs.
The ETL pulls in from some redshift views, and eventually writes back to a redshift table using the aforementioned method. In general this works for other datatypes, but adding a statement as simple as SELECT CAST(12.34 AS DECIMAL(4, 2)) AS decimal_test results in a column full of NULLs. In pyspark, I can print out the column and see the decimal values immediately before they are written as NULLs to the redshift table. When I look at the schema on redshift, I can see the column decimal_test has a type of NUMERIC(4, 2). What could be causing this failure?
I need to move data from PostgreSQL's jsonb to Redshift's SUPER types.
However, some of the values from the jsonb are higher than the 65535 limit on VARCHAR values that exists on Redshift (and SUPER types also for that matter).
Is there a way to select from jsonb truncating values? E.g. applying a map function (to truncate up to a number of characters) to each value on jsonb?
Bonus: Can this be done recursively?
I am creating an Athena external table on a CSV that I generated from my PostgreSQL database.
The csv contains a columns that has a jsonb datatype.
If possible, I want to exclude this column from the table created in Athena, or kindly suggest a way to include this datatype.
I tried to load the redshift table but failed on one column- The length of the data column 'column_description'is longer than the length defined in the table. Table: 65535, Data: 86555.
I tried to increase the length of column in RS table, looks like 65535 is the max length RS supports.
Do we have any alternatives to store value in Redshift?
The answer is that Redshift doesn't support anything larger and that one shouldn't store large artifacts in an analytic database. If you are using Redshift for its analytic powers to find specific artifacts (images, files, etc) then these should be stored in S3 and the object key (pointer) should be stored in redshift.
I am using PostgreSQL backend with linked tables in Access. On using the wizard to link to the linked tables, I get errors:
Scaling of decimal value resulted in data truncation
This appears to be the wrong scale for numeric data types being chosen as the default by Access: the Postgresql data type being linked is Numeric with no precision or scale defined, and is being linked as Decimal with precision 28 and scale 6 as default.
How can I get Access to link it as Double?
I see here MS Access linked tables automatically long integers that the self-answer was:
Figured it out (and I feel dumb): When linking tables you can choose
the desired format for each field when going through the linked table
wizard steps.
But, I see no option in Access to choose the desired format during linking.
If there is anything like a "default" data type when creating an ODBC linked table in Access, that type would be Text(255). That is, if the ODBC driver reports a column with a data type that Access does not support (e.g. TIME in SQL Server) then Access will include it as a Text(255) column in the linked table.
In this case, for a PostgreSQL table
CREATE TABLE public.numeric_test_table
(
id integer NOT NULL,
text_col character varying(50),
numeric_col numeric,
CONSTRAINT numeric_test_table_pk PRIMARY KEY (id)
)
WITH (
OIDS=FALSE
);
the PostgreSQL ODBC driver is actually reporting the numeric column as being numeric(28,6) as confirmed by calling OdbcConnection#GetSchema("columns") from C#
so that is what Access uses as the column type for its linked table. It is only when Access goes to retrieve the actual data that the PostgreSQL ODBC driver sends back values that won't "fit" in the corresponding column of the linked table.
So no, there is almost certainly no overall option to tell Access to treat all numeric (i.e., Decimal) columns as Double. The "best" solution would be to alter the PostgreSQL table definitions to explicitly state the precision and scale, as suggested in the PostgreSQL documentation:
If you're concerned about portability, always specify the precision and scale [of a numeric column] explicitly.
If modifying the PostgreSQL database is not feasible then another option would be to use a pass-through query in Access to explicitly convert the column to Double ...
SELECT id, text_col, numeric_col::double precision FROM public.numeric_test_table
... bearing in mind that pass-through queries always return read-only recordsets.