I have loaded the excel data into a table using SSIS.
Table Structure :
Monthly_Budget
SEQUENCE INT IDENTITY,
TRANSACTION_DATE VARCHAR(100),
TRANSACTION_REMARKS VARCHAR(1000),
WITHDRAWL_AMOUNT VARCHAR(100),
DEPOSIT_AMOUNT VARCHAR(100),
BALANCE_AMOUNT VARCHAR(100)
Values in WITHDRAWL_AMOUNT Column:
7,987.00
1,500.00
7,000.00
50.00
NULL
253.00
4,700.00
2,000.00
148.00
2,000.00
64.00
1,081.00
2,000.00
NULL
NULL
7,000.00
Now, I am trying to run a query to get the summation of values under WITHDRWAL_AMOUNT but I am getting an error :
Error converting data type varchar to numeric.
My Query :
SELECT SUM(CAST(ISNULL(LTRIM(RTRIM(WITHDRAWL_AMOUNT)),0) AS NUMERIC(6,2))) AS NUM FROM MONTHLY_BUDGET
Try converting them like this:
select SUM(CAST(ltrim(rtrim(replace(WITHDRAWL_AMOUNT, ',', ''))) as numeric(6, 2)) )
It is much, much preferable to store the values in the proper types that you want. I can, however, understand putting external data into a staging table and then using logic such as the above to load the data into the final table.
Related
i will import data csv to postgresql via pgAdmin 4. But, there are problem
ERROR: invalid input syntax for type integer: ""
CONTEXT: COPY films, line 1, column gross: ""
i understand about the error that is line 1 column gross there is null value and in some other columns there are also null values. My questions, how to import file csv but in the that file there is null value. I've been search in google but not found similar my case.
CREATE TABLE public.films
(
id int,
title varchar,
release_year float4,
country varchar,
duration float4,
language varchar,
certification varchar,
gross int,
budget int
);
And i try in this code below, but failed
CREATE TABLE public.films
(
id int,
title varchar,
release_year float4 null,
country varchar null,
duration float4 null,
language varchar null,
certification varchar null,
gross float4 null,
budget float4 null
);
error message in image
I've searched on google and on the stackoverflow forums. I hope that someone will help solve my problem
There is no difference between the two table definitions. A column accepts NULL by default.
The issue is not a NULL value but an empty string:
select ''::integer;
ERROR: invalid input syntax for type integer: ""
LINE 1: select ''::integer;
select null::integer;
int4
------
NULL
Create a staging table that has data type of varchar for the fields that are now integer. Load the data into that table. Then modify the empty string data that will be integer using something like:
update table set gross = nullif(trim(gross), '');
Then move the data to the production table.
This is not a pgAdmin4 issue it is a data issue. Working in psql because it is easier to follow:
CREATE TABLE public.films_text
(
id varchar,
title varchar,
release_year varchar,
country varchar,
duration varchar,
language varchar,
certification varchar,
gross varchar,
budget varchar
);
\copy films_text from '~/Downloads/films.csv' with csv
COPY 4968
CREATE TABLE public.films
(
id int,
title varchar,
release_year float4,
country varchar,
duration float4,
language varchar,
certification varchar,
gross int,
budget int
);
-- Below done because of this value 12215500000 in budget column
alter table films alter COLUMN budget type int8;
INSERT INTO films
SELECT
id::int,
title,
nullif (trim(release_year), '')::real, country, nullif(trim(duration), '')::real,
LANGUAGE,
certification,
nullif (trim(gross), '')::float, nullif(trim(budget), '')::float
FROM
films_text;
INSERT 0 4968
It worked for me:
https://learnsql.com/blog/how-to-import-csv-to-postgresql/
a small workaround but it works
I created a table
I added headers to csv file
Right click on the newly created table-> Import/export data, select csv file to upload, go to tab2 - select Header and it should work
SQL server db has a time column datatype of
[start_time] time NULL,
[end_time] time NULL,
but dataflows doesn't have a function for this.
The only way I can think of doing this is a post query (if you fully recreate the table)
alter table dbo.[testTable]
alter column [start_time] time(0)
alter column [end_time] time(0)
I tried using timestamp but again it's not a matching datatype
toTimestamp(substring(start_date,12,9),'HH:mm:ss')
so this doesn't work.
Any help on understanding this would be great
** updating with screens shots
So this issue is for parquet or csv to Sql db tables.
if you have a column that looks like DateTime you need to keep it as a string as there is no toDateTime function only toTimestamp. Neither string or Timestamp can be converted to DateTime datatype in SQLdb sink. You will end up with nulls in your column
Sample befor using expression to change the start_date to yyyy-mm-dd THH:mm:ss
You can simply map the DATETIME column to the target TIME column in the sink activity.
Make sure the option "Allow schema drift" in sink activity is unchecked.
My test schema:
-- source table
DROP TABLE IF EXISTS [dbo].[tempSourceTable]
CREATE TABLE [dbo].[tempSourceTable](
[id] int IDENTITY(1,1) NOT NULL,
[key] nvarchar(max) NULL,
[start_date] datetime NULL
)
INSERT INTO [dbo].[tempSourceTable] VALUES ('key1', '2021-10-14 12:34:56')
SELECT * FROM [dbo].[tempSourceTable]
-- target table
DROP TABLE IF EXISTS [dbo].[tempTargetTable]
CREATE TABLE [dbo].[tempTargetTable](
[id] int IDENTITY(1,1) NOT NULL,
[key] nvarchar(max) NULL,
[start_time] time NULL
)
result after execute the dataflow in a pipeline:
Here is my testing CSV input:
start_date,end_date,start_date_time,end_date_time,start_time,end_time
09/01/2020,09/01/2020,09/01/2020 11:01,09/01/2020 11:01,11:01:46,11:01:52
09/01/2020,,09/01/2020 11:01,,11:01:47,
09/01/2020,09/01/2020,09/01/2020 11:01,09/01/2020 11:50,11:01:49,11:50:41
09/01/2020,09/01/2020,09/01/2020 11:01,09/01/2020 11:01,11:01:51,11:01:55
09/01/2020,09/01/2020,09/01/2020 11:01,09/01/2020 11:01,11:01:52,11:01:56
You may specify the data/time/datetime format for CSV source data:
You can see the correct parsing result in data preview:
After that, a simple sink activity should achieve what OP wants to do:
The sink table schema I used for testing:
CREATE TABLE [dbo].[tempTargetTable](
[start_date] date NULL,
[end_date] date NULL,
[start_date_time] datetime NULL,
[end_date_time] datetime NULL,
[start_time] time NULL,
[end_time] time NULL
)
Result in DB:
I would like to have a generated column, which value will be the concated string from two other values:
CREATE TABLE public.some_data (
user_name varchar NULL,
domain_name serial NOT NULL,
email GENERATED ALWAYS AS (user_name ||'#'||domain_name) stored
);
But that gives SQL Error [42601]: ERROR: syntax error at or near "ALWAYS"
You need to provide the data type for the column as #Belayer commented.
And then you need to explicitly cast domain_name as text (or some varchar). Otherwise you'll get an error that the expression isn't immutable as #nbk commented. serial is translated to be basically an integer and for whatever reason implicit casts of an integer in concatenations are considered not immutable by the engine. We had that just recently here.
So overall, using the given types for the columns, you want something like:
CREATE TABLE public.some_data
(user_name varchar NULL,
domain_name serial NOT NULL,
email text GENERATED ALWAYS AS (user_name || '#' || domain_name::text) STORED);
But it's a little weird that a domain name is a serial? Shouldn't that be a text or similar? Then you wouldn't need the cast of course.
You need to create an IMMUTABLE function to achieve the generate column, for example:
CREATE OR REPLACE FUNCTION generate_email_concat(varchar,int) returns text as
$$
select $1 ||'#'||$2::text;
$$
LANGUAGE SQL IMMUTABLE ;
CREATE TABLE public.some_data (
user_name varchar NULL,
domain_name serial NOT NULL,
email text GENERATED ALWAYS AS (generate_email_concat(user_name,domain_name)) stored
);
INSERT into some_data(user_name) values ('hello');
You try to concatenate varchar and integer. You have to cast domain_name. This works for me
CREATE TABLE public.some_data (
user_name varchar NULL,
domain_name serial NOT NULL,
email varchar GENERATED ALWAYS AS (CASE WHEN user_name IS NULL THEN 'noname'||'#'||domain_name::text ELSE user_name ||'#'||domain_name::text END) STORED
);
I get an error trying to insert data into my tables ... but I don't know why?
Syntax is correct.
column "population" is of type integer but expression is of type record
create table states(name varchar(25), population int );
create table countries(name varchar(25), population int );
insert into states values (('tn',54945),('ap',2308));
select name from states;
insert into countries values (('india',3022),('america',30902));
select * from countries;
There are extra parentheses around the tuples of values to insert, which turns the whole thing to a single record of records.
Instead:
insert into countries(name, population) values ('india',3022),('america',30902);
I have a table created like
CREATE TABLE data
(value1 smallint references labels,
value2 smallint references labels,
value3 smallint references labels,
otherdata varchar(32)
);
and a second 'label holding' table created like
CREATE TABLE labels (id serial primary key, name varchar(32));
The rationale behind it is that value1-3 are a very limited set of strings (6 options) and it seems inefficient to enter them directly in the data table as varchar types. On the other hand these do occasionally change, which makes enum types unsuitable.
My question is, how can I execute a single query such that instead of the label IDs I get the relevant labels?
I looked at creating a function for it and stumbled at the point where I needed to pass the label holding table name to the function (there are several such (label holding) tables across the schema). Do I need to create a function per label table to avoid that?
create or replace function translate
(ref_id smallint,reference_table regclass) returns varchar(128) as
$$
begin
select name from reference_table where id = ref_id;
return name;
end;
$$
language plpgsql;
And then do
select
translate(value1, labels) as foo,
translate(value2, labels) as bar
from data;
This however errors out with
ERROR: relation "reference_table" does not exist
All suggestions welcome - at this point a can still alter just about anything...
CREATE TABLE labels
( id smallserial primary key
, name varchar(32) UNIQUE -- <<-- might want this, too
);
CREATE TABLE data
( value1 smallint NOT NULL REFERENCES labels(id) -- <<-- here
, value2 smallint NOT NULL REFERENCES labels(id)
, value3 smallint NOT NULL REFERENCES labels(id)
, otherdata varchar(32)
, PRIMARY KEY (value1,value2,value3) -- <<-- added primary key here
);
-- No need for a function here.
-- For small sizes of the `labels` table, the query below will always
-- result in hash-joins to perform the lookups.
SELECT l1.name AS name1, l2.name AS name2, l3.name AS name3
, d.otherdata AS the_data
FROM data d
JOIN labels l1 ON l1.id = d.value1
JOIN labels l2 ON l2.id = d.value2
JOIN labels l3 ON l3.id = d.value3
;
Note: labels.id -> labels.name is a functional dependency (id is the primary key), but that doesn't mean that you need a function. The query just acts like a function.
You can pass the label table name as string, construct a query as string and execute it:
sql = `select name from ` || reference_table_name || `where id = ` || ref_id;
EXECUTE sql INTO name;
RETURN name;