Why does Azure Data Flow convert varbinary(1000) to varbinary(max) and how can I prevent this? - type-conversion

My source table, which is located in an Azure SQL Server Data Warehouse, has a column named Upline with the data type varbinary(1000). In the destination table, located in the same Azure SQL Server Data Warehouse, the data type and column name are the same. My issue is in the Azure Data Flow that is populating the destination table.
Instead of inserting the data into the sink table in the data flow, it is creating a new table in my data warehouse. Here is the create statement for the table that is being created with
CREATE TABLE [Common].[T_7be15bb497654f0c8eeb82459912f178]
(
[EmployeeSK] [int] NULL,
[EmployeeLastName] [nvarchar](max) NULL,
[EmployeeFirstName] [nvarchar](max) NULL,
[EmploymentStatus] [nvarchar](max) NULL,
[HireDate] [date] NULL,
[OriginalHireDate] [date] NULL,
[TerminationDate] [date] NULL,
[CurrentPosition] [nvarchar](max) NULL,
[PreviousPosition] [nvarchar](max) NULL,
[WorkAssignmentEffectiveStart] [date] NULL,
[Region] [nvarchar](max) NULL,
[District] [nvarchar](max) NULL,
[Site] [nvarchar](max) NULL,
[OnSiteDepartment] [nvarchar](max) NULL,
[DepartmentName] [nvarchar](max) NULL,
[ManagerDayForceEmployeeNumber] [nvarchar](max) NULL,
[Upline] [varbinary](max) NULL,
[Lvl] [int] NULL,
[dimStartDate] [date] NULL,
[dimEndDate] [date] NULL,
[dimIsCurrent] [int] NULL,
[dimHash] [nvarchar](max) NULL,
[r7ace46966877481a90d6f8039c6524b5] [int] NULL
)
As you can see from the picture (if you can see the picture), the table is giving the column Upline a varbinary(max) data type. Why is this happening? How can I stop it from happening? When I take this column out of the source and destination tables it works successfully. However, I need the column. The data flow has these activities: source, select, derived column, surrogate key, and sink. It is doing very simple stuff and the Upline is not being changed in the derived column.
Here is the error message from the pipeline that runs the data flow:
"Found an implicit conversion from VarBinary(Max) to VarBinary(1000) that requires ANSI truncation warning. This is not supported. Use the CONVERT function explicitly to execute this request." There is no convert function in the derived column activity so I can't do the suggestion it gives.
The max length of the data in the column is 24 (found using Select len(max(Upline))FROM [source table]).
Any help would be appreciated. Thanks.

I found a work around. In the create statement of the destination table I changed varbinary(1000) to be varbinary(max) and, at the end, I replaced this ending:
WITH
(
DISTRIBUTION = ROUND_ROBIN,
CLUSTERED COLUMNSTORE INDEX
)
GO
with this:
WITH
(
DISTRIBUTION = ROUND_ROBIN,
HEAP
)
GO
Now, the whole create statement looks like this:
CREATE TABLE [Common].[dimEmployee_temp]
(
[EmployeeSK] [int] IDENTITY(1,1) NOT NULL,
[DayForceEmployeeNumber] [nvarchar](255) NOT NULL,
[ConaEmployeeNumber] [char](10) NULL,
[EmployeeLastName] [nvarchar](255) NULL,
[EmployeeFirstName] [nvarchar](255) NULL,
[EmploymentStatus] [nvarchar](255) NULL,
[HireDate] [date] NULL,
[OriginalHireDate] [date] NULL,
[TerminationDate] [date] NULL,
[CurrentPosition] [nvarchar](255) NULL,
[PreviousPosition] [nvarchar](255) NULL,
[WorkAssignmentEffectiveStart] [date] NULL,
[Region] [nvarchar](255) NULL,
[District] [nvarchar](255) NULL,
[Site] [nvarchar](255) NULL,
[OnSiteDepartment] [nvarchar](255) NULL,
[DepartmentName] [nvarchar](255) NULL,
[UnionName] [nvarchar](255) NULL,
[ManagerDayForceEmployeeNumber] [nvarchar](255) NULL,
[Upline] [varbinary](max) NULL,
[Lvl] [int] NULL,
[dimStartDate] [date] NOT NULL,
[dimEndDate] [date] NULL,
[dimIsCurrent] [int] NOT NULL,
[dimHash] [nvarchar](256) NULL
)
WITH
(
DISTRIBUTION = ROUND_ROBIN,
HEAP
)
GO
This doesn't help me understand what is going on in the Azure data flow, but it does run successfully and insert into the table.

Related

SQL Server Wizard says my source datetime's are datetime2's. Process cannot be run

I am trying to export some records from ServerA.DatabaseA..Anchor to ServerB.DatabaseB..Anchor. The source is active. The destination is a development database. The destination has one additional field, a tenancy id.
SQL for Source table
CREATE TABLE [dbo].[Anchor](
[AnchorId] [int] IDENTITY(130,1) NOT NULL,
[NodeId] [int] NOT NULL,
[UnitName] [varchar](200) NOT NULL,
[SetPosition] [varchar](5) NOT NULL,
[CreateDate] [datetime] NOT NULL,
[StartDate] [datetime] NOT NULL,
[EndDate] [datetime] NOT NULL,
[Latitude] [float] NULL,
[Longitude] [float] NULL,
[RadiusMeters] [int] NOT NULL,
[Creator] [varchar](100) NULL,
[Cellnumber1] [varchar](20) NULL,
[Cellnumber2] [varchar](20) NULL,
[Cellnumber3] [varchar](20) NULL,
[EmailTo] [varchar](255) NULL,
[UseMsgFwdContacts] [bit] NOT NULL,
CONSTRAINT [PK_Anchor] PRIMARY KEY CLUSTERED
(
[AnchorId] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO
ALTER TABLE [dbo].[Anchor] ADD CONSTRAINT [DF_Anchor_UseMsgFwdContacts] DEFAULT ((0)) FOR [UseMsgFwdContacts]
GO
SQL Query for selecting records
Datetime's are cast to datetime for testing
SELECT [AnchorId]
,'MXM' as OrgCode -- the new tenancy field
,[NodeId]
,[UnitName]
,[SetPosition]
,CAST([CreateDate] as DATETIME) as [CreateDate]
,CAST([StartDate] as DATETIME) as [StartDate]
,CAST([EndDate] as DATETIME) as [EndDate]
,[Latitude]
,[Longitude]
,[RadiusMeters]
,[Creator]
,[Cellnumber1]
,[Cellnumber2]
,[Cellnumber3]
,[EmailTo]
,[UseMsgFwdContacts]
FROM [client_Maxam].[dbo].[Anchor]
WHERE AnchorId >335
Click Ok and Next, and now...
Why is my source datatype being picked up as datetime2, preventing me from running the export.

Return values from left join only if it exists, otherwise from the main table

I have 2 tables
CREATE TABLE [dbo].[Owners]
(
[OwnerId] [int] NOT NULL,
[AccessToken] [nvarchar](50) NULL,
[TokenSecret] [nvarchar](50) NULL
)
CREATE TABLE [dbo].[Tweets]
(
[TweetId] [int] IDENTITY(1,1) NOT NULL,
[ReferenceId] [int] NULL,
[TweetContent] [nvarchar](max) NULL,
[ReferenceType] [int] NOT NULL,
[AccessToken] [nvarchar](50) NULL,
[TokenSecret] [nvarchar](50) NULL,
)
I would like to return all fields of tweets, and based on if there is an owners for the tweet, the AccessToken/TokenSecret will come from the owner table, if not then it will come from the tweets table.
I am having a hard time figuring out the best/efficient way to write this query.
This is what I have so far (only returns from main table, not owner if there is one)
SELECT *
FROM Tweets t
LEFT JOIN Owners o ON t.ReferenceId = o.OwnerId
WHERE t.ReferenceType = 1
I am using SQL Server 2017, in the tweet table, ReferenceId is the ownerid correspondence to he tweets table
Perhaps coalesce would be helpful to select the first non-null value:
select coalesce(o.AccessToken, t.AccessToken) as actualAccessToken [...]

postgresql cannot insert data to newly added column

In postgresql I have a table which I need to add a new column. the original table ddl is belowing:
CREATE TABLE survey.survey_response (
id uuid NOT NULL DEFAULT uuid_generate_v4(),
survey_id uuid NOT NULL,
survey_question_id uuid NULL,
user_id varchar(256) NULL,
device_id varchar(256) NULL,
user_country varchar(100) NULL,
client_type varchar(100) NULL,
product_version varchar(100) NULL,
answer text NULL,
response_date timestamptz NOT NULL DEFAULT now(),
survey_category varchar(100) NULL,
tags varchar(250) NULL,
tracking_id uuid NULL,
CONSTRAINT survey_response_pkey PRIMARY KEY (id)
)
WITH (
OIDS=FALSE
) ;
Then I alter the table to add a new column:
alter table survey.survey_response add column system_tags varchar(30) ;
But after that I found my instert statement cannot make change to this new column, for all the original columns it works fine:
INSERT INTO survey.survey_response
(id, survey_id, user_id, tags, system_tags)
VALUES(uuid_generate_v4(), uuid_generate_v4(),'1123','dsfsd', 'dsfsd');
select * from survey.survey_response where user_id = '1123';
The "tags" columns contains inserted value, however, system_tags keeps null.
I tested the above scenario in my local postgreSQL 9.6, any ideas about this strange behavior? Thanks a lot
-----------------update----------
I found this survey.survey_response table has been partitioning based on month, So my inserted record will also be displayed in survey.survey_response_y2017m12. but the new system_tags column is also NULL
CREATE TABLE survey.survey_response_y2017m12 (
id uuid NOT NULL DEFAULT uuid_generate_v4(),
survey_id uuid NOT NULL,
survey_question_id uuid NULL,
user_id varchar(256) NULL,
device_id varchar(256) NULL,
user_country varchar(100) NULL,
client_type varchar(100) NULL,
product_version varchar(100) NULL,
answer text NULL,
response_date timestamptz NOT NULL DEFAULT now(),
survey_category varchar(100) NULL,
tags varchar(250) NULL,
tracking_id uuid NULL,
system_tags varchar(30) NULL,
CONSTRAINT survey_response_y2017m12_response_date_check CHECK (((response_date >= '2017-12-01'::date) AND (response_date < '2018-01-01'::date)))
)
INHERITS (survey.survey_response)
WITH (
OIDS=FALSE
) ;
If I run the same scenario in a non-partition table then the insert works fine.
So do I need any special settings for alter table for partition table?
Old thread but you need to drop and create again the RULE to fix the issue.

How to DROP and CREATE a table in SQL Server 2016 Stored Procedure

I am trying to create a stored procedure that will drop a table and then create a new table with the same name.
However, when I right clicked on the table and did Script Table as ... DROP and CREATE To ... New Query Editor Window and then copied the script into a blank stored procedure, it wouldn't let me create the procedure because
There is already an object named 'MyTable' in the database
What should I do?
NOTE: the new table will not be the same structure as the old table. I make some transformations using other stored procedures to the table, so at the end, I want to drop it and create from scratch.
Here is the actual script:
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE PROCEDURE [dbo].[spDropAndCreate]
AS
BEGIN
SET NOCOUNT ON;
IF OBJECT_ID ('dbo.MyTable','U') IS NOT NULL
DROP TABLE [dbo].[MyTable]
CREATE TABLE [dbo].[MyTable]
(
[ID] [int] IDENTITY(1,1) NOT NULL,
[Col1] [nvarchar](50) NULL,
[Col2] [nvarchar](50) NULL,
[Col3] [nvarchar](50) NULL,
[Col4] [nvarchar](50) NULL,
[Col5] [nvarchar](50) NULL,
[Col6] [nvarchar](50) NULL,
[Col7] [nvarchar](50) NULL,
[Col8] [nvarchar](50) NULL,
[Col9] [nvarchar](50) NULL,
[Col10] [nvarchar](50) NULL
)
GO
END
Here is the error message:
Msg 102, Level 15, State 1, Procedure spDropAndCreate, Line 17 [Batch Start Line 9]
Incorrect syntax near 'MyTable'
Msg 2714, Level 16, State 6, Line 35
There is already an object named 'MyTable' in the database
Msg 102, Level 15, State 1, Line 156
Incorrect syntax near 'END'
You need to remove the GO that is within the BEGIN and END block. GO is a batch separator which is like telling the block to execute before it's even ended. Check out this previous post for more info.
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
CREATE PROCEDURE [dbo].[spDropAndCreate]
AS
BEGIN
SET NOCOUNT ON;
IF OBJECT_ID ('dbo.MyTable','U') IS NOT NULL
DROP TABLE [dbo].[MyTable]
CREATE TABLE [dbo].[MyTable]
(
[ID] [int] IDENTITY(1,1) NOT NULL,
[Col1] [nvarchar](50) NULL,
[Col2] [nvarchar](50) NULL,
[Col3] [nvarchar](50) NULL,
[Col4] [nvarchar](50) NULL,
[Col5] [nvarchar](50) NULL,
[Col6] [nvarchar](50) NULL,
[Col7] [nvarchar](50) NULL,
[Col8] [nvarchar](50) NULL,
[Col9] [nvarchar](50) NULL,
[Col10] [nvarchar](50) NULL
)
END
Add a GO after the DROP TABLE statement to separate the batches.
This won’t work for a procedure though. Instead uses exists when sys objects.
If exists(select 1 from sys.objects where name = 'mytable')
drop mytable

Creating composite foreign key constraint

I am trying to create a composite foreign key relationship/constraint. All tables are empty. I have this table:
CREATE TABLE [dbo].[ChemSampleValueTest](
[SampleNumber] [int] NOT NULL,
[ParameterID] [int] NOT NULL,
[Value] [numeric](18, 6) NOT NULL,
[Accuracy] [varchar](50) NULL,
[ResultGroupID] [int] NOT NULL,
[QAState] [nvarchar](32) NOT NULL,
CONSTRAINT [PK_SampleValueTest] PRIMARY KEY CLUSTERED
(
[SampleNumber] ASC,
[ParameterID] ASC,
[ResultGroupID] ASC
)
) ON [PRIMARY]
and this table:
CREATE TABLE [dbo].[ChemSampleValueEventLinkTest](
[Event] [int] NOT NULL,
[SampleNumber] [int] NOT NULL,
[ResultGroupID] [int] NOT NULL,
[ParameterID] [int] NOT NULL,
[QAState] [nvarchar](32) NULL
) ON [PRIMARY]
and I want to link them like this:
alter table [ChemSampleValueEventLinkTest] add
constraint FK_ChemSampleValueEvent_ChemSampleValue_test
foreign key ([SampleNumber], [ResultGroupID], [ParameterID])
references ChemSampleValueTest ([SampleNumber], [ResultGroupID], [ParameterID])
As far as I can tell all column types are the same, but it keeps on saying
There are no primary or candidate keys in the referenced table
'ChemSampleValueTest' that match the referencing column list in the foreign key
'FK_ChemSampleValueEvent_ChemSampleValue_test'.
Where am I going wrong?
It looks like you need to have your FK/references list in the same order they appear in the PK definition.
This should work:
CREATE TABLE [dbo].[ChemSampleValueTest](
[SampleNumber] [int] NOT NULL,
[ParameterID] [int] NOT NULL,
[Value] [numeric](18, 6) NOT NULL,
[Accuracy] [varchar](50) NULL,
[ResultGroupID] [int] NOT NULL,
[QAState] [nvarchar](32) NOT NULL,
CONSTRAINT [PK_SampleValueTest] PRIMARY KEY CLUSTERED
(
[SampleNumber] ASC,
[ParameterID] ASC,
[ResultGroupID] ASC
)
) ON [PRIMARY]
CREATE TABLE [dbo].[ChemSampleValueEventLinkTest](
[Event] [int] NOT NULL,
[SampleNumber] [int] NOT NULL,
[ResultGroupID] [int] NOT NULL,
[ParameterID] [int] NOT NULL,
[QAState] [nvarchar](32) NULL
) ON [PRIMARY]
alter table [ChemSampleValueEventLinkTest] add
constraint FK_ChemSampleValueEvent_ChemSampleValue_test
foreign key ([SampleNumber], [ParameterID], [ResultGroupID])
references ChemSampleValueTest ([SampleNumber], [ParameterID], [ResultGroupID])