In Azure Data Factory, I'm doing a pretty vanilla 'Copy Data' operation. One dataset to another.
Can I view the query being used to perform the copy operation? Apparently, it's a syntax error, but I've only used drag-and-drop menus. Here's the error:
ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A
database operation failed. Please search error to get more
details.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Data.SqlClient.SqlException,Message=Incorrect
syntax near the keyword 'from'.,Source=.Net SqlClient Data
Provider,SqlErrorNumber=156,Class=15,ErrorCode=-2146232060,State=1,Errors=[{Class=15,Number=156,State=1,Message=Incorrect
syntax near the keyword 'from'.,},],'
Extra context
1.Clear Schema and import Schema again.
2. This mostly happens when there are some changes to table schema after creating pipeline and datasets. verify once.
3.The schema and datasets should be refreshed when there are some changes in the SQL table schema.
4. For table name or view name used in the query, use []. ex: [dbo].[persons]
5. In datasets select table name
6. try to publish before testing.
I followed the same scenario and reproduced the same thing. I didn’t get any error as you can see, the above error mainly happens because of schema.
Source dataset:
In source dataset, I manually added and connected to SQL database with Sample SQL table.
Sink dataset:
In sink dataset, I added another SQL database with auto create table, write behavior: Upset, key column: PersonID.
Before execution there is no table on SQL database, then after the execution was successful, I got this sample output in azure SQL database.
Related
I am trying to get a geography data type from a production DB to another DB on a nightly occurrence. I really wanted to leverage upsert as the write activity, but it seems that geography is not supported with this method. I was reading a similar post about bringing the data through ADF as a well known text data type and then changing it, but I keep getting confused on what to do with the data once it is brought over as a well known data type. I would appreciate any advice, thank you.
Tried to utilize ADF pipelines and data flows. Tried to convert the data type once it was in the destination, but then I was not able to run the pipeline again.
I tried to upsert the data with geography datatype from one Azure SQL database to another using copy activity and got error message.
Then, I did the upsert using dataflow activity. Below are the steps.
A source table is taken in dataflow as in below image.
CREATE TABLE SpatialTable
( id int ,
GeogCol1 geography,
GeogCol2 AS GeogCol1.STAsText() );
INSERT INTO SpatialTable (id,GeogCol1)
VALUES (1,geography::STGeomFromText('LINESTRING(-122.360 46.656, -122.343 46.656 )', 4326));
INSERT INTO SpatialTable (id,GeogCol1)
VALUES (2,geography::STGeomFromText('POLYGON((-122.357 47.653 , -122.348 47.649, -122.348 47.658, -122.358 47.658, -122.358 47.653))', 4326));
Then Alter Row transformation is taken and in Alter Row Conditions, Upsert if isNull(id)==false()is given. (Based on the column id, sink table upserted)
Then, in Sink dataset for target table is given. In sink settings, Update method is selected as Allow Upsert and required Key column is given. (Here column id is selected)
When pipeline is run for the first time, data is inserted into target table.
When pipeline is run for the second time by updating the existing data and inserting new records to source, data is upserted correctly.
Source Data is changed for id=1 and new row is inserted with id=3
Sink data is reflecting the changes done in source.
I'm trying to get basic plain SQL example working in Slick 3, on Postgres but with custom DB schema, say local instead of default public one. I have hard time inserting the row as executing the following
sqlu"INSERT INTO schedule(user_id, product_code, run_at) VALUES ($userId, $code, $nextRun)"
says
org.postgresql.util.PSQLException: ERROR: relation "schedule" does not exist
The table is in place because when I prefix schedule with local. in the insert statement it works as expected. How can I get correct schema provided to this query?
I'm using it as part of akka-projection handler and all the projection internals like maintaining offsets work as expected on local schema.
I cannot simply put schema as a variable as it errors while resolving parameters:
sqlu"INSERT INTO ${schema}.schedule(user_id, product_code, run_at) VALUES ($userId, $code, $nextRun)"
You can insert schema name using #${value}:
sqlu"INSERT INTO #${schema}.table ..."
I have to create a table in DB2 and read the query from file in Before/After Sql tab in Datastage.
I am using DB2 connector for this.
I have also parameterized the query but getting below error-
an unexpected token was found '/'.
create table Temp as(#Query#) with data
can u help in suggesting how can i achieve this successfully. Thanks in advance
Try loading the entire query into the parameter, rather than just the table name.
I have Azure Data Factory copy activity which loads parquet files to Azure Synapse. Sink is configured as shown below:
After data loading completed I had a staging table structure like this:
Then I create temp table based on stg one and it has been working fine until today when new created tables suddenly received nvarchar(max) type instead of nvarchar(4000):
Temp table creation now is failed with obvious error:
Column 'currency_abbreviation' has a data type that cannot participate in a columnstore index.'
Why the AutoCreate table definition has changed and how can I return it to the "normal" behavior without nvarchar(max) columns?
I've got exactly the same problem! I'm using a data factory to read csv-files into my Azure datawarehouse and this used to result in nvarchar(4000) columns, but now they are all nvarchar(max). I also get the error
Column xxx has a data type that cannot participate in a columnstore index.
My solution for now is to change my SQL code and use a CAST to change the formats, but there must be a setting in the data factory to get the former results back...
I am trying to tune the performance of a stored procedure that contains a temp table in Toad for SQL Server. After selecting "Include Actual Execution Plan" from the 'Editor' menu, I run the query. The Results Set returns values as expected, however, the Execution Plan tab shows the following error:
Invalid object name '#temp'.
I have tried creating the temp tables first then just executing the SELECT statement that references it, I tried creating the temp tables as global temp tables and running the SELECT statement in another window, and I have messed with the SHOWPLAN_TEXT and STATISTICS PROFILE (as mentioned in this question) but I keep receiving the same error. The only thing I have not tried is using a table variable, but the changes I will be making cannot be done on table variables, so this is not really an option for me at this time.
Has anyone else come across this or have any ideas as to what I might be doing wrong?
You'll want to use the ISQL command line utility on a machine that has SQL Server client package installed. Or any other utility that can submit a query to SQL Server.
ISQL Docs and How to get an execution plan (2nd part of the post)