I have created in pipeline parameter and i wanted to pass array of tables to perform multiple copy activity. BUT before that i am testing with only one table as a parameter.
I have created below query in expression (sink side of copy activity).
(Please note i am copying from synapse to synapse (from one schema to another))
#concat('select * from DW_GL.',pipeline().parameters.p_param_input_table,' where updated_on >',activity('Old_Lookup1').output.firstRow.date_value,' and updated_on <=',activity('Old_Lookup1').output.firstRow.date_value_new)
My table is in the schema "DW_GL" so i tried to remove it from above expression and still its not working.
i am getting error:
Failure happened on 'Source' side. ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A database operation failed with the following error: 'Parse error at line: 1, column: 75: Incorrect syntax near '03'.',Source=,''Type=System.Data.SqlClient.SqlException,Message=Parse error at line: 1, column: 75: Incorrect syntax near '03'.,Source=.Net SqlClient Data Provider,SqlErrorNumber=103010,Class=16,ErrorCode=-2146232060,State=1,Errors=[{Class=16,Number=103010,State=1,Message=Parse error at line: 1, column: 75: Incorrect syntax near '03'.,},],'
i tried removing empty white spaces from expression .
also tried to use direct parameter without concatenate as '#{pipeline().parameters.p_param_input_table}' still not worked.
I think the error is because this concat statement does not include single quotes (') around the date values. You need to use '' to include them, which translates to ''' to end (or begin) a string with ', and '''' to insert JUST a single ':
#concat('select * from DW_GL.',pipeline().parameters.p_param_input_table,' where updated_on > ''',activity('Old_Lookup1').output.firstRow.date_value,''' and updated_on <= ''',activity('Old_Lookup1').output.firstRow.date_value_new, '''')
Below query can also work :
select * from DW_GL.#{pipeline().parameters.p_param_input_table} where updated_on > '#{activity('Old_Lookup1').output.firstRow.date_value}' and updated_on <=' #{activity('Old_Lookup1').output.firstRow.date_value_new}'
Related
I am working on adf from snowflake to adls using data flow:
I am using Pipeline expression:
#concat('SELECT * FROM mySchema.myTable WHERE loadDate >= ', '''', '2022-07-01', '''')
It failed with the error message:
Operation on target Copy Data failed:
{"StatusCode":"DF-Executor-StoreIsNotDefined","Message":"Job failed
due to reason: The store configuration is not defined. This error is
potentially caused by invalid parameter assignment in the
pipeline.","Details":""}
(It worked when I directly run the query below in snowflake:
SELECT * FROM mySchema.myTable WHERE loadDate >= '2022-07-01')
But when I used Pipeline expression below (removing Where clause):
#concat('SELECT * FROM mySchema.myTable')
It worked.
Or if I used Pipeline expression below (using a different Where clause without timestamp comparison:
#concat('SELECT * FROM mySchema.myTable WHERE loadDate is not null')
This also worked.
So, my question is: why the first expression failed? How should I fix it?
Thanks in advance.
You can give the date in the string itself as suggested by #Sally Dabbah.
This is my repro with Azure SQL Database for your reference:
Give the query in the double quotes and date in single quotes in the dynamic content pipeline expression like below.
"select * from dbo.myschema where loaddate >= '2022-07-01'"
Give this parameter $query in the dataflow expression.
Pipeline succeded:
Data in the csv file in the sink (to single file):
I have a Lookup that retrieves a few records from a MS SQL table containing schema, table name and a whole where clause. These values are passed to a copy data (within a ForEach) In the copy data i use a Dynamic query statement like:
#concat('select a.*, current_date as crt_tms from ',item().shm_nam,'.',item().tab_nam,
item().where_clause )
This construction works fine without the where_clause or with a where clause with an integer. But it goes wrong with strings like:
'a where a.CODSYSBRN ='XXX' ;'
it's about the quote (')
How can i pass it through?
I know that the where clause as a fixed string in the dynamic query works when i use double quotes (to escape the single quote):
'a where a.CODSYSBRN =''XXX'' ;'
Point is i need the where clause to be completely dynamic because it differ per table
whatever i try i get this kind of error:
Syntax error or access violation;257 sql syntax error: incorrect syntax near "where a"
ps i also tested this, but with the same result:
select a.*, current_date as crt_tms from #{item().shm_nam}.#{item().tab_nam} a #{item().where_clause}
As you have mentioned you are getting whole where clause from the lookup table, the query must have included the column values in where clause for string and integer types separately.
Example lookup table:
In your copy activity, you can use Concat() function as you were already doing it, to combine static values & parameters.
#concat('select * from ',item().schma_name,'.',item().table_name,' ',item().where_clause)
For debugging purposes, I have added the expression in set variable activity, to see the value of the expression.
Iteration1:
Iteration2:
I am trying to query from a table within a copy command however, I have continually gotten errors. Here is the example SQL statement.
copy schema.table
from 's3://bucket/folder`
iam_role (select value from roles.iam where key = 'IAMRole');
The inner select statement on its own returns a value however, when I run the above, I get the following error:
SQL Error [500310] [42601]: [Amazon](500310) Invalid operation: syntax error at or near "("
The COPY command, as you must suspect, does not support embedded SQL.
If you want to do something like this, you can, but you'll need a procedure.
%sql select Name_of_School, Safety_Score from SCHOOLS where \
Safety_Score= (select MAX(Safety_Score) from SCHOOLS)
i am trying to execute this query i got the message.
ibm_db_sa://rbm44299:***#dashdb-txn-sbox-yp-lon02-04.services.eu-gb.bluemix.net:50000/BLUDB
(ibm_db_dbi.ProgrammingError) ibm_db_dbi::ProgrammingError: SQLNumResultCols failed: [IBM][CLI Driver][DB2/LINUXX8664] SQL0206N "SAFETY_SCORE" is not valid in the context where it is used. SQLSTATE=42703 SQLCODE=-206
[SQL: select Name_of_School, Safety_Score from SCHOOLS where Safety_Score= (select MAX(Safety_Score) from SCHOOLS)]
(Background on this error at: http://sqlalche.me/e/f405)
SQL0206N is this error message https://www.ibm.com/support/knowledgecenter/SSEPGG_11.5.0/com.ibm.db2.luw.messages.sql.doc/com.ibm.db2.luw.messages.sql.doc-gentopic1.html#sql0206n
SQL0206N name is not valid in the context where it is used.
This error can occur in the following cases:
For an INSERT or UPDATE statement, the specified column is not a column of the table, or view that was specified as the object of the insert or update.
For a SELECT or DELETE statement, the specified column is not a column of any of the tables or views identified in a FROM clause in the statement.
among other cases.
I.e. Column SAFETY_SCORE does not exist in your table. Maybe the column is "Safety_Score" or "Safety Score" or some other name.
If the column name is not in UPPER CASE in your table, you will need to surround it in double quotes.
I could fix by using the %%sql structure and the double quotes :
%%sql
select MAX("Safety Score") AS MAX_SAFETY_SCORE from Chicago_SCHOOLS;
I run a postgres database with postgis extention. With imposm3 openstreetmap data is loaded into the scheme import. The tables are called osm_point, osm_linestring and osm_polygon.
Imposm is storing NULL as empty fields.
To query a specific attribute of the table, I usually write
select * from import.osm_point where highway <> ''
But only with the attribute natural this query is not working.
As seen in this picture, the attribute natural contains some filled lines and some empty lines.
When running a query to select only natural without empt field, the following is happen.
ERROR: syntax error at or near "<>"
LINE 1: select * from import.osm_point where natural <> ''
^
SQL state: 42601
Character: 46```