I am working on adf from snowflake to adls using data flow:
I am using Pipeline expression:
#concat('SELECT * FROM mySchema.myTable WHERE loadDate >= ', '''', '2022-07-01', '''')
It failed with the error message:
Operation on target Copy Data failed:
{"StatusCode":"DF-Executor-StoreIsNotDefined","Message":"Job failed
due to reason: The store configuration is not defined. This error is
potentially caused by invalid parameter assignment in the
pipeline.","Details":""}
(It worked when I directly run the query below in snowflake:
SELECT * FROM mySchema.myTable WHERE loadDate >= '2022-07-01')
But when I used Pipeline expression below (removing Where clause):
#concat('SELECT * FROM mySchema.myTable')
It worked.
Or if I used Pipeline expression below (using a different Where clause without timestamp comparison:
#concat('SELECT * FROM mySchema.myTable WHERE loadDate is not null')
This also worked.
So, my question is: why the first expression failed? How should I fix it?
Thanks in advance.
You can give the date in the string itself as suggested by #Sally Dabbah.
This is my repro with Azure SQL Database for your reference:
Give the query in the double quotes and date in single quotes in the dynamic content pipeline expression like below.
"select * from dbo.myschema where loaddate >= '2022-07-01'"
Give this parameter $query in the dataflow expression.
Pipeline succeded:
Data in the csv file in the sink (to single file):
Related
I have created in pipeline parameter and i wanted to pass array of tables to perform multiple copy activity. BUT before that i am testing with only one table as a parameter.
I have created below query in expression (sink side of copy activity).
(Please note i am copying from synapse to synapse (from one schema to another))
#concat('select * from DW_GL.',pipeline().parameters.p_param_input_table,' where updated_on >',activity('Old_Lookup1').output.firstRow.date_value,' and updated_on <=',activity('Old_Lookup1').output.firstRow.date_value_new)
My table is in the schema "DW_GL" so i tried to remove it from above expression and still its not working.
i am getting error:
Failure happened on 'Source' side. ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A database operation failed with the following error: 'Parse error at line: 1, column: 75: Incorrect syntax near '03'.',Source=,''Type=System.Data.SqlClient.SqlException,Message=Parse error at line: 1, column: 75: Incorrect syntax near '03'.,Source=.Net SqlClient Data Provider,SqlErrorNumber=103010,Class=16,ErrorCode=-2146232060,State=1,Errors=[{Class=16,Number=103010,State=1,Message=Parse error at line: 1, column: 75: Incorrect syntax near '03'.,},],'
i tried removing empty white spaces from expression .
also tried to use direct parameter without concatenate as '#{pipeline().parameters.p_param_input_table}' still not worked.
I think the error is because this concat statement does not include single quotes (') around the date values. You need to use '' to include them, which translates to ''' to end (or begin) a string with ', and '''' to insert JUST a single ':
#concat('select * from DW_GL.',pipeline().parameters.p_param_input_table,' where updated_on > ''',activity('Old_Lookup1').output.firstRow.date_value,''' and updated_on <= ''',activity('Old_Lookup1').output.firstRow.date_value_new, '''')
Below query can also work :
select * from DW_GL.#{pipeline().parameters.p_param_input_table} where updated_on > '#{activity('Old_Lookup1').output.firstRow.date_value}' and updated_on <=' #{activity('Old_Lookup1').output.firstRow.date_value_new}'
I am executing this query:
select * from schema.country where country_code='abc' and country='INDIA'
inside ADF dataflow. I am getting this query from the database so I don't want to hardcode it inside the data flow. All the data flow inputs are adding a single quotes while executing. The dataflow is failing because of parsing error.
How can I escape single quotes ?
You can just pass your query in double quotes ("") in the dataflow to escape single quotes.
Please go through the below 3 approaches based on your usage of query in the dataflow.
This is my sample data from database:
If you are using Query in the source of dataflow, you can give the below expression in the Expression builder.
"select * from [dbo].[country] where country_code='abc' and country_name='INDIA'"
Output Data preview:
If you want to get your query from your database, then create a stored procedure for the query and access it inside the dataflow.
My stored procedure in the database:
create or alter procedure dbo.proc2
as
begin
select * from [dbo].[country] where country_code='abc' and country_name='INDIA'
end;
Use this query inside stored procedure in the dataflow like below. You will get the same output as above.
If you are using pipeline paramter to pass query to the dataflow, then create a parameter in the dataflow with default value. Give your query in double quotes in the pipeline parameters expression. Check the Expression check box after this.
Then give the $parameter1 in the expression builder of query and execute. Please check this SO thread to learn more it.
My csv in sink after pipeline execution in this process:
I have a Lookup that retrieves a few records from a MS SQL table containing schema, table name and a whole where clause. These values are passed to a copy data (within a ForEach) In the copy data i use a Dynamic query statement like:
#concat('select a.*, current_date as crt_tms from ',item().shm_nam,'.',item().tab_nam,
item().where_clause )
This construction works fine without the where_clause or with a where clause with an integer. But it goes wrong with strings like:
'a where a.CODSYSBRN ='XXX' ;'
it's about the quote (')
How can i pass it through?
I know that the where clause as a fixed string in the dynamic query works when i use double quotes (to escape the single quote):
'a where a.CODSYSBRN =''XXX'' ;'
Point is i need the where clause to be completely dynamic because it differ per table
whatever i try i get this kind of error:
Syntax error or access violation;257 sql syntax error: incorrect syntax near "where a"
ps i also tested this, but with the same result:
select a.*, current_date as crt_tms from #{item().shm_nam}.#{item().tab_nam} a #{item().where_clause}
As you have mentioned you are getting whole where clause from the lookup table, the query must have included the column values in where clause for string and integer types separately.
Example lookup table:
In your copy activity, you can use Concat() function as you were already doing it, to combine static values & parameters.
#concat('select * from ',item().schma_name,'.',item().table_name,' ',item().where_clause)
For debugging purposes, I have added the expression in set variable activity, to see the value of the expression.
Iteration1:
Iteration2:
I have source query in Copy data activity of Azure Data Factory. Currently it is static query, but I would like to get entire statement from database and use that in Query in Activity.
I try to pass SQL statement as variable there is issue.
Lookup SQL Query for fetching Select statement:
SELECT [source_query] FROM [SalesLT].[configuration] WHERE [Id] = '1'
Query returns and set variable:
SELECT Count(CustomerId) Source_Rows FROM [SalesLT].[Customer]
Error with Copy data:
Type=System.Data.SqlClient.SqlException,Message=Incorrect syntax near
')'.,Source=.Net SqlClient Data
You can't directly use the pipeline parameter in Query instead need to use #concat function to concatenate your SQL Query with parameters. Please refer the example below:
I suggest you to please go through this tutorial from #TechBrothersIT to get more clarification on the related issue.
I got the requirement like this,
Create a single Talend job with reads multiple tables and writes to multiple files dynamically(when we give a tablename via context variable the job should take that table as select * from tablename and writes to file tablename.txt)
My oracle query given in toracle input stage-
"SELECT * FROM '"+context.Table_Name+"'"
In Context Variable part given as
Table_Name- String- checked Prompt for value for dynamic table name
In the metadata definition for Oracle table I gave as
Type="dynamic" db type="varchar2"
Issues Facing:
The context variable is not been identified by the job
ORA-00903: invalid table name
Exception in component tOracleInput_1
java.sql.SQLSyntaxErrorException: ORA-00903: invalid table name
at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:91)
at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:112)
at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:173)
But when I hard code the tablename, job is running fine
The target file path I gave as
"C:/Talend/OutputFIles/context.Table_Name.txt"
Instead of printing value of context variable, I am getting as context.Table_Name.txt as filename!!!
Please help me on this
Did you try to remove ' around the table name?Try this:
"SELECT * FROM " + context.Table_Name
Same for filename construction, you should write:
"C:/Talend/OutputFIles/" + context.Table_Name + ".txt"
Better, you should have pathname defined by a context variable, giving:context.OutputPath + context.Table_Name + ".txt"TRF