I have Lookup Activity in Azure Data Factory.
I have parameter "offset", which have initial value 5.
I want to use parameter value as Integer value in Lookup query, but failing. Please advice.
Original Static Lookup Query:
SELECT *
FROM sales.[Customers]
ORDER BY CustomerId OFFSET 5 ROWS FETCH NEXT 10 ROWS ONLY
--Parameterized Lookup Query:
SELECT *
FROM sales.[Customers]
ORDER BY CustomerId #concat('OFFSET ', pipeline().parameters.offset,' ROWS FETCH NEXT 10 ROWS ONLY')
Error of ADF for parameterized Lookup:
A database operation failed with the following error: 'Incorrect syntax near
'#concat'.',Source=,''Type=System.Data.SqlClient.SqlException,Message=Incorrect syntax near
'#concat'.,Source=.Net SqlClient Data
Provider,SqlErrorNumber=102,Class=15,ErrorCode=-2146232060,State=1,Errors=
[{Class=15,Number=102,State=1,Message=Incorrect syntax near '#concat'.,},],'
Put the entire SQL statement in an expression (using the Expression Builder):
#concat('SELECT * FROM sales.[Customers] ORDER BY CustomerId OFFSET ', pipeline().parameters.offset, ' ROWS FETCH NEXT 10 ROWS ONLY')
You can directly call the parameter as well
SELECT *
FROM sales.[Customers]
ORDER BY CustomerId OFFSET #{pipeline().parameters.offset} ROWS FETCH NEXT 10 ROWS ONLY
Related
Im relatively new and would like to redenominate some values in my current database. This means going into my jasonb column in my database, selecting a key value and dividing it by a 1000. I know how to select values but update after I have performed a calculation has failed me. My table name is property_calculation and has two columns as follows: * dynamic_fields is my jasonb column
ID
dynamic_fields
1
{"totalBaseValue": 4198571.230720645844841865113039602874778211116790771484375,"surfaceAreaValue": 18.108285497586717127660449477843940258026123046875,"assessedAnnualValue": 1801819.534798908603936834409607936611000776616631213755681528709828853607177734375}
2
{"totalBaseValue": 7406547.28939837918763178237213651300407946109771728515625,"surfaceAreaValue": 31.94416993248973568597648409195244312286376953125,"assessedAnnualValue": 9121964.022681592442116216621222042691512210677018401838722638785839080810546875}
I would like to update the dynamic_fields.totalBaseValue by dividing it by 1000 and committing it back as the new value. I have tried the following with no success:
update property_calculation
set dynamic_fields = (
select jsonb_agg(case
when jsonb_typeof(elem -> 'totalBaseValue') = 'number'
then jsonb_set(elem, array['totalBaseValue'], to_jsonb((elem ->> 'totalBaseValue')::numeric / 1000))
else elem
end)
from jsonb_array_elements(dynamic_fields::jsonb) elem)::json;
I get the following error:
ERROR: cannot extract elements from an object
SQL state: 22023
My json column has no zero string or null values.
Move the jsonb_typeof() check into the where clause:
update property_calculation
set dynamic_fields =
jsonb_set(
dynamic_fields,
'{totalBaseValue}',
to_jsonb((dynamic_fields->>'totalBaseValue')::numeric / 1000)
)
where jsonb_typeof(dynamic_fields->'totalBaseValue') = 'number';
db<>fiddle here
I have HTTP access log data in a Druid data source, and I want to see access patterns based on certain identifiers in the URL path. I wrote this query, and it works fine:
select regexp_extract(path, '/id/+([0-9]+)', 1) as "id",
sum("count") as "request_count"
from "access-logs"
where __time >= timestamp '2022-01-01'
group by 1
The only problem is that not all requests match that pattern, so I get one row in the result with an empty "id". I tried adding an extra condition in the where clause:
select regexp_extract(path, '/id/+([0-9]+)', 1) as "id",
sum("count") as "request_count"
from "access-logs"
where __time >= timestamp '2022-01-01' and "id" != ''
group by 1
But when I do that, I get this error message:
Error: Plan validation failed: org.apache.calcite.runtime.CalciteContextException:
From line 4, column 46 to line 4, column 49: Column 'id' not found in any table
So it doesn't let me reference the result of the expression in the where clause. I could of course just copy the entire regexp_extract expression, but is there a cleaner way of doing this?
Since id is an aggregated column, you would need a HAVING clause to filter on it.
Hello I have created a view, but want to pivot it.
OUTPUT before pivoting:
expected output:
my full query:
SELECT *
FROM CROSSTAB(
'SELECT DISTINCT GROUP_DEST::TEXT,DEST::TEXT,TIER::TEXT,RATE::TEXT FROM VBB_TIER ORDER BY 1,2')
AS CT(ROW_NAME TEXT, TIER_1 TEXT, TIER_2 TEXT )
I getting this error and unable to resolve:
ERROR: invalid source data SQL statement
DETAIL: The provided SQL must return 3 columns: rowid, category, and values.
SQL state: 22023
Using filtered aggregation is typically a lot easier than the somewhat convoluted crosstab() function:
select group_dest,
dest,
max(rate) filter (where tier in ('0-100', ('0-150')) as tier_1,
max(rate) filter (where tier in ('101-200', '151-350') as tier_2
from vbb_tier
group by group_dest, dest;
I have this redshift SQL query. I extracted a number with decimal from the comment using the "REGEXP_SUBSTR" function. I also need to convert it from string to number/decimal. Then, I need to subtract that number from the total.
This is my query
SELECT sc.comment,
sm.subtotal,
to_number(REGEXP_SUBSTR(sc.comment, '[0.00-9]+..[0.00-9]+', 1),'9999999D99')
FROM "sales_memo_comments" sc INNER JOIN "sales_memo" sm ON sc.foreign_id = sm.parent_id
I tried using the "to_number" function on Redshift SQL, but its giving me the following: ERROR: invalid input syntax for type numeric: " "
This is the current output Before extracting the number refund amount from the comment column:
comment
"SAR719.00 Refund transaction executed successfully, Refund Request ID:504081288877953603 \n , Authorization Code:095542 "
"AUD52.07 Refund transaction executed successfully, Refund Request ID:6J45695858A90833"
Canceled by : ron.johnd#company.co.us
refund amount is [MYR197.41]
"Please Ignore Order refunded by Refund Request ID:5002758809696048 , Authorization Code:2587759"
OMR37.83($98.23) Refund transaction executed successfully
This is it after using the above SQL query with REGEXP. I still get some anomalies.
comment
719
52.07
.co.
197.41
5.0027621
37.83($98.23
Two questions
How do I edit the REGEXP to take account for the anomalies seen above
How do I convert my string REGEXP to a numeric value to do a subtraction with another numeric column?
Any help would be appreciated.
Here is a way - you need to be able to test whether a string is numeric and for that you need a UDF - so just run this once to define that function
create or replace function isnumeric (aval VARCHAR(20000))
returns bool
IMMUTABLE
as $$
try:
x = int(aval);
except:
return (1==2);
else:
return (1==1);
$$ language plpythonu;
Then, you could change your code as follows
SELECT sc.comment,
sm.subtotal,
to_number(
case when isnumeric(REGEXP_SUBSTR(sc.comment, '[0.00-9]+..[0.00-9]+', 1))
then REGEXP_SUBSTR(sc.comment, '[0.00-9]+..[0.00-9]+', 1)
else 0 end
,'9999999D99')
FROM "sales_memo_comments" sc INNER JOIN "sales_memo" sm ON sc.foriegn_id = sm.parent_id
My approach would be to first add two columns: one for string length and the other counting allowed characters. From this table, you could filter for only rows where the two matched (ie no non-allowed characters) and then just cast the remaining values to floats or decimals or whatever.
with temp as (
SELECT '719' as comment
UNION SELECT '52.07'
UNION SELECT '.co.'
UNION SELECT '197.41'
UNION SELECT '5.0027621'
UNION SELECT '37.83($98.23'
),
temp2 as (
SELECT *
,regexp_count(comment, '[0-9.]') as good_char_length
,len(comment) as str_length
FROM
temp
)
SELECT *
,comment::float as comment_as_float
FROM
temp2
WHERE
good_char_length = str_length
Hi I tried to get a count of rows from my below query:
select count(substring(wsresult_question FROM '[0-9]+') as pumporder) AS totals,
job_id,
job_siteid,
job_completed
from webserviceresults w, jobs s
where job_siteid = '1401'
and job_id = wsresult_jobid
and job_completed is not null
and wsresult_question LIKE 'job.job_site_data.site_meters.pump.%'
and wsresult_category = 'Job'
group by pumporder,job_id,job_siteid,job_completed order by job_completed desc
I tried this and i got the error like
There was an SQL error:
ERROR: syntax error at or near "as" LINE 1: ... count(substring(wsresult_question FROM '[0-9]+') as pumpord... ^
In this line substring(wsresult_question FROM '[0-9]+') as pumporder I just tired to get only a number from some concatenate strings. The concatenate string is being like
1.job.job_site_data.site_meters.pump.0.meter_calibration_record.meter_adjustedtofast
2.job.job_site_data.site_meters.pump.0.meter_calibration_record.meter_adjustedtoslow
3.job.job_site_data.site_meters.pump.1.meter_calibration_record.meter_adjustedtofast
So substring(wsresult_question FROM '[0-9]+') as pumporder is return the numbers like 0,1 in array. I need to total the count of rows now. So Kindly help me on this.
Please let me know if you have any queries.
Thanks in advance!
your error means you should not create an alias for the function - only for the column, so if you remove as pumporder from count(substring(wsresult_question FROM '[0-9]+') as pumporder) , error will go away
Your approach though is very doubtful. If you want to count number of rows with substring(wsresult_question FROM '[0-9]+'), you better instead:
select count(1) AS totals,
job_id,
job_siteid,
job_completed
from webserviceresults w, jobs s
where job_siteid = '1401'
and job_id = wsresult_jobid
and job_completed is not null
and wsresult_question ~ '^(job.job_site_data.site_meters.pump.)[0-9]'
and wsresult_category = 'Job'
group by pumporder,job_id,job_siteid,job_completed order by job_completed desc
and lastly the string job.job_site_data.site_meters.pump.0 looks like json path, so it would be more appropriate using json array length function, not count on rows