Azure Data Factory: For each item() value does not exist for a particular attribute - azure-data-factory

I have a for each activity which has a stored procedure (SP) wherein I am inputing values using item() evidently.
Now suppose SP's input values are item().a, item().b and item().c
Question: For some of the iteration of foreach, item().b does not exist which is expected. So how should i deal with it in the Stored procedure? Because at this point of time it is giving me an error when it executed SP by saying:
"The template language expression 'item().b' cannot be evaluated because property 'b' doesn't
exist, available properties are 'a, c'
or how should I overcome this failure in the data factory?
Apparently, data factory has the check for empty() but it does not have the check for exist().

You could use “?”. I.e., item()?.b
Please reference question mark and a related post.

I don't think you can solve this in the Data Factory. You could use the String(Item()) to convert it to a Json string in the format:
{
'a':'value',
'b':'value',
'c':'value'
}
Then you can handle that in your stored procedure with some creative SQL:
DECLARE #jsonParams NVARCHAR(255) = '
{
"a":"a value",
"c":"b value"
}'
DECLARE #paramA VARCHAR(10) = (SELECT JSON_VALUE(#jsonParams,'$.a'))
DECLARE #paramB VARCHAR(10) = (SELECT JSON_VALUE(#jsonParams,'$.b'))
DECLARE #paramC VARCHAR(10) = (SELECT JSON_VALUE(#jsonParams,'$.c'))

Related

xmlcast is returing null value in Db2

I am always getting a null value from using XMLCAST function in Db2. But when using a normal function call it is returning value in xml format... as below query proper xml data is coming. function Getdatavalue is returning xml datatype. using Linux 8.2 and db2 11.5
db2 "select sample.Getdatavalue(1,4504)) from sysibm.sysdummy1"
in the above result we have some double quotes in output xml.
but once we have used xmlcast function as below, it is returning the hexa value of null. i am using MD5 hashing mechanism.
db2 "select HEX(HASH(xmlcast(sample.Getdatavalue(1,4504) as varchar(32672)),0)) as hash_val from sysibm.sysdummy1";
even once i tried without Hash it was returning null result for below query. it looks xmlcast is not working.
db2 "select xmlcast(sample.Getdatavalue(1,4504) as varchar(32672)) as hash_val from sysibm.sysdummy1";
is there any way to convert xml datatype to clob or varchar.
body of Getdatavalue as:
CREATE FUNCTION sample.Getdatavalue (v1 smallint, v2 integer)
RETURNS xml
LANGUAGE SQL
BEGIN ATOMIC
DECLARE v_Data xml;
set v_Data = (SELECT XMLELEMENT( NAME "rt_ky_cmpnnt",
XMLAGG(XMLELEMENT(NAME "rt", XMLAttributes(rt.rt_field_add_val AS
"rt_add_val",
rt.rt_name AS
"rt_name",
rt.rt_name AS
"rt_name",
rt.rt_val1 AS "rt_val1",
rt_lim AS "rt_lim",
rt.rt_val2 AS "rt__val2",
rt.rt_whole_val AS "rt_whole_val" ))order by
rf.rt_num asc)OPTION null ON NULL)
FROM sample.cmpnnt_val rt inner join sample.field_val rf
on rf.abc=rt.abc
and rf.pqr=rt.pqr
and rt.xyz=rf.xyz
where rt.abc = v1
AND rt.bcd = v2 );
RETURN v_Data;
END!
can anyone please explain how to handle xmlcast.
To retrieve a string from an XML value, use the function XMLSERIALIZE.
The XMLSERIALIZE function returns a serialized XML value of the specified data type generated from the XML-expression argument.

With PostgREST, convert a column to and from an external encoding in the API

We are using PostgREST to automatically generate a REST API for a Postgres database. Our primary keys have an external representation that's different from how we store them internally. For simplicity's sake lets pretend the ids are stored as integers but we represent them as hexadecimal strings outwardly.
It's simple enough to get PostgREST to convert to the external representation for read operations:
CREATE DOMAIN hexid AS bigint;
CREATE TABLE fruits (
fruit_id hexid PRIMARY KEY,
name text
);
CREATE OR REPLACE VIEW api_fruits AS
SELECT to_hex(fruit_id) as fruit_id, name FROM fruits;
INSERT INTO fruits(fruit_id, name) VALUES('51955', 'avocado');
PostgREST generates the expected representation when we GET api_fruits:
[
{
"fruit_id": "caf3",
"name": "avocado"
}
]
But that's about as far as we get with this solution. It's a one way transformation so we won't be able to POST/PATCH records this way. The way PostgREST works is to transform such requests into equivalent INSERT and UPDATE statements. But this view with its custom formatting is not updatable. This is what would happen if we tried:
ERROR: cannot insert into column "fruit_id" of view "api_fruits"
DETAIL: View columns that are not columns of their base relation are not updatable.
STATEMENT: WITH pgrst_source AS (WITH pgrst_payload AS (SELECT $1::json AS json_data), pgrst_body AS ( SELECT CASE WHEN json_typeof(json_data) = 'array' THEN json_data ELSE json_build_array(json_data) END AS val FROM pgrst_payload) INSERT INTO "api_x"."api_fruits"("fruit_id", "name") SELECT "fruit_id", "name" FROM json_populate_recordset (null::"api_x"."api_fruits", (SELECT val FROM pgrst_body)) _ RETURNING "api_x"."api_fruits".*) SELECT '' AS total_result_set, pg_catalog.count(_postgrest_t) AS page_total, CASE WHEN pg_catalog.count(_postgrest_t) = 1 THEN coalesce((
WITH data AS (SELECT row_to_json(_) AS row FROM pgrst_source AS _ LIMIT 1)
SELECT array_agg(json_data.key || '=eq.' || json_data.value)
FROM data CROSS JOIN json_each_text(data.row) AS json_data
WHERE json_data.key IN ('')
), array[]::text[]) ELSE array[]::text[] END AS header, '' AS body, nullif(current_setting('response.headers', true), '') AS response_headers, nullif(current_setting('response.status', true), '') AS response_status FROM (SELECT * FROM pgrst_source) _postgrest_t
We can't INSERT into "View columns that are not columns of their base relation".
The obvious workaround is to serve fruit_id as a straight column, just an integer. With some post and preprocessing at the nginx level we can hex encode it there (and hex decode incoming ids). I'm wondering if we can do better than that though. For large API operations, re-encoding the JSON will use a lot of memory and CPU time and it seems so unnecessary.
It would have been great to be able to use a custom CREATE CAST to take the incoming hexadecimal strings and turn them back into integers, something like this:
CREATE CAST (json AS hexid) WITH FUNCTION json_to_hexid AS ASSIGNMENT;
But alas custom casts are ignored on CREATE DOMAIN types. And we can't make a true custom column type because our cloud Postgres host (Google Cloud SQL) doesn't allow custom extensions.
It feels like some combination of INSTEAD OF triggers or rules could work. But when using query parameters to filter results using query parameters (e.g. select a fruit by id), I don't think there's an appropriate trigger to use. INSTEAD OF doesn't work for straight SELECT does it?
For example I've tested doing something like this to take care of INSERT and allow POST with PostgREST. It works:
CREATE OR REPLACE FUNCTION api_fruits_insert()
RETURNS trigger AS
$$
BEGIN
INSERT INTO fruits(fruit_id, name) VALUES (('x' || lpad(NEW.fruit_id, 16, '0'))::bit(64)::bigint::hexid, NEW.name);
RETURN NEW;
END
$$ LANGUAGE 'plpgsql';
CREATE TRIGGER api_fruits_insert
INSTEAD OF INSERT
ON api_fruits
FOR EACH ROW
EXECUTE PROCEDURE api_fruits_insert();
The trouble is in the WHERE clause. Let's PATCH api_fruits?fruit_id=in.(7b,caf3) with {"name": "pear"}. This works out of the box since the name column is updatable but look at the query:
WITH pgrst_source AS (WITH pgrst_payload AS (SELECT $1::json AS json_data), pgrst_body AS ( SELECT CASE WHEN json_typeof(json_data) = 'array' THEN json_data ELSE json_build_array(json_data) END AS val FROM pgrst_payload) UPDATE "api_x"."api_fruits" SET "name" = _."name" FROM (SELECT * FROM json_populate_recordset (null::"api_x"."api_fruits" , (SELECT val FROM pgrst_body) )) _ WHERE "api_x"."api_fruits"."fruit_id" = ANY ($2) RETURNING 1) SELECT '' AS total_result_set, pg_catalog.count(_postgrest_t) AS page_total, array[]::text[] AS header, '' AS body, nullif(current_setting('response.headers', true), '') AS response_headers, nullif(current_setting('response.status', true), '') AS response_status FROM (SELECT * FROM pgrst_source) _postgrest_t
DETAIL: parameters: $1 = '{
"name": "pear"
}', $2 = '{7b,caf3}'
So we have essentially UPDATE api_fruits SET name='berry' WHERE fruit_id IN ('7b', 'caf3');. Surprisingly this works but it's a full table scan so Postgres can evaluate to_hex(fruit_id) for each row looking for matches. The same happens if we try to GET a record by fruit_id. How would we rewrite the WHERE clauses?
It really feels like some combination of just the right Postgres and PostgREST features should be able to get us to a point where it's all happening in Postgres without nginx's help and without excessive complexity. Any ideas?

c# using Enterprise-Library

How we can pass data table to stored procedure in c# using Enterprise library ?
I am using command
cmd.AddInParameter(objDbCommand, DBAccessDetails.dataTablePrm, SqlDbType.Structured, curatedListprm);
then coming issue as :
Conversion failed when converting the nvarchar value 'ZAQ' to data type int.
The data for table-valued parameter "#data" doesn't conform to the table type of the parameter. SQL Server error is: 245, state: 1
Plz tell me what I do and what is issue ?
First you need to have created your user defined table type
CREATE TYPE [schema].[tableName] AS TABLE(
table columns and types )
GO
then you need to create your stored proc to take the table valued parameted
create procedure procName
(
#tvp dbo.tableNamereadonly
)
then you need to pass your datatable in via .Net with something like this
var param = new SqlParameter("#tvp" Sqldbtype.Structured);
param.Value = {your data table}

Postgresql function doesn't return anything when UUID argument is used in WHERE clause

I'm pretty new to Postgresql. The issue I'm having is that I have a function that returns a table, but when I pass an UUID which is used in the where clause, it returns nothing. The funny thing is that if I take the SQL statement inside the function and run it by itself in PgAdmin, it gives me the right result.
The function looks like the following:
CREATE OR REPLACE FUNCTION get_service (
service_id uuid ) RETURNS TABLE(id uuid,title text,description text,category text,photo_url text,address text,
created_by uuid,created_on timestamp,service_rating float,rating_count bigint) AS $func$
Select
service.id,
service.title,
service.description,
service.category,
service.photo_url,
service.address,
service.created_by,
service.created_on,
CAST(AVG(rating.rating) AS float) as service_rating,
Count(rating.rating) as rating_count
from service
left join rating_service_map map
on service.id = map.service_id
left join rating
on rating.id = map.rating_id
where service.id = service_id
group by service.id,service.title,service.description,service.category,service.photo_url,service.address,service.created_by,service.created_on;
$func$ LANGUAGE SQL;
I have two records in my service table. The ID is of the type uuid and has a default value of uuid_generate_v4(). One of the records has an id of '2af3f03e-b2e5-44fd-89e8-3dc5fb641732'
If I run this I get no result:
select * from get_service('2af3f03e-b2e5-44fd-89e8-3dc5fb641732')
But if I run the following statement (the SQL portion of the function), then I get my right result:
Select
service.id,
service.title,
service.description,
service.category,
service.photo_url,
service.address,
service.created_by,
service.created_on,
CAST(AVG(rating.rating) AS float) as service_rating,
Count(rating.rating) as rating_count
from service
left join rating_service_map map
on service.id = map.service_id
left join rating
on rating.id = map.rating_id
where service.id = '2af3f03e-b2e5-44fd-89e8-3dc5fb641732'
group by service.id,service.title,service.description,service.category,service.photo_url,service.address,service.created_by,service.created_on;
I've also tried to cast the service_id (I've tried "where service.id = sevice_id::uuid" and "where service.id = CAST(service_id AS uuid)") but none of them worked.
I really appreciate it if you can tell me what I'm doing wrong. I've been at this for a couple of hours now.
Thank you.
I suspect that it's because the identifier service_id is ambiguous, being present as both a function parameter and a column in the map table.
Unlike a plain query, where such ambiguity would result in an error, conflicts in SQL functions are resolved by giving precedence to the column, so service_id in your case is actually referring to map.service_id.
You can either qualify it in your function body using the name of your function (i.e. get_service.service_id), or simply choose another name for the parameter.

Avoid COUNT to CAST to BIGINT

Using Open JPA 2.0, Database is DB2 9.7. For query like SELECT COUNT(1) FROM USER WHERE FNAME := fname, JPA is converting the query to SELECT COUNT(CAST(? AS BIGINT)) FROM TABLENAME.
How to avoid the CAST to BIGINT?
Code sample below:
query = entityManager.createNamedQuery("qry.checkuser");
query.setParameter("fname", fname);
Long count = (Long)query.getSingleResult();
Which one is the problem in the CAST?
I think you can't avoid it since is SQL generated by the JPA provider.
BTW, I allways use Number super class instead of specific subclass:
query = entityManager.createNamedQuery("qry.checkuser", Number.class);
query.setParameter("fname", fname);
Number count = query.getSingleResult();
// Do whatever is needed
if (count.longValue()...
This way there's no problem if the JPA provider returns a integer, long or BigXXXX.
What worked for me is:
SELECT COUNT(USER_ID) FROM USER WHERE FNAME := fname
Basically we need to use a non nullable column like Primary key column and with this change the CAST can be avoided which consumes additional CPU, a minor gain.