IndexError: string index out of range ADF - Databricks - Python - azure-data-factory

I'm passing a UDF value from a JSON table through Azure Data Factory as the following:
"schema_array": "[('creationdate','timestamp'),('agent_name_txt','string'),('email','string'),('agent_hire_date','date'),('days_since_hire_text','int'),('department_auto','string')]"
But when my databricks notebooks accepts it and the indexing is interpreted as a string (which it needs to be set as an array.
This will work in my notebook by accepting the value when hardcoded, but not when passing from ADF. Here's the data type when pass from ADF: print(type(schema_array))
<class 'str'>
In need to know how to transform this (I'm in python) so my notebook will accept it as an array. I've tried to set it as an array in ADF many different ways, but will get the following error: "The variable 'udf' of type 'String' cannot be initialized or updated with value of type 'Array'. The variable 'udf' only supports values of types 'String'".
I'm running it through a schema mapping function to transform my data frame:
def string_to_datatype(datatype):
if datatype == 'timestamp':
final_datatype = t.TimestampType()
elif datatype == 'integer' or datatype == 'int':
final_datatype = t.IntegerType()
elif datatype == 'date':
final_datatype = t.DateType()
else: final_datatype = t.StringType()
return final_datatype
schema = t.StructType([])
for s in schema_array:
schema = schema.add(t.StructField(s[0], string_to_datatype(s[1])))

AFAIK, Currently in ADF it is not supported to pass the arrays as parameters to Databricks Notebook.
This is the reason for the above error as it is doing the list operations with string.
So, pass it as a string from ADF to Notebook, convert it to list in Notebook and use as per your requirement.
Use eval('variable_name') for it.
My Repro for your reference:
Passing String variable to Notebook:
Notebook run:

Related

Is there any difference between initializing with empty values or null value?

Is there any difference between these two part of code? (I am specially using Flutter/Dart but also interested to know about this in any other popular languages like C/C++, Java, JS, Python, etc.)
Code1:
String a = null; (in Flutter: a = null as String; )
List<T> = null; (in Flutter: List<T> = null as List<T>;)
Code 2:
String a = '';
List<T> = [];
With Dart nullsafety your first examples are invalid (assuming you meant to have a variable name on your List). If a variable can contain null the type needs a '?' suffix:
String? a = null;
List<T>? b = null;
But with or without nullsafety or '?', use of the variables in your second example won't result in runtime errors. For example a.trim(): if a is null a runtime error will occur because you're trying to call (null).trim(). In your second example no error because there is a String object to access, even though that String object is empty of characters. Same for your List: b.forEach((e) {}) when b==null a runtime error occurs because there's no Object to find forEach() - i.e. the runtime doesn't know what to do with (null).forEach(). In your second example the forEach() doesn't execute the function because List b is empty, but there's no runtime error because there's an Object to call forEach() on.
In C: NULL for pointers is usually synonymous with 0. The result of trying to access memory address NULL is "undefined" in C, because it could be valid as 0x00 in some instances like embedded systems or low-level system code, but will usually result in your process being terminated (crash).

How to enumerate over columns with tokio-postgres when the field types are unknown at compile-time?

I would like a generic function that converts the result of a SQL query to JSON. I would like to build a JSON string manually (or use an external library). For that to happen, I need to be able to enumerate the columns in a row dynamically.
let rows = client
.query("select * from ExampleTable;")
.await?;
// This is how you read a string if you know the first column is a string type.
let thisValue: &str = rows[0].get(0);
Dynamic types are possible with Rust, but not with the tokio-postgres library API.
The row.get function of tokio-postgres is designed to require generic inference according to the source code
Without the right API, how can I enumerate rows and columns?
You need to enumerate the rows and columns, doing so you can get the column reference while enumerating, and from that get the postgresql-type. With the type information it's possible to have conditional logic to choose different sub-functions to both: i) get the strongly typed variable; and, ii) convert to a JSON value.
for (rowIndex, row) in rows.iter().enumerate() {
for (colIndex, column) in row.columns().iter().enumerate() {
let colType: string = col.type_().to_string();
if colType == "int4" { //i32
let value: i32 = row.get(colIndex);
return value.to_string();
}
else if colType == "text" {
let value: &str = row.get(colIndex);
return value; //TODO: escape characters
}
//TODO: more type support
else {
//TODO: raise error
}
}
}
Bonus tips for tokio-postgres code maintainers
Ideally, tokio-postgres would include a direct API that returns a dyn any type. The internals of row.rs already use the database column type information to confirm that the supplied generic type is valid. Ideally a new API uses would use the internal column information quite directly with improved FromSQL API, but a simpler middle-ground exists:-
It would be possible for an extra function layer in row.rs that uses the same column type conditional logic used in this answer to then leverage the existing get function. If a user such as myself needs to handle this kind of conditional logic, I also need to maintain this code when new types are handled by tokio-postgresql, therefore, this kind of logic should be included inside the library where such functionality can be better maintained.

How can I use a string to load a variable stored in a .mat-file? [duplicate]

If for instance I have a variable xa=2, and then I construct a string by joining 'x' and 'a', how can I make this new string have the value 2?
xa=2;
var=strcat('x','a');
The result of this is var=xa, but what I want is var=2.
Thank you
Use eval():
var = eval(strcat('x','a'));
It will "evaluate" the string 'xa' and translate it to the value of the variable xa.
Source : MATLAB documentation

How to pass parameters to a Progress program using database field dynamic-based rules?

I have in my database a set of records that concentrates information about my .W's, e.g. window name, parent directory, file name, procedure type (for internal treatments purposes), used to build my main menu. With this data I'm developing a new start procedure for the ERP that I maintain and using the opportunity in order to rewrite some really outdated functions and programs and implement new functionalities. Until now, I hadn't any problems but when I started to develop the .P procedure which will check the database register of a program that was called from the menu of this new start procedure - to check if it needs to receive fixed parameters to be run and its data types - I found a problem that I can't figure out a solution.
In this table, I have stored in one of the fields the parameters needed by the program, each with his correspondent data type. The problem is on how to pass different data types to procedures based only on the stored data. I tried to pre-convert data using a CASE clause and an include to check the parameter field for correct parameter sending but the include doesn't work as I've expected.
My database field is stored as this:
Description | DATATYPE | Content
I've declared some variables and converted properly the stored data into their correct datatype vars.
DEF VAR c-param-exec AS CHAR NO-UNDO EXTENT 9 INIT ?.
DEF VAR i-param-exec AS INT NO-UNDO EXTENT 9 INIT ?.
DEF VAR de-param-exec AS DEC NO-UNDO EXTENT 9 INIT ?.
DEF VAR da-param-exec AS DATE NO-UNDO EXTENT 9 INIT ?.
DEF VAR l-param-exec AS LOG NO-UNDO EXTENT 9 INIT ?.
DEF VAR i-count AS INT NO-UNDO.
blk-count:
DO i-count = 0 TO 8:
IF TRIM(programa.parametro[i-count]) = '' THEN
LEAVE blk-count.
i-count = i-count + 1.
CASE ENTRY(2,programa.parametro[i-count],CHR(1)):
WHEN 'CHARACTER' THEN
c-param-exec[i-count] = ENTRY(3,programa.parametro[i-count],CHR(1)).
WHEN 'INTEGER' THEN
i-param-exec[i-count] = INT(ENTRY(3,programa.parametro[i-count],CHR(1))).
WHEN 'DECIMAL' THEN
de-param-exec[i-count] = DEC(ENTRY(3,programa.parametro[i-count],CHR(1))).
WHEN 'DATE' THEN
da-param-exec[i-count] = DATE(ENTRY(3,programa.parametro[i-count],CHR(1))).
WHEN 'LOGICAL' THEN
l-param-exec[i-count] = (ENTRY(3,programa.parametro[i-count],CHR(1)) = 'yes').
OTHERWISE
c-param-exec[i-count] = ENTRY(3,programa.parametro[i-count],CHR(1)).
END CASE.
END.
Then I tried to run the program using an include to pass parameters (in this example, the program have 3 INPUT parameters).
RUN VALUE(c-prog-exec) ({util\abrePrograma.i 1},
{util\abrePrograma.i 2},
{util\abrePrograma.i 3}).
Here is my abrePrograma.i
/* abrePrograma.i */
(IF ENTRY(2,programa.parametro[{1}],CHR(1)) = 'CHARACTER' THEN c-param-exec[{1}] ELSE
IF ENTRY(2,programa.parametro[{1}],CHR(1)) = 'INTEGER' THEN i-param-exec[{1}] ELSE
IF ENTRY(2,programa.parametro[{1}],CHR(1)) = 'DECIMAL' THEN de-param-exec[{1}] ELSE
IF ENTRY(2,programa.parametro[{1}],CHR(1)) = 'DATE' THEN da-param-exec[{1}] ELSE
IF ENTRY(2,programa.parametro[{1}],CHR(1)) = 'LOGICAL' THEN l-param-exec[{1}] ELSE
c-param-exec[{1}])
If I suppress the 2nd, 3rd, 4th and 5th IF's from the include or use only one data type in all IF's (e.g. only CHAR, only DATE, etc.) the program works properly and executes like a charm but I need to call some old programs, which expects different datatypes in its INPUT parameters and using the programs as described OpenEdge doesn't compile the caller, triggering the error number 223.
---------------------------
Erro (Press HELP to view stack trace)
---------------------------
** Tipos de dados imcompativeis em expressao ou atribuicao. (223)
** Nao entendi a linha 86. (196)
---------------------------
OK Ajuda
---------------------------
Can anyone help me with this ?
Thanks in advance.
Looks as if you're trying to use variable parameter definitions.
Have a look at the "create call" statement in the ABL reference.
http://documentation.progress.com/output/ua/OpenEdge_latest/index.html#page/dvref/call-object-handle.html#wwconnect_header
Sample from the documentation
DEFINE VARIABLE hCall AS HANDLE NO-UNDO.
CREATE CALL hCall.
/* Invoke hello.p non-persistently */
hCall:CALL-NAME = "hello.p".
/* Sets CALL-TYPE to the default */
hCall:CALL-TYPE = PROCEDURE-CALL-TYPE
hCall:NUM-PARAMETERS = 1.
hCall:SET-PARAMETER(1, "CHARACTER", "INPUT", "HELLO WORLD").
hCall:INVOKE.
/* Clean up */
DELETE OBJECT hCall.
The best way to get to the bottom of those kind of preprocessor related issues is to do a compile with preprocess listing followed by a syntax check on the preprocessed file. Once you know where the error is in the resulting preprocessed file you have to find out which include / define caused the code that won't compile .
In procedure editor
compile source.w preprocess source.pp.
Open source.pp in the procedure editor and do syntax check
look at original source to find include or preprocessor construct that resulted in the code that does not compile.
Okay, I am getting a little bit lost (often happens to me with lots of preprocessors) but am I missing that on the way in and out of the database fields you are storing values as characters, right? So when storing a parameter in the database you have to convert it to Char and on the way out of the database you have convert it back to its correct data-type. To not do it one way or the other would cause a type mismatch.
Also, just thinking out loud (without thinking it all the way through) wonder if using OOABL (Object Oriented ABL) depending on if you Release has it available wouldn't make it easier by defining signatures for the different datatypes and then depending on which type of input or output parameter you call it with, it will use the correct signature and correct conversion method.
Something like:
METHOD PUBLIC VOID storeParam(input cParam as char ):
dbfield = cParam.
RETURN.
END METHOD.
METHOD PUBLIC VOID storeParam(input iParam as int ):
dbfield = string(iParam).
RETURN.
END METHOD.
METHOD PUBLIC VOID storeParam(input dParam as date ):
dbfield = string(dParam).
RETURN.
END METHOD.
just a thought.

How to pass a byte[] parameter type in a sql query in MyBatis?

I need to match against an encrypted column in the DB. I need to pass the encrypted value for matching as a byte[]. The hashcode of the byte[] is passed instead of the actual value stored in the byte[]. Because the hash code is passed, it does not match the value correctly. Below is my query and the function call in the Mapper.java.
AccBalNotificationBean selectAccBalNotificationBean(#Param("acctIdByteArray") byte[] acctIdByteArray);
SELECT toa.accounts_id from tbl_transactions_other_accounts toa WHERE other_account_number = #{acctIdByteArray}
Thank you for your help.
I assume the datatype of your other_account_number column is of type string (char, varchar etc). Mybatis will use the StringDataTypeHandler by default and call the .toString() method of your byte array. Give MyBatis a hint that you want the content of your array to be used, by specifying the typeHandler.
.. WHERE other_account_number = #{acctIdByteArray, typeHandler=org.apache.ibatis.type.ByteArrayTypeHandler}