Iterate within the payload in mule - db2

I am retrieving some rows from the database. For example, after retrieving I have two rows as the payload. I would like to iterate within the rows. The retrieved two rows are as follows,
[{PAYLOAD_MSG=PU, SENSITIVEDATAINDICATOR=Y, ERRORMESSAGE=User generated exception test,
INTERFACEID=I0826, EXCEPTIONID=73, PAYLOADMSGID=I0826MessTesting0002, SEVERITY=2, INTERFACENAME=replay,
SOURCEPROTOCOL=MQ, CREATIONTIME=2016-03-01 08:29:36.211319, EVENTSOURCE=MQ_Input.transaction.Rollback},
{PAYLOAD_MSG=UvdjI, SENSITIVEDATAINDICATOR=N, ERRORMESSAGE=User generated exception test, INTERFACEID=I0826,
EXCEPTIONID=72, PAYLOADMSGID=I0826MessTesting0001, SEVERITY=2, INTERFACENAME=replay, SOURCEPROTOCOL=MQ,
CREATIONTIME=2016-03-01 08:29:36.211319, EVENTSOURCE=MQ_Input.transaction.Rollback}]
I want to access the Payload_Msg of the first row and as well as the second row. Specifically how can this be done? I have tried the following ways,
[payload[0].Payload_Msg], #[payload.Payload_Msg]. But they were not working. Could someone help to overcome this.


You can use ForEach loop to iterate and access objects.
<foreach collection="payload">
<logger message="#[payload.PAYLOAD_MSG]"/>
</foreach>


#[payload[0].PAYLOAD_MSG] must work else check with #[message.payload[0] PAYLOAD_MSG]


You can use Foreach component, but first set it to a flow var so you can still access it within the Foreach component. Another way to handle this is to use the Collection Splitter component.
Hope this helps.


You could use a Collection Splitter after the database connector to process each records one by one. Then you can use #[payload.Payload_Msg] in the Logger to retrieve the value.

Related

Constructing list in globalMap and retrieving it as flow

I have 3 levels of folders and I am using them as below:
tFileList_1 --iterate--> tFileList_2 --iterate--> tFileList_3 --> tJava ...
And tJava has multiple branches after it.
Based on certain conditions which are written in tJava (as seen above) and also other tJava (not seen above, but present in multiple branches), I want to collect list of files that satisfy particular condition.
So, I am thinking of putting them to globalMap like below:
Initialise:
List<String> filesMetCondition = new ArrayList<String>();
globalMap.put("filesMetCondition", filesMetCondition);
Putting into array: (from multiple places)
((List<String>)globalMap.get("filesMetCondition")).add("file2");
Populating globalMap should not be a problem.
My question is how to access this globalMap as a flow. I should be able to use something like tHashOutput or tFixedFlow or tForEach that links to above globalMap variable so that I can connect components to it.

You can used a tLoop to iterate your ArrayList stored in the globalMap.
then you can access the current value with
String value =
((List<String>)globalMap.get("filesMetCondition")).
get(((Integer)globalMap.get("tLoop_1_CURRENT_VALUE")));
System.out.println(value);
To use the value in a Flow, add a tIterateToFlow

How to get column title for each row entry

I am using a row id to obtain the cells for a single row. However, the response returns the column id but not the title of the column. In an attempt to make the code readable for others it would be helpful to also obtain the column title. I was thinking of doing this by using the column id that is obtained in the getRow function but I am not entirely sure how to catch it. Below is the basic getRow function for reference. I appreciate any assistance. Thank you in advance all.
smartsheet.sheets.getRow(options)
.then(function(row) {
console.log(row);
})
.catch(function(error) {
console.log(error);
});

My preferred way of addressing this is to dynamically create a column map on my first GET /sheets/{sheetId} request.
Let's say we have a sheet with three columns: Japan, Cat, and Cafe. Here is one way to make a column map.
const columnMap = makeColumnMap(<your sheet data>);
function makeColumnMap(sheetData){
const colMap = {};
sheetData.columns.map( column => colMap[column.title] = column.id);
return colMap;
}
Now, you can reference your specific columns like this: columnMap["Japan"], columnMap["Cat"], and columnMap["Cafe"] or you can use dot notation if you prefer.
Basically, what we're doing is creating a dictionary to map the column titles to the corresponding column id.

Posting this as a separate answer based on your response (and for easier formatting).
I have a couple specific recommendations that will help you.
Try to consolidate your API calls.
I then want to use that columnID to call getColumns(columnId) to obtain the title.
This is 'work' that you don't need to do. A single GET /sheets/{sheetId} will include all the data you need in one call. It's just a matter of parsing the JSON response.
Use this as an opportunity to improve your ability to work with JSON.
I do not know how to catch the columnId once getRow() is called.
The response is a single object with nested arrays and objects. Learning to navigate the JSON in a way that makes sense to you will come in really handy.
I would recommend saving the response from a GET sheet call as it's own JSON file. From there, you can bring it into your script and play with your logic to reference the values you want.

Is PapaParse adding an empty string to the end of its data array?

Papa Parse seems wise, but I think he might be giving me null. I'm just:
Papa.parse(countries);
Where countries is a string containing the XMLHttpRequest of the countries csv file from a timezone database here:
https://timezonedb.com/download
But Papa Parse seems to have added an empty array to the end of it's data array. So when I'm searching and sorting through the array, that one empty guy at the end is giving me troubles. I can write around it but it's not ideal, and I thought Papa Parse was supposed to make those kind of csv parsing problems go away. Am I Parsing wrong?
Here is the end of the PapaParsed Array in console:

You need to use skipEmptyLines: true in parse config. For example:
Papa.parse(this.csvData, {skipEmptyLines: true,})

it was adding empty line to my iteration as well. i decided to skip it by doing loop:
for(let i=0;i<data.length -1;i++){

We can also use below syntax to remove empty lines from the record.
For example, in order to remove empty values from header, we can use the below code snippet.
headers.filter(Boolean);

dataFrame keying using pandas groupby method

I new to pandas and trying to learn how to work with it. Im having a problem when trying to use an example I saw in one of wes videos and notebooks on my data. I have a csv file that looks like this:
filePath,vp,score
E:\Audio\7168965711_5601_4.wav,Cust_9709495726,-2
E:\Audio\7168965711_5601_4.wav,Cust_9708568031,-80
E:\Audio\7168965711_5601_4.wav,Cust_9702445777,-2
E:\Audio\7168965711_5601_4.wav,Cust_7023544759,-35
E:\Audio\7168965711_5601_4.wav,Cust_9702229339,-77
E:\Audio\7168965711_5601_4.wav,Cust_9513243289,25
E:\Audio\7168965711_5601_4.wav,Cust_2102513187,18
E:\Audio\7168965711_5601_4.wav,Cust_6625625104,-56
E:\Audio\7168965711_5601_4.wav,Cust_6073165338,-40
E:\Audio\7168965711_5601_4.wav,Cust_5105831247,-30
E:\Audio\7168965711_5601_4.wav,Cust_9513082770,-55
E:\Audio\7168965711_5601_4.wav,Cust_5753907026,-79
E:\Audio\7168965711_5601_4.wav,Cust_7403410322,11
E:\Audio\7168965711_5601_4.wav,Cust_4062144116,-70
I loading it to a data frame and the group it by "filePath" and "vp", the code is:
res = df.groupby(['filePath','vp']).size()
res.index
and the output is:
[E:\Audio\7168965711_5601_4.wav Cust_2102513187,
Cust_4062144116, Cust_5105831247,
Cust_5753907026, Cust_6073165338,
Cust_6625625104, Cust_7023544759,
Cust_7403410322, Cust_9513082770,
Cust_9513243289, Cust_9702229339,
Cust_9702445777, Cust_9708568031,
Cust_9709495726]
Now Im trying to approach the index like a dict, as i saw in examples, but when im doing
res['Cust_4062144116']
I get an error:
KeyError: 'Cust_4062144116'
I do succeed to get a result when im putting the filepath, but as i understand and saw in previouse examples i should be able to use the vp keys as well, isnt is so?
Sorry if its a trivial one, i just cant understand why it is working in one example but not in the other.

Rutger you are not correct. It is possible to "partial" index a multiIndex series. I simply did it the wrong way.
The index first level is the file name (e.g. E:\Audio\7168965711_5601_4.wav above) and the second level is vp. Meaning, for each file name i have multiple vps.
Now, this is correct:
res['E:\Audio\7168965711_5601_4.wav]
and will return:
Cust_2102513187 2
Cust_4062144116 8
....
but trying to index by the inner index (the Cust_ indexes) will fail.

You groupby two columns and therefore get a MultiIndex in return. This means you also have to slice using those to columns, not with a single index value.
Your .size() on the groupby object converts it into a Series. If you force it in a DataFrame you can use the .xs method to slice a single level:
res = pd.DataFrame(df.groupby(['filePath','vp']).size())
res.xs('Cust_4062144116', level=1)
That works. If you want to keep it as a series, boolean indexing can help, something like:
res[res.index.get_level_values(1) == 'Cust_4062144116']
The last option is a bit less readable, but sometimes also more flexibile, you could test for multiple values at once for example:
res[res.index.get_level_values(1).isin(['Cust_4062144116', 'Cust_6073165338'])]

Data Processing, how to approach

I have the following Problem, given this XML Datastructure:
<level1>
<level2ElementTypeA></level2ElementTypeA>
<level2ElementTypeB>
<level3ElementTypeA>String1Ineed<level3ElementTypeB>
</level2ElementTypeB>
...
<level2ElementTypeC>
<level3ElementTypeB attribute1>
<level4ElementTypeA>String2Ineed<level4ElementTypeA>
<level3ElementTypeB>
<level2ElementTypeC>
...
<level2ElementTypeD></level2ElementTypeD>
</level1>
<level1>...</level1>
I need to create an Entity which contain: String1Ineed and String2Ineed.
So every time I came across a level3ElementTypeB with a certain value in attribute1, I have my String2Ineed. The ugly part is how to obtain String1Ineed, which is located in the first element of type level2ElementTypeB above the current level2ElementTypeC.
My 'imperative' solution looks like that that I always keep an variable with the last value of String1Ineed and if I hit criteria for String2Ineed, I simply use that. If we look at this from a plain collection processing point of view. How would you model the backtracking logic between String1Ineed and String2Ineed? Using the State Monad?

Isn't this what XPATH is for? You can find String2Ineed and then change the axis to search back for String1Ineed.