Widgets with non-duplicate values in PySpark - pyspark

I am trying to build a widget from a list that has non-duplicate values based on the below scenario. Could someone please help me with getting the right way?
fileInfoList = list(filter(lambda f: f.name.endswith("") , dbutils.fs.ls(srcPath)))
for fileNames in fileInfoList:
print(fileNames.name)
This prints:
Employee
EmployeeHistory
Contractor
ContractorHistory
What I wanted is only the values without History.
Tried this but returns error:
dbutils.widgets.dropdown("FileName", "Employee", [str(fileNames.name) for fileNames in fileInfoList])

Why don't you simply filter your list before feeding into the dropdown function?
>> fileList = ['Employee', 'Contractor', 'EmployeeHistory', 'ContractorHistory']
>> print(fileList)
['Employee', 'Contractor', 'EmployeeHistory', 'ContractorHistory']
>> filteredFileList = [item for item in fileList if 'History' not in item]
>> print(filteredFileList)
['Employee', 'Contractor']

Related

Matlab : cannot get the values of a field in a struct

I am encountering a rather weird problem. I have a big struct imported from a .mat file (it's an EEG recording):
Now let's assume I want to plot one field, I need to get the values in this field.
However, when I do this:
fieldE1 = EEG.('00 E1');
fieldE1 only becomes the last value of the field : .
If I just write in the console EEG.('00 E1'), it returns this :
ans =
-12.5850
ans =
-12.5790
ans =
-12.5760
ans =
-12.5820
ans =
-12.5890
ans =
-12.5880
ans =
-12.5880
ans =
-12.5860
On and on and on for all the values. Which explains why fieldE1 only returned the last value. I have the same behaviour when I use getfield(EEG, '00 E1') .
Any help would be appreciated.
Subscript references to a field in a struct array will return a comma-separated list. The list must be captured in array delimiters upon assignment to be used as an array:
fieldE1 = [EEG.('00 E1')];

getting list of values with list of keys for dictionary in Matlab

Suppose I use containers map to create a dictionary in MATLAB which has the following map:
1-A;
2-B;
3-C;
Denote the dictionary as D.
Now I have an input list [2,1,3], and what I am expecting is [B,A,C]. The problem is, I can't just use [2,1,3] as the input list for D, but only input 2,1 and 3 one by one for D and get B, A, C each time.
This can get the job done but as you can see, it's a bit less efficient.
So my question is: is there anything else I can do to let the dictionary return the whole list at the same time?
As far as I can find there is no one-step solution like python's dict.items. You can, however, get in a few lines. mydict.keys() gives you the keys of the dict as a cell array, and mydict.values() gives you the values as a cell array, so you can (in theory) combine those:
>> mykeys = mydict.keys();
>> myvals = mydict.values();
>> mypairs = [mykeys',myvals']
mypairs =
3×2 cell array
'A' [1]
'B' [2]
'C' [3]
However, in principle maps are unordered, and I can't find anything in the MATLAB documentation that says that the order returns by keys and the order returned by values is necessarily consistent (unlike Python). So if you want to be extra safe, you can call values with a cell array of the keys you want, which in this case would be all the keys:
>> mykeys = mydict.keys();
>> myvals = mydict.values(mykeys);
>> mypairs = [mykeys',myvals']
mypairs =
3×2 cell array
'A' [1]
'B' [2]
'C' [3]

search and delete cell elements in matlab

I have a cell array of doubles and string in which a particular column looks like this
abc = {[110;10];[20;110];[10];[220];[380];[15];[220];[110;15];[110;20];[110]};
I would like to delete all elements which are less than 110 and I tried this statement abc(cellfun(#(x) any(x<110),abc),1) = [];
I got an error A null assignment can have only one non-colon index. Could someone please explain and rectify this?
I expect the output to be like this
abc = {[110];[110];[];[220];[380];[];[220];[110];[110];[110]};
Thanks!
abc = abc(~cellfun(#(x) any(x<110),abc),1)?
That will invert the logical indices and then select the corresponding entries.
EDIT: After your comment was provided, that should do it:
abc_out = cellfun(#(x) x(x>=110), abc, 'UniformOutput', false)

How do I iterate through an imported excel doc?

what I need to do for this code is import a giant (302x11) excel doc, and then ask the user for an input. Then, I need to iterate through each element in the 5th column of the excel array, and if that element matches the user input, save the entire row to a new array. After going through all 302 rows, I need to display the new array.
So far, I have this:
Vin = input('Vin: ');
filename='MagneticCore.xlsx';
sheet=2;
xlRange='B2:L305';
[ndata, text, alldata] = xlsread(filename,sheet,xlRange,'basic');
After this, I'm not sure how to iterate through the alldata array.
alldata is a cell, to select the fifth column you can use alldata{:,5}. Searching in Cells is done this way without iterating
Try it on your own, if you get stuck update your question with code and error message
Daniel R is right, you can do this without iterating through the cell array. Here's how you could iterate through the array if you needed to:
[ndata, text, alldata] = xlsread('Book1.xlsx');
target = 12;
newArray = {};
for r = 1:size(alldata, 1)
% get the element in the fifth column of the current row
e = raw{r,5};
if e == target
% add to newArray
newArray{end + 1} = alldata(r,:);
end
end
% display newArray
for r = 1:size(newArray, 1)
disp(newArray{r})
end

How can I dynamically access a field of a field of a structure in MATLAB?

I'm interested in the general problem of accessing a field which may be buried an arbitrary number of levels deep in a containing structure. A concrete example using two levels is below.
Say I have a structure toplevel, which I define from the MATLAB command line with the following:
midlevel.bottomlevel = 'foo';
toplevel.midlevel = midlevel;
I can access the midlevel structure by passing the field name as a string, e.g.:
fieldnameToAccess = 'midlevel';
value = toplevel.(fieldnameToAccess);
but I can't access the bottomlevel structure the same way -- the following is not valid syntax:
fieldnameToAccess = 'midlevel.bottomlevel';
value = toplevel.(fieldnameToAccess); %# throws ??? Reference to non-existent field 'midlevel.bottomlevel'
I could write a function that looks through fieldnameToAccess for periods and then recursively iterates through to get the desired field, but I am wondering if there's some clever way to use MATLAB built-ins to just get the field value directly.
You would have to split the dynamic field accessing into two steps for your example, such as:
>> field1 = 'midlevel';
>> field2 = 'bottomlevel';
>> value = toplevel.(field1).(field2)
value =
foo
However, there is a way you can generalize this solution for a string with an arbitrary number of subfields delimited by periods. You can use the function TEXTSCAN to extract the field names from the string and the function GETFIELD to perform the recursive field accessing in one step:
>> fieldnameToAccess = 'midlevel.bottomlevel';
>> fields = textscan(fieldnameToAccess,'%s','Delimiter','.');
>> value = getfield(toplevel,fields{1}{:})
value =
foo