I'm currently building a dashboard with powerbi and several CSV as sources.
Those CSV will have regular updates and i want to get a visualisation with the date of the last source update (for every CSV)
Is there a way to use the metadata csv file in powerbi to visualise it ?
Or a better way to get what i'm seeking for ?
Regards,
I tried the solution proposed:
Which mean the function is not well orthographied , any idea?
You can use Folder.Files with appropriate filters, to get to file metadata.
It's simple enough to use a function to get the data you want:
fnFileData:
(FullFilename as text, OutputType as text) =>
let
Filepath = Text.BeforeDelimiter(FullFilename, "\", {0, RelativePosition.FromEnd}) & "\",
Filename = Text.AfterDelimiter(FullFilename, "\", {0, RelativePosition.FromEnd}),
Filtered = Table.SelectRows(Folder.Files(Filepath), each [Folder Path] = Filepath and [Name] = Filename),
Filedata = Table.TransformColumnNames(Filtered, Text.Lower),
Output = Table.Column(Filedata,Text.Lower(OutputType)){0}
in
Output
In this case, if you want the Date Modified, then invoke as
= fnFileData("C:\MyPath\MyFile.txt", "Date Modified")
Related
I have a MS SQL table which contains a list of files that are stored within an ADLS gen2 account. All files have the same schema and structure.
I have concatenated the results of the table into a string.
mystring = ""
for index, row in files.iterrows():
mystring += "'"+ row["path"] + "',"
mystring = mystring[:-1]
print(mystring)
OUTPUT
'abfss://[file]#[container].dfs.core.windows.net/ARCHIVE/2021/08/26/003156/file.parquet','abfss:/[file]#[container].dfs.core.windows.net/ARCHIVE/2021/08/30/002554/file.parquet','abfss:/[file]#[container].dfs.core.windows.net/ARCHIVE/2021/09/02/003115/file.parquet'
I am now attempting to pass the string using
sdf = spark.read.parquet(mystring)
however I am getting the error
IllegalArgumentException: java.net.URISyntaxException: Illegal character in scheme name at index 0: 'abfss://[file]#[container].dfs.core.windows.net/ARCHIVE/2021/08/26/003156/file.parquet','abfss:/[file]#[container].dfs.core.windows.net/ARCHIVE/2021/08/30/002554/file.parquet','abfss:/[file]#[container].dfs.core.windows.net/ARCHIVE/2021/09/02/003115/file.parquet','abfss:/[file]#[container].dfs.core.windows.net/ARCHIVE/2021/09/24/003516/file.parquet','abfss:/[file]#[container].dfs.core.windows.net/ARCHIVE/2021/10/07/002659/file.parquet'
When I manually copy and past mystring into read.parquet the code executes with no errors.
Maybe I'm going down a rabbit hole but some feedback would be much appreciated
After reproducing from my end, I could able to achieve following the below.
paths = []
for index,row in files.iterrows():
paths.append(row["path"])
df = spark.read.parquet(paths)
RESULTS:
NOTE: Make sure you have same schema in all the files.
%% File Names reading and label generation
dataFolder= 'allcontent';
fileNames = dir([dataFolder 'c*.*']);
lbl = sscanf(cat(1,'fileNames.name'),'co2%c%d.rd.%d');
status = lbl(1:3:end);
id = lbl(2:3:end);
ids = unique(id);
trial = lbl(3:3:end);
I want to concatenate the names of all the files in the folder titled all content , at the moment, matlab doesn't understand what allcontent is. Can someone help me get the contents of the folder ' all content' which are of the form 'c*.*' and then concatenate them?
You can use fullfile to concatenate paths in Matlab, i.e.
fileNames = dir(fullfile(dataFolder, 'c*.*'));
Also, I don't think fileNames.name should be in quotes. As #Wolfie mentioned, you can concatenate the filenames into a cell array using {fileNames.name}
filenames_array = {fileNames.name}
Then you can iterate over filenames_array using for or cellfun
I have a series of DICOM Images which I want to anonymize, I found few Matlab codes and some programs which do the job, but none of them export a .txt file of removed personal information. I was wondering if there is a function which can also save removed personal information of a DICOM images in .txt format for features uses. Also, I am trying to create a table which shows the corresponding new images ID to their real name.(subjects real name = personal-information-removed image ID)
Any thoughts?
Thanks for considering my request!
I'm guessing you only want to output to your text file the fields that are changed by anonymization (either modified, removed, or added). First, you may want to modify some dicomanon options to reduce the number of changes, in particular passing the arguments 'WritePrivate', true to ensure private extensions are kept.
First, you can perform the anonymization, saving structures of pre- and post-anonymization metadata using dicominfo:
preAnonData = dicominfo('input_file.dcm');
dicomanon('input_file.dcm', 'output_file.dcm', 'WritePrivate', true);
postAnonData = dicominfo('output_file.dcm');
Then you can use fieldnames and setdiff to find fields that are removed or added by anonymization, and add them to the post-anonymization or pre-anonymization data, respectively, with a nan value as a place holder:
preFields = fieldnames(preAnonData);
postFields = fieldnames(postAnonData);
removedFields = setdiff(preFields, postFields);
for iField = 1:numel(removedFields)
postAnonData.(removedFields{iField}) = nan;
end
addedFields = setdiff(postFields, preFields);
for iField = 1:numel(addedFields)
preAnonData.(addedFields{iField}) = nan;
end
It will also be helpful to use orderfields so that both data structures have the same ordering for their field names:
postAnonData = orderfields(postAnonData, preAnonData);
Finally, now that each structure has the same fields in the same order we can use struct2cell to convert their field data to a cell array and use cellfun and isequal to find any fields that have been modified by the anonymization:
allFields = fieldnames(preAnonData);
preAnonCell = struct2cell(preAnonData);
postAnonCell = struct2cell(postAnonData);
index = ~cellfun(#isequal, preAnonCell, postAnonCell);
modFields = allFields(index);
Now you can create a table of the changes like so:
T = table(modFields, preAnonCell(index), postAnonCell(index), ...
'VariableNames', {'Field', 'PreAnon', 'PostAnon'});
And you could use writetable to easily output the table data to a text file:
writetable(T, 'anonymized_data.txt');
Note, however, that if any of the fields in the table contain vectors or structures of data, the formatting of your output file may look a little funky (i.e. lots of columns, most of them empty, except for those few fields).
One way to do this is to store the tags before and after anonymisation and use these to write your text file. In Matlab, dicominfo() will read the tags into a structure:
% Get tags before anonymization
tags_before = dicominfo(file_in);
% Anoymize
dicomanon(file_in, file_out); % Need to set tags values where required
% Get tags after anonymization
tags_after = dicominfo(file_out);
% Do something with the two structures
disp(['Patient ID:', tags_before.PatientID ' -> ' tags_after.PatientID]);
disp(['Date of Birth:', tags_before.PatientBirthDate ' -> ' tags_after.PatientBirthDate]);
disp(['Family Name:', tags_before.PatientName.FamilyName ' -> ' tags_after.PatientName.FamilyName]);
You can then write out the before/after fields into a text file. You'd need to modify dicomanon() to choose your own values for the removed fields, since by default they are set to empty.
this question about matlab:
i'm running a loop and each iteration a new set of data is produced, and I want it to be saved in a new file each time. I also overwrite old files by changing the name. Looks like this:
name_each_iter = strrep(some_source,'.string.mat','string_new.(j).mat')
and what I#m struggling here is the iteration so that I obtain files:
...string_new.1.mat
...string_new.2.mat
etc.
I was trying with various combination of () [] {} as well as 'string_new.'j'.mat' (which gave syntax error)
How can it be done?
Strings are just vectors of characters. So if you want to iteratively create filenames here's an example of how you would do it:
for j = 1:10,
filename = ['string_new.' num2str(j) '.mat'];
disp(filename)
end
The above code will create the following output:
string_new.1.mat
string_new.2.mat
string_new.3.mat
string_new.4.mat
string_new.5.mat
string_new.6.mat
string_new.7.mat
string_new.8.mat
string_new.9.mat
string_new.10.mat
You could also generate all file names in advance using NUM2STR:
>> filenames = cellstr(num2str((1:10)','string_new.%02d.mat'))
filenames =
'string_new.01.mat'
'string_new.02.mat'
'string_new.03.mat'
'string_new.04.mat'
'string_new.05.mat'
'string_new.06.mat'
'string_new.07.mat'
'string_new.08.mat'
'string_new.09.mat'
'string_new.10.mat'
Now access the cell array contents as filenames{i} in each iteration
sprintf is very useful for this:
for ii=5:12
filename = sprintf('data_%02d.mat',ii)
end
this assigns the following strings to filename:
data_05.mat
data_06.mat
data_07.mat
data_08.mat
data_09.mat
data_10.mat
data_11.mat
data_12.mat
notice the zero padding. sprintf in general is useful if you want parameterized formatted strings.
For creating a name based of an already existing file, you can use regexp to detect the '_new.(number).mat' and change the string depending on what regexp finds:
original_filename = 'data.string.mat';
im = regexp(original_filename,'_new.\d+.mat')
if isempty(im) % original file, no _new.(j) detected
newname = [original_filename(1:end-4) '_new.1.mat'];
else
num = str2double(original_filename(im(end)+5:end-4));
newname = sprintf('%s_new.%d.mat',original_filename(1:im(end)-1),num+1);
end
This does exactly that, and produces:
data.string_new.1.mat
data.string_new.2.mat
data.string_new.3.mat
...
data.string_new.9.mat
data.string_new.10.mat
data.string_new.11.mat
when iterating the above function, starting with 'data.string.mat'
How do I create a csv file with date as the name of the csv file. I tried doing it, but the date won't appear only the name does. The language is Scilab which is similar to Matlab.
I do not understand your question fully. But following the csvWrite documentation and the date documentation. You could do someting like this
filename_with_date_string = date() + ".csv";
directory_path = TMPDIR;
// Some matrix you want to save
M = [1:10] * 0.1;
// Create the file
file = fullfile(directory_path, filename_with_date_string);
// Fill it with your matrix
csvWrite(M, file);