I have a large matrix of 102730 rows in the form of text file (sample text file is attached) with some header files in it. The first column show year, the next the month, followed by the day, and value1, value2 and value 3. Some of the cells are missing/empty. I want to fill these empty cells with NaN, so that they don't interefere with the next value.
This is the input matrix:
1970 01 13 21.0 6.1 06 000.0
1970 01 14 22.4 8.1 03 000.0
1970 01 15 21.2 8.1 04 000.0
1970 01 16 22.6 9.1 04 000.0
1970 01 17 22.8 9.1 02 000.0
1970 01 18 22.9 8.9 07 000.0
1970 01 19 23.8 10.8 04 000.0
1970 01 20 21.8 12.1 10 010.5
1970 01 21 19.8 06 012.9
1970 01 22 15.3 8.5 07 000.0
1974 06 28 39.2 25.6 03 000.0
1974 06 29 41.2 30.5 05 000.0
1974 06 30 40.3 31.2 07 000.0
1974 07 01 41.3 31.5 12 000.0
1974 07 02 43.3 31.3 20 000.0
1974 07 03 41.2 16 041.6
1974 07 04 34.3 21.4 14 054.5
1974 07 05 33.1 23.8 05 000.0
1974 07 06 36.2 28.9 06 000.0
1975 04 18 36.6 20.8 12 000.0
1975 04 19 37.4 21.1 05 000.0
1975 04 20 39.9 27.0 07 000.0
1975 04 21 39.5 27.3 09 000.0
1975 04 22
1975 04 23 39.5 27.1 08 000.0
1975 04 24 37.7 26.0 10 000.0
1975 04 25 38.7 27.2 15 000.0
The desired output matrix:
1970 01 13 21.0 6.1 06 000.0
1970 01 14 22.4 8.1 03 000.0
1970 01 15 21.2 8.1 04 000.0
1970 01 16 22.6 9.1 04 000.0
1970 01 17 22.8 9.1 02 000.0
1970 01 18 22.9 8.9 07 000.0
1970 01 19 23.8 10.8 04 000.0
1970 01 20 21.8 12.1 10 010.5
1970 01 21 19.8 Nan 06 012.9
1970 01 22 15.3 8.5 07 000.0
1974 06 28 39.2 25.6 03 000.0
1974 06 29 41.2 30.5 05 000.0
1974 06 30 40.3 31.2 07 000.0
1974 07 01 41.3 31.5 12 000.0
1974 07 02 43.3 31.3 20 000.0
1974 07 03 41.2 Nan 16 041.6
1974 07 04 34.3 21.4 14 054.5
1974 07 05 33.1 23.8 05 000.0
1974 07 06 36.2 28.9 06 000.0
1975 04 18 36.6 20.8 12 000.0
1975 04 19 37.4 21.1 05 000.0
1975 04 20 39.9 27.0 07 000.0
1975 04 21 39.5 27.3 09 000.0
1975 04 22 Nan Nan Nan Nan
1975 04 23 39.5 27.1 08 000.0
1975 04 24 37.7 26.0 10 000.0
1975 04 25 38.7 27.2 15 000.0
As an attempt, first I tried with this:
T = readtable('sample.txt') ;
Above code didn't work since it meshed up and gave the wrong number of columns when there 2 digits before the decimal. Secondly, I found this link: Creating new matrix from cell with some empty cells disregarding empty cells
The foll. code snippet may be useful from this link, but I don't know how to read the data directly from the text pad inorder to apply this code & subsequent retrieval process:
inds = ~cellfun('isempty', elem); %elem to be replaced as sample
I also find out the method to detect empty cells here: How do I detect empty cells in a cell array?
but I couldn't figure out how to read the data from a text file considering these empty cells.
Could anyone please help?
Since R2019a, you can simply use readmatrix:
>> myMat = readmatrix('sample.txt')
From the docs:
For delimited text files, the importing function converts empty fields in the file to either NaN (for a numeric variable) or an empty character vector (for a text variable). All lines in the text file must have the same number of delimiters. The importing function ignores insignificant white space in the file.
For previous releases, you can use detectImportOptions object when calling readtable:
% Detect options.
>> opts = detectImportOptions('sample.txt');
% Read table.
>> myTable = readtable('sample.txt',opts);
% Visualise last rows of table.
>> tail(myTable)
ans =
8×7 table
Var1 Var2 Var3 Var4 Var5 Var6 Var7
____ ____ ____ ____ ____ ____ ____
1975 4 18 36.6 20.8 12 0
1975 4 19 37.4 21.1 5 0
1975 4 20 39.9 27 7 0
1975 4 21 39.5 27.3 9 0
1975 4 22 NaN NaN NaN NaN
1975 4 23 39.5 27.1 8 0
1975 4 24 37.7 26 10 0
1975 4 25 38.7 27.2 15 0
For your text file, detectImportOptions is filling missing values with NaN :
>> opts.VariableOptions
If the desired output is a matrix, you can then use table2array:
>> myMat = table2array(myTable)
Related
I have a bunch of csv files in a folder in following format, I want to extract complete time series from each (the numeric part from line #17), identify duplicate record and merge them in a ascending order according to year and date.
Specific csv file is accessible via google drive link below
wnsnum 1
paroms Waterhoogte
loccod HOEKVHLD
locoms Hoek van Holland
rks_begdat 1993 07 09
rks_begtyd 00:00
rks_enddat 2014 31 12
rks_endtyd 23:50
begdat begtyd enddat endtyd rkssta
1993 07 09 00:00 2007 31 12 23:50 D
2008 01 01 00:00 2009 30 12 23:50 G
2009 31 12 00:00 2009 31 12 23:50 O
2010 01 01 00:00 2011 17 06 18:40 G
2011 17 06 18:50 2011 18 06 18:50 O
2011 18 06 19:00 2014 31 12 23:50 G
datum tijd bpgcod waarde kwlcod
1993 07 09 00:00 -70 0
1993 07 09 00:10 -69 0
1993 07 09 00:20 -68 0
1993 07 09 00:30 -67 0
1993 07 09 00:40 -68 0
1993 07 09 00:50 -70 0
1993 07 09 01:00 -69 0
1993 07 09 01:10 -69 0
1993 07 09 01:20 -68 0
1993 07 09 01:30 -67 0
1993 07 09 01:40 -65 0
1993 07 09 01:50 -64 0
1993 07 09 02:00 -62 0
1993 07 09 02:10 -61 0
1993 07 09 02:20 -61 0
1993 07 09 02:30 -59 0
1993 07 09 02:40 -58 0
1993 07 09 02:50 -55 0
Now my code is working in a following way:
SL_files = dir(sprintf('%s%s%s',fullfile(dirName),'\','*.csv'));
for idx = 1:size(SL_files,1)
disp(SL_files(idx,1).name)
fid = fopen(sprintf('%s%s%s',fullfile(dirName),'\',SL_files(idx,1).name));
data = textscan(fid, '%s %f %f %f %f %f %f', ...
'Delimiter',',', 'MultipleDelimsAsOne',1,'headerlines',16);
fclose(fid);
end
Now I could read the file. Now my problem is how to combine multiple files' data into one matrix and arrange them in a ascending order according to year and day values. Thanks!
I finally solve my problem. Here is the code:
numMat_All = [];
for idx = 1:size(SL_files,1)
disp(SL_files(idx,1).name)
fid = fopen(sprintf('%s%s%s',fullfile(dirName),'\',SL_files(idx,1).name));
data = textscan(fid, '%s %f %f %f %f %f %f', ...
'Delimiter',',', 'MultipleDelimsAsOne',1,'headerlines',16);
fclose(fid);
CharCell = data{1,1};
result = regexprep(CharCell,'[\s;:]+',' ');
numMat = cell2mat(cellfun(#str2num, result(:,1:end), 'UniformOutput', false));
numMat_All = [numMat_All;numMat];
data = []; CharCell = []; result = []; numMat = [];
end
dt = datetime([numMat_All(:,1:5), repmat(0,length(numMat_All),1)]);
T = table(dt,numMat_All(:,[6:7]));
T1 = sortrows(T,'dt');
Please help me.
I looked for the way to graph the following table and I could not.
Take the column "HR" for the X axis and "VALUE" for Y axis, the problem is that the graph is cut. Column "RNO" only for sort.
For example: the value graph X Axis 10,11, ..., 21,22,23 when reaching the value or 00 does not continue the graph in the X axis, what happens is that values less than 10 puts them first. I attach the image that appears to me:
HR RATE_TIMEWAITED_PER_CLASS RNO
11 1
12 135.083333 2
13 232.916667 3
14 130.611111 4
15 155.111111 5
16 186.472222 6
17 166.805556 7
18 110.916667 8
19 89.3055556 9
20 198.166667 10
21 56.0277778 11
22 32.0277778 12
23 29.4722222 13
00 501.111111 14
01 18.6944444 15
02 16.0555556 16
03 14.1666667 17
04 375.892811 18
05 16.0833333 19
06 29.8611111 20
07 79.6666667 21
08 131.25 22
09 332.666667 23
this is my database table attendence fields are id studid ,adate,amonth ,ayear,acls_id, attendence and field values are below.
id studid adate amonth ayear acls_id attendence
1 28 02 07 2015 10 1
2 31 02 07 2015 10 1
4 32 02 07 2015 10 1
5 28 13 07 2015 10 1
6 31 13 07 2015 10 1
7 32 13 07 2015 10 1
9 28 14 07 2015 10 1
10 31 14 07 2015 10 1
11 32 14 07 2015 10 1
13 28 15 07 2015 10 1
14 31 15 07 2015 10 0
15 32 15 07 2015 10 1
17 28 16 07 2015 10 0
18 31 16 07 2015 10 1
19 32 16 07 2015 10 1
21 28 17 07 2015 10 1
22 31 17 07 2015 10 1
23 32 17 07 2015 10 0
24 28 20 08 2015 10 1
25 31 20 08 2015 10 1
26 32 20 08 2015 10 0
I want to display like pivot table format.
when im select month and year and check that month and year in database.if yes display values like this pivot table form
eg:july 2015
studid date1 date2 date3..date13 date14 date15 date16 date17.....date31
28 x 1 x 1 1 1 0 1 x x x..x
31 x 1 x 1 1 1 1 1 x x x..x
32 x 1 x 1 1 1 1 0 x x x..x
my controller code is
$amonth = Input::get('amonth');
$ayear = Input::get('ayear');
$dayCount = date('t', strtotime('01-'. $amonth . '-' . $ayear));
$days = [];
for($i = 1; $i <= $dayCount; $i++) {
$days[] = $i; }
$attendence_tbl = WysAttendancename::where('cls_id',$id)->first();
$wys_attendence_table = $attendence_tbl->attendance_name;
$attendance = DB::table($wys_attendence_table)
->where('amonth','=',$amonth)
->where('ayear','=',$ayear)
->get();`
my view .blade.php
<td>Teacher Name</td>
#for($i = 1; $i <= $dayCount; $i++)
<td>{{$i}}</td>
#endfor
</tr>
#foreach($teacher as $teachers)
<tr>
<td>{{$teachers->tname}}</td>
#foreach($attendance as $attendances)
#if($teachers->id == $attendances->t_auserid)
#if($attendances->t_attendance == 1)
<td><font color="green">p</font></td>
#elseif($attendances->t_attendance == 0)
<td><font color="red">a</font></td>
#endif
#endif
#endforeach
#endforeach
its work fine but problem is couldn't check the month day(eg :adate-2) is in exist in database or not .
iam using display full month in column ,code is
$dayCount = date('t', strtotime('01-'. $amonth . '-' . $ayear));
$days = [];
for($i = 1; $i <= $dayCount; $i++) {
$days[] = $i; } `
in my database 1/07/2015 is not but in view page display 2/70/2015 value in 1/07/2015 and also 13/07/2015 value in 3/107/2015 ...
I want to dislpay like above pivot table form
how to change my code to check databse adate and month day($i)???
Im taking weather day from wunderground.com and I and then trimming down the data for use on gnuplot. Im having trouble replacing the second column only data from number to months abbreviations. Only interested in the second column.
I want to go from this;
>2013 08 02 23 37 00 73.3
>2013 08 02 23 42 00 73.4
>2013 08 02 23 45 00 73.3
>2013 08 02 23 47 00 73.1
>2013 08 02 23 52 00 73.1
>2013 08 02 23 57 00 73.1
To this:
>2013 AUG 02 23 37 00 73.3
>2013 AUG 02 23 42 00 73.4
>2013 AUG 02 23 45 00 73.3
>2013 AUG 02 23 47 00 73.1
>2013 AUG 02 23 52 00 73.1
>2013 AUG 02 23 57 00 73.1
i am trying to use sed to change the numbers into the correct month and i keep getting this. I only want the correct sed expression to execute not all of them. This is the command i am trying to use.
sed -e 's/01/JAN/' -e 's/02/FEB/' -e 's/03/MAR/' -e 's/04/APR/' -e 's/05/MAY/' -e 's/06/JUN/' -e 's/07/JUL/' -e 's/08/AUG/' -e 's/09/SEP/' -e 's/10/OCT/' -e 's/11/NOV/' -e 's/12/DEC/'
How would i go about this.
This might work for you (GNU sed):
sed -nri 'G;s/$/01JAN02FEB03MAR04APR05MAY06JUN07JUL08AUG09SEP10OCT11NOV12DEC/;s/ (..)(.*)\1(...)/ \3\2/;P' file
This adds a lookup table to the end of each line and substitutes the key for the value.
I would use awk for this:
$ awk 'BEGIN{split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec",a)} {$2=a[$2+0]}1' a
>2013 Aug 02 23 37 00 73.3
>2013 Aug 02 23 42 00 73.4
>2013 Aug 02 23 45 00 73.3
>2013 Aug 02 23 47 00 73.1
>2013 Aug 02 23 52 00 73.1
>2013 Aug 02 23 57 00 73.1
To update the field with the new content, just redirect and then move:
awk .... file > temp_file && mv temp_file file
Explanation
What we do is to give awk a list of strings with the months names. Once we convert it into an array, a[1] will be Jan, a[2] Feb and so on. So then it is just a matter of replacing the 2nd field with a[2nd field].
BEGIN{split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec",a)} fetches the data and inserts into the a[] array.
{$2=a[$2+0]} sets the 2nd field as a[2nd field]. The $2+0 is done to convert 08 to 8.
Finally 1 evaluates as true and makes awk perform its default action: {print $0}.
Workaround that works for this problem (since your first column is very predictable) but not the general question:
sed -E -e 's/^([0-9]{4}) 01/\1 JAN/' -e 's/^([0-9]{4}) 02/\1 FEB/' etc.
awk has a sub function that could get unwieldy for many options you have here.
Perl scripts might be the best way to go.
$ awk '{$2=substr("JanFebMarAprMayJunJulAugSepOctNovDec",(3*$2)-2,3)}1' file
>2013 Aug 02 23 37 00 73.3
>2013 Aug 02 23 42 00 73.4
>2013 Aug 02 23 45 00 73.3
>2013 Aug 02 23 47 00 73.1
>2013 Aug 02 23 52 00 73.1
>2013 Aug 02 23 57 00 73.1
Since it came up in a comment:
The idiomatic awk way to map from a month number to a name is:
number = (match("JanFebMarAprMayJunJulAugSepOctNovDec",<name>)+2)/3
and the above is just the natural inverse of that:
name = substr("JanFebMarAprMayJunJulAugSepOctNovDec",(3*<number>)-2,3)
Like with anything in awk there's various ways to get the output you want but IMHO the symmetry here makes it an attractive solution:
awk 'BEGIN{
months = "JanFebMarAprMayJunJulAugSepOctNovDec"
name = "Jul"
number = (match(months,name)+2)/3
print name " -> " number
name = substr(months,(3*number)-2,3)
print number " -> " name
}'
Jul -> 7
7 -> Jul
Notice that the script uses the same definition for months no matter which direction the conversion is being done and it's a similar math calculation in both directions.
Nothing wrong with doing it this way too of course:
awk 'BEGIN{
split("Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec",num2name)
for (number in num2name) {
name2num[num2name[number]] = number
}
name = "Jul"
number = name2num[name]
print name " -> " number
name = num2name[number]
print number " -> " name
}'
Jul -> 7
7 -> Jul
Just a few more lines of code, nbd.
Using gnu awk's function strftime() and mktime()
awk '{$2=strftime("%b",mktime("2014 " $2 " 1 1 0 0"))}1' file
>2013 Aug 02 23 37 00 73.3
>2013 Aug 02 23 42 00 73.4
>2013 Aug 02 23 45 00 73.3
>2013 Aug 02 23 47 00 73.1
>2013 Aug 02 23 52 00 73.1
>2013 Aug 02 23 57 00 73.1
Explanation
mktime("2014 " $2 " 1 1 0 0") fake the epoch time, use the column 2 as month
strftime("%b",mktime("2014 " $2 " 1 1 0 0")) convert the epoch back to the date, with %b, export the abbreviated month name (Jan, Feb, etc)
The benefit with this awk:
It is shorter, of course. Second, you can control/adjust the format in strftime() to export any date format you like.
For example, if change to full month name %B. You needn't rewrite the code.
awk '{$2=strftime("%B",mktime("2014 " $2 " 1 1 0 0"))}1' file
In the matrix shown below how can I select elements 01, 09, 17 and 25. From Egon's answer to my earlier question Select Diagonal Elements of a Matrix in MATLAB I am able to select the central value 25 using c = (size(A)+1)/2; but I am wondering how to select above mentioned elements in NW direction.
A = [01 02 03 04 05 06 07
08 09 10 11 12 13 14
15 16 17 18 19 20 21
22 23 24 25 26 27 28
29 30 31 32 33 34 35
36 37 38 39 40 41 42
43 44 45 46 47 48 49];
Use diag to get elements on the diagonal.
diagA = diag(A)
You can restrict this to the elements from the top left to the middle with
n = ceil(size(A, 1) / 2)
diagA(1:n)
Another way to do this is with linear indexing. If you have an N-by-N matrix, you can select the elements you want as follows:
values = A(1:N+1:ceil((N^2)/2));