Extracting netCDF time series for each lat/long in Matlab - matlab

I'm currently working with netCDF output from climate models and would like to obtain a text file of the time series for each latitude/longitude combination in the netCDF. For example, if the netCDF has 10 latitudes and 10 longitudes I would obtain 100 text files, each with a time series in a column format. I'm fairly familiar with the Matlab/netCDF language, but I can't seem to wrap my head around this. Naming the text files is not important; I will rename them "latitude_longitude_PCP.txt", where PCP is precipitation at the latitude and longitude location.
Any help would be appreciated. Thanks.
--Darren

There are several ways this problem could be solved.
Method 1. If you were able to put your netcdf file on a THREDDS Data Server, you could use the NetCDF Subset Service Grid as Point to specify a longitude/latitude point and get back the data in CSV or XML format. Here's an example from Unidata's THREDDS Data Server: http://thredds.ucar.edu/thredds/ncss/grid/grib/NCEP/GFS/Global_0p5deg/best/pointDataset.html
Method 2. If you wanted to use Matlab to extract a time series at a specific longitude/latitude location you could use the "nj_tseries" function from NCTOOLBOX, available at: http://nctoolbox.github.io/nctoolbox/
Method 3. If you really want to write an ASCII time series at every i,j location in your [time,lon,lat] grid using Matlab, you could do something like this (using NCTOOLBOX):
url='http://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GFS/Global_2p5deg/best';
nc = ncgeodataset(url);
nc.variables
var='Downward_Short-Wave_Radiation_Flux_surface_12_Hour_Average';
lon = nc.data('lon');
lat = nc.data('lat');
jd = nj_time(nc,var);
ncvar = nc.variable(var);
for j=1:length(lat)
for i=1:length(lon)
v=ncvar.data(:,j,i);
outfile=sprintf('%6.2flon%6.2flat.csv',lon(i),lat(j))
fid=fopen(outfile,'wt')
data= [datevec(jd) v]
fprintf(fid,'%2.2d %2.2d %2.2d %2.2d %2.2d %2.2d %7.2f\n',data')
fclose(fid)
disp([outfile ' created.'])
end
end
If you had enough memory to read all the data into matlab, you could read outside the double loop, which would be a lot faster. But writing ASCII is slow anyway, so it might not matter that much.

%% Create demo data
data = reshape(1:20*30*40,[20 30 40]);
nccreate('t.nc','data','Dimensions',{'lat', 20, 'lon',30, 'time', inf});
ncwrite('t.nc', 'data',data);
ncdisp('t.nc');
%% Write timeseries to ASCII files
% Giving an idea of the size of your data can help people
% recommend different approaches tailored to the data size.
% For smaller data, it might be faster to read in the full
% 3D data into memory
varInfo = ncinfo('t.nc','data');
disp(varInfo);
for latInd =1:varInfo.Size(1)
for lonInd =1:varInfo.Size(2)
fileName = ['t_ascii_lat',num2str(latInd),'_lon',num2str(lonInd),'.txt'];
tSeries = ncread('t.nc','data',[latInd, lonInd, 1],[1,1,varInfo.Size(3)]);
dlmwrite(fileName,squeeze(tSeries));
end
end
%% spot check
act = dlmread('t_ascii_lat10_lon29.txt');
exp = squeeze(data(10,29,:));
assert(isequal(act,exp));

Related

MATLAB: How can I efficiently read in these data files?

I have 100 data files in a folder called "Experiment1", and I need to take all of the data from them and put them into a single matrix. Each data file contains 15 columns and 40 rows of data.
The order in which the files are in the folder is arbitrary. It doesn't matter in what order they get put into the combined matrix.
I've written some code using dlmread that will do the job:
for i = 1:100
%% Read in the relevant file.
filename = ['File_' int2str(i) '.dat']
Data = dlmread(fullfile(pwd, 'Experiment1',filename));
%% Put all the data I need in a separate matrix
NeededData(1+((i-1)*40):i+((i-1)*40)-i+40,1:15) = Data(:,1:15);
end
However, there are two things I don't like about my code.
The files have random names at present, and I'd need to manually change all their names to "File_1.dat", "File_2.dat", etc.
The code is cumbersome and hard to read.
How could I do things better?
Since you've fixed the problem of defining the name of the files to be read with dir, you can improve the way you add the read data (Data) to the output matrix (NeededData).
You can sumultaneously read the input files and add the data to the output matrix by inserting the call to dlmread directly in the assignment statement:
files=dir('*.dat');
n_files=length(files)
% Initialize the output matrix as empty
NeededData_0=[]
for i=1:n_files
% Simultaneously read input file and assign data to the output matrinx
NeededData_0=[NeededData_0;dlmread(files(i).name)]
end
In case you prefer working with the inides (as in your origina approach), since you know in advance that all the files have the same (40) number of rows) you can simplify the notation as follows:
files=dir('*.dat');
n_files=length(files)
% Define the number of rows in each inout file
n_rows=40;
% Define the number of colums in each inout file
n_col=15;
NeededData_2=nan(n_rows*n_files,n_col)
% Define the sequence of rows
r_list=1:n_rows:n_rows*n_files
for i=1:3
Data=dlmread(files(i).name)
NeededData_2(r_list(i):r_list(i)+n_rows-1,:)=Data
end
Hope this helps.
Using the suggestion to use dir present in the answers I have made the following code, which is clearly an improvement on my earlier effort. I would welcome further improvements, or suggestions for alternative approaches.
files = dir('*.dat');
for i = 1:length({files.name})
%% Read in the relevant file.
Data = dlmread(files(i).name);
%% Put all the data I need in a separate matrix
NeededData(1+((i-1)*40):i+((i-1)*40)-i+40,1:15) = Data(:,1:15);
end

MATLAB loading data from multiple .mat files

My data is x,y co-ordinates in multiple files
a=dir('*.mat')
b={a(:).name}
to load the filenames in a cell array
How do I use a loop to sequentially load one column of data from each file into consecutive rows of a new/separate array......?
I've been doing it individually using e.g.
Load(example1.mat)
A(:,1)=AB(:,1)
Load(example2.mat)
A(:,2)=AB(:,1)
Load(example3.mat)
A(:,3)=AB(:,1)
Obviously very primitive and time consuming!!
My Matlab skills are weak so any advice gratefully received
Cheers
Many thanks again, I'm still figuring out how to read the code but I used it like this;
a=dir('*.mat');
b={a(:).name};
test1=zeros(numel(b),1765);
for k=1:numel(b) S=load(b{k});
I then used the following code to create a PCA cluster plot
test1(k,:)=S.AB(:,2); end [wcoeff,score,latent,tsquared,explained] = pca(test1,... 'VariableWeights','variance');
c3 = wcoeff(:,1:3) coefforth = inv(diag(std(test1)))*wcoeff; I = c3'*c3 cscores = zscore(test1)*coefforth;
figure() plot(score(:,1),score(:,2),'+') xlabel('1st Principal Component') ylabel('2nd Principal Component') –
I was using 'gname' to label the points on the cluster plot but found that the point were simply labelled from 1 to the number of rows in the array.....I was going to ask you about this but I found out simply through trial and error if I used 'gname(b)' this labels the points with the .names listed in b.....
However the clusterplot starts to look very busy/messy once I have labelled quite a few points so now I am wondering is is possible to extract the filenames into a list by dragging round or selecting a few points, I think it is possible as I have read a few related topics.....but any tips/advice around gname or labelled/extracting labels from clusterplots would be greatly appreciated. Apologies again for my formatting I'm still getting used to this website!!!
Here is a way to do it. Hopefully I got what you wanted correctly :)
The code is commented but please ask any questions if something is unclear.
a=dir('*.mat');
b={a(:).name};
%// Initialize the output array. Here SomeNumber depends on the size of your data in AB.
A = zeros(numel(b),SomeNumber);
%// Loop through each 'example.mat' file
for k = 1:numel(b)
%// ===========
%// Here you could do either of the following:
1)
%// Create a name to load with sprintf. It does not require a or b.
NameToLoad = sprintf('example%i.mat',k);
%// Load the data
S = load(NameToLoad);
2)
%// Load directly from b:
S = load(b{k});
%// ===========
%// Now S is a structure containing every variable from the exampleX.mat file.
%// You can access the data using dot notation.
%// Store the data into rows of A
A(k,:) = S.AB(:,1);
end
Hope that is what you meant!

How to create array data-structures in MATLAB?

I basically have a large data set file and I want to write a MATLAB script that creates a data structure for it. I have tried to read about using structured arrays in MATLAB, but I haven't found a solution of how to do this. I don't really have a lot of experience in writing scripts on MATLAB.
Edited: My data set is a large list of items with, say, 10 different characteristics of each item written down. So for example, say 100,000 listings of houses and characteristics given could be price, county, state, date when sold, etc. This file is in a txt., xls., or any format you like to play with.
I would like to write a MATLAB script that creates a data structure of it say in the format:
house(i).price
house(i).county
house(i).state
house(i).date
etc
Any suggestions to the right direction or examples of teaching how to do this would be greatly appreciated.
This seems like a very reasonable question, and one that can be easily addressed.
The format of the file, really makes this problem easy or hard. I really don't like .xls files for this kind of work myself, but I realize, you get what you get. Let's assume it's in a tab delimited text file like:
Price County State Date
100000 Sherlock London 2001-10-01
134000 Holmes Dartmoor 2011-12-30
123456 Watson Boston 2003-04-15
IfI would just read the whole thing into an parse the field name row and use dynamic structure naming to make the array of structures.
fid = fopen('data.txt','r');
tline = fgetl(fid);
flds = regexp(tline,'\s*','split');
% initialize the first prototype struct
data = struct();
for ii=1:length(flds)
data.(flds{ii}) = [];
end
ii = 1;
% get the first line of data
tline = fgetl(fid);
while ischar(tline)
% parse the data
rowData = regexp(tline,'\s*','split');
% we're assuming no missing data, etc
% populate the structure
for jj=1:length(flds)
data(ii).(flds{jj}) = rowData{jj};
end
% since we don't know how many lines we have
% we could figure that out, but we won't now
% we'll just use the size extending feature of
% matlab arrays, even though it's slow, just
% to show how we would do it
tline = fgetl(fid);
ii = ii + 1;
end
fclose(fid)
Hope this gets you started!

MATLAB error message when plotting big data files?

I have written code to plot data from very large .txt files (20Gb to 60Gb). The .txt files contain two columns of data, that represent the outputs of two sensors from an experiment that I did. The reason the data files are so large is that the data was recorded at 4M samples/s.
The code works well for plotting relatively small .txt files (10Gb), however when I try to plot my larger data files (60Gb) I get the following error message:
Attempted to access TIME(0); index must be a
positive integer or logical.
Error in textscan_loop (line 17)
TIME =
((TIME(end)+sample_rate):sample_rate:(sample_rate*(size(d,1)))+(TIME(end)));%shift
Time along
The basic idea behind my code is to conserve RAM by reading Nlines of data from .txt on disk to Matlab variable C in RAM, plotting C then clearing C. This process occurs in loop so the data is plotted in chunks until the end of the .txt file is reached. The code can be found below:
Nlines = 1e6; % set numbe of lines to sample per cycle
sample_rate = (1); %sample rate
DECE= 1000;% decimation factor
TIME = (0:sample_rate:sample_rate*((Nlines)-1));%first inctance of time vector
format = '%f\t%f';
fid = fopen('H:\PhD backup\Data/ONK_PP260_G_text.txt');
while(~feof(fid))
C = textscan(fid, format, Nlines, 'CollectOutput', true);
d = C{1}; % immediately clear C at this point you need the memory!
clearvars C ;
TIME = ((TIME(end)+sample_rate):sample_rate:(sample_rate*(size(d,1)))+(TIME(end)));%shift Time along
plot((TIME(1:DECE:end)),(d(1:DECE:end,:)))%plot and decimate
hold on;
clearvars d;
end
fclose(fid);
I think the while loop does around 110 cycles before the code stops executing and the error message is displayed, I know this because the graph shows around 110e7 data points and the loop processes 1e6 data points at a time.
If anyone knows why this error might be occurring please let me know.
Cheers,
Jim
The error that you encounter is in fact not in the plotting, but in the line of reference.
Though I have been unable to reproduce the exact error, I suspect it to be related to this:
Time = 1:0
Time(end)
In any case, the way forward is clear. You need to run this code with dbstop if error and observe all relevant variables in the line that throws the error.
From here you will likely figure out what is causing the problem, hopefully just something simple like your code being unable to deal with data size that is an exact multiple of 1000 or so.
Trying to use plot for big data is problematic as matlab is trying to plot every single data point.
Obviously the screen will not display all of these points (many will overlap), and therefore it is recommended to plot only the relevant points. One could subsample and do this manually as you seem to have tried, but fortunately we have a ready to use solution for this:
The Plot (Big) File Exchange Submission
Here is the introduction:
This simple tool intercepts data going into a plot and reduces it to
the smallest possible set that looks identical given the number of
pixels available on the screen. It then updates the data as a user
zooms or pans. This is useful when a user must plot a very large
amount of data and explore it visually.
This works with MATLAB's built-in line plot functions, allowing the
functionality of those to be preserved.
Instead of:
plot(t, x);
One could use:
reduce_plot(t, x);
Most plot options, such as multiple series and line properties, can be
passed in too, such that 'reduce_plot' is largely a drop-in
replacement for 'plot'.
h = reduce_plot(t, x(1, :), 'b:', t, x(2, :), t, x(3, :), 'r--*');
This function works on plots where the "x" data is always increasing,
which is the most common, such as for time series.

MATLAB query about for loop, reading in data and plotting

I am a complete novice at using matlab and am trying to work out if there is a way of optimising my code. Essentially I have data from model outputs and I need to plot them using matlab. In addition I have reference data (with 95% confidence intervals) which I plot on the same graph to get a visual idea on how close the model outputs and reference data is.
In terms of the model outputs I have several thousand files (number sequentially) which I open in a loop and plot. The problem/question I have is whether I can preprocess the data and then plot later - to save time. The issue I seem to be having when I try this is that I have a legend which either does not appear or is inaccurate.
My code (apolgies if it not elegant):
fn= xlsread(['tbobserved' '.xls']);
time= fn(:,1);
totalreference=fn(:,4);
totalreferencelowerci=fn(:,6);
totalreferenceupperci=fn(:,7);
figure
plot(time,totalrefrence,'-', time, totalreferencelowerci,'--', time, totalreferenceupperci,'--');
xlabel('Year');
ylabel('Reference incidence per 100,000 population');
title ('Total');
clickableLegend('Observed reference data', 'Totalreferencelowerci', 'Totalreferenceupperci','Location','BestOutside');
xlim([1910 1970]);
hold on
start_sim=10000;
end_sim=10005;
h = zeros (1,1000);
for i=start_sim:end_sim %is there any way of doing this earlier to save time?
a=int2str(i);
incidenceFile =strcat('result_', 'Sim', '_', a, 'I_byCal_total.xls');
est_tot=importdata(incidenceFile, '\t', 1);
cal_tot=est_tot.data;
magnitude=1;
t1=cal_tot(:,1)+1750;
totalmodel=cal_tot(:,3)+cal_tot(:,5);
h(a)=plot(t1,totalmodel);
xlim([1910 1970]);
ylim([0 500]);
hold all
clickableLegend(h(a),a,'Location','BestOutside')
end
Essentially I was hoping to have a way of reading in the data and then plot later - ie. optimise the code.
I hope you might be able to help.
Thanks.
mp
Regarding your issue concerning
I have a legend which either does not
appear or is inaccurate.
have a look at the following extracts from your code.
...
h = zeros (1,1000);
...
a=int2str(i);
...
h(a)=plot(t1,totalmodel);
...
You are using a character array as index. Instead of h(a) you should use h(i). MATLAB seems to cast the character array a to double as shown in the following example with a = 10;.
>> double(int2str(10))
ans = 49 48
Instead of h(10) the plot handle will be assigned to h([49 48]) which is not your intention.