If I create a h5 file larger than 2GB, I get an error. Smaller than 2GB things work fine. I didn't think this filesize was a problem with the h5 format. This seems to not be a problem on my windows 8.1 machine, but is on my Ubuntu 20.04 machine.
MWE:
fname = 'tmp001.h5';
h5create(fname,'/DS1',[195 2048 1500],'DataType','single'); % doesn't work
h5write(fname,'/DS1',1,[1 1 1],[1 1 1]);
fname = 'tmp002.h5';
h5create(fname,'/DS1',[195 2048 1400],'DataType','single');% doesn't work
h5write(fname,'/DS1',1,[1 1 1],[1 1 1]);
fname = 'tmp003.h5';
h5create(fname,'/DS1',[195 2048 1300],'DataType','single');% does work
h5write(fname,'/DS1',1,[1 1 1],[1 1 1]);
This only seems to be a problem if I give it only a segment of the data set, and even then it works providing the size of the data is more than 2^14! See files 4 and 5:
fname = 'tmp004.h5';
h5create(fname,'/DS1',[536886273],'DataType','single');% does work
h5write(fname,'/DS1',rand(536886273,1)); % filesize > 2gb but length > 2^14
fname = 'tmp005.h5';
h5create(fname,'/DS1',[536886273],'DataType','single');% doesn't work
h5write(fname,'/DS1',1,1,1); % filesize > 2gb but length < 2^14
fname = 'tmp006.h5';
h5create(fname,'/DS1',[536886272],'DataType','single');% does work
h5write(fname,'/DS1',1,1,1); % length > 2^14 but okay since filesize < 2gb
Error:
Warning: The following error was caught while executing 'H5ML.id' class destructor:
Error using hdf5lib2
The HDF5 library encountered an error and produced the following stack trace information:
H5FD_truncate driver truncate request failed
H5F_dest low level truncate failed
H5F_try_close problems closing file
H5O_close problem attempting file close
H5D_close unable to release object header
H5I_dec_app_ref can't decrement ID ref count
H5I_dec_app_ref_always_close can't decrement ID ref count
H5Dclose can't decrement count on dataset ID
Error in H5ML.id/close (line 47)
H5ML.hdf5lib2(obj.callback, obj.identifier);
Error in H5ML.id/delete (line 36)
obj.close();
Related
I block on my problem that I will wrote in details below. During 3 days I tried a lot of differents things, none worked..
If anyone have an idea of what to do !
Here is my message error :
the call to "ft_selectdata" took 0 seconds
preprocessing
Out of memory. Type "help memory" for your options.
Error in ft_preproc_dftfilter (line 187)
tmp = exp(2*1i*pi*freqs(:)*time); % complex sin and cos
Error in ft_preproc_dftfilter (line 144)
filt = ft_preproc_dftfilter(filt, Fs, Fl(i), 'dftreplace', dftreplace, 'dftbandwidth',
dftbandwidth(i), 'dftneighbourwidth', dftneighbourwidth(i)); % enumerate all options
Error in preproc (line 464)
dat = ft_preproc_dftfilter(dat, fsample, cfg.dftfreq, optarg{:});
Error in ft_preprocessing (line 375)
[dataout.trial{i}, dataout.label, dataout.time{i}, cfg] = preproc(data.trial{i}, data.label,
data.time{i}, cfg, begpadding, endpadding);
Error in EEG_Prosocial_script (line 101)
data_intpl = ft_preprocessing(cfg, allData_preprosses);
187 tmp = exp(2*1i*pi*freqs(:)*time); % complex sin and cos
There is some informations from matlab about my computer and about the caracteristics of the calcul
Maximum possible array: 7406 MB (7.766e+09 bytes)*
Memory available for all arrays: 7406 MB (7.766e+09 bytes) *
Memory used by MATLAB: 4195 MB (4.398e+09 bytes)
Physical Memory (RAM): 12206 MB (1.280e+10 bytes)
Limited by System Memory (physical + swap file) available.
K>> whos Name
Size Bytes Class Attributes
freqs 7753x1 62024 double
li 1x1 16 double complex
time 1x1984512 15876096 double
So there the config of the computer which failed to run the script (Alienware aurora R4) :
Ram : 4gb free / 12 # 1,6Ghz --> 2x (4Gb 1600Mhz) - 2x (2Gb 1600 MHz)
Intel core i7-3820 4 core 8 threads 3,7 GHz 1 CPU
NVIDIA GeForce GTX 690 2gb
RAM : Kingston KVT8FP HYC
Hard disk : SSD kingston 250Go SATA 3"
This code work on this computer (Dell inspiron 14-500) : config
Ram 4 Go of memory DDR4 2 666 MHz (4 Go x 1)
Intel® Core™ i5-8265U 8e generatio, (6 Mo memory, 3,9 GHz)
Intel® UHD Graphics 620
Hard disk SATA 2,5" 500 Go 5 400 tr/min
Thank you
Kind regards,
by doing freqs(:)*time you are trying to create a 7753*1984512 size array, and you dont have memory for that... (you will need ~123 gigabytes for that, or 1.2309e+11 bytes, where your computer has ~7e+09)
see for example the case for :
f=rand(7,1);
t=rand(1,19);
size(f(:)*t)
ans =
7 19
what you want do to is probably a for loop per time element etc.
I recently connected a reactor control tower through a serial 'COMMS' port to my computer (serial to USB). It seems to create a connection on the COM4 port (as indicated on the control panel 'devices and printers' section). However, it always gives me the following message when i try to 'fwrite(s)' or 'fscanf(s)'.
Warning: Unsuccessful read: The specified amount of data was not returned within the Timeout period..
%My COM4
s = serial('COM4');
s.BaudRate = 9600;
s.DataBits = 8;
s.Parity ='none';
s.StopBits = 1;
s.FlowControl='none';
s.Terminator = ';';
s.ByteOrder = 'LittleEndian';
s.ReadAsyncMode = 'manual';
% Building write message.
devID = '02'; % device ID
cmd = 'S'; % command read or write; S for write
readM = cell(961,3);% Read at most 961-by-3 values filling a 961–by–3 matrix in column order
strF = num2str(i);
strF = '11'; %pH parameter
strP = '15'; %pH set point
val = '006.8'; %pH set value
msg_ = strcat('!', devID, cmd, strF, strP, val);%output the string
chksum = dec2hex(mod(sum(msg_),256)); %conversion to hexdec
msg = strcat(msg_,':', char(chksum), ';');
fopen(s); %connects s to the device using fopen , writes and reads text data
fwrite(s, uint8(msg)); %writes the binary data/ Convert to 8-bit unsigned integer (unit8) to the instrument connected to s.
reply=fscanf(s); %reads ASCII data from the device connected to the serial port object and returns it to reply, for binary data use fread
fclose(s); %Disconnect s from the scope, and remove s from memory and the workspace.
This leads me to believe that the device is connected but is not sending or receiving information and I am unsure as to how I can configure this or even really check if there is communication occurring between the tower and my computer.
I am trying to read from a file and display the data in rows 6, 11, 111 and 127 in Matlab. I could not figure out how to do it. I have been searching Matlab forums and this platform for an answer. I used fscanf, textscan and other functions but they did not work as intended. I also used a for loop but again the output was not what I wanted. I can now only read one row and display it. Simply I want to display all of them(data in rows given above) at the same time. How can I do that?
matlab code
n = [0 :1: 127];
%% Problem 1
figure
x1 = cos(0.17*pi*n)
%it creates file and writes content of x1 to the file
fileID = fopen('file.txt','w');
fprintf(fileID,'%d \n',x1);
fclose(fileID);
%line number can be changed in order to obtain wanted values.
fileID = fopen('file.txt');
line = 6;
C = textscan(fileID,'%s',1,'delimiter','\n', 'headerlines',line-1);
celldisp(C)
fclose(fileID);
and this is the file
1
8.607420e-01
4.817537e-01
-3.141076e-02
-5.358268e-01
-8.910065e-01
-9.980267e-01
-8.270806e-01
-4.257793e-01
9.410831e-02
5.877853e-01
9.177546e-01
9.921147e-01
7.901550e-01
3.681246e-01
-1.564345e-01
-6.374240e-01
-9.408808e-01
-9.822873e-01
-7.501111e-01
-3.090170e-01
2.181432e-01
6.845471e-01
9.602937e-01
9.685832e-01
7.071068e-01
2.486899e-01
-2.789911e-01
-7.289686e-01
-9.759168e-01
-9.510565e-01
-6.613119e-01
-1.873813e-01
3.387379e-01
7.705132e-01
9.876883e-01
9.297765e-01
6.129071e-01
1.253332e-01
-3.971479e-01
-8.090170e-01
-9.955620e-01
-9.048271e-01
-5.620834e-01
-6.279052e-02
4.539905e-01
8.443279e-01
9.995066e-01
8.763067e-01
5.090414e-01
-4.288121e-15
-5.090414e-01
-8.763067e-01
-9.995066e-01
-8.443279e-01
-4.539905e-01
6.279052e-02
5.620834e-01
9.048271e-01
9.955620e-01
8.090170e-01
3.971479e-01
-1.253332e-01
-6.129071e-01
-9.297765e-01
-9.876883e-01
-7.705132e-01
-3.387379e-01
1.873813e-01
6.613119e-01
9.510565e-01
9.759168e-01
7.289686e-01
2.789911e-01
-2.486899e-01
-7.071068e-01
-9.685832e-01
-9.602937e-01
-6.845471e-01
-2.181432e-01
3.090170e-01
7.501111e-01
9.822873e-01
9.408808e-01
6.374240e-01
1.564345e-01
-3.681246e-01
-7.901550e-01
-9.921147e-01
-9.177546e-01
-5.877853e-01
-9.410831e-02
4.257793e-01
8.270806e-01
9.980267e-01
8.910065e-01
5.358268e-01
3.141076e-02
-4.817537e-01
-8.607420e-01
-1
-8.607420e-01
-4.817537e-01
3.141076e-02
5.358268e-01
8.910065e-01
9.980267e-01
8.270806e-01
4.257793e-01
-9.410831e-02
-5.877853e-01
-9.177546e-01
-9.921147e-01
-7.901550e-01
-3.681246e-01
1.564345e-01
6.374240e-01
9.408808e-01
9.822873e-01
7.501111e-01
3.090170e-01
-2.181432e-01
-6.845471e-01
-9.602937e-01
-9.685832e-01
-7.071068e-01
-2.486899e-01
2.789911e-01
Assuming the file is not exceedingly large, the simplest way would probably be read the entire file & index the output to your desired lines.
line = [6 11 111 127];
fileID = fopen('file.txt');
C = textscan(fileID,'%s','delimiter','\n');
fclose(fileID);
disp(C{1}(line))
I'm trying to read the data from the COM3 port.
I'm using this code:
in = fscanf(s);
if(in == 'A')
fclose(s);
break;
end
The problem is that when no data is sent to the com3 port, the fscanf() will wait for a certain time interval and then give a timeout.
Is there a way to read data only when it is present?
Read only when data present
You can read out the BytesAvailable-property of the serial object s to know how many bytes are in the buffer ready to be read:
bytes = get(s,'BytesAvailable'); % using getter-function
bytes = s.BytesAvailable; % using object-oriented-addressing
Then you can check the value of bytes to match your criteria. Assuming a char is 1 byte, then you can check for this easily before reading the buffer.
if (bytes >= 1)
in = fscanf(s);
% do the handling of 'in' here
end
Minimize the time to wait
You can manually set the Timeout-property of the serial object s to a lower value to continue execution earlier as the default timeout.
set(s,'Timeout',1); % sets timeout to 1 second (default is 10 seconds)
Most likely you will get the following warning:
Unsuccessful read: A timeout occurred before the Terminator was
reached..
It can be suppressed by executing the following command before fscanf.
warning('off','MATLAB:serial:fscanf:unsuccessfulRead');
Here is an example:
s = serial('COM3');
set(s,'Timeout',1); % sets timeout to 1 second (default is 10 seconds)
fopen(s);
warning('off','MATLAB:serial:fscanf:unsuccessfulRead');
in = fscanf(s);
warning('on','MATLAB:serial:fscanf:unsuccessfulRead');
if(in == 'A')
fclose(s);
break;
end
I traced an oracle process, and find it first open a file /etc/netconfig as file handle 11, and then duplicate it as 256 by calling fcntl with parameter F_DUPFD, and then close the original file handle 11. Later it read using file handle 256. So what's the point to duplicate the file handle? Why not just work on the original file handle?
12931: 0.0006 open("/etc/netconfig", O_RDONLY|O_LARGEFILE) = 11
12931: 0.0002 fcntl(11, F_DUPFD, 0x00000100) = 256
12931: 0.0001 close(11) = 0
12931: 0.0002 read(256, " # p r a g m a i d e n".., 1024) = 1024
12931: 0.0003 read(256, " t s t p i _ c".., 1024) = 215
12931: 0.0002 read(256, 0x106957054, 1024) = 0
12931: 0.0001 lseek(256, 0, SEEK_SET) = 0
12931: 0.0002 read(256, " # p r a g m a i d e n".., 1024) = 1024
12931: 0.0003 read(256, " t s t p i _ c".., 1024) = 215
12931: 0.0003 read(256, 0x106957054, 1024) = 0
12931: 0.0001 close(256) = 0
On some systems, like Solaris, standard I/O with FILE only works with file descriptors 0-255 because its implementation of the FILE structure uses an 8-bit integer instead of int. If a program uses a lot of file descriptors, it's useful to reserve file descriptors 3-255 using fnctl(fd, F_DUPFD, 256). Otherwise, functions like fopen(), freopen() and fdopen() will fail if you have 256 files open.
As an aside, they're file descriptors rather than file handles. The latter are a C feature used with fopen and its brethren while descriptors are more UNIXy, for use with open et al.
Interesting. The only reason that comes to mind is that some other piece of code has a specific need for the file descriptor to be 256. I suspect only Oracle would know the bizarre reasons for that. In any case, you're not guaranteed to get 256, you get the file first available file descriptor greater than or equal to that number.
From a bit of investigation (I don't know every little thing about the innards of UNIX off the top of my head), there are attributes that belong to a group of duplicated descriptors such as file position and access mode. There are other attributes that belong to a single file descriptor, even when duplicated, such as the close-on-exec flag in GNULib.
Doing a duplicate (either with dup, dup2 or your fcntl) could be a way to create two descriptors, one with different file descriptor attributes, but I can't see that being the case in your question since the first descriptor is closed anyway. As you say, why not just use the low descriptor?
Interestingly enough, if you google for netconfig f_dupfd, you will see similar traces where the fcntl fails and it continues to read that file with the low descriptor so my thoughts on the matter are that this is an attempt to preserve low file descriptors as much as possible. For example:
4327: open("/etc/netconfig", O_RDONLY|O_LARGEFILE) = 4
4327: fcntl(4, F_DUPFD, 0x00000100) Err#22 EINVAL
4327: read(4, " # p r a g m a i d e n".., 1024) = 1024
4327: read(4, " t s t p i _ c".., 1024) = 215
4327: read(4, 0x00296B80, 1024) = 0
4327: lseek(4, 0, SEEK_SET) = 0
4327: read(4, " # p r a g m a i d e n".., 1024) = 1024
4327: read(4, " t s t p i _ c".., 1024) = 215
4327: read(4, 0x00296B80, 1024) = 0
4327: close(4) = 0
Maybe the software has a byte array of file descriptors somewhere that's limited so it attempts to move other files above the 255-limit.
But really, that's just guesswork on my part (although I'd like to think it's relatively intelligent guesswork). Also keep in mind that it may not be Oracle itself doing this. The netconfig stuff is used in a lot of places so it may be some underlying library doing that, especially in light of the fact that most of the afore-mentioned web hits weren't Oracle-specific (ftp, remsh and so on).
Here is another example when a technique of reserving low-numbered file descriptors is needed.
Assume that a process opens a large number of file descriptor e.g. it accepts more than 1024 simultaneous socket connections. At the same time the process also uses third party library that opens socket connections and uses select() to see if sockets are ready for reading or writing. Additionally the third party library was compiled with __FD_SETSIZE set to 1024 (default value).
If the library opens a socket when all file descriptors below 1024 are in use then it will get a descriptor that select() and associated FD_* macros can not cope with. This will result in process crashing or undefined behaviour.