I am looking to utilize the labSend, labReceive functionality with spmd in MATLAB to execute perform the following:
Lab1, run a global optimization routine and pass an intermediate result to Lab2
Lab2, wait for the intermediate result from Lab1 (using labProbe), once received use this result and begin a new optimization routine.
Lab3,4,..., n wait for the previous result from Lab_n-1, once received use this result and begin a new optimization routine.
Problem:
Warning: An incoming message was discarded from lab 1 (tag: 1)
Warning: An incoming message was discarded from lab 1 (tag: 1)
Warning: An incoming message was discarded from lab 1 (tag: 1)
Warning: An incoming message was discarded from lab 1 (tag: 1)
Warning: An incoming message was discarded from lab 1 (tag: 1)
Data from labSend:
0.4907 0.3328 0.3625 0.5843 0.3159 0.5065 0.5100 0.4984 0.3336 0.5055
0.5216 0.5268 0.5002 0.4828 0.4907 0.3328 0.3625 0.5843 0.3159 0.5065
0.5100 0.4984 0.3336 0.5055 0.5216 0.5268 0.5002 0.4828 0.5010
which is in order with 0.4907 being the first message sent via labSend.
Last value received from labReceive:
0.5055
meaning the last 5 messages from labSend were ignored.
Now, the spmd routine is asynchronous as it 1) has to wait for the intermediate result of the previous lab and 2) the optimization routine speeds up as it progresses (searching a smaller domain)
Therefore, the previous labs may send multiple messages before lab_n has the chance to process them (executing something else).
Question:
Is there a way to immediately process (receive) data from Lab1 if I am looking at Lab2 and just store it somewhere? Or is there a way to process only the most recent message? and ignore any queued messages?
Thanks for your help!
Related
I have two perl processes that communicate over System V IPC message Q on Ubuntu.
The receiver runs successfully for a time, receives messages in a poller function like this
sub getCompleteRecord {
while( msgrcv($q, $buff, $size, $msgType, &IPC_NOWAIT) ) {
# assemble record from messages and return
}
# print $! error code
#`here if nothing ready (42) or other error`
After some time I eventually get an error code 22 (EINVAL), which means invalid argument, and then subsequently all calls to msgrcv() fail with 22 and the separate sender process also cannot msgsnd(), again getting EIVAL.
When I restart the processes the queue can again be used.
Any suggestions for reasons, or how to approach diagnosing this?
As noted in the comments the meaning of the error code 22 is that either the msqid ($q) or the buffer size ($size) are incorrect. However, this is all happening in a loop, and those two values never change. I log the values before each call and I see many successes and then suddenly a failure for seemingly the same values.
masterQueue 360448 read buffer size:5000 msgType1
Message received
--- many similar successes, then:
masterQueue 360448 read buffer size:5000 msgType1
read error 22=Invalid argument
And from this point both reader process and writer process fail. If I restart, then everything works for about 30 minutes and then fails again.
I have a DUT were the writes takes 2 clock cycles and reads consume 2 clock cycles before it could actually happen, I use regmodel and tried using inbuilt sequence uvm_reg_bit_bash_seq but it seems that the writes and reads happens at 1 clock cycle delay, could anyone tell what is the effective way to model 2 clock cycle delays and verify this, so that the DUT delays are taken care.
Facing the following error now,
Writing a 1 in bit #0 of register "ral_pgm.DIFF_STAT_CORE1" with initial value 'h0000000000000000 yielded 'h0000000000000000 instead of 'h0000000000000001
I have found one way of doing it, took the existing uvm_reg_single_bit_bash_seq and modified by adding p_sequencer and added 2 clock cycle delays after write and read method calls as per the DUT latency, this helped me in fixing the issue as well added a get call after write method to avoid fetching old value after read operation.
...
`uvm_declare_p_sequencer(s_master_sequencer)
rg.write(status, val, UVM_FRONTDOOR, map, this);
wait_for_clock(2); // write latency
rg.read(status, val, UVM_FRONTDOOR, map, this);
wait_for_clock(2); // read latency
if (status != UVM_IS_OK) begin
`uvm_error("mtm_reg_bit_bash_seq", $sformatf("Status was %s when reading register \"%s\" through map \"%s\".",
status.name(), rg.get_full_name(), map.get_full_name()));
end
val = rg.get(); // newly added method call (get) to fetch value after read
...
task wait_for_clock( int m = 1 );
repeat ( m ) begin
#(posedge p_sequencer.vif.CLK);
end
endtask: wait_for_clock
...
When an out-of-memory error is raised in a parfor, is there any way to kill only one Matlab slave to free some memory instead of having the entire script terminate?
Here is what happens by default when an out-of-memory error occurs in a parfor: the script terminated, as shown in the screenshot below.
I wish there was a way to just kill one slave (i.e. removing a worker from parpool) or stop using it to release as much memory as possible from it:
If you get a out of memory in the master process there is no chance to fix this. For out of memory on the slave, this should do it:
The simple idea of the code: Restart the parfor again and again with the missing data until you get all results. If one iteration fails, a flag (file) is written which let's all iterations throw an error as soon as the first error occurred. This way we get "out of the loop" without wasting time producing other out of memory.
%Your intended iterator
iterator=1:10;
%flags which indicate what succeeded
succeeded=false(size(iterator));
%result array
result=nan(size(iterator));
FLAG='ANY_WORKER_CRASHED';
while ~all(succeeded)
fprintf('Another try\n')
%determine which iterations should be done
todo=iterator(~succeeded);
%initialize array for the remaining results
partresult=nan(size(todo));
%initialize flags which indicate which iterations succeeded (we can not
%throw erros, it throws aray results)
partsucceeded=false(size(todo));
%flag indicates that any worker crashed. Have to use file based
%solution, don't know a better one. #'
delete(FLAG);
try
parfor falseindex=1:sum(~succeeded)
realindex=todo(falseindex);
try
% The flag is used to let all other workers jump out of the
% loop as soon as one calculation has crashed.
if exist(FLAG,'file')
error('some other worker crashed');
end
% insert your code here
%dummy code which randomly trowsexpection
if rand<.5
error('hit out of memory')
end
partresult(falseindex)=realindex*2
% End of user code
partsucceeded(falseindex)=true;
fprintf('trying to run %d and succeeded\n',realindex)
catch ME
% catch errors within workers to preserve work
partresult(falseindex)=nan
partsucceeded(falseindex)=false;
fprintf('trying to run %d but it failed\n',realindex)
fclose(fopen(FLAG,'w'));
end
end
catch
%reduce poolsize by 1
newsize = matlabpool('size')-1;
matlabpool close
matlabpool(newsize)
end
%put the result of the current iteration into the full result
result(~succeeded)=partresult;
succeeded(~succeeded)=partsucceeded;
end
After quite bit of research, and a lot of trial and error, I think I may have a decent, compact answer. What you're going to do is:
Declare some max memory value. You can set it dynamically using the MATLAB function memory, but I like to set it directly.
Call memory inside your parfor loop, which returns the memory information for that particular worker.
If the memory used by the worker exceeds the threshold, cancel the task that worker was working on. Now, here it get's a bit tricky. Depending on the way you're using parfor, you'll either need to delete or cancel either the task or worker. I've verified that it works with the code below when there is one task per worker, on a remote cluster.
Insert the following code at the beginning of your parfor contents. Tweak as necessary.
memLimit = 280000000; %// This doesn't have to be in parfor. Everything else does.
memData = memory;
if memData.MemUsedMATLAB > memLimit
task = getCurrentTask();
cancel(task);
end
Enjoy! (Fun question, by the way.)
One other option to consider is that since R2013b, you can open a parallel pool with 'SpmdEnabled' set to false - this allows MATLAB worker processes to die without the whole pool being shut down - see the doc here http://www.mathworks.co.uk/help/distcomp/parpool.html . Of course, you still need to arrange somehow to shutdown the workers.
i'm trying to do some serial communication with matlab.
Settings are:
9600 baude
8 databits
0 paritybit
1 stopbit
main .m-file
SYN = 1;
FIN = 2;
ACK = 4;
NAK = 8;
PWM = 16;
MAG = 32;
txByte = 0;
rxByte = 0;
serobj = serial('COM3');
set(serobj,'Baudrate', 9600);
set(serobj,'Parity', 'none');
set(serobj,'Databits', 8);
set(serobj,'StopBits', 1);
set(serobj,'FlowControl', 'none');
set(serobj,'ErrorFcn',#TimeOutErrorFcn);
set(serobj,'Timeout',1);
fopen(serobj);
% Start the handshake
fwrite(serobj, SYN, 'uint8');
% wait for answer
rxByte = fread(serobj,1,'uint8');
if(isempty(rxByte))
rxByte = 255;
end
% what is the answer?
switch (rxByte(1))
case ACK+SYN
fwrite(serobj, ACK, 'uint8');
case NAK
fclose(serobj);
otherwise
fwrite(serobj, NAK, 'uint8');
fclose(serobj);
end
while(strcmp(get(serobj, 'Status'),'open'))
% Do some communication
end
callback function as .m-file
function TimeOutErrorFcn(obj, event)
%TIMEOUTERRORFCN Summary of this function goes here
% Detailed explanation goes here
disp('Error');
% close the connection here, send FIN and do fclose(serobj)
end
After the timeout is exceeded, the TimeOutErrorFcn is not called! All i get is the following output on the command promt:
Warning: The specified amount of data was not returned within the Timeout period.
Mathsworks help says:
Timeout
Waiting time to complete a read or write operation
You configure Timeout to be the maximum time (in seconds) to wait to complete a read or write operation.
If a time-out occurs, the read or write operation aborts. Additionally, if a time-out occurs during an asynchronous read or write operation, then:
An error event is generated.
The callback function specified for ErrorFcn is executed.
ErrorFcn
Specify the callback function to execute when an error event occurs
expand all in page
Description
You configure ErrorFcn to execute a callback function when an error event occurs.
Note An error event is generated only for asynchronous read and write operations.
An error event is generated when a time-out occurs. A time-out occurs if a read or write operation does not successfully complete within the time specified by the Timeout property. An error event is not generated for configuration errors such as setting an invalid property value.
If the RecordStatus property value is on, and an error event occurs, the record file records this information:
The event type as Error
The error message
The time the event occurred using the format day-month-year hour:minute:second:millisecond
To learn how to create a callback function, see Creating and Executing Callback Functions.
So, I really don't see my mistake.
You aren't doing an asynchronous read operation, so therefore an error event doesn't occur. Instead, it is a warning. To force your error function to run, you need to tell Matlab to label that warning as an error. Try adding this to the top of your code:
warning('error', 'instrument:fread:unsuccessfulRead');
If there are other warnings that you want to flag as errors, you can get the command by issuing "verbose on".
I'm writing a little VOIP app like Skype, which works quite good right now, but I've run into a very strange problem.
In one thread, I'm calling within a while(true) loop the winsock recv() function twice per run to get data from a socket.
The first call gets 2 bytes which will be casted into a (short) while the second call gets the rest of the message which looks like:
Complete Message: [2 Byte Header | Message, length determined by the 2Byte Header]
These packets are round about 49/sec which will be round about 3000bytes/sec.
The content of these packets is audio-data that gets converted into wave.
With ioctlsocket() I determine wether there is some data on the socket or not at each "message" I receive (2byte+data). If there's something on the socket right after I received a message within the while(true) loop of the thread, the message will be received, but thrown away to work against upstacking latency.
This concept works very well, but here's the problem:
While my VOIP program is running and when I parallely download (e.g. via browser) a file, there always gets too much data stacked on the socket, because while downloading, the recv() loop seems actually to slow down. This happens in every download/upload situation besides the actual voip up/download.
I don't know where this behaviour comes from, but when I actually cancel every up/download besides the voip traffic of my application, my apps works again perfectly.
If the program runs perfectly, the ioctlsocket() function writes 0 into the bytesLeft var, defined within the class where the receive function comes from.
Does somebody know where this comes from? I'll attach my receive function down below:
std::string D_SOCKETS::receive_message(){
recv(ClientSocket,(char*)&val,sizeof(val),MSG_WAITALL);
receivedBytes = recv(ClientSocket,buffer,val,MSG_WAITALL);
if (receivedBytes != val){
printf("SHORT: %d PAKET: %d ERROR: %d",val,receivedBytes,WSAGetLastError());
exit(128);
}
ioctlsocket(ClientSocket,FIONREAD,&bytesLeft);
cout<<"Bytes left on the Socket:"<<bytesLeft<<endl;
if(bytesLeft>20)
{
// message gets received, but ignored/thrown away to throw away
return std::string();
}
else
return std::string(buffer,receivedBytes);}
There is no need to use ioctlsocket() to discard data. That would indicate a bug in your protocol design. Assuming you are using TCP (you did not say), there should not be any left over data if your 2byte header is always accurate. After reading the 2byte header and then reading the specified number of bytes, the next bytes you receive after that constitute your next message and should not be discarded simply because it exists.
The fact that ioctlsocket() reports more bytes available means that you are receiving messages faster than you are reading them from the socket. Make your reading code run faster, don't throw away good data due to your slowness.
Your reading model is not efficient. Instead of reading 2 bytes, then X bytes, then 2 bytes, and so on, you should instead use a larger buffer to read more raw data from the socket at one time (use ioctlsocket() to know how many bytes are available, and then read at least that many bytes at one time and append them to the end of your buffer), and then parse as many complete messages are in the buffer before then reading more raw data from the socket again. The more data you can read at a time, the faster you can receive data.
To help speed up the code even more, don't process the messages inside the loop directly, either. Do the processing in another thread instead. Have the reading loop put complete messages in a queue and go back to reading, and then have a processing thread pull from the queue whenever messages are available for processing.