Im using matlab TCP connection to read the values of a stream data come from external device, the problem is when I used data=fscanf(t , "%d") where t is a TCP object, it reads a different number of values at each time, but I don't want it to do this, I want to read a 46 values at the same time.
Anyone have an idea?
To receive fixed size chunks of 46 values, use the size parameter of fscanf to receive not more than 46 values. Now this receiving has to be triggered only if enough data is available. Best practice is to use the BytesAvailableFcn. If configured with the correct size, it is triggered every time 46 values are available.
Related
how can I transfer large data without splitting. Am using tcp socket. Its for a game. I cant use udp and there might be 1200 values in an array. Am sending array in json format. But the server receiving it like splitted.
Also is there any option to send http request like tcp? I need the response in order. Also it should be faster.
Thanks,
You can't.
HTTP may chunk it
TCP will segment it
IP will packetize it
routers will fragment it ...
and TCP will reassemble it all at the other end.
There isn't a problem here to solve.
You do not have much control over splitting packets/datagrams. The network decides about this.
In the case of IP, you have the DF (don't fragment) flag, but I doubt it will be of much help here. If you are communicating over Ethernet, then 1200 element array may not fit into an Ethernet frame (payload size is up to the MTU of 1500 octets).
Why does your application depend on the fact that the whole data must arrive in a single unit, and not in a single connection (comprised potentially of multiple units)?
how can I transfer large data without splitting.
I'm interpreting the above to be roughly equivalent to "how can I transfer my data across a TCP connection using as few TCP packets as possible". As others have noted, there is no way to guarantee that your data will be placed into a single TCP packet -- but you can do some things to make it more likely. Here are some things I would do:
Keep a single TCP connection open. (HTTP traditionally opens a separate TCP connection for each request, but for low-latency you can't afford to do that. Instead you need to open a single TCP connection, keep it open, and continue sending/receiving data on it for as long as necessary).
Reduce the amount of data you need to send. (i.e. are there things that you are sending that the receiving program already knows? If so, don't send them)
Reduce the number of bytes you need to send. (The easiest way to do this is to zlib-compress your message-data before you send it, and have the receiving program decompress the message after receiving it. This can give you a size-reduction of 50-90%, depending on the content of your data)
Turn off Nagle's algorithm on your TCP socket. That will reduce latency by 200mS and discourage the TCP stack from playing unnecessary games with your data.
Send each data packet with a single send() call (if that means manually copying all of the data items into a separate memory buffer before calling send(), then so be it).
Note that even after you do all of the above, the TCP layer will still sometimes spread your messages across multiple packets, etc -- that's just the way TCP works. And even if your local TCP stack never did that, the receiving computer's TCP stack would still sometimes merge the data from consecutive TCP packets together inside its receive buffer. So the receiving program is always going to "receive it like splitted" sometimes, because TCP is a stream-based protocol and does not maintain message boundaries. (If you want message boundaries, you'll have to do your own framing -- the easiest way is usually to send a fixed-size (e.g. 1, 2, or 4-byte) integer byte-count field before each message, so the receiver knows how many bytes it needs to read in before it has a full message to parse)
Consider the idea that the issue may be else where or that you may be sending too much unnecessary data. In example with PHP there is the isset() function. If you're creating an internet based turn based game you don't (need to send all 1,200 variables back and forth every single time. Just send what changed and when the other player receives that data only change the variables are are set.
I used a ts analyzer for a .ts file i have with mpeg-2 codec and i found out that it splits in 7311 packets.
I m trying to find this through matlab by using fopen to open the ts file in binary and fread to read the file but all i get is a column with a huge collection of numbers(way above the number of packets). Does anyone know how can i determine which of these data are the packets? Or if someone knows another way to find the packets would help me a lot.
Thank you in advance
From some quick googling, the MPEG-2 transport stream ('ts') format consists of packets 188-bytes in length, each having a 4-byte header followed by a 184-byte payload. Essentially, you can count the number of packets by counting the number of headers you find - but beware that, if you are only interested in counting the number of, e.g., video packets in the stream, then you will need some deeper analysis of the headers, because the stream may contain any number of interleaved "elementary streams" (which can be video, audio, or arbitrary data). Each elementary packet type in the stream is denoted by a unique "PID" which is contained in the header.
Aside from the above, you will also have to handle synchronisation - each header begins with the "synchronisation byte", which has a value 0x47 (or 01000111 in binary). According to this resource, decoders begin by looking for this synchronisation byte; once they find one, they may have found a packet header. To make sure, they try to find three consecutive synchronisation bytes (188 bytes apart in the stream); if three are found, synchronisation can occur and the packet boundaries may from then on be assumed at 188-byte intervals. Note, however, that the first byte of each assumed header should be checked to see if it is a synchronisation byte - if it is not, then this is called "sync loss" and the syncrhonisation process must start again.
Once you have some code to syncrhonise to a stream, it should be fairly easy to extract the PIDs from the header of each packet and count the number of packets associated with each unique PID you find. You should probably also check the first bit after the synchronisation byte as, if set to 1, this indicates a transport error, and the packet's payload is invalid. Detailed information on the format of packet headers can be found here.
I am currently writing code to transfer data to a remote vendor. The transfer will take place over a TCP socket. The problem I have is the data is variable length and there are no framing or size markers. Sending the data is no problem, but I am unsure of the best way to handle the returned data.
The data is comprised of distinct "messages" but they do not have a fixed size. Each message has an 8 or 16 byte bitmap that indicates what components are included in this message. Some components are fixed length and some are variable. Each variable length component has a size prefix for that portion of the overall message.
When I first open the socket I will send over messages and each one should receive a response. When I begin reading data I should be at the start of a message. I will need to interpret the bitmap to know what message fields are included. As the data arrives I will have to validate that each field indicated by the bitmap is present and of the correct size.
Once I have read all of the first message, the next one starts. My concern is if the transmission gets cut partway through a message, how can I recover and correctly find the next message start?
I will have to simulate a connection failure and my code needs to automatically retry a set number of times before canceling that message.
I have no control over the code on the remote end and cannot get framing bytes or size prefixes added to the messages.
Best practices, design patterns, or ideas on the best way to handle this are all welcomed.
From a user's point of view, TCP is a stream of data, just like you might receive over a serial port. There are no packets and no markers.
A non-blocking read/recv call will return you what has currently arrived at which point you can parse that. If, while parsing, you run out of data before reaching the end of the message, read/recv more data and continue parsing. Rinse. Repeat. Note that you could get more bytes than needed for a specific message if another has followed on its heels.
A TCP stream will not lose or re-order bytes. A message will not get truncated unless the connection gets broken or the sender has a bug (e.g. was only able to write/send part and then never tried to write/send the rest). You cannot continue a TCP stream that is broken. You can only open a new one and start fresh.
A TCP stream cannot be "cut" mid-message and then resumed.
If there is a short enough break in transmission then the O/S at each end will cope, and packets retransmitted as necessary, but that is invisible to the end user application - as far as it's concerned the stream is contiguous.
If the TCP connection does drop completely, both ends will have to re-open the connection. At that point, the transmitting system ought to start over at a new message boundary.
For something like this you would probably have a lot easier of a time using a networking framework (like netty), or a different IO mechansim entirely, like Iteratee IO with Play 2.0.
I'm attempting to read ocean depth values at multiple frequencies which are being broadcast via UDP packets. What I'm doing is telling the logging program to return the depth values to a specific UDP port, then use the DatagramReceivedFcn to run a function when data is received and essentially save that depth.
u1 = udp(remoteip,dataport18,'ByteOrder','littleEndian','LocalPort',dataport18,'DatagramTerminateMode','off');
set(u1,'InputBufferSize',6000);
u1.DatagramReceivedFcn = {#receivedata18};
fopen(u1);
Thus, when data is received on the port specified in 'dataport18', it will run the function receivedata18(). However, I'm trying to read depth data for multiple frequencies, so I create additional UDP objects:
u2 = udp(remoteip,dataport38,'ByteOrder','littleEndian','LocalPort',dataport38,'DatagramTerminateMode','off');
set(u2,'InputBufferSize',6000);
u2.DatagramReceivedFcn = {#receivedata38};
fopen(u2);
What I'm finding though is that only data for u1 (18 kHz) is being saved. My guess is that since both frequencies ping at the same time, they both send their UDP packets and try to evaluate their respective functions at the same time, which Matlab is not capable of doing.
Is this indeed what is going on? If so, is there any way around this issue so that I can concurrently read depth data that is being sent at the same time from two separate UDP packets?
Thanks!
Update
I'm wondering if I would need the Parallel Computing Toolbox in order to perform this. I have a similar program in Python that is performed in essentially the same way, however it has no issues. I'm assuming that it must be that Matlab can't run simultaneous functions without the Parallel Computing Toolbox
Thought I should update this in case anyone cares. It's not really an answer to my question, but what I'm currently doing that's working.
Instead of having the data sent to different UDP port, I simply have them sent to the same port and then read them sequentially. Thus I'm not reading them synchronously, although that doesn't really slow down the operation much at all.
I have been having a heck of a time getting Udp sockets working correctly on Windows Phone 7 (Mango). First I had this problem udp async receive and now that I figured it out, I am seeing a weird behavior where the end of the data I send over the socket is all zero.
At first, I thought there was a weird size cap. All my packets were user 1380 bytes. I was seeing that for some reason, after ~byte 1220 it was all zeros, but according to the socket, I was still receiving all ~1380 bytes. I matched up the sizes with my server application, and I was receiving the correct number of byte. So I printed the bytes out on both sides of the connection and saw this issue with much of the last 200 bytes or so being zero.
So I reduced the size of my packet data to ~1200 bytes, and I was still seeing the issue. I even reduced it to 1000 bytes and still!
Any ideas?
Update - I have done some testing, and it seems that the last 144 bytes are FUBAR. Sometimes they are zero, sometimes they are garbage. Think this is a bug?
You need to check how many bytes were transferred in the async operation. Check SocketAsyncEventArgs.BytesTransferred to see how many bytes in the buffer are actually valid.
Sorry, I had a bug in my code where I was using an array over, overwriting my own data.