I used a ts analyzer for a .ts file i have with mpeg-2 codec and i found out that it splits in 7311 packets.
I m trying to find this through matlab by using fopen to open the ts file in binary and fread to read the file but all i get is a column with a huge collection of numbers(way above the number of packets). Does anyone know how can i determine which of these data are the packets? Or if someone knows another way to find the packets would help me a lot.
Thank you in advance
From some quick googling, the MPEG-2 transport stream ('ts') format consists of packets 188-bytes in length, each having a 4-byte header followed by a 184-byte payload. Essentially, you can count the number of packets by counting the number of headers you find - but beware that, if you are only interested in counting the number of, e.g., video packets in the stream, then you will need some deeper analysis of the headers, because the stream may contain any number of interleaved "elementary streams" (which can be video, audio, or arbitrary data). Each elementary packet type in the stream is denoted by a unique "PID" which is contained in the header.
Aside from the above, you will also have to handle synchronisation - each header begins with the "synchronisation byte", which has a value 0x47 (or 01000111 in binary). According to this resource, decoders begin by looking for this synchronisation byte; once they find one, they may have found a packet header. To make sure, they try to find three consecutive synchronisation bytes (188 bytes apart in the stream); if three are found, synchronisation can occur and the packet boundaries may from then on be assumed at 188-byte intervals. Note, however, that the first byte of each assumed header should be checked to see if it is a synchronisation byte - if it is not, then this is called "sync loss" and the syncrhonisation process must start again.
Once you have some code to syncrhonise to a stream, it should be fairly easy to extract the PIDs from the header of each packet and count the number of packets associated with each unique PID you find. You should probably also check the first bit after the synchronisation byte as, if set to 1, this indicates a transport error, and the packet's payload is invalid. Detailed information on the format of packet headers can be found here.
Related
I have a question regarding the TCPStream package in Rust. I want to read data from a server. The problem is that it is not guaranteed that the data is sent in one TCP package.
And here comes my question:
Is the read message capable of reading more than one package, or do I have to call it more than one? Is there any "best practice"?
From the user space TCP packets are not visible and their boundaries don't matter. Instead user space reads only a byte stream and writes only to a byte stream. Packetizing is done at a lower level in a way to be optimal for latency and bandwidth. It might well happen that multiple write from user space end up in the same packet and it might also happen that a single write will result in multiple packets. And the same is true with read: it might get part of a packet, it might get the payload taken from multiple consecutive packets ...
Any packet boundaries from the underlying transport are no longer visible from user space. Thus protocols using TCP must implement their own message semantic on top of the byte stream.
All of this is not specific to Rust, but applies to other programming languages too.
Assume I have defined my own application layer protocol on top of TCP for Instant Messaging. I have used a packet structure for the messages. As I am using symmetric (AES) and asymmetric (RSA) encryption, I obtain a different
packet size for different message types. Now to my questions.
How to read from a socket that I receive a single application layer packet?
What size should I specify?
Thanks in advance.
I have two approaches in mind.
Read from the TCP stream a fixed amount of bytes that represents the
actual packet size, and finally re-read from the stream the former gathered size of bytes.
Read the maximal packet size from the stream. Verify the actual size of
obtained bytes and decide so which message type it was.
Now, a more general question. Should I provide metadata like the packet size, encryption method, receiver, sender, etc.? If yes, should I encrypt these meta data as well?
Remember that with TCP, when reading from the network, there is no guarantee about the number of bytes received at that point in time. That is, a client might send a full packet in its write(), but that does not mean that your read() will receive the same number of bytes. Thus your code will always need to read some number of bytes from the network, then verify (based on the accumulated data) that you have received the necessary number of bytes, and then you can verify the packet (type, contents, etc) from there.
Some applications use state machine encoders/decoders and fixed size buffers for reading/writing their network data; other applications dynamically allocate buffers large enough for the "full packet", then continue reading bytes from the network until the "full packet" buffer is full. Which approach you take depends on your application. Thus the size you use for reading is not as important as how your code ensures that it has received a full packet.
As for whether you should encrypt additional metadata, that depends very much on your threat model (i.e. what threats your protocol wants to guard against, what assurances your protocol needs to provide to its clients/users). There's no easy way to answer that question without more context/details.
Hope this helps!
So far I thought they are the same as bytes are made of bits and that both side needs to know byte size and endiannes of the other side and transform stream accordingly. However Wikipedia says that byte stream != bit stream (https://en.wikipedia.org/wiki/Byte_stream ) and that bit streams are specifically used in video coding (https://en.wikipedia.org/wiki/Bitstream_format). In this RFC https://www.rfc-editor.org/rfc/rfc107 they discuss these 2 things and describe Two separate kinds of inefficiency arose from bit streams.. My questions are:
what's the real difference between byte stream and bit stream?
how bit stream works if it's different from byte stream? How does a receiving side know how many bits to process at a given time?
why is bit stream better than byte stream in some cases?
This is a pretty broad question, I'll have to give the 10,000 feet view. Bit streams are common in two distinct usages:
very low-level, it is the fundamental way that lots of hardware operates. Best examples are the data stream that comes off a hard disk or a optical disk or the data sent across a transmission line, like a USB cable or the coax cable or telephone line through which you received this post. The RFC you found applies here.
high-level, they are common in data compression, a variable number of bits per token allows packing data tighter. Huffman coding is the most basic way to compress. The video encoding subjects you found applies here.
what's the real difference between byte stream and bit stream?
Byte streams are highly compatible with computers which are byte-oriented devices and the ones you'll almost always encounter in programming. Bit streams are much more low-level, only system integration engineers ever worry about them. While the payload of a bit stream is often the bytes that a computer is interested in, more overhead is typically required to ensure that the receiver can properly interpret the data. There are usually a lot more bits than necessary to encode the bytes in the data. Extra bits are needed to ensure that the receiver is properly synchronized and can detect and perhaps correct bit errors. NRZ encoding is very common.
The RFC is quite archeological, in 1971 they were still hammering out the basics of getting computers to talk to each other. Back then they were still close to the transmission line behavior, a bit stream, and many computers did not yet agree on 8 bits in a byte. They are fretting over the cost of converting bits to local bytes on very anemic hardware and the need to pack as many bits into a message as possible.
How does a receiving side know how many bits to process at a given time?
The protocol determines that, like that RFC does. In the case of a variable length bit encoding it is bit values themselves that determine it, like Huffman coding does.
why is bit stream better than byte stream in some cases?
Covered already I think, because it is better match for its purpose. Either because the hardware is bit-oriented or because variable bit-length coding is useful.
A bit is a single 1 or 0 in computer code, also known as a binary digit.
The most common use for the bit stream is with the transmission control protocol, or TCP. This series of guidelines tells computers how to send and receive messages between each other. The World Wide Web and e-mail services, among others, rely on TCP guidelines to send information in an orderly fashion. Sending through the bit stream ensures the pieces arrive in the proper order and the message isn't corrupted during delivery, which could make it unreadable.So a bit stream sends one bit after another.
Eight bits make up a byte, and the byte stream transmits these eight-bit packets from computer to computer.
The packets are decoded upon arrival so the computer can interpret them.Thus a byte stream is a special case of bits sent together as a group in sequential order.For a byte stream to be most effective, it flows through a dedicated and reliable path sometimes referred to as a pipe, or pipeline.
When it comes to sending a byte stream over a computer network, a reliable bi-directional transport layer protocol, such as the transmission control protocol (TCP) used on the Internet, is required. These are referred to as a byte stream protocol. Other serial data protocols used with certain types of hardware components, such as the universal asynchronous receiver/transmitter (UART) technique, is a serial data channel that also uses a byte stream for communication. In this case, the byte, or character, is packaged up in a frame on the transmitting end, where an extra starting bit and some optional checking bits are attached and then separated back out of the frame on the receiving end. This technique is sometimes referred to as a byte-oriented protocol.
Taking a general life example,suppose you have a lot of match sticks to send.Then you could send them one stick after the other,one at a
time.. or you could pack a few of them in a match box and send them
together ,one matchbox after the other in sequence.the first is like
bitstream and the latter like bytestream.
Thus it all depends on what the hardware wants or is best suited for..If your hand is small and you cant accept matchboxes but you still want matchsticks then you take them one at a time or else take the box.Also byte streams are better in a sense that every bit does not need to be checked and data can be sent in batches of 8,.if any of it fails the entire 8bits can be re sent.
To add to the other good answers here:
A byte stream is a type of bit stream. A byte stream describes the bits as meaningful "packages" that are 8 bits wide.
Certain (especially low-level) streams may be agnostic of meaning in each 8 bit sequence. It would be a poor description to call these "byte streams"
Similar to how every Honda Civic is a car, but not every car is a Honda Civic...
So I'm reading trying to find more about sockets and I found that the documentation kind of repeats itself without going in any further detail in regards to the way sockets send data:
So what does the {packet, N} do when specifying this in the options (when opening a socket) ? Does it include a header of N bytes before the data or does it break the message into N packets? Or does it include a header of N to all the packets that the message ends up being broken into?
I was reading Joe Armstrong's Software for a concurrent world and I found this paragraph:
The word packet refers to the length of an application request or response message, not to the physical packet seen on the wire.
I can't get my head round the meaning of this. What is meant by the packet seen on the wire.
I tried to look into the documentation and I found little referring to what the option does. The brief explanation that I found is that it prepends the message with a N header however I also found this comment in a code written as an example:
%% Usually, it's a good idea to give up in case of a
%% send timeout, as you never know how much actually
%% reached the server, maybe only a packet header?!
According to this, then, the message gets broken into pieces (a random number I presume?) and each gets sent with a N byte header.
My question is how does the {header, N} option affect the way data is sent.
The excerpt from Joe's book you quote refers to the fact that applications are typically blissfully unaware of how networks arrange data for transmission. Depending on the network type, configuration, and protocols in use, the blocks of data sent over it will vary in size and will be framed with different metadata. Applications typically don't see raw packets, framing information, or the fact that sometimes raw packets are retransmitted due to problems that cause them to be dropped or corrupted.
Your question mentions two options: {packet, N} and {header, N}. These are quite different from each other.
The {packet, N} option allows N to be 1, 2, or 4. It attaches a header of N bytes to the front the message. The header specifies, in network order, the length of the message, i.e. the number of bytes in the message.
If you send a message consisting of X bytes, Erlang will prepend to the data N bytes containing the network order value of X, and send the whole thing. Assuming the receiver has also been configured with the same {packet, N} option, it will read the N-byte header to determine how many bytes to expect, wait to receive that many bytes, and then deliver those bytes, without the length header, to the receiving application. How the underlying networking software and hardware breaks the data into chunks for transmission across the network is a separate matter hidden from your application.
The {header, Size} option delivers the message to the receiver as a list of Size bytes followed by the remainder of the data as a binary. This option makes sense only when the binary option is in effect for the socket.
I am currently writing code to transfer data to a remote vendor. The transfer will take place over a TCP socket. The problem I have is the data is variable length and there are no framing or size markers. Sending the data is no problem, but I am unsure of the best way to handle the returned data.
The data is comprised of distinct "messages" but they do not have a fixed size. Each message has an 8 or 16 byte bitmap that indicates what components are included in this message. Some components are fixed length and some are variable. Each variable length component has a size prefix for that portion of the overall message.
When I first open the socket I will send over messages and each one should receive a response. When I begin reading data I should be at the start of a message. I will need to interpret the bitmap to know what message fields are included. As the data arrives I will have to validate that each field indicated by the bitmap is present and of the correct size.
Once I have read all of the first message, the next one starts. My concern is if the transmission gets cut partway through a message, how can I recover and correctly find the next message start?
I will have to simulate a connection failure and my code needs to automatically retry a set number of times before canceling that message.
I have no control over the code on the remote end and cannot get framing bytes or size prefixes added to the messages.
Best practices, design patterns, or ideas on the best way to handle this are all welcomed.
From a user's point of view, TCP is a stream of data, just like you might receive over a serial port. There are no packets and no markers.
A non-blocking read/recv call will return you what has currently arrived at which point you can parse that. If, while parsing, you run out of data before reaching the end of the message, read/recv more data and continue parsing. Rinse. Repeat. Note that you could get more bytes than needed for a specific message if another has followed on its heels.
A TCP stream will not lose or re-order bytes. A message will not get truncated unless the connection gets broken or the sender has a bug (e.g. was only able to write/send part and then never tried to write/send the rest). You cannot continue a TCP stream that is broken. You can only open a new one and start fresh.
A TCP stream cannot be "cut" mid-message and then resumed.
If there is a short enough break in transmission then the O/S at each end will cope, and packets retransmitted as necessary, but that is invisible to the end user application - as far as it's concerned the stream is contiguous.
If the TCP connection does drop completely, both ends will have to re-open the connection. At that point, the transmitting system ought to start over at a new message boundary.
For something like this you would probably have a lot easier of a time using a networking framework (like netty), or a different IO mechansim entirely, like Iteratee IO with Play 2.0.