I'm running some software that sniffs local packets, then encapsulates the full, raw packet data in a TCP packet and sends it to the server.
What I'd like to do is have the server receive the packet data and put it into a PCAP file for download by anyone who is connected to the server.
Basically what I need is some inspired information regarding the PCAP file format and how to make entries into a PCAP file using raw packet data without using a packet sniffer.
Can this be done and if so how?
Impossible to tell without looking at your data and knowing what tool you are using to capture the packet.
It might be as simple as adding a global header.
The global header contains the magic number, GMT offset, timestamp
precision, the maximum length of captured packets (in octets), and the
data link type. This information is followed by zero or more records
of captured packet data.
If you are using tcpdump then the output might already be pcap compatible. If not, just install tcpdump-libpcap and then use that version.
EXAMPLE PCAP STRUCTURE:
SOURCE: https://www.lesliesikos.com/pcap/
For some example code with PCAP headers, check out this Python program:
https://www.codeproject.com/Tips/612847/Generate-a-quick-and-easy-custom-pcap-file-using-P
Related
I have two apps sending tcp packages, both written in python 2. When client sends tcp packets to server too fast, the packets get concatenated. Is there a way to make python recover only last sent package from socket? I will be sending files with it, so I cannot just use some character as packet terminator, because I don't know the content of the file.
TCP uses packets for transmission, but it is not exposed to the application. Instead, the TCP layer may decide how to break the data into packets, even fragments, and how to deliver them. Often, this happens because of the unterlying network topology.
From an application point of view, you should consider a TCP connection as a stream of octets, i.e. your data unit is the byte, not a packet.
If you want to transmit "packets", use a datagram-oriented protocol such as UDP (but beware, there are size limits for such packets, and with UDP you need to take care of retransmissions yourself), or wrap them manually. For example, you could always send the packet length first, then the payload, over TCP. On the other side, read the size first, then you know how many bytes need to follow (beware, you may need to read more than once to get everything, because of fragmentation). Here, TCP will take care of in-order delivery and retransmission, so this is easier.
TCP is a streaming protocol, which doesn't expose individual packets. While reading from stream and getting packets might work in some configurations, it will break with even minor changes to operating system or networking hardware involved.
To resolve the issue, use a higher-level protocol to mark file boundaries. For example, you can prefix the file with its length in octets (bytes). Or, you can switch to a protocol that already handles this kind of stuff, like http.
First you need to know if the packet is combined before it is sent or after. Use wireshark to check it the sender is sending one packet or two. If it is sending one, then your fix is to call flush() after each write. I do not know the answer if the receiver is combining packets after receiving them.
You could change what you are sending. You could send bytes sent, followed by the bytes. Then the other side would know how many bytes to read.
Normally, TCP_NODELAY prevents that. But there are very few situations where you need to switch that on. One of the few valid ones are telnet style applications.
What you need is a protocol on top of the tcp connection. Think of the TCP connection as a pipe. You put things in one end of the pipe and get them out of the other. You cannot just send a file through this without both ends being coordinated. You have recognised you don't know how big it is and where it ends. This is your problem. Protocols take care of this. You don't have a protocol and so what you're writing is never going to be robust.
You say you don't know the length. Get the length of the file and transmit that in a header, followed by the number of bytes.
For example, if the header is a 64bits which is the length, then when you receive your header at the server end, you read the 64bit number as the length and then keep reading until the end of the file which should be the length.
Of course, this is extremely simplistic but that's the basics of it.
In fact, you don't have to design your own protocol. You could go to the internet and use an existing protocol. Such as HTTP.
Assume I have defined my own application layer protocol on top of TCP for Instant Messaging. I have used a packet structure for the messages. As I am using symmetric (AES) and asymmetric (RSA) encryption, I obtain a different
packet size for different message types. Now to my questions.
How to read from a socket that I receive a single application layer packet?
What size should I specify?
Thanks in advance.
I have two approaches in mind.
Read from the TCP stream a fixed amount of bytes that represents the
actual packet size, and finally re-read from the stream the former gathered size of bytes.
Read the maximal packet size from the stream. Verify the actual size of
obtained bytes and decide so which message type it was.
Now, a more general question. Should I provide metadata like the packet size, encryption method, receiver, sender, etc.? If yes, should I encrypt these meta data as well?
Remember that with TCP, when reading from the network, there is no guarantee about the number of bytes received at that point in time. That is, a client might send a full packet in its write(), but that does not mean that your read() will receive the same number of bytes. Thus your code will always need to read some number of bytes from the network, then verify (based on the accumulated data) that you have received the necessary number of bytes, and then you can verify the packet (type, contents, etc) from there.
Some applications use state machine encoders/decoders and fixed size buffers for reading/writing their network data; other applications dynamically allocate buffers large enough for the "full packet", then continue reading bytes from the network until the "full packet" buffer is full. Which approach you take depends on your application. Thus the size you use for reading is not as important as how your code ensures that it has received a full packet.
As for whether you should encrypt additional metadata, that depends very much on your threat model (i.e. what threats your protocol wants to guard against, what assurances your protocol needs to provide to its clients/users). There's no easy way to answer that question without more context/details.
Hope this helps!
I used a ts analyzer for a .ts file i have with mpeg-2 codec and i found out that it splits in 7311 packets.
I m trying to find this through matlab by using fopen to open the ts file in binary and fread to read the file but all i get is a column with a huge collection of numbers(way above the number of packets). Does anyone know how can i determine which of these data are the packets? Or if someone knows another way to find the packets would help me a lot.
Thank you in advance
From some quick googling, the MPEG-2 transport stream ('ts') format consists of packets 188-bytes in length, each having a 4-byte header followed by a 184-byte payload. Essentially, you can count the number of packets by counting the number of headers you find - but beware that, if you are only interested in counting the number of, e.g., video packets in the stream, then you will need some deeper analysis of the headers, because the stream may contain any number of interleaved "elementary streams" (which can be video, audio, or arbitrary data). Each elementary packet type in the stream is denoted by a unique "PID" which is contained in the header.
Aside from the above, you will also have to handle synchronisation - each header begins with the "synchronisation byte", which has a value 0x47 (or 01000111 in binary). According to this resource, decoders begin by looking for this synchronisation byte; once they find one, they may have found a packet header. To make sure, they try to find three consecutive synchronisation bytes (188 bytes apart in the stream); if three are found, synchronisation can occur and the packet boundaries may from then on be assumed at 188-byte intervals. Note, however, that the first byte of each assumed header should be checked to see if it is a synchronisation byte - if it is not, then this is called "sync loss" and the syncrhonisation process must start again.
Once you have some code to syncrhonise to a stream, it should be fairly easy to extract the PIDs from the header of each packet and count the number of packets associated with each unique PID you find. You should probably also check the first bit after the synchronisation byte as, if set to 1, this indicates a transport error, and the packet's payload is invalid. Detailed information on the format of packet headers can be found here.
I know that it is possible that multiple packets would be stacked to the buffer to be read from and that a long packet might require a loop of multiple send attempts to be fully sent. But I have a question about packaging in these cases:
If I call recv (or any alternative (low-level) function) when there are multiple packets awaiting to be read, would it return them all stacked into my buffer or only one of them (or part of the first one if my buffer is insufficient)?
If I send a long packet which requires multiple iterations to be sent fully, does it count as a single packet or multiple packets? It's basically a question whether it marks that the package sent is not full?
These questions came to my mind when I thought about web sockets packaging. Special characters are used to mark the beginning and end of a packet which sorta leads to a conclusion that it's not possible to separate multiple packages.
P.S. All the questions are about TCP/IP but you are welcomed to share information (answers) about UDP as well.
TCP sockets are stream based. The order is guaranteed but the number of bytes you receive with each recv/read could be any chunk of the pending bytes from the sender. You can layer a message based transport on top of TCP by adding framing information to indicate the way that the payload should be chunked into messages. This is what WebSockets does. Each WebSocket message/frame starts with at least 2 bytes of header information which contains the length of the payload to follow. This allows the receiver to wait for and re-assemble complete messages.
For example, libraries/interfaces that implement the standard Websocket API or a similar API (such as a browser), the onmessage event will fire once for each message received and the data attribute of the event will contain the entire message.
Note that in the older Hixie version of WebSockets, each frame was started with '\x00' and terminated with '\xff'. The current standardized IETF 6455 (HyBi) version of the protocol uses the header information that contains the length which allows much easier processing of the frames (but note that both the old and new are still message based and have basically the same API).
TCP connection provides for stream of bytes, so treat it as such. No application message boundaries are preserved - one send can correspond to multiple receives and the other way around. You need loops on both sides.
UDP, on the other hand, is datagram (i.e. message) based. Here one read will always dequeue single datagram (unless you mess with low-level flags on the socket). Event if your application buffer is smaller then the pending datagram and you read only a part of it, the rest of it is lost. The way around it is to limit the size of datagrams you send to something bellow the normal MTU of 1500 (less IP and UDP headers, so actually 1472).
Is there any tool existing or any shortcut that can be used to extract print job payload from IPP packets(captured via wireshark utility)?
Write now I'm manually inspecting/dissecting the packets byte per byte which is kind of excruciating especially with requests which spans several packet exchanges (HTTP Continue).
Thanks