I can read UDP packet using
void my_callback(u_char *useless, const struct pcap_pkthdr* pkthdr, const u_char* packet)
I have hexa output of my packet:
08 00 27 E5 B5 3B 52 54 00 12 35 02 08 00 45 00 00 4C 7C E7 00 00 40 11 3C 28 5B BD 59 C6 0A 00 02 0F 00 7B 00 7B 00 38 B7 9D 24 02 03 E8 00 00 04 A8 00 00 07 51 83 BC 03 DC DC C5 CC 47 F1 F1 69 C3 DC C5 CF 37 D2 5F A7 F5 DC C5 CF 38 3C 2D C2 CF DC C5 CF 38 3C 32 0B 9A
I know, that it is NTP packet.
How can I extrath data? Cut ethernet frames, etc..
Thank you for your help.
I am using pcap c++.
If you read pcap you get raw packet from the network device. Several options may be there:
Packet is read from ethernet device
Packet is read from vlan device
Packet is read from some other device
What kind of device is used during pcap defines what protocol header is first in your packet. To know it you can look at link layer type field of global pcap header.
Once you defined first protocol header you need to open protocol specification and find:
Size of header (in your case it looks like regular ethenet header - 14 bytes 08 00 27 E5 B5 3B 52 54 00 12 35 02 08 00)
How to find encapsulated packet type (in your case last 08 00 means IP)
Once you found IP header (45 00 00 4C 7C E7 00 00 40 11 ...) you can determine IP header length:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Version| IHL |Type of Service| Total Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Identification |Flags| Fragment Offset |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Time to Live | Protocol | Header Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Source Address |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Destination Address |
Here you need:
IHL defines sizeof IP header. This is lower 4 bits of first byte of IP header. In your case it is 0x5. This means 5 words or 20 bytes.
Protocol defines what data is encapsulated in IP header. In your case 0x11 (IPPROTO_UDP)
After that you can get UDP header (8 bytes) check ports if you need it and parse NTP header in according to NTP specification.
In your example total shift of the NTP header will be 14+20+8 bytes.
Related
I need some help to understand the avcC atom structure of a particular mp4 sample I am trying to analyze.
Hex dump:
00 00 00 38 61 76 63 43 01 64 00 1F FF E1 00 1C 67 64 00 1F AC D9 80
50 05 BB 01 6A 02 02 02 80 00 00 03 00 80 00 00 1E 07 8C 18 CD 01 00
05 68 E9 7B 2C 8B FD F8 F8 00 00 00 00 13 63 6F 6C 72
This is what I understand from the above:
00 00 00 38 Size of avcC atom
61 76 63 43 avcC signature
01 configurationVersion
64 AVCProfileIndication
00 profile_compatibility
1F AVCLevelIndication
FF 111111b + lengthSizeMinusOne
E1 111b + numOfSequenceParameterSets (in this case, 1 SPS)
00 1C SPS length (in this case, 28 bytes)
67 64 00 1F AC D9 80 50 05 BB 01 6A 02 02 02 80 00 00 03 00 80 00 00 1E 07 8C 18 CD SPS data (28 bytes as per above)
01 numOfPictureParameterSets (in this case, 1 PPS)
00 05 PPS length
This is where the problem begins. Based on the PPS length given by the previous bytes, the next 5 bytes should be the PPS data: 68 E9 7B 2C 8B
However according to the avcC header, the total length of the atom is 56 bytes (0x38), which means that the following 4 bytes should be included: FD F8 F8 00
But the problem is that the PPS length is given as 5 bytes (0x05). So what exactly are these final 4 bytes?
Then follows the header of the colr atom:
00 00 00 13 size of colr atom
63 6F 6C 72 colr signature
Which I have checked and is indeed 19 bytes in length (0x13).
The problem is with the avcC atom and with that particular mp4 sample I am analyzing (I've checked other samples too and they didn't have this peculiarity).
You can find the sample here.
EDIT
mp4info tool from the bento4 suite reports the following as the avcC atom's size: 8+48
And mp4dump reports:
AVC SPS: [6764001facd9805005bb016a02020280000003008000001e078c18cd]
AVC PPS: [68e97b2c8b]
So it correctly reports the total size of the atom as 56 bytes (0x38) based on what is found in the avcC header, but the SPS/PPS data are analyzed the same way as above. I still don't understand what the final 4 bytes are or where do they belong.
I dind't get any answer but fortunately a bit more careful reading of ISO 14496-15 solved this issue:
if( profile_idc == 100 || profile_idc == 110 ||
profile_idc == 122 || profile_idc == 144 )
{
bit(6) reserved = ‘111111’b;
unsigned int(2) chroma_format;
bit(5) reserved = ‘11111’b;
unsigned int(3) bit_depth_luma_minus8;
bit(5) reserved = ‘11111’b;
unsigned int(3) bit_depth_chroma_minus8;
unsigned int(8) numOfSequenceParameterSetExt;
for (i=0; i< numOfSequenceParameterSetExt; i++) {
unsigned int(16) sequenceParameterSetExtLength;
bit(8*sequenceParameterSetExtLength) sequenceParameterSetExtNALUnit;
}
}
Apparently a sequence of 4+ bytes may exist at the end of an avcC atom depending on the profile used. In my sample above the profile is 100 (0x64), hence it meets the criteria. So the last 4 bytes are:
FD = bits 111111 are reserved, remaining 01 means chroma subsampling 4:2:0
F8 = bits 11111 are reserved, remaining 000 means luma bit depth is 8
F8 = bits 11111 are reserved, remaining 000 means chroma bit depth is 8
00 = zero SPS extensions
I'm currently working with NFC/NDEF and I'm running into an issue where I'm unable to understand the data coming in. I have a general understanding of the NDEF standard and have looked over the MIFARE datasheet, so I'm able to pick out a few things, but there are a few bytes that are seemingly out of place and are puzzling me.
Here is the hexdump of the data on the tag, collected via nfc-mfultralight r:
00000000 04 02 2f a1 d2 11 5f 81 1d 48 00 00 e1 10 12 00 |../..._..H......|
00000010 01 03 a0 0c 34 03 1b 91 01 05 54 02 65 6e 68 69 |....4.....T.enhi|
00000020 11 01 05 54 02 65 6e 68 69 51 01 05 54 02 65 6e |...T.enhiQ..T.en|
00000030 68 69 fe 00 00 00 00 00 00 00 00 00 00 00 00 00 |hi..............|
I know the first 16 bytes (04 02 2f a1 d2 11 5f 81 1d 48 00 00 e1 10 12 00) are the NFC/MIFARE header (first 9 being the serial number/check bytes, 1 byte for internal, 2 for lock, and then final 4 are OTP bytes.)
Starting at byte 21 I can see the start of a TLV record with the Terminator TLV flag at the end (03 1b ... fe), indicating a record of NDEF type with length 27. This matches the length of the expected NDEF record.
However, I'm confused by bytes 16..20 (01 03 a0 0c 34). What are these?
It appears these are a part of the Lock Control TLV, a part of the NFC Type 2 Tag standard (pages 10-11).
The bytes are laid out as such:
0x01 - Lock Control TLV block name
0x03 - Length is 3 bytes
0xa0 - Encodes the position within the tag the lock area is at, composed of two nibbles:
0b0000 - Higher 4 bits represent the number of pages, while the lower 4 bits are the number of bytes
0b1100 - The number of bits used in the lock area.
0x0c - Indicates size in bits of the lock area
0x34 - Provides number of bytes in a page and the number of bytes each dynamic lock bit is able to lock.
I am using a REINER SCT cyberJack RFID standard card reader and an estonian ID card.
pcsc_scan correctly identifies the card:
$ pcsc_scan
PC/SC device scanner
V 1.5.2 (c) 2001-2017, Ludovic Rousseau <ludovic.rousseau#free.fr>
Using reader plug'n play mechanism
Scanning present readers...
0: REINER SCT cyberJack RFID standard (9084002233) 00 00
Wed Mar 13 14:02:39 2019
Reader 0: REINER SCT cyberJack RFID standard (9084002233) 00 00
Card state: Card inserted,
ATR: 3B DB 96 00 80 B1 FE 45 1F 83 00 12 23 3F 53 65 49 44 0F 90 00 F1
ATR: 3B DB 96 00 80 B1 FE 45 1F 83 00 12 23 3F 53 65 49 44 0F 90 00 F1
+ TS = 3B --> Direct Convention
+ T0 = DB, Y(1): 1101, K: 11 (historical bytes)
TA(1) = 96 --> Fi=512, Di=32, 16 cycles/ETU
250000 bits/s at 4 MHz, fMax for Fi = 5 MHz => 312500 bits/s
TC(1) = 00 --> Extra guard time: 0
TD(1) = 80 --> Y(i+1) = 1000, Protocol T = 0
-----
TD(2) = B1 --> Y(i+1) = 1011, Protocol T = 1
-----
TA(3) = FE --> IFSC: 254
TB(3) = 45 --> Block Waiting Integer: 4 - Character Waiting Integer: 5
TD(3) = 1F --> Y(i+1) = 0001, Protocol T = 15 - Global interface bytes following
-----
TA(4) = 83 --> Clock stop: state H - Class accepted by the card: (3G) A 5V B 3V
+ Historical bytes: 00 12 23 3F 53 65 49 44 0F 90 00
Category indicator byte: 00 (compact TLV data object)
Tag: 1, len: 2 (country code, ISO 3166-1)
Country code: 23 3F
Tag: 5, len: 3 (card issuer's data)
Card issuer data: 65 49 44
Mandatory status indicator (3 last bytes)
LCS (life card cycle): 0F (unknown)
SW: 9000 (Normal processing.)
+ TCK = F1 (correct checksum)
Possibly identified card (using /home/mag/.cache/smartcard_list.txt):
3B DB 96 00 80 B1 FE 45 1F 83 00 12 23 3F 53 65 49 44 0F 90 00 F1
Estonia ID-card (eID)
https://id.ee
however pkcs11-tool does not see the card:
$ pkcs11-tool --module /usr/lib/x86_64-linux-gnu/opensc-pkcs11.so -L
Available slots:
Slot 0 (0x0): REINER SCT cyberJack RFID standard (9084002233) 00 00
(empty)
What can be the cause of the problem? What do I miss?
Apparently the toolchain of the estonian card is not compatible with pkcs-11. However the chrome-token-signing package contains the needed code at least for authentication to their service both for chromium and firefox, and the qdigidoc4 package contains a tool to create and check signed/encrypted documents.
Their repo is here:
deb https://installer.id.ee/media/ubuntu/ bionic main
Another issue I have encountered that my cert is said to be invalid. That was because the certificate won't get enabled right away when you receive your ID.
I have a database filled with a lot of logged IPV4 messages. It is used to get queries like: "give me all messages from MacAddress ... that were logged in the period ... to ... that have ..."
Some queries will result in a huge amount of logged messages. Therefore we decided to make a PCAP file if such a request was made.
"Please create a PCAP file containing all logged messages from your
database that ..."
So upon request, my service should fetch the requested data from the database (in pages) and create a PCAP file filled with the data fetched from the database. Later callers can ask for a read-only OWIN stream to this file
The service can create such a file. The problem is that it is not recognized as a proper WireShark file.
I've read Libcap File Format. Whenever I have to create a file filled with LoggedMessages I fill a binary file as follows.
Global Header
Per logged message:
A packet header
Packet data with:
Ethernet Frame: Destination Mac, Source Mac, EtherType (0x800)
IPV4 header
Logged Data
Wireshark starts complaining about the file when it attempts to read the Ethertype. It says this is a Length. Definition of Ethernet Frame with EtherType
So below I show the start of my file. Hexadecimal format per byte + my interpretation of it. After that the comments from wireshark
The created stream starts with the Global Header: a 32 bytes structure. First the hexadecimal values then the interpretation:
=== Global Header ====
D4 C3 B2 A1 02 00 04 00
00 00 00 00 00 00 00 00
FF FF 00 00 01 00 00 00
Magic number A1B2C3D4 (Original Time Precision)
Version: 2 - 4
ThisZone 0
sigFigs 0
snapLen 0000FFFF
datalinkType 1
Note that the magic number has the LSB first, indicating that every multi-byte number will have the least significant byte first. So a 2 byte value of 0x1234 will have in memory first 34 then 12.
After that the Packets should come. Every time one Packet Header, followed by one Packet Data
=== Packet header ===
09 89 58 5A C8 85 0B 00
6B 00 00 00 6B 00 00 00
Timestamp: 1515751689.7551446 (usec precision)
Number of saved bytes (incl_len) 107 bytes (0x006b)
Actual packet length (orig_len) 107 bytes (0x006b)
=== Packet Data ===
CF 31 59 D3 E7 98 53 39 - 17 F0 A9 9C 00 08 45 00
5D 00 00 00 00 00 FF 00 - E0 0D 8A 84 77 44 E0 2B
9C FB 4D 43 D5 8A 00 00 - 00 00 41 41 41 41 41 41
41 41 41 41 41 41 41 41 - 41 41 41 41 41 41 41 41
// etc, until total 107 bytes
The packet data consists of a Mac Header, IPV4 header and a couple of 0x41 as data
=== Mac Header ===
Destination Mac: CF:31:59:D3:E7:98
Source Mac: 53:39:17:F0:A9:9C
Ether type: 0800
Note that the magic number showed that every multi-byte number has the LSB first, so the two bytes 00 08 will have a 16-bit meaning of 0x0800
If you look at the PCAP file interpretation I show below, then the problem starts here: the Ether Type is not interpreted as Ether Type, but as length.
After remark in one of the answers, I tried to reverse the two byte ether type from 00 08 into 08 00 (MSB first), but that made the problems worse.
=== IPV4 header ===
- 45 00 5D 00
- 00 00 00 00
- FF 00 E0 0D
- 8A 84 77 44
- E0 2B 9C FB
Specification of the IPV4 header structure
DWORD 0
- bits 00..04: version; bits 04..07 IP Header Length: 04 05
- bits 08..13 DSCP; bits 14..15 ECN: 00
- bits 16..31 Total Length (header + Payload): 93 (005D)
DWORD 1
- bits 00..15 Identification: 0000
- bits 16..18 Flags; bits 19..31 offset: 0000
DWORD 2
- bits 00..07 Time to Live FF
- bits 08..15 Protocol; used protocol 00
- bits 16..31 Header Checksum 3552 (0DE0)
DWORD 3 and 4
Source IP: 138.132.119.68
Destination IP: 224.43.156.251
Bacause wireshark complains about checksum, I verify as follows:
Verify checksum:
Header: 0045 005D 0000 0000 00FF 0DE0 848A 4477 2BE0 FB9C
69 + 93 + 0 + 0 + 255 + 3552 + 33930 + 17527 + 11232 + 64412 = 131070 (01FFFE)
0001 + FFFE = FFFF
1's complement: 0000 (checksum ok)
This is what WireShark (version 2.4.4) makes of it:
The following seems normal:
Frame 1: 107 bytes on wire (856 bits), 107 bytes captured (856 bits)
Encapsulation type: Ethernet (1)
Arrival Time: Jan 12, 2018 11:08:09.755144000 W. Europe Standard Time
[Time shift for this packet: 0.000000000 seconds]
Epoch Time: 1515751689.755144000 seconds
[Time delta from previous captured frame: 0.000000000 seconds]
[Time delta from previous displayed frame: 0.000000000 seconds]
[Time since reference or first frame: 0.000000000 seconds]
Frame Number: 1
Frame Length: 107 bytes (856 bits)
Capture Length: 107 bytes (856 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: eth:llc:data]
[Coloring Rule Name: Checksum Errors]
[Coloring Rule String [truncated]: eth.fcs.status=="Bad" ||
ip.checksum.status=="Bad" || tcp.checksum.status=="Bad" ||
udp.checksum.status=="Bad" || sctp.checksum.status=="Bad" ||
mstp.checksum.status=="Bad" || cdp.checksum.status=="Bad" ||]
Here comes the first problem: EtherType is interpreted as Length
IEEE 802.3 Ethernet
Destination: cf:31:59:d3:e7:98 (cf:31:59:d3:e7:98)
Source: 53:39:17:f0:a9:9c (53:39:17:f0:a9:9c)
Length: 8
Padding: ff00e00d8a847744e02b9cfb4d43d58a0000000041414141...
Trailer: 414141414141414141414141414141414141414141414141...
Frame check sequence: 0x41414141 incorrect, should be 0xe19cae36
[FCS Status: Bad]
After the length, which I meant as an EtherType, comes a lot of padding, instead of interpretation of my 5 DWORDs.
The link to the Ethernet Frame in wikipedia I showed says:
The EtherType field is two octets long and it can be used for two
different purposes. Values of 1500 and below mean that it is used to
indicate the size of the payload in octets, while values of 1536 and
above indicate that it is used as an EtherType, to indicate which
protocol is encapsulated in the payload of the frame.
My value if 0x0800 = 2048. This certainly is above 1536
For example, an EtherType value of 0x0800 signals that the frame
contains an IPv4 datagram.
If value 0x0800 the incorrect value? Or is my error somewhere else?
Looks like your ethertype has the wrong byte order. It should be:
=== Packet Data ===
CF 31 59 D3 E7 98 53 39 - 17 F0 A9 9C 08 00 XX XX
Out of curiosity, how much data is actually sent when establishing a connection to a port (using Java sockets). Is it the size of a Socket object? SocketConnection object?
Your understanding of TCP network connections seems to conflate them with electrical circuits. (Understandable, given your background.)
From a physical standpoint, there's no such thing as a connection, only data packets. Through the TCP protocol, two devices agree to establish a logical (that is, software) connection. A connection is established by a client first sending data to the remote host (SYN), the server sending data back to the client (SYN-ACK), and the client sending a final acknowledgement (ACK). All of this negotation necessarily consumes bandwidth, and when you terminate a connection, you must negotiate a completely new connection to begin sending data again.
For example, I'll connect from my machine to a local web server, 192.168.1.2:80.
First, my machine sends a TCP SYN. This sends 66 bytes over the wire: (headers deliniated with |)
0000 00 24 8c a9 4c b4 00 1e 68 66 20 79 08 00|45 00 .$..L... hf y..E.
0010 00 34 53 98 40 00 80 06 00 00 c0 a8 01 0b c0 a8 .4S.#... ........
0020 01 02|36 0a 00 50 09 ef 3a a7 00 00 00 00 80 02 ..6..P.. :.......
0030 20 00 50 c8 00 00 02 04 05 b4 01 03 03 02 01 01 .P..... ........
0040 04 02 ..
The first 14 bytes are the Ethernet frame, specifying that this packet's destination MAC address. This will typically be an upstream router, but in this case, the server happens to be on the same switch, so it's the machine's MAC address, 00:24:8c:a9:4c:b4. The source (my) MAC follows, along with the payload type (IP, 0x0800). The next 20 bytes are the IPv4 headers, followed by 32 bytes of TCP headers.
The server responds with a 62-byte SYN-ACK:
0000 00 1e 68 66 20 79 00 24 8c a9 4c b4 08 00|45 00 ..hf y.$ ..L...E.
0010 00 30 69 b9 40 00 80 06 0d b1 c0 a8 01 02 c0 a8 .0i.#... ........
0020 01 0b|00 50 36 0a d3 ae 9a 73 09 ef 3a a8 70 12 ...P6... .s..:.p.
0030 20 00 f6 9d 00 00 02 04 05 b4 01 01 04 02 ....... ......
Again, 14 bytes of Ethernet headers, 20 bytes of IP headers, and 28 bytes of TCP headers. I send an ACK:
0000 00 24 8c a9 4c b4 00 1e 68 66 20 79 08 00|45 00 .$..L... hf y..E.
0010 00 28 53 9a 40 00 80 06 00 00 c0 a8 01 0b c0 a8 .(S.#... ........
0020 01 02|36 0a 00 50 09 ef 3a a8 d3 ae 9a 74 50 10 ..6..P.. :....tP.
0030 fa f0 83 78 00 00 ...x..
14 + 20 + 20 = 54 bytes over the wire (this is the smallest possible TCP packet size, by the way – the SYN and SYN-ACK packets were larger because they included options).
This adds up to 182 bytes over the wire to establish a connection; now I can begin sending actual data to the server:
0000 00 24 8c a9 4c b4 00 1e 68 66 20 79 08 00 45|00 .$..L... hf y..E.
0010 01 9d 53 9d 40 00 80 06 00 00 c0 a8 01 0b c0 a8 ..S.#... ........
0020 01 02|36 0a 00 50 09 ef 3a a8 d3 ae 9a 74 50 18 ..6..P.. :....tP.
0030 fa f0 84 ed 00 00|47 45 54 20 2f 20 48 54 54 50 ......GE T / HTTP
0040 2f 31 2e 31 0d 0a 48 6f 73 74 3a 20 66 73 0d 0a /1.1..Ho st: fs..
...
14 Ethernet + 20 IP + 20 TCP + data, in this case HTTP.
So we can see that it costs ~182 bytes to establish a TCP connection, and an additional 162-216 bytes to terminate a TCP connection (depending on whether a 4-way FIN ACK FIN ACK or more common 3-way FIN FIN-ACK ACK termination handshake is used), adding up to nearly 400 bytes to "pulse" a connection by disconnecting and reconnecting.
Compared to the 55 bytes you'd use to send one byte of data over an already established connection, this is obviously wasteful.
What you want to do is establish one connection and then send data as-needed. If you're really bandwidth constrained, you could use UDP (which requires no handshaking at all and has an overhead of only 14 Ethernet + 20 IP + 8 UDP bytes per packet), but then you face the problem of using an unreliable transport, and having to handle lost packets on your own.
The minimum size of a TCP packet is 40 bytes. It takes three exchanged packets to create a connection, two from the client and one from the server, and four more to close it, two in each direction. The last packet in a connect exchange can also contain data, as can the first in the close exchange in each direction, which can amortize it a bit, as can combining the outgoing FIN and ACK as per #josh3736's comment below.