How to determine data block encoding method? - encoding

It is known that the HEX code:
08 45 DB 3C 7E D0 05
means the timestamp range (192 lines)
"2021-11-11 10:24:49 013000us"
"2021-11-11 10:24:49 113000us"
...
"2021-11-11 10:25:08 113000us"
HEX code:
07 45 DB 3C 7E D0 05
means the timestamp range (192 lines)
"2021-11-11 10:24:49 012999us"
"2021-11-11 10:24:49 112999us"
...
"2021-11-11 10:25:08 112999us"
HEX code:
00 45 DB 3C 7E D0 05
means the timestamp range (192 lines)
"2021-11-11 10:24:49 012992us"
"2021-11-11 10:24:49 112992us"
...
"2021-11-11 10:25:08 112992us"
How to determine the encoding method for timestamp range?

It looks like a Unix time multiplied by 1,000,000, and then microseconds added. In little-endian order. That gives the first time of each of your ranges, sort of. For example, for your first one I get 13000 microseconds after 2021-11-11 07:24:49 GMT. So you are showing three hours after GMT, e.g. Moscow time.
The range must be implied.

Related

DateTime format or coding

i have 3 parameters with date/time and status of phases of electricity meter:
28.11.2019 4:18 - all phases have power
28.11.2019 4:18 - all phases don't have power
28.11.2019 4:23 - all phases don't have power
But these data coded to hex like this:
FA 22 7C 27 07 FF E1
49 22 7C 27 07 08 F3
4E 22 7C 27 00 08 1D
And i don't know how it was did? I mean maybe there is some method to keep and transfer date/time or what?
34 [48 83 27] 07 C6 EA - 3.12.2019 09:01, </b>
38 [48 83 27] 00 E7 74 - 3.12.2019 09:01, </b>
89 [49 83 27] 07 10 38 - 3.12.2019 09:12, </b>
8E [49 83 27] 00 10 D6 - 3.12.2019 09:12, </b>
0E [4B 83 27] 07 21 EA - 3.12.2019 09:24, </b>
13 [4B 83 27] 00 42 74 - 3.12.2019 09:24,</b>
19 [4B 83 27] 07 63 2A - 3.12.2019 09:24,</b>
1D [4B 83 27] 00 63 A4 - 3.12.2019 09:24,</b>
21 [4B 83 27] 07 84 5A - 3.12.2019 09:25,</b>
25 [4B 83 27] 00 84 D4 - 3.12.2019 09:25,</b>
square brackets - i guess it's date. 4th byte is state of phases. I mean low halfbyte(or semibyte?) of 4th byte. 0x7(0b0111) - all phases have power, 0x0(0b0000) - al phases doesn't have power.
TL;DR: The first 4 bytes of each line is a little-endian count of time units since some epoch that is either January 1 or January 6, 1980.
Each timestamp is 4 bytes. When we look at the third last and the second last line, they both have 4B 83 27 within the square brackets that you have put, yet the time changes fro 9:24 to 9:25. The times are obviously truncated, there needs not be more than a second or two between them, but they are not the same. The solution I found was to include the byte before the square bracket in the timestamp.
Timestamps are little endian. The byte before the square bracket varies the fastest as time moves. The next byte varies a little. The remaining two bytes stay constant within the 24 minutes range you are giving. The obvious explanation is that 34 48 83 27 really means 0x27834834, that is, the bytes are reversed.
It took some experimentation, but two common epochs would fit:
January 1, 1980. In this case the time unit used for counting the time since the epoch is around 1 900 440 700 nanoseconds.
January 6 the same year. In this case the unit must be a little smaller, around 1 899 789 030 nanoseconds.
Both January 1 and January 6, 1980 are commonly used epochs. See the link at the bottom for documentation.
I have searched somewhat for an explanation for the size of the unit. Could it be some nice fraction of a second or of a day, for example? I haven’t found any reasonable explanation. I am probably missing something.
I have ignored the issue of time zone and UTC offset. The epoch is likely defined in UTC, and if the times given in the question are in your local time zone, for example, this adds a slight inaccuracy to my analysis.
In code
To convert from hex number to date and time we may use a simple method like the following in Java.
private static LocalDateTime epoch = LocalDateTime.of(1980, Month.JANUARY, 6, 0, 0);
private static int nanosPerUnit = 1_899_789_030;
private static LocalDateTime convert(int n) {
return epoch.plusNanos((long) n * (long) nanosPerUnit);
}
Demonstration:
int[] numbers = {
0x27834834,
0x27834838,
0x27834989,
0x2783498e,
0x27834b0e,
0x27834b13,
0x27834b19,
0x27834b1d,
0x27834b21,
0x27834b25
};
for (int n : numbers) {
System.out.format("%x %s%n", n, convert(n));
}
Output:
27834834 2019-12-03T09:01:20.396289720
27834838 2019-12-03T09:01:27.995445840
27834989 2019-12-03T09:12:08.224348950
2783498e 2019-12-03T09:12:17.723294100
27834b0e 2019-12-03T09:24:27.242281620
27834b13 2019-12-03T09:24:36.741226770
27834b19 2019-12-03T09:24:48.139960950
27834b1d 2019-12-03T09:24:55.739117070
27834b21 2019-12-03T09:25:03.338273190
27834b25 2019-12-03T09:25:10.937429310
The dates and times agree with those from the question. I leave it to you to insert January 1 and 1_900_440_700 nanoseconds and see that this too gives results that agree with the question.
Links
Endianness on Wikipedia
Epoch (computing) on Wikipedia

How to get the maximum value of HID report according to mouse sensor parameters?

I'm making a mouse to joystick converter.The solution is map the maximum and minimum value that the mouse can report to the joystick.How can I get the theoretically maximum or minimum value(not logically,regardless of error) of the mouse HID report according to the parameters of the mouse sensor?
My mouse is logitech G403
I dump the HID descriptor:
Usage Page (Generic Desktop) 05 01
Usage (Mouse) 09 02
Collection (Application) A1 01
Usage (Pointer) 09 01
Collection (Physical) A1 00
Usage Page (Button) 05 09
Usage Minimum (Button 1) 19 01
Usage Maximum (Button 16) 29 10
Logical Minimum (0) 15 00
Logical Maximum (1) 25 01
Report Count (16) 95 10
Report Size (1) 75 01
Input (Data,Var,Abs,NWrp,Lin,Pref,NNul,Bit) 81 02
Usage Page (Generic Desktop) 05 01
Logical Minimum (-32767) 16 01 80
Logical Maximum (32767) 26 FF 7F
Report Size (16) 75 10
Report Count (2) 95 02
Usage (X) 09 30
Usage (Y) 09 31
Input (Data,Var,Rel,NWrp,Lin,Pref,NNul,Bit) 81 06
Logical Minimum (-127) 15 81
Logical Maximum (127) 25 7F
Report Size (8) 75 08
Report Count (1) 95 01
Usage (Wheel) 09 38
Input (Data,Var,Rel,NWrp,Lin,Pref,NNul,Bit) 81 06
Usage Page (Consumer Devices) 05 0C
Usage (AC Pan) 0A 38 02
Report Count (1) 95 01
Input (Data,Var,Rel,NWrp,Lin,Pref,NNul,Bit) 81 06
End Collection C0
End Collection
I can only know the logical maximum and minimum of X and Y from this descriptor,No physical values or units are given here.
Through some rough tests I got the following data:
12000dpi,1000 report rate: the maximum value I measured was about 900
6000dpi,1000 report rate: the maximum value I measured was about 450
12000dpi,125 report rate: the maximum value I measured was about 8000
Although you can see here that the report values and DPI/report rate vary linearly but I still don't know what the report value really means.
This mouse fps(sampling rate?) is 12000,That's up to 12000 counts per second?
But obviously the report value is not number of counts.

A Script To Convert Numbers

I was hoping someone could assist me in making a linux script that can take a number and convert it to this format.
Here is a few of examples:
number = 753082360700386 Converted = 7A 35 80 32 06 07 30 68
number = 653082360700387 Converted = 6A 35 80 32 06 07 30 78
number = 453082360700389 Converted = 4A 35 80 32 06 07 30 98
The number will always have one A and last pair of numbers
swapped from 89 to 98 and next set swapped from 03 to 30
going from right to left and adding spaces between each pair

Unrecognized status byte in Midi file

I've been working on Midi file for some time and I stuck on some kind of status byte of thing. According to the standard Midi file format there is no such a things. So, Can someone tell what is this 3 bytes information "00 a040". I know that "00" is the byte stands for delta time and 0xa0 should be status byte, If only I understood it correctly. Last 3 bytes located at line 18 is the only part I don't understand so far. After those 3 bytes, then comes the text meta event bytes lead by "00 ff01".
Midi File Line 18th to 19th:
ff 51 03 09 cc 90 00 c0 00 00 b0 07 64 00 0a 40
00 ff 01 20 62 64 63 61 34 32 36 64 31 30 34 61
The SMF specification says:
Running status is used: status bytes of MIDI channel messages may be omitted if the preceding event is a MIDI channel message with the same status.
So these bytes can be decoded as follows:
ff 51 03 09 cc 90: meta event: set tempo, 9CC90h = 642192 µs per quarter note
00: delta time
c0 00: set program 0 (piano) on channel 0
00: delta time
b0 07 64: set controller 7 (volumn) to value 100
00: delta time
  0a 40: running status (repeat B0h); set controller 10 (expression) to value 64
00: delta time
ff 01 20 ...: meta event: text: "bdca426d104a..."

Read ASC file into MATLAB using textscan - variable column lengths

I am trying to read the following data into MATLAB:
'0.000000 1 18EFFA59x Rx D 8 AD 09 02 00 00 00 00 30'
'0.004245 1 14EFF01Cx Rx D 6 DB 00 FF FF 00 71'
'0.004640 1 CEF801Cx Rx D 3 3F 00 3B'
'0.005130 1 14EF131Cx Rx D 6 DB 00 FF FF 00 71'
'0.005630 1 CEF801Cx Rx D 3 3F 00 C3'
'0.010015 1 18EFFA59x Rx D 8 AD 07 01 00 00 00 00 30'
'0.014145 1 CF004F0x Rx D 8 F0 FF 7D 00 00 FF FF FF'
'0.015060 1 18EFFA59x Rx D 8 AD 07 02 00 00 00 00 30'
'0.018235 1 18EF1CF0x Rx D 8 F2 1E 05 FF FF 00 71 FF'
'0.018845 1 18EA5941x Rx D 3 09 FF 00'
I can easily read in each line as a string - but to make post-processing more efficient I'd like to separate each line by its delimiter - which is whitespace. In other words, the end result should be a non-singleton cell array. I can't seem to find a very efficient way of doing this. Efficiency is important because these files are several million lines long and processing in MATLAB with strings/cells takes a long time.
Any help would be appreciated. Thanks.
You appear to have fixed-width fields, so I would treat it as such and let textscan do the most of the pre-processing for you by turning off delimiters and whitespace and defining the field widths and types explicitly:
test = {...
'0.000000 1 18EFFA59x Rx D 8 AD 09 02 00 00 00 00 30'
'0.004245 1 14EFF01Cx Rx D 6 DB 00 FF FF 00 71'
'0.004640 1 CEF801Cx Rx D 3 3F 00 3B'
'0.005130 1 14EF131Cx Rx D 6 DB 00 FF FF 00 71'
'0.005630 1 CEF801Cx Rx D 3 3F 00 C3'
'0.010015 1 18EFFA59x Rx D 8 AD 07 01 00 00 00 00 30'
'0.014145 1 CF004F0x Rx D 8 F0 FF 7D 00 00 FF FF FF'
'0.015060 1 18EFFA59x Rx D 8 AD 07 02 00 00 00 00 30'
'0.018235 1 18EF1CF0x Rx D 8 F2 1E 05 FF FF 00 71 FF'
'0.018845 1 18EA5941x Rx D 3 09 FF 00'};
test = strjoin(test', '\n');
C = textscan(test, '%8.6f %2u %11s %4s %2s %2u %33s', 'delimiter', '','whitespace','');
col1 = C{1};
col2 = C{2};
col3 = strtrim(C{3});
col3 = cellfun(#(x)hex2dec(x(1:end-1)), col3); % for instance.
col4 = strtrim(C{4});
col5 = strtrim(C{5});
col6 = C{6};
col7 = strtrim(C{7});
In the real world, you'd substitute the text string for a file id. For the last variable-length field, just read the whole thing in, making sure you specify the maximum possible length. MATLAB will read a field until it gets to the end or reaches a newline character (in fact, I made the last field width 1 larger, just to make sure). Each field is then aggregated into a cell. I also took the liberty of converting the third field from hex to decimal to show how you might post-process the numbers further.
As a further note, if you really do have gigantic files and need maximum speed, you could skip the strtrim step on the character fields by specifying %*ns where n is the desired field width, for any known gaps such as the 2 character gap between columns 3 and 4. The star says to ignore that field. I find this way of doing things a bit more readable and intuitive, however, and leaves a small margin of error in case one of the fields, such as the 4th, occasionally has a 3 character entry.