Why isn't my pack() packing my data? - perl

What I am trying to do is simple. Here below.
my #arr = split(/\s+/,"50 00 9F 11 00 28 82 48 21 84 BC 00 01 02 01 00 09 01 38 00 23 05 08 01 01 02 00 00 18 00 50 05 00 00 00 00 00 00 00 00 02 00 0C FE CE 00 0F 00 FD FF 2D 00 00 00 00 00 04 01 0C FE");
my #hexData;
my $i=0;
foreach my $elem(#arr){
$hexData[$i]=hex($elem);
$i++;
}
my $data= pack ('C', #hexData);
print $data;
And its not working :( Would you please help?

TLP's solution is quite right, but pack actually has the ability to deal with hex.
my $data = "50 00 9F 11 00 28 82 48 21 84 BC 00 01 02 01 00 09 01 38 00 23 05 08 01 01 02 00 00 18 00 50 05 00 00 00 00 00 00 00 00 02 00 0C FE CE 00 0F 00 FD FF 2D 00 00 00 00 00 04 01 0C FE";
$data =~ tr/ //d; # Remove the spaces
print pack "H*", $data;
does the whole thing without the intermediate array.

I'm not terribly familiar with the pack function, but it seems to me that your template expects only one value.
Perhaps you should try
my $data = pack ('C*', #hexData);
And while you are at it, upgrade your code to something more perlish:
my #arr = qw(50 00 9F 11 00 28 82 48 21 84 BC 00 01 02 01 00 09 01 38 00 23 05
08 01 01 02 00 00 18 00 50 05 00 00 00 00 00 00 00 00 02 00 0C FE
CE 00 0F 00 FD FF 2D 00 00 00 00 00 04 01 0C FE);
my #hexData;
foreach my $elem (#arr) {
push #hexData, hex($elem);
}
my $data = pack ('C*', #hexData);
print $data;
Or even:
my $data = pack("C*", map(hex, #arr));

Related

How to filter not simple binary data column in PySpark?

Sample of my df is:
+------------------------------------------------------------------------------------------------------------------+
|id | binary_col |
+------------------------------------------------------------------------------------------------------------------+
| 1 | [08 01 10 0D 00 0E CC 93 01 00 00 00 01 00 00 00 00 00 00 00 80 FF BF 40 00 00 00 00 00 00 F0 3F BE 2B 00 00]|
| 2 | [08 01 10 0D 00 0E CC 93 01 00 00 00 01 00 00 00 00 00 00 00 F0 FF BF 40 00 00 00 00 00 00 F0 3F 57 66 00 00]|
| 3 | [08 01 10 0D 00 0E CC 93 01 00 00 00 01 00 00 00 00 00 00 00 C0 FF BF 40 00 00 00 00 00 00 F0 3F D5 69 00 00]|
| 4 | [08 01 10 0D 00 0E CC 93 01 00 00 00 01 00 00 00 00 00 00 00 80 FF BF 40 00 00 00 00 00 00 F0 3F 5A 60 00 00]|
+------------------------------------------------------------------------------------------------------------------+
with these schema (df.printSchema())
|-- id: int (nullable = true)
|-- binary_col: binary (nullable = true)
And I want to filter only the values with [08 01 10 0D 00 0E CC 93 01 00 00 00 01 00 00 00 00 00 00 00 80 FF BF 40 00 00 00 00 00 00 F0 3F BE 2B 00 00] (It doesn't work filtering id=1 because there are other ids in the df)
I've tried to cast binary to bigint to filter later like here: Spark: cast bytearray to bigint
by doing df.withColumn('casted_bin', F.conv(F.hex(F.col("binary_col")), 16, 10).cast("bigint")).show(truncate=False) but it didn't work.
How can I filter any kind of binary data type?
Note: I had asked previously here (How to filter Pyspark column with binary data type?) but it was a very simple binary data and the answer generated the binary from a numeric value while now I don't know how to generate the numeric value.

Reverse engineer 8bit checksum from a car

I'm trying to reverse engineer the CAN bus codes generated by the steering wheel of a car. I'm interested in a single message, the one that holds the information about what button of the steering wheel has been pressed. I have done that for many different cars but this one is special (?)
I found the message that holds the data has for ID 0x3D4. So I logged this specific message many times with/without a different combination of the steering wheel buttons.
The way car works (most of the time) is that a single bit will toggle from 0 to 1 when a button is held down. And go back to 0 when the button is released. This message has a DLC of 8 so there would be potentially enough room for 64 buttons.
Now for this car, they have used 4 bits of the second byte for a counter counting from 0 to F. The first byte is the one that is causing a problem. I don't understand how it is generated. It seems to be dependant on the value of the other bytes but does not follow a standard CRC algo... It seems that it also depends on the position of the bytes... Pressing on buttons in the car will change a bit in the data which then changes the value of the first byte. If I generate a fake command to the car without changing the first byte, the car will not react to it.
I would like to find how this first byte is generated so I can generate it myself.
The DLC: 8
CAN bus high speed 500kb/s
CA 0E 80 00 00 04 00 00
1F 0F 80 00 00 04 00 00
ED 00 80 00 00 04 00 00
4A 01 81 00 00 04 00 00
01 02 81 00 00 04 00 00
4C 03 81 00 00 04 00 00
DC 04 81 00 00 04 00 00
37 05 81 00 00 04 00 00
1E 06 81 00 00 04 00 00
8B 07 80 00 00 04 00 00
25 08 80 00 00 04 00 00
70 09 80 00 00 04 00 00
6B 0A 80 00 00 04 00 00
9B 0B 81 00 00 04 00 00
BE 0C 81 00 00 04 00 00
41 0D 81 00 00 04 00 00
3E 0E 81 00 00 04 00 00
1F 0F 80 00 00 04 00 00
ED 00 80 00 00 04 00 00
BE 01 80 00 00 04 00 00
F5 02 80 00 00 04 00 00
B8 03 80 00 00 04 00 00
I also found these following messages that share the same first byte:
7F 2D 84 00 00 04 00 00
7F 07 81 00 00 04 00 00
7F 03 82 00 00 04 00 00
4C 03 81 00 00 04 00 00
4C 00 84 00 00 04 00 00
4C 07 82 00 00 04 00 00
BE 0C 81 00 00 04 00 00
BE 01 80 00 00 04 00 00
BE 0F 84 00 00 04 00 00
BE 23 88 00 00 04 00 00
More data:
70 09 80 00 00 04 00 00
6B 0A 80 00 00 04 00 00
9B 0B 81 00 00 04 00 00
BE 0C 81 00 00 04 00 00
41 0D 81 00 00 04 00 00
3E 0E 81 00 00 04 00 00
EB 0F 81 00 00 04 00 00
ED 00 80 00 00 04 00 00
BE 01 80 00 00 04 00 00
F5 02 80 00 00 04 00 00
B8 03 80 00 00 04 00 00
28 04 80 00 00 04 00 00
04 05 82 00 00 04 00 00
4B 06 84 00 00 04 00 00
2A 07 84 00 00 04 00 00
26 18 84 00 00 04 00 00
73 19 84 00 00 04 00 00
68 1A 84 00 00 04 00 00
6C 1B 84 00 00 04 00 00
49 1C 84 00 00 04 00 00
B6 1D 84 00 00 04 00 00
C9 1E 84 00 00 04 00 00
BE 0F 84 00 00 04 00 00
4C 00 84 00 00 04 00 00
1F 01 84 00 00 04 00 00
54 02 84 00 00 04 00 00
19 03 84 00 00 04 00 00
89 04 84 00 00 04 00 00
09 25 84 00 00 04 00 00
20 26 84 00 00 04 00 00
41 27 84 00 00 04 00 00
EF 28 84 00 00 04 00 00
BA 29 84 00 00 04 00 00
A1 2A 84 00 00 04 00 00
A5 2B 84 00 00 04 00 00
80 2C 84 00 00 04 00 00
7F 2D 84 00 00 04 00 00
00 2E 84 00 00 04 00 00
D5 2F 84 00 00 04 00 00
27 20 84 00 00 04 00 00
74 21 84 00 00 04 00 00
3F 22 84 00 00 04 00 00
19 03 84 00 00 04 00 00
2B 14 84 00 00 04 00 00
C0 15 84 00 00 04 00 00
E9 16 84 00 00 04 00 00
88 17 84 00 00 04 00 00
26 18 84 00 00 04 00 00
73 19 84 00 00 04 00 00
68 1A 84 00 00 04 00 00
6C 1B 84 00 00 04 00 00
49 1C 84 00 00 04 00 00
B6 1D 84 00 00 04 00 00
C9 1E 84 00 00 04 00 00
1C 1F 84 00 00 04 00 00
EE 10 84 00 00 04 00 00
BD 11 84 00 00 04 00 00
F6 12 84 00 00 04 00 00
BB 13 84 00 00 04 00 00
2B 14 84 00 00 04 00 00
0C 15 88 00 00 04 00 00
25 16 88 00 00 04 00 00
44 17 88 00 00 04 00 00
EA 18 88 00 00 04 00 00
BF 19 88 00 00 04 00 00
A4 1A 88 00 00 04 00 00
02 0B 88 00 00 04 00 00
27 0C 88 00 00 04 00 00
D8 0D 88 00 00 04 00 00
CC 2E 88 00 00 04 00 00
19 2F 88 00 00 04 00 00
EB 20 88 00 00 04 00 00
B8 21 88 00 00 04 00 00
F3 22 88 00 00 04 00 00
BE 23 88 00 00 04 00 00
2E 24 88 00 00 04 00 00
C5 25 88 00 00 04 00 00
EC 26 88 00 00 04 00 00
8D 27 88 00 00 04 00 00
23 28 88 00 00 04 00 00
76 29 88 00 00 04 00 00
6D 2A 88 00 00 04 00 00
69 2B 88 00 00 04 00 00
4C 2C 88 00 00 04 00 00
B3 2D 88 00 00 04 00 00
CC 2E 88 00 00 04 00 00
72 0F 88 00 00 04 00 00
80 00 88 00 00 04 00 00
D3 01 88 00 00 04 00 00
98 02 88 00 00 04 00 00
D5 03 88 00 00 04 00 00
45 04 88 00 00 04 00 00
AE 05 88 00 00 04 00 00
4B 06 84 00 00 04 00 00
2A 07 84 00 00 04 00 00
E2 08 82 00 00 04 00 00
B7 09 82 00 00 04 00 00
AC 0A 82 00 00 04 00 00
A8 0B 82 00 00 04 00 00
8D 0C 82 00 00 04 00 00
72 0D 82 00 00 04 00 00
0D 0E 82 00 00 04 00 00
D8 0F 82 00 00 04 00 00
2A 00 82 00 00 04 00 00
79 01 82 00 00 04 00 00
32 02 82 00 00 04 00 00
7F 03 82 00 00 04 00 00
EF 04 82 00 00 04 00 00
04 05 82 00 00 04 00 00
2D 06 82 00 00 04 00 00
4C 07 82 00 00 04 00 00
E2 08 82 00 00 04 00 00
B7 09 82 00 00 04 00 00
AC 0A 82 00 00 04 00 00
A8 0B 82 00 00 04 00 00
8D 0C 82 00 00 04 00 00
72 0D 82 00 00 04 00 00
0D 0E 82 00 00 04 00 00
D8 0F 82 00 00 04 00 00
2A 00 82 00 00 04 00 00
79 01 82 00 00 04 00 00
32 02 82 00 00 04 00 00
7F 03 82 00 00 04 00 00
EF 04 82 00 00 04 00 00
04 05 82 00 00 04 00 00
2D 06 82 00 00 04 00 00
Following data have been logged while I tried all the buttons available. + a few combinations of the buttons.
76 00 A0 00 00 04 00 00
25 01 A0 00 00 04 00 00
6E 02 A0 00 00 04 00 00
23 03 A0 00 00 04 00 00
B3 04 A0 00 00 04 00 00
58 05 A0 00 00 04 00 00
71 06 A0 00 00 04 00 00
10 07 A0 00 00 04 00 00
BE 08 A0 00 00 04 00 00
EB 09 A0 00 00 04 00 00
04 0A A1 00 00 04 00 00
00 0B A1 00 00 04 00 00
25 0C A1 00 00 04 00 00
DA 0D A1 00 00 04 00 00
A5 0E A1 00 00 04 00 00
84 0F A0 00 00 04 00 00
76 00 A0 00 00 04 00 00
25 01 A0 00 00 04 00 00
6E 02 A0 00 00 04 00 00
23 03 A0 00 00 04 00 00
B3 04 A0 00 00 04 00 00
9F 05 A2 00 00 04 00 00
D0 06 A4 00 00 04 00 00
B1 07 A4 00 00 04 00 00
D3 08 A8 00 00 04 00 00
86 09 A8 00 00 04 00 00
9D 0A A8 00 00 04 00 00
99 0B A8 00 00 04 00 00
70 0C A4 00 00 04 00 00
8F 0D A4 00 00 04 00 00
F0 0E A4 00 00 04 00 00
25 0F A4 00 00 04 00 00
B1 00 A2 00 00 04 00 00
E2 01 A2 00 00 04 00 00
A9 02 A2 00 00 04 00 00
23 03 A0 00 00 04 00 00
B3 04 A0 00 00 04 00 00
C3 05 80 00 00 04 00 00
30 06 90 00 00 04 00 00
51 07 90 00 00 04 00 00
25 08 80 00 00 04 00 00
70 09 80 00 00 04 00 00
6B 0A 80 00 00 04 00 00
6F 0B 80 00 00 04 00 00
4A 0C 80 00 00 04 00 00
B5 0D 80 00 00 04 00 00
CA 0E 80 00 00 04 00 00
1F 0F 80 00 00 04 00 00
ED 00 80 00 00 04 00 00
BE 01 80 00 00 04 00 00
01 02 81 00 00 04 00 00
4C 03 81 00 00 04 00 00
DC 04 81 00 00 04 00 00
37 05 81 00 00 04 00 00
1E 06 81 00 00 04 00 00
7F 07 81 00 00 04 00 00
D1 08 81 00 00 04 00 00
70 09 80 00 00 04 00 00
6B 0A 80 00 00 04 00 00
6F 0B 80 00 00 04 00 00
4A 0C 80 00 00 04 00 00
B5 0D 80 00 00 04 00 00
CA 0E 80 00 00 04 00 00
1F 0F 80 00 00 04 00 00
2A 00 82 00 00 04 00 00
79 01 82 00 00 04 00 00
32 02 82 00 00 04 00 00
7F 03 82 00 00 04 00 00
EF 04 82 00 00 04 00 00
04 05 82 00 00 04 00 00
2D 06 82 00 00 04 00 00
2A 07 84 00 00 04 00 00
84 08 84 00 00 04 00 00
D1 09 84 00 00 04 00 00
CA 0A 84 00 00 04 00 00
CE 0B 84 00 00 04 00 00
27 0C 88 00 00 04 00 00
D8 0D 88 00 00 04 00 00
A7 0E 88 00 00 04 00 00
72 0F 88 00 00 04 00 00
4C 00 84 00 00 04 00 00
1F 01 84 00 00 04 00 00
32 02 82 00 00 04 00 00
B8 03 80 00 00 04 00 00
28 04 80 00 00 04 00 00
C3 05 80 00 00 04 00 00
71 06 A0 00 00 04 00 00
10 07 A0 00 00 04 00 00
BE 08 A0 00 00 04 00 00
69 09 C0 00 00 04 00 00
72 0A C0 00 00 04 00 00
F4 0B A0 00 00 04 00 00
D1 0C A0 00 00 04 00 00
2E 0D A0 00 00 04 00 00
51 0E A0 00 00 04 00 00
84 0F A0 00 00 04 00 00
76 00 A0 00 00 04 00 00
25 01 A0 00 00 04 00 00
6E 02 A0 00 00 04 00 00
B8 03 80 00 00 04 00 00
28 04 80 00 00 04 00 00
C3 05 80 00 00 04 00 00
EA 06 80 00 00 04 00 00
8B 07 80 00 00 04 00 00
25 08 80 00 00 04 00 00
0F 09 80 04 00 04 00 00
14 0A 80 04 00 04 00 00
B8 0B 80 06 00 04 00 00
9D 0C 80 06 00 04 00 00
62 0D 80 06 00 04 00 00
1D 0E 80 06 00 04 00 00
C8 0F 80 06 00 04 00 00
3A 00 80 06 00 04 00 00
C1 01 80 04 00 04 00 00
8A 02 80 04 00 04 00 00
C7 03 80 04 00 04 00 00
57 04 80 04 00 04 00 00
BC 05 80 04 00 04 00 00
95 06 80 04 00 04 00 00
F4 07 80 04 00 04 00 00
5A 08 80 04 00 04 00 00
0F 09 80 04 00 04 00 00
14 0A 80 04 00 04 00 00
10 0B 80 04 00 04 00 00
35 0C 80 04 00 04 00 00
CA 0D 80 04 00 04 00 00
B5 0E 80 04 00 04 00 00
60 0F 80 04 00 04 00 00
92 00 80 04 00 04 00 00
C1 01 80 04 00 04 00 00
8A 02 80 04 00 04 00 00
C7 03 80 04 00 04 00 00
57 04 80 04 00 04 00 00
BC 05 80 04 00 04 00 00
95 06 80 04 00 04 00 00
F4 07 80 04 00 04 00 00
5A 08 80 04 00 04 00 00
0F 09 80 04 00 04 00 00
14 0A 80 04 00 04 00 00
10 0B 80 04 00 04 00 00
35 0C 80 04 00 04 00 00
CA 0D 80 04 00 04 00 00
B5 0E 80 04 00 04 00 00
60 0F 80 04 00 04 00 00
92 00 80 04 00 04 00 00
C1 01 80 04 00 04 00 00
8A 02 80 04 00 04 00 00
C7 03 80 04 00 04 00 00
57 04 80 04 00 04 00 00
BC 05 80 04 00 04 00 00
95 06 80 04 00 04 00 00
23 07 80 02 00 04 00 00
8D 08 80 02 00 04 00 00
D8 09 80 02 00 04 00 00
C3 0A 80 02 00 04 00 00
C7 0B 80 02 00 04 00 00
E2 0C 80 02 00 04 00 00
1D 0D 80 02 00 04 00 00
62 0E 80 02 00 04 00 00
B7 0F 80 02 00 04 00 00
45 00 80 02 00 04 00 00
16 01 80 02 00 04 00 00
5D 02 80 02 00 04 00 00
10 03 80 02 00 04 00 00
80 04 80 02 00 04 00 00
6B 05 80 02 00 04 00 00
42 06 80 02 00 04 00 00
23 07 80 02 00 04 00 00
25 08 80 00 00 04 00 00
70 09 80 00 00 04 00 00
6B 0A 80 00 00 04 00 00
6F 0B 80 00 00 04 00 00
4A 0C 80 00 00 04 00 00
1D 0D 80 02 00 04 00 00
B5 0E 80 04 00 04 00 00
60 0F 80 04 00 04 00 00
3A 00 80 06 00 04 00 00
69 01 80 06 00 04 00 00
22 02 80 06 00 04 00 00
6F 03 80 06 00 04 00 00
FF 04 80 06 00 04 00 00
14 05 80 06 00 04 00 00
3D 06 80 06 00 04 00 00
5C 07 80 06 00 04 00 00
F2 08 80 06 00 04 00 00
A7 09 80 06 00 04 00 00
BC 0A 80 06 00 04 00 00
B8 0B 80 06 00 04 00 00
9D 0C 80 06 00 04 00 00
62 0D 80 06 00 04 00 00
1D 0E 80 06 00 04 00 00
C8 0F 80 06 00 04 00 00
3A 00 80 06 00 04 00 00
69 01 80 06 00 04 00 00
22 02 80 06 00 04 00 00
6F 03 80 06 00 04 00 00
FF 04 80 06 00 04 00 00
14 05 80 06 00 04 00 00
3D 06 80 06 00 04 00 00
5C 07 80 06 00 04 00 00
F2 08 80 06 00 04 00 00
A7 09 80 06 00 04 00 00
BC 0A 80 06 00 04 00 00
10 0B 80 04 00 04 00 00
35 0C 80 04 00 04 00 00
1D 0D 80 02 00 04 00 00
62 0E 80 02 00 04 00 00
B7 0F 80 02 00 04 00 00
45 00 80 02 00 04 00 00
16 01 80 02 00 04 00 00
5D 02 80 02 00 04 00 00
10 03 80 02 00 04 00 00
80 04 80 02 00 04 00 00
6B 05 80 02 00 04 00 00
42 06 80 02 00 04 00 00
23 07 80 02 00 04 00 00
8D 08 80 02 00 04 00 00
D8 09 80 02 00 04 00 00
C3 0A 80 02 00 04 00 00
C7 0B 80 02 00 04 00 00
E2 0C 80 02 00 04 00 00
1D 0D 80 02 00 04 00 00
62 0E 80 02 00 04 00 00
B7 0F 80 02 00 04 00 00
45 00 80 02 00 04 00 00
16 01 80 02 00 04 00 00
5D 02 80 02 00 04 00 00
B8 03 80 00 00 04 00 00
28 04 80 00 00 04 00 00
C3 05 80 00 00 04 00 00
EA 06 80 00 00 04 00 00
4C 07 82 00 00 04 00 00
E2 08 82 00 00 04 00 00
B7 09 82 00 00 04 00 00
AC 0A 82 00 00 04 00 00
A8 0B 82 00 00 04 00 00
8D 0C 82 00 00 04 00 00
72 0D 82 00 00 04 00 00
0D 0E 82 00 00 04 00 00
D8 0F 82 00 00 04 00 00
2A 00 82 00 00 04 00 00
79 01 82 00 00 04 00 00
32 02 82 00 00 04 00 00
7F 03 82 00 00 04 00 00
EF 04 82 00 00 04 00 00
04 05 82 00 00 04 00 00
2D 06 82 00 00 04 00 00
4C 07 82 00 00 04 00 00
E2 08 82 00 00 04 00 00
B7 09 82 00 00 04 00 00
AC 0A 82 00 00 04 00 00
A8 0B 82 00 00 04 00 00
8D 0C 82 00 00 04 00 00
72 0D 82 00 00 04 00 00
0D 0E 82 00 00 04 00 00
D8 0F 82 00 00 04 00 00
2A 00 82 00 00 04 00 00
79 01 82 00 00 04 00 00
32 02 82 00 00 04 00 00
7F 03 82 00 00 04 00 00
EF 04 82 00 00 04 00 00
04 05 82 00 00 04 00 00
2D 06 82 00 00 04 00 00
4C 07 82 00 00 04 00 00
E2 08 82 00 00 04 00 00
B7 09 82 00 00 04 00 00
AC 0A 82 00 00 04 00 00
A8 0B 82 00 00 04 00 00
8D 0C 82 00 00 04 00 00
72 0D 82 00 00 04 00 00
0D 0E 82 00 00 04 00 00
1F 0F 80 00 00 04 00 00
ED 00 80 00 00 04 00 00
BE 01 80 00 00 04 00 00
57 12 80 00 00 04 00 00
1A 13 80 00 00 04 00 00
8A 14 80 00 00 04 00 00
61 15 80 00 00 04 00 00
48 16 80 00 00 04 00 00
29 17 80 00 00 04 00 00
87 18 80 00 00 04 00 00
D2 19 80 00 00 04 00 00
C9 1A 80 00 00 04 00 00
CD 1B 80 00 00 04 00 00
E8 1C 80 00 00 04 00 00
17 1D 80 00 00 04 00 00
68 1E 80 00 00 04 00 00
BD 1F 80 00 00 04 00 00
4F 10 80 00 00 04 00 00
1C 11 80 00 00 04 00 00
57 12 80 00 00 04 00 00
1A 13 80 00 00 04 00 00
8A 14 80 00 00 04 00 00
61 15 80 00 00 04 00 00
48 16 80 00 00 04 00 00
29 17 80 00 00 04 00 00
87 18 80 00 00 04 00 00
D2 19 80 00 00 04 00 00
C9 1A 80 00 00 04 00 00
CD 1B 80 00 00 04 00 00
E8 1C 80 00 00 04 00 00
17 1D 80 00 00 04 00 00
CA 0E 80 00 00 04 00 00
1F 0F 80 00 00 04 00 00
ED 00 80 00 00 04 00 00
BE 01 80 00 00 04 00 00
F5 02 80 00 00 04 00 00
B8 03 80 00 00 04 00 00
43 24 80 00 00 04 00 00
A8 25 80 00 00 04 00 00
81 26 80 00 00 04 00 00
E0 27 80 00 00 04 00 00
4E 28 80 00 00 04 00 00
1B 29 80 00 00 04 00 00
00 2A 80 00 00 04 00 00
04 2B 80 00 00 04 00 00
21 2C 80 00 00 04 00 00
DE 2D 80 00 00 04 00 00
A1 2E 80 00 00 04 00 00
74 2F 80 00 00 04 00 00
86 20 80 00 00 04 00 00
D5 21 80 00 00 04 00 00
9E 22 80 00 00 04 00 00
D3 23 80 00 00 04 00 00
43 24 80 00 00 04 00 00
A8 25 80 00 00 04 00 00
81 26 80 00 00 04 00 00
E0 27 80 00 00 04 00 00
4E 28 80 00 00 04 00 00
1B 29 80 00 00 04 00 00
00 2A 80 00 00 04 00 00
04 2B 80 00 00 04 00 00
21 2C 80 00 00 04 00 00
DE 2D 80 00 00 04 00 00
A1 2E 80 00 00 04 00 00
74 2F 80 00 00 04 00 00
86 20 80 00 00 04 00 00
D5 21 80 00 00 04 00 00
F5 02 80 00 00 04 00 00
B8 03 80 00 00 04 00 00
28 04 80 00 00 04 00 00
C3 05 80 00 00 04 00 00
EA 06 80 00 00 04 00 00
8B 07 80 00 00 04 00 00
F3 48 80 00 00 04 00 00
A6 49 80 00 00 04 00 00
BD 4A 80 00 00 04 00 00
B9 4B 80 00 00 04 00 00
9C 4C 80 00 00 04 00 00
63 4D 80 00 00 04 00 00
1C 4E 80 00 00 04 00 00
C9 4F 80 00 00 04 00 00
3B 40 80 00 00 04 00 00
68 41 80 00 00 04 00 00
23 42 80 00 00 04 00 00
6E 43 80 00 00 04 00 00
FE 44 80 00 00 04 00 00
15 45 80 00 00 04 00 00
3C 46 80 00 00 04 00 00
5D 47 80 00 00 04 00 00
F3 48 80 00 00 04 00 00
A6 49 80 00 00 04 00 00
BD 4A 80 00 00 04 00 00
B9 4B 80 00 00 04 00 00
9C 4C 80 00 00 04 00 00
63 4D 80 00 00 04 00 00
1C 4E 80 00 00 04 00 00
C9 4F 80 00 00 04 00 00
3B 40 80 00 00 04 00 00
68 41 80 00 00 04 00 00
23 42 80 00 00 04 00 00
6E 43 80 00 00 04 00 00
FE 44 80 00 00 04 00 00
15 45 80 00 00 04 00 00
3C 46 80 00 00 04 00 00
5D 47 80 00 00 04 00 00
F3 48 80 00 00 04 00 00
A6 49 80 00 00 04 00 00
BD 4A 80 00 00 04 00 00
6F 0B 80 00 00 04 00 00
4A 0C 80 00 00 04 00 00
B5 0D 80 00 00 04 00 00
CA 0E 80 00 00 04 00 00
1F 0F 80 00 00 04 00 00
ED 00 80 00 00 04 00 00
3D 81 80 00 00 04 00 00
76 82 80 00 00 04 00 00
3B 83 80 00 00 04 00 00
AB 84 80 00 00 04 00 00
40 85 80 00 00 04 00 00
69 86 80 00 00 04 00 00
08 87 80 00 00 04 00 00
A6 88 80 00 00 04 00 00
F3 89 80 00 00 04 00 00
E8 8A 80 00 00 04 00 00
EC 8B 80 00 00 04 00 00
C9 8C 80 00 00 04 00 00
36 8D 80 00 00 04 00 00
49 8E 80 00 00 04 00 00
9C 8F 80 00 00 04 00 00
6E 80 80 00 00 04 00 00
3D 81 80 00 00 04 00 00
76 82 80 00 00 04 00 00
3B 83 80 00 00 04 00 00
AB 84 80 00 00 04 00 00
40 85 80 00 00 04 00 00
69 86 80 00 00 04 00 00
08 87 80 00 00 04 00 00
A6 88 80 00 00 04 00 00
F3 89 80 00 00 04 00 00
E8 8A 80 00 00 04 00 00
EC 8B 80 00 00 04 00 00
C9 8C 80 00 00 04 00 00
36 8D 80 00 00 04 00 00
49 8E 80 00 00 04 00 00
9C 8F 80 00 00 04 00 00
ED 00 80 00 00 04 00 00
BE 01 80 00 00 04 00 00
F5 02 80 00 00 04 00 00
B8 03 80 00 00 04 00 00
28 04 80 00 00 04 00 00
04 05 82 00 00 04 00 00
4B 06 84 00 00 04 00 00
2A 07 84 00 00 04 00 00
84 08 84 00 00 04 00 00
D1 09 84 00 00 04 00 00
CA 0A 84 00 00 04 00 00
CE 0B 84 00 00 04 00 00
EB 0C 84 00 00 04 00 00
14 0D 84 00 00 04 00 00
6B 0E 84 00 00 04 00 00
BE 0F 84 00 00 04 00 00
4C 00 84 00 00 04 00 00
1F 01 84 00 00 04 00 00
54 02 84 00 00 04 00 00
19 03 84 00 00 04 00 00
89 04 84 00 00 04 00 00
62 05 84 00 00 04 00 00
4B 06 84 00 00 04 00 00
2A 07 84 00 00 04 00 00
84 08 84 00 00 04 00 00
D1 09 84 00 00 04 00 00
CA 0A 84 00 00 04 00 00
CE 0B 84 00 00 04 00 00
EB 0C 84 00 00 04 00 00
14 0D 84 00 00 04 00 00
6B 0E 84 00 00 04 00 00
BE 0F 84 00 00 04 00 00
4C 00 84 00 00 04 00 00
1F 01 84 00 00 04 00 00
54 02 84 00 00 04 00 00
19 03 84 00 00 04 00 00
89 04 84 00 00 04 00 00
62 05 84 00 00 04 00 00
4B 06 84 00 00 04 00 00
2A 07 84 00 00 04 00 00
48 08 88 00 00 04 00 00
1D 09 88 00 00 04 00 00
06 0A 88 00 00 04 00 00
02 0B 88 00 00 04 00 00
27 0C 88 00 00 04 00 00
D8 0D 88 00 00 04 00 00
A7 0E 88 00 00 04 00 00
72 0F 88 00 00 04 00 00
80 00 88 00 00 04 00 00
D3 01 88 00 00 04 00 00
98 02 88 00 00 04 00 00
D5 03 88 00 00 04 00 00
45 04 88 00 00 04 00 00
AE 05 88 00 00 04 00 00
87 06 88 00 00 04 00 00
E6 07 88 00 00 04 00 00
48 08 88 00 00 04 00 00
1D 09 88 00 00 04 00 00
06 0A 88 00 00 04 00 00
CE 0B 84 00 00 04 00 00
EB 0C 84 00 00 04 00 00
72 0D 82 00 00 04 00 00
0D 0E 82 00 00 04 00 00
D8 0F 82 00 00 04 00 00
ED 00 80 00 00 04 00 00
BE 01 80 00 00 04 00 00
F5 02 80 00 00 04 00 00
B8 03 80 00 00 04 00 00
28 04 80 00 00 04 00 00
C3 05 80 00 00 04 00 00
EA 06 80 00 00 04 00 00
8B 07 80 00 00 04 00 00
Thanks!
If your goal is to send an arbitrary known code to the car, a simple way is to use a map to store all the valid 1st bytes, and send it accordingly.
If your goal is to utilize those other bits previously not used (~5 bytes of them), then unfortunately it is not possible from the data you provided. One reason is that there is no way for us to know the ordering of the ~40 currently unused bits even if we knew the algorithm used to generate the 1st byte.
The space of 1 byte values is small, only 256. For any code that you want to send, you could just try 256 times each with a different 1st byte. If any one does anything, then it's probably it. If none does, then maybe that code is invalid altogether?
I can say what it is not. The first byte is not a CRC, nor is it any linear combination of the provided remaining bits over GF(2). It may be some other function of the following bits/bytes.

perl seek and remove at varying offsets in binmode

This is my script I am writing.
#usr/bin/perl
use warnings;
open(my $infile, '<', "./file1.bin") or die "Cannot open file1.bin: $!";
binmode($infile);
open(my $outfile, '>', "./extracted data without 00's.bin") or die "Cannot create extracted data without 00's.bin: $!";
binmode($outfile);
local $/; $infile = <STDIN>;
print substr($infile, 0, 0x840, '');
$infile =~ s/\0{16}//;
print $outfile;
I'm loading a binary file in perl.
I have been able to seek and patch at certain offsets, but what I would like to do is, now be able to find any instance of "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00" (16 bytes?) and remove it from the file, but no less than 16 bytes. Anything less than that I would want to leave. In some of the files the offset where the 00's start will be at different offsets, but if I am thinking correctly, if I can just search for 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 and remove any instance of it, then it won't matter what offset the 00's are at. I would extract the data first from specific offsets, then search the file and prune 00's from it. I can already extract the specific offsets I need, I just need to open the extracted file and shave off 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
EF 39 77 5B 14 9D E9 1E 94 A9 97 F2 6D E3 68 05
6F 7B 77 BB C4 99 67 B5 C9 71 12 30 9D ED 31 B6
AB 1F 81 66 E1 DD 29 4E 71 8D 54 F5 6C C8 86 0D
5B 72 AF A8 1F 26 DD 05 AF 78 13 EF A5 E0 76 BB
8A 59 9B 20 C5 58 95 7C E0 DB 44 6A EC 7E D0 10
09 42 B1 12 65 80 B3 EC 58 1A 2F 92 B9 32 D9 07
96 DE 32 51 4B 5F 3B 50 9A D1 09 37 F4 6D 7C 01
01 4A A4 24 04 DC 83 08 17 CB 34 2C E5 87 26 C1
35 38 F4 C4 E4 78 FE FC A2 BE 99 48 C9 CA 69 90
33 87 09 A8 27 BA 91 FC 4B 77 FA AB F5 1E 4E C0 I want to leave everything from
F2 78 6E 31 7D 16 3B 53 04 8A C1 A8 4B 70 39 22 <----- here up
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <----- I want to prune everything
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 from here on
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00<---- this IS the end of the file, and
just need to prune these few rows
of 00's
Say that "F2 78 6E" from the example above, is at offset 0x45000 BUT in another file the 00 00's will start at a different offset, how could I code it so the 00 00's would get pruned. In any file that I am opening?
If I need to be more specific, just ask.
Seems like I would peekk so far into the file until I hit a long 00 00 string, then prune any remaining lines. Does that make sense at all?
All I want to do is search the file for any instances of 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 and delete/prune/truncate it. I want to save everything but the 00's
EDIT #2
this did it:
open($infile, '<', './file1') or die "cannot open file1: $!";
binmode $infile;
open($outfile, '>', './file2') or die "cannot open file2: $!";
binmode $outfile;
local $/; $file = <$infile>;
$file =~ s/\0{16}//g;
print $outfile $file;
close ($infile);
close ($outfile);
Thank you ikegami for all your help and patience :)
No such thing as removing from a file. You have to either
copy the file without the undesired bits, or
read the rest of the file, seek back, print over the undesired bits, then truncate the file.
I went with option 1.
$ perl -e'
binmode STDIN;
binmode STDOUT;
local $/; $file = <STDIN>;
$file =~ s/\0{16}//;
print $file;
' <file.in >file.out
I'm loading the entire file into memory. Either option can be done in chunks, but it complicates things because your NULs could span two chunks.
In a poorly phrased update, you seem to have asked to avoid changes in the first 0x840 bytes. Two solutions:
$ perl -e'
binmode STDIN;
binmode STDOUT;
local $/; $file = <STDIN>;
substr($file, 0x840) =~ s/\0{16}//;
print $file;
' <file.in >file.out
$ perl -e'
binmode STDIN;
binmode STDOUT;
local $/; $file = <STDIN>;
print substr($file, 0, 0x840, '');
$file =~ s/\0{16}//;
print $file;
' <file.in >file.out

Extracting "plaintext" header from HEX file using Perl

Have a file that appears to have plaintext headers in them that I would like to extract and convert to plaintext.
Using HEXedit, this is what I'm seeing, which is in a file:
3a40 - 31 65 33 38 00 00 00 00 00 00 00 00 00 00 00 00 - 1e38............
3a50 - 00 00 00 00 00 00 00 00 00 00 0a 00 74 00 65 00 - ............t.e.
3a60 - 78 00 74 00 2f 00 61 00 73 00 63 00 69 00 69 00 - x.t./.a.s.c.i.i.
3a70 - 00 00 18 00 61 00 66 00 66 00 79 00 6d 00 65 00 - ....a.f.f.y.m.e
3a80 - 74 00 72 00 69 00 78 00 2d 00 61 00 72 00 72 00 - t.r.i.x.-.a.r.r
3a90 - 61 00 79 00 2d 00 62 00 61 00 72 00 63 00 6f 00 - a.y.-.b.a.r.c.o.
3aa0 - 64 00 65 00 00 00 64 00 40 00 35 00 32 00 30 00 - d.e...d.#.5.2.0.
3ab0 - 38 00 32 00 36 00 30 00 30 00 39 00 31 00 30 00 - 8.2.6.0.0.9.1.0.
3ac0 - 37 00 30 00 36 00 31 00 31 00 31 00 38 00 31 00 - 7.0.6.1.1.1.8.1.
3ad0 - 31 00 34 00 31 00 32 00 31 00 33 00 34 00 35 00 - 1.4.1.2.1.3.4.5.
3ae0 - 35 00 30 00 39 00 38 00 39 00 00 00 00 00 00 00 - 5.0.9.8.9.......
3af0 - 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 - ................
3b00 - 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0a 00 - ................
and this is the output I'd like to get:
text/ascii affymetrix-array-barcode d#52082600910706111811412134550989
Try with the iconv command. Something like this should work:
tail -c +6 input.txt | iconv -f UTF16 -t ASCII >output.txt
Then split on the null bytes.
Granted, I'm no wiz, but this does the job if all your files look very similar to the one you just posted:
use strict;
open FILE, 'file.dat';
binmode FILE;
my ($chunk, $buf, $n);
seek FILE, 28, 0;
while (($n=read FILE, $chunk, 16)) { $buf .= $chunk; }
my #s=split(/\0\0/, $buf, 4);
print "$s[0] $s[1] $s[2]\n";
close (FILE);
A perl solution might be interesting, but wouldn't the unix strings command give you the plaintext portion of the file?

How to convert/manipulate BINARY file to ASCII file?

I'm looking for a way to take the TEXT characters from a 4byte BINARY file to array or TEXT file,
Lets say my input file is:
00000000 2e 00 00 00 01 00 00 00 02 00 00 00 03 00 00 00 |................|
00000010 04 00 00 00 05 00 00 00 06 00 00 00 07 00 00 00 |................|
00000020 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000070 00 00 00 00 00 00 00 00 |........|
00000078
And my desired output is:
46,1,2,3,4,5,6,7,8,9,0,0...
The output can be a TEXT file or an array.
I notice that the pack/unpack functions may help here, but I couldn't figure how to use them properly,
An example would be nice.
Use unpack:
local $/;
#_=unpack("V*", <>);
gets you an array. So as an inefficient (don't try on huge files) example:
perl -e 'local$/;print join(",",map{sprintf("%d",$_)}unpack("V*",<>))' thebinaryfile
The answer is dependent on what you consider an ASCII character. Anything below 128 is technically an ASCII character, but I am assuming you mean characters you normally find in a text file. In that case, try this:
#!/usr/bin/perl
use strict;
use warnings;
use bytes;
$/ = \1024; #read 1k at a time
while (<>) {
for my $char (split //) {
my $ord = ord $char;
if ($char > 31 and $char < 127 or $char =~ /\r\n\t/) {
print "$ord,"
}
}
}
od -t d4 -v <filename>