packing an incrementing counter into hex with perl - perl

i am trying to create a 12 byte header and i am using pack. everything was going awesome until i needed to include an incrementing value in the pack function..
EDIT:
i have an open ffmpeg pipe that i am capturing the output from. the captured data is going to be a payload that will be encrypted and appended to the end of the header. my loop is not a for loop but i thought i would make it a little easier for the readers to understand.
so let me update the question here in hopes it clears any confusion. if i need to provide any more information let me know.
my $m = 00;
my $n = 00;
while( <$source_audio> ) {
if(($n % 16) == 0){
$m++; #this should increment when n reaches 0xff
}
my $header = pack( 'C C n n C*', 0x80, 0x78, $m, $n, 0x12, 0x34, 0x98, 0x76, 0x01, 0x02, 0x03, 0x04 );
print $file $header ;
$n++;
}
i just need to be able to pack $m and $n so i get the expected output:
80 78 00 01 12 34 98 76 01 02 03 04
80 78 00 02 12 34 98 76 01 02 03 04
80 78 00 03 12 34 98 76 01 02 03 04
80 78 00 04 12 34 98 76 01 02 03 04
80 78 00 05 12 34 98 76 01 02 03 04
... iterating 0x0000 to 0xFFFF ...
80 78 FF FB 12 34 98 76 01 02 03 04
80 78 FF FC 12 34 98 76 01 02 03 04
80 78 FF FD 12 34 98 76 01 02 03 04
80 78 FF FE 12 34 98 76 01 02 03 04
80 78 FF FF 12 34 98 76 01 02 03 04
EDIT2:
this gives me the desired output, but i get errors with strict and warnings enabled:
my $m = 0x00;
my $n = 0x00;
my $message = "this";
while ( <$source_audio> ) {
my $header = pack('C*', 0x80, 0x78, $m, $n, 0x12, 0x34, 0x98, 0x76, 0x01, 0x02, 0x03, 0x04);
if(($n % 0xFF) == 0 and $n != 0x00){
$m++;
}
$n++;
print $temp $header;
}
Character in 'C' format wrapped in pack
Solution was exactly as ikegami pointed out:
pack('C C n C*', 0x80, 0x78, $n, 0x12, 0x34, 0x98, 0x76, 0x01, 0x02, 0x03, 0x04);

The problem has nothing to do with pack or your use of it. The problem is that you are miscalculating $m and $n. The warning is the result of $m exceeding 0xFF, which clearly should not happen.
The following properly calculates $m and $n (which I renamed to $hi and $lo for reasons that will soon become obvious):
my $hi = 0x00;
my $lo = 0x00;
while ( <$source_audio> ) {
my $header = pack( 'C*',
0x80, 0x78, $hi, $lo, 0x12, 0x34, 0x98, 0x76, 0x01, 0x02, 0x03, 0x04 );
...
if ( $lo == 0xFF ) {
$lo = 0;
++$hi;
} else {
++$lo;
}
}
This could also be written as follows:
$lo = ( $lo + 1 ) & 0xFF;
++$hi if !$lo;
But we're really just incrementing one number which is stored across two octets.
my $n = 0;
while ( <$source_audio> ) {
my $hi = $n >> 8;
my $lo = $n & 0xFF;
my $header = pack( 'C*',
0x80, 0x78, $hi, $lo, 0x12, 0x34, 0x98, 0x76, 0x01, 0x02, 0x03, 0x04 );
...
++$n;
}
But let's leave the packing to pack! We're trying to store the value as a 16 bit unsigned int using big-endian byte order. Consulting Formats for Packing and Unpacking Numbers or the documentation for pack shows we can use S> or n.
my $n = 0;
while ( <$source_audio> ) {
my $header = pack( 'C C n C*',
0x80, 0x78, $n, 0x12, 0x34, 0x98, 0x76, 0x01, 0x02, 0x03, 0x04 );
...
++$n;
}

Related

Perl printf line breaks

Looking to add a line break after every 16 bytes of the 100 bytes printed from j. All 100 bytes are currently printed on one line at the moment.
Any assistance would be great full, thanks.
for ($j = 6; $j < 106; $j++)
{
printf("%02X ", hex(#bytes[$j]));
}
printf("\t");
Thanks all. I approached it another way. In the end the visual aspect was not a concern as I ended up printing to file anyway. This data then gets pasted in a hex editor so formatting was a forethought rather than a need be. The below is now outputting all the data I was looking for in the end rather than a predetermined length.
for ($j = 6; $j < #bytes; $j = $j + 1)
{
printf outfile ("%02X ", hex(#bytes[$j]));
}
printf("\n");
Use splice() to grab 16 elements at a time from your array.
I've also switched to using say(), join(), map() and sprintf().
#!/usr/bin/perl
use strict;
use warnings;
use feature 'say';
my #bytes = map { int(rand(255)) } 0..200;
# As splice is destructive, take a copy of the array.
# Also, let's remove those first six elements and any
# after index 106.
my #data = #bytes[6 .. 105];
while (my #row = splice #data, 0, 16) {
say join ' ', map { sprintf '%02X', $_ } #row;
}
Update: A few more Perl tips for you.
Always add use strict and use warnings to the top of your Perl programs. And fix the problems they will show you.
#bytes[$j] is better written as $bytes[$j] (as it's a single value). That's one of the things use warnings will tell you about.
for ($j = 6; $j < 106; $j++) is probably better written as for my $j (6 .. 105). Less chance of "off-by-one" errors.
Actually, as you're just using $j to get the element from the array, you probably want to just iterate across the elements of the array directly - for my $elem (#bytes[6 .. 105]) (and then using $elem in place of $bytes[$j]).
You can utilize some $counter and print \n once $counter % 16 equal 0.
use strict;
use warnings;
my #bytes = map { int(rand(255)) } 0..200;
my $counter = 1;
for ( #bytes[6..106] ) {
printf "%02X ", $_;
print "\n" unless $counter++ % 16;
}
print "\n";
Output sample
C9 CA E7 66 13 F5 56 BE 08 68 E4 22 93 77 E0 14
08 4F F3 AD CC F4 66 DE 6C BB 1B E6 CE F3 13 DD
AE 6A CD 9B 5E 98 1F D4 2E C5 80 4B 3E 8E BC BF
5B 27 F9 0D 97 AB 26 C0 11 2D 1D 95 CE 26 3C D8
3C D8 A4 06 0A 48 0D 45 53 28 7E 5D D2 AD 90 5C
03 32 95 48 F6 DB 20 90 A7 62 41 3D D7 AB 7C 3B
CF 3D 0D C2 DA
Note: index from 6 to 106 gives 101 element
Short and fast:
sub hex_dump {
my $i = 0;
my $d = uc unpack 'H*', pack 'C*', #_;
$d =~ s{ ..\K(?!\z) }{ ++$i % 16 ? " " : "\n" }xseg;
return "$d\n";
}
print hex_dump(#bytes[6..105]);
There's also the very clean splice loop approach.
sub hex_dump {
my $d = '';
$d .= join(" ", map sprintf("%02X", $_), splice(#_, 0, 16)) . "\n" while #_;
return $d;
}
print hex_dump(#bytes[6..105]);
Because this approach produces a line at a time, it's great for recreating the traditional hex dump format (with offset and printable characters shown).
$ hexdump -C a.pl
00000000 75 73 65 20 73 74 72 69 63 74 3b 0a 75 73 65 20 |use strict;.use |
00000010 77 61 72 6e 69 6e 67 73 3b 0a 0a 73 75 62 20 68 |warnings;..sub h|
00000020 65 78 5f 64 75 6d 70 20 7b 0a 20 20 20 6d 79 20 |ex_dump {. my |
...
000000a0 74 65 73 20 3d 20 30 2e 2e 32 35 35 3b 0a 0a 70 |tes = 0..255;..p|
000000b0 72 69 6e 74 20 68 65 78 5f 64 75 6d 70 28 40 62 |rint hex_dump(#b|
000000c0 79 74 65 73 5b 36 2e 2e 31 30 35 5d 29 3b 0a |ytes[6..105]);.|
000000cf
You could also use a for loop. You would iterate over the indexes, and print either a space or a line feed depending on the index of the byte you're printing.
sub hex_dump {
my $d = '';
for my $i (0..$#_) {
$d .= sprintf("%02X%s", $_[$i], $i == $#_ || $i % 16 == 15 ? "\n" : " ");
}
return $d;
}
print hex_dump(#bytes[6..105]);

Problems with sending UDP packets (milight, limitlessled)

I trying to write a menubar app to get control over my lights via Mac.
I'm using the system of milight (limitless, easybulbs...).
They have an open system were you can send commands via UDP.
I'm able to control my lights via python-limitless library in python, so I know the networking thing such as IP and port is right.
So I think I do anything wrong with this UDP stuff I never worked with.
I'm trying to use SwiftSocket library to send my commands but nothing happens, I've been trying it since 2 days.
Here ist what I'm trying:
let host = "192.168.2.102"
let port = 5987
var client: UDPClient!
#IBAction func lightOn(_ sender: NSButton) {
let bridgeon: [UInt8] = [0x31, 0x00, 0x00, 0x00, 0x03, 0x03, 0x00, 0x00, 0x00, 0x01]
let rgbwon: [UInt8] = [0x31, 0x00, 0x00, 0x07, 0x03, 0x01, 0x00, 0x00, 0x00, 0]
print("Licht an")
print(client.send(data: bridgeon))
sleep(1)
print(client.send(data: rgbwon))
sleep(1)
}
#IBAction func lightOff(_ sender: NSButton) {
print("Licht aus")
}
override func viewDidLoad() {
super.viewDidLoad()
client = UDPClient(address: host, port: Int32(port))
}
When I compare this with the complexity of the pythonlibrary I'm sure I forget something important. I haven't worked with networks yet so be lenient with me.
thanks and greetings.
I'm a bit late, but I hope it can help you :
Before sending your lighton request you have to send a first request to get, the wifi bridge session. You also need to compute what Milight called the "checksum" based on your request.
You also make sure about what kind of bulb you have, is it WW bulb or CW? I was stuck for severals days because I was sending wrong request..
I made an implementation, It's in php but you can use it same way in Objective-C.
Check it out : https://github.com/winosaure/MilightAPI
UPDATE :
According to limitlessled "documentation" (http://www.limitlessled.com/dev/) this is how a request is composed :
UDP Hex Send Format: 80 00 00 00 11 {WifiBridgeSessionID1}
{WifiBridgeSessionID2} 00 {SequenceNumber} 00 {COMMAND} {ZONE NUMBER}
00 {Checksum}
This is why you must get wifibridge session first then you need to calculate the checksum.
Let me take one example about how to turn on the light.
The documentation says that :
31 00 00 08 04 01 00 00 00 = Light ON
31 00 00 08 04 01 00 00 00 refer to {COMMAND} above.
So far the full request must be :
80 00 00 00 11 {WifiBridgeSessionID1} {WifiBridgeSessionID2} 00 {SequenceNumber} 00 31 00 00 08 04 01 00 00 00 {ZONE NUMBER} 00 {Checksum}
Now let's get the Wifibridge session. The documention says :
to get the WifiBridgeSessionID1 and WifiBridgeSessionID2 send this
command UDP.
SEND hex bytes: 20 00 00 00 16 02 62 3A D5 ED A3 01 AE 08
2D 46 61 41 A7 F6 DC AF (D3 E6) 00 00 1E <-- Send this to the ip
address of the wifi bridge v6
That's why I'm doing this:
private function getWifiBridgeSession()
{
$command = array (
0x20,0x00, 0x00,
0x00, 0x16, 0x02,
0x62, 0x3A, 0xD5,
0xED, 0xA3, 0x01,
0xAE, 0x08, 0x2D,
0x46, 0x61, 0x41,
0xA7, 0xF6, 0xDC,
0xAF, 0xD3, 0xE6,
0x00, 0x00, 0x1E);
return $this->sendCommand($command);
}
Once you send a UDP request with this command, you will get a result.
The Wifi Bridge session1 refers to the 20th byte of the response and WifiBridge Session2 will refer to the 21th byte response (Don't forget that we start to count from 0, so you must take something like "response[19]" and "response[20]").
Let's say, after sending this request I get this response :
28 00 00 00 11 00 02 AC CF 23 F5 7A D4 69 F0 3C 23 00 01 05 00
So my "WifiBridgesession1" is 0x05 and "Wifibridgesession2" is 0x00
So now our request to "turn on" the light is :
80 00 00 00 11 0x05 0x00 00 {SequenceNumber} 00 31 00 00 08 04 01 00 00 00 {ZONE NUMBER} 00 {Checksum}
So now we need to find out {SequenceNumber} {Zone Number} and {Checksum}
What is a "Sequence Number"?
The doc says :
Sequential byte just helps with keeping commands in the correct order,
and it helps to ignore duplicate packets already received. increment
this byte for each new command by 1.
So put what you want and increase this value to 1 for each request. (Personnally I always send 0x01).
"Zone number" refers to which zone you synchronized your light.
Valid List for {ZONE NUMBER} 0x00 All 0x01 Zone1 0x02 Zone2 0x03
Zone3 0x04 Zone4
Let's say, our "zone" is 0x01.
Almost done. we just need now to calculate the "checksum".
The doc says :
take the 9 bytes of the command, and 1 byte of the zone, and add the 0
= the checksum = (checksum & 0xFF) e.g. SUM((31 00 00 08 04 01 00 00 00)(command) 01(zone) 00) = 3F(chksum)
So the checksum for our command is :
31+00+00+08+04+01+00+00+00+01+00 = 0x54
I add all byte of the command (turn on) + 0x01 for the zone + 0x00
So now we have everything and the full request to turn on the light is :
80 00 00 00 11 05 00 00 01 00 31 00 00 08 04 01 00 00 00 01 00 54
That's it.
Note : Do not just copy and paste the request, I calculated the value based on example, the request to turn on the light will change each time, based on what you will calculate.
Maybe you got noticed that I wrote "00 31 00 00 08 04 01 00 00 00" to do the "turn on" command, this will work only for CW bulb. The doc does not specify that...
The same Command for WW bulb is 00 31 00 00 07 03 01 00 00 00
So the full command for WW bulb will be :
80 00 00 00 11 05 00 00 01 00 31 00 00 07 03 01 00 00 00 01 00 54
What is the difference between CW and WW bulb?
I can tell that CW refers to "Cold White" and WW to "Warm White". But as I am not an expert in "led bulb" I cannot explain more, I don't know why we need to write a different request for both, either.
Anyway I wish I was clear enough.
Let me know how things are working.

How to decode hex into separate bits in Perl?

A byte stored as two hex numbers contains a set of flags. I need to extract these flags, as 0's and 1's. Iterating over entries in a file with:
foreach(<>)
{
#line = split(/ /,$_);
$torR = !!((hex $line[4]) & 0x3); # bit 0 or 1
$torY = !!((hex $line[4]) & 0x4); # bit 2
$torG = !!((hex $line[4]) & 0x8); # bit 3
print "$torR,$torY,$torG\n";
}
run on data file:
796.129 [1f/01] len:7< 02 01 D5 01 8B 0A 8E
796.224 [1f/01] len:7< 02 01 D4 03 09 A9 B8
796.320 [1f/01] len:7< 00 01 D4 03 07 49 5A
796.415 [1f/01] len:7< 00 01 D4 00 11 A0 EE
796.515 [1f/01] len:7< 00 01 D4 00 00 31 4C
796.627 [1f/01] len:7< 02 01 D4 01 89 C1 FD
796.724 [1f/01] len:7< 02 01 D3 03 06 39 FD
796.820 [1f/01] len:7< 08 01 D4 03 08 40 6F
796.915 [1f/01] len:7< 08 01 D5 00 13 3D A4
797.015 [1f/01] len:7< 08 01 D4 00 00 34 04
Actual Result -
1,,
1,,
,,
,,
,,
1,,
1,,
,,1
,,1
,,1
Desired result:
1,0,0
1,0,0
0,0,0
0,0,0
0,0,0
1,0,0
1,0,0
0,0,1
0,0,1
0,0,1
Seems like 'false' gets stored as empty string instead of '0'.
Is there a neat trick to get this right at once, or do I need to convert empty strings to zeros "manually"?
If you want the true/false values to be numeric, you need to coerce them to be numeric:
$torR = 0 + !!((hex $line[4]) & 0x3); # bit 0 or 1
$torY = 0 + !!((hex $line[4]) & 0x4); # bit 2
$torG = 0 + !!((hex $line[4]) & 0x8); # bit 3
Keep in mind that the empty string '' is also a false value.
On the other hand, I might be inclined to write that as:
my (#ryg) = map 0 + !!((hex $line[4]) & $_), 0x3, 0x4, 0x5;
print join(', ', #ryg), "\n";
In addition, you would probably benefit from not using plain numbers in your program. Consider, for example, having a %FLAG structure that gives names to these constants, and a %COL structure that gives names to the columns you are interested in. Using the data you posted:
use Const::Fast;
const my %FLAG => (
torR => 0x3,
torY => 0x4,
torG => 0x5,
);
const my %COL => (
# ...
tor => 4,
);
while (my $line = <DATA>) {
my #line = split ' ', $line;
my %set_flags = map +($_ => 0 + !!((hex $line[$COL{tor}]) & $FLAG{$_})), qw(torR torY torG);
print join(', ', #set_flags{qw(torR torY torG)}), "\n";
}
__DATA__
796.129 [1f/01] len:7< 02 01 D5 01 8B 0A 8E
796.224 [1f/01] len:7< 02 01 D4 03 09 A9 B8
796.320 [1f/01] len:7< 00 01 D4 03 07 49 5A
796.415 [1f/01] len:7< 00 01 D4 00 11 A0 EE
796.515 [1f/01] len:7< 00 01 D4 00 00 31 4C
796.627 [1f/01] len:7< 02 01 D4 01 89 C1 FD
796.724 [1f/01] len:7< 02 01 D3 03 06 39 FD
796.820 [1f/01] len:7< 08 01 D4 03 08 40 6F
796.915 [1f/01] len:7< 08 01 D5 00 13 3D A4
797.015 [1f/01] len:7< 08 01 D4 00 00 34 04
I think I would use split and unpack to turn each value into an array of zeroes and ones, and then examine them individually
use strict;
use warnings 'all';
for my $val ( qw/ 02 02 00 00 00 01 01 08 08 08 / ) {
my #bits = split //, unpack 'b8', chr hex $val;
my $torR = $bits[0] || $bits[1] ? 1 : 0;
my $torY = $bits[2] ? 1 : 0;
my $torG = $bits[3] ? 1 : 0;
print "$torR,$torY,$torG\n";
}
output
1,0,0
1,0,0
0,0,0
0,0,0
0,0,0
1,0,0
1,0,0
0,0,1
0,0,1
0,0,1
Or here's a way using the Bit::Vector which produces the same result
use strict;
use warnings 'all';
use Bit::Vector;
for my $val ( qw/ 02 02 00 00 00 01 01 08 08 08 / ) {
my $vec = Bit::Vector->new_Hex(8, $val);
my $torR = $vec->Chunk_Read(2, 0) ? 1 : 0;
my $torY = $vec->Chunk_Read(1, 2) ? 1 : 0;
my $torG = $vec->Chunk_Read(1, 3) ? 1 : 0;
print "$torR,$torY,$torG\n";
}

HMAC implementation failure

I hope this is the right forum; I was not sure if I should ask this in stackoverflow, cryptography or security.
So my problem is that php's hash_hmac function is only available with php >=5.1.2. Because some servers are not updated to this version I wrote my own HMAC-implementaion based on php's hash function. But the code doesn't produce the same output as hash_hmac...
So where is my mistake?
define("HASH_ALGO", "sha512");
define("HMAC_BLOCKSIZE", 64);
function computeHMAC($message, $key) {
$ikey;
$okey;
$zero = hex2bin("00");
$ipad = hex2bin("36");
$opad = hex2bin("5C");
/*
* HMAC construction scheme:
* $ikey = $key padded with zeroes to blocksize and then each byte xored with 0x36
* $okey = $key padded with zeroes to blocksize and then each byte xored with 0x5C
* hash($okey . hash($ikey . $message))
*/
//Hash key if it is larger than HMAC_BLOCKSIZE
if (strlen($key) > HMAC_BLOCKSIZE) {
$key = hash(HASH_ALGO, $key, true);
}
//Fill ikey with zeroes
for ($i = 0; $i < HMAC_BLOCKSIZE; $i++) {
$ikey[$i] = $zero;
}
//Fill ikey with the real key
for ($i = 0; $i < strlen($key); $i++) {
$ikey[$i] = $key[$i];
}
//Until they get xored both keys are equal
$okey = $ikey;
//Xor both keys
for ($i = 0; $i < HMAC_BLOCKSIZE; $i++) {
$ikey[$i] ^= $ipad;
$okey[$i] ^= $opad;
}
//Build inner hash
$innerHash = hash(HASH_ALGO, $ikey . $message, true);
//Build outer hash
$outerHash = hash(HASH_ALGO, $okey . $innerHash, true);
return $outerHash;
}
The function was tested with the following code:
echo hexDump(computeHMAC("Testolope", "Testkeyolope"));
echo hexDump(hash_hmac(HASH_ALGO, "Testolope", "Testkeyolope", true));
The output is the following:
HexDump (64 Bytes):
65 a8 81 af 49 f2 49 c5 64 7a 7a b7 a6 ac a0 4e 9e 9b 1a 3c 76 fc 48 19 13 33 e0 f8 82 be 48 52 1a 50 49 09 1e fe bf 94 63 5f 9d 36 82 3f 2f a1 43 b4 60 9f 9f e5 d1 64 c6 5b 32 22 45 07 c9 cb
HexDump (64 Bytes):
d2 e9 52 d2 ab f0 db a7 60 e0 52 b0 5c 23 5a 73 d9 8c 78 8e 9e fb 26 82 54 7e f9 c8 f1 65 df 7f 97 44 fe 2b 1e 2b 6d d5 cb a4 ba c6 73 35 06 9c 0f c8 2d 36 8c b3 9b c4 48 01 5c c2 9f ce b4 08
The problem is that you've mixed up the digest size and block size; SHA-512 has a digest size of 64, but a block size of 128.
Secondly, both $ikey and $okey are arrays and not strings, so you need to convert them both into a string first:
$innerHash = hash(HASH_ALGO, join($ikey) . $message, true);
$outerHash = hash(HASH_ALGO, join($okey) . $innerHash, true);
That said, both hash() and hash_hmac() are documented as being available since 5.1.2, so I'm not sure what this will achieve :)

Alesis QS MIDI Sysex Data Conversion

My aim is to convert a stream of byte code sent from an Alesis synthesizer to a human readable format. I need to be able to take a "Program Dump" and read the 10 character string that makes up the patch name.
In order to receive the "Program Dump" from the synth, I sent the synth the following command via MIDI-OX:
F0 00 00 0E 0E 01 73 F7
I requested that it send me the dump for program 73.
I received this:
F0 00 00 0E 0E 00 73 00 60 24 0B 27 27 01 64 1E 19 19 05 23 19 1E 2A 41 0D 23 46 19 1E 06 00 47 0D 23 30 6C 18 63 30 6C 18 40 3F 0A 67 1B 16 20 40 00 60 18 00 18 06 05 0C 2B 41 13 70 05 30 40 31 63 70 05 00 40 31 63 70 05 00 40 31 63 00 4C 2A 51 00 46 7F 78 18 40 0F 40 31 40 31 04 30 0C 00 30 6C 03 30 3C 0F 00 00 05 0A 0F 14 19 1E 23 28 2D 72 00 76 34 3C 54 42 19 46 0C 33 3C 0C 00 0E 1B 46 60 58 31 46 61 58 31 00 7F 14 4E 37 6C 74 13 00 40 31 00 30 0C 0A 18 56 02 27 60 0B 60 00 63 46 61 0B 00 00 63 46 61 0B 00 00 63 46 01 18 55 22 01 0C 7F 71 31 00 1F 00 63 00 63 08 60 18 00 60 58 07 60 18 1E 00 00 0A 14 1E 28 32 3C 46 50 5A 64 01 0C 2D 15 29 05 36 0C 19 66 78 18 00 1C 36 0C 41 31 63 0C 43 31 63 00 7E 29 1C 6F 58 00 01 02 00 63 00 60 18 14 30 2C 05 4E 40 17 40 01 46 0D 43 17 00 00 46 0D 43 17 00 00 46 0D 03 30 2A 45 02 18 7E 63 63 00 3E 00 46 01 46 11 40 31 00 40 31 0F 40 71 3D 00 00 14 28 3C 50 64 78 0C 21 35 49 03 58 4C 71 31 1C 6C 18 32 4C 71 31 00 38 6C 18 02 63 46 19 06 63 46 01 7C 53 00 60 18 53 37 6C 70 0D 03 40 31 28 60 58 0A 1C 01 2F 00 03 0C 1B 06 2F 00 00 0C 1B 06 2F 00 00 0C 1B 06 60 54 0A 05 30 7C 47 47 01 7C 00 0C 03 0C 23 00 63 00 00 63 1E 3C 63 18 00 00 28 50 78 20 49 71 19 42 6A 12 07 F7
MIDI-OX told me that it received 408 bytes.
This jives with the specification:
"There are 400 data bytes sent for a single program dump, which corresponds to 350
bytes of program data. With the header, the total number of bytes transmitted with
a program dump is 408. The location of each parameter within a program dump is
shown in the next section."
The "Program Dump" should be composed of these values:
F0 00 00 0E 0E 00 <program#> <data> F7
That means the data should begin with "00 60" and end with "07 F7".
Now I should be able to convert these 400 bytes to the "350 bytes of packed parameter data" for this program. Following the "Program Data Format", 10 digits of the program name should be located within the packed data somewhere. Patch 73 is called either "BlowDeTune" or "PanBristle", not totally sure if it starts at 0 or 1.
So how do you go about make the conversion? Page 1 of the specification gives the transmission format, but I don't understand how to unpack it.
Can anyone help?
The Alesis QS MIDI Sysex Specification is here:
http://www.midiworld.com/quadrasynth/qs_swlib/qs678r.pdf
MIDI-OX can be found here:
http://www.midiox.com/
You are lucky, because some years ago I played a bit with Midi (with my Atari ST 520) so I had enough interest in the topic to investigate a bit...
For the record, I found the MIDI System Exclusive Message format, in accordance with the reference you give for your synth.
I first thought the packing algorithm was described in this page, but after implementing its decoding and founding garbage, I saw I was wrong... I will give this code just in case it could be useful to you elsewhere...
This first try was useful because when I re-read the spec in the PDF file, I understood the A7 to G0 symbols were actually bits...
Data is "packed" because Midi non-control words must be 7bit clean (high bit always unset).
They take 7 bytes of raw data, see it as a big word of 56 bits, and split that word every 7 bits, leaving the high bit always at 0:
Original data (using a different notation):
0 - b07 b06 b05 b04 b03 b02 b01 b00
1 - b17 b16 b15 b14 b13 b12 b11 b10
2 - b27 b26 b25 b24 b23 b22 b21 b20
3 - b37 b36 b35 b34 b33 b32 b31 b30
4 - b47 b46 b45 b44 b43 b42 b41 b40
5 - b57 b56 b55 b54 b53 b52 b51 b50
6 - b67 b66 b65 b64 b63 b62 b61 b60
Transmitted/encoded data:
0 - 0 b06 b05 b04 b03 b02 b01 b00
1 - 0 b15 b14 b13 b12 b11 b10 b07
2 - 0 b24 b23 b22 b21 b20 b17 b16
3 - 0 b33 b32 b31 b30 b27 b26 b25
4 - 0 b42 b41 b40 b37 b36 b35 b34
5 - 0 b51 b50 b47 b46 b45 b44 b43
6 - 0 b60 b57 b56 b55 b54 b53 b52
7 - 0 b67 b66 b65 b64 b63 b62 b61
So we have:
0 - 00000000 0x00
1 - 01100000 0x60
2 - 00100100 0x24
3 - 00001011 0x0B
4 - 00100111 0x27
5 - 00100111 0x27
6 - 00000001 0x01
7 - 01100100 0x64
0 - 00011110 0x1E
1 - 00011001 0x19
2 - 00011001 0x19
3 - 00000101 0x05
4 - 00100011 0x23
5 - 00011001 0x19
6 - 00011110 0x1E
7 - 00101010 0x2A
and once decoded, we should have:
0 - 00000000 0x00
1 - 00110000 0x30
2 - 01101001 0x69
3 - 01110001 0x71
4 - 00111010 0x3A
5 - 00000101 0x05
6 - 11001000 0xC8
0 - 10011110 0x9E
1 - 01001100 0x4C
2 - 10100110 0xA6
3 - 00110000 0x30
4 - 11001010 0xCA
5 - 01111000 0x78
6 - 01010100 0x54
I believe I decoded correctly the data, but still have garbage (ie. non readable strings)...
Perhaps you will see a logic error in my code, which might be a starting point anyway.
I saw that MIDI-OX can be scripted with WSH, so I wrote a JS script which I ran with WSH, with output on console:
var midiData =
[
0xF0, 0x00, 0x00, 0x0E, 0x0E, 0x00, 0x73,
0x00, 0x60, 0x24, 0x0B, 0x27, 0x27, 0x01, 0x64, 0x1E, 0x19, 0x19, 0x05, 0x23, 0x19, 0x1E, 0x2A,
0x41, 0x0D, 0x23, 0x46, 0x19, 0x1E, 0x06, 0x00, 0x47, 0x0D, 0x23, 0x30, 0x6C, 0x18, 0x63, 0x30,
0x6C, 0x18, 0x40, 0x3F, 0x0A, 0x67, 0x1B, 0x16, 0x20, 0x40, 0x00, 0x60, 0x18, 0x00, 0x18, 0x06,
0x05, 0x0C, 0x2B, 0x41, 0x13, 0x70, 0x05, 0x30, 0x40, 0x31, 0x63, 0x70, 0x05, 0x00, 0x40, 0x31,
0x63, 0x70, 0x05, 0x00, 0x40, 0x31, 0x63, 0x00, 0x4C, 0x2A, 0x51, 0x00, 0x46, 0x7F, 0x78, 0x18,
0x40, 0x0F, 0x40, 0x31, 0x40, 0x31, 0x04, 0x30, 0x0C, 0x00, 0x30, 0x6C, 0x03, 0x30, 0x3C, 0x0F,
0x00, 0x00, 0x05, 0x0A, 0x0F, 0x14, 0x19, 0x1E, 0x23, 0x28, 0x2D, 0x72, 0x00, 0x76, 0x34, 0x3C,
0x54, 0x42, 0x19, 0x46, 0x0C, 0x33, 0x3C, 0x0C, 0x00, 0x0E, 0x1B, 0x46, 0x60, 0x58, 0x31, 0x46,
0x61, 0x58, 0x31, 0x00, 0x7F, 0x14, 0x4E, 0x37, 0x6C, 0x74, 0x13, 0x00, 0x40, 0x31, 0x00, 0x30,
0x0C, 0x0A, 0x18, 0x56, 0x02, 0x27, 0x60, 0x0B, 0x60, 0x00, 0x63, 0x46, 0x61, 0x0B, 0x00, 0x00,
0x63, 0x46, 0x61, 0x0B, 0x00, 0x00, 0x63, 0x46, 0x01, 0x18, 0x55, 0x22, 0x01, 0x0C, 0x7F, 0x71,
0x31, 0x00, 0x1F, 0x00, 0x63, 0x00, 0x63, 0x08, 0x60, 0x18, 0x00, 0x60, 0x58, 0x07, 0x60, 0x18,
0x1E, 0x00, 0x00, 0x0A, 0x14, 0x1E, 0x28, 0x32, 0x3C, 0x46, 0x50, 0x5A, 0x64, 0x01, 0x0C, 0x2D,
0x15, 0x29, 0x05, 0x36, 0x0C, 0x19, 0x66, 0x78, 0x18, 0x00, 0x1C, 0x36, 0x0C, 0x41, 0x31, 0x63,
0x0C, 0x43, 0x31, 0x63, 0x00, 0x7E, 0x29, 0x1C, 0x6F, 0x58, 0x00, 0x01, 0x02, 0x00, 0x63, 0x00,
0x60, 0x18, 0x14, 0x30, 0x2C, 0x05, 0x4E, 0x40, 0x17, 0x40, 0x01, 0x46, 0x0D, 0x43, 0x17, 0x00,
0x00, 0x46, 0x0D, 0x43, 0x17, 0x00, 0x00, 0x46, 0x0D, 0x03, 0x30, 0x2A, 0x45, 0x02, 0x18, 0x7E,
0x63, 0x63, 0x00, 0x3E, 0x00, 0x46, 0x01, 0x46, 0x11, 0x40, 0x31, 0x00, 0x40, 0x31, 0x0F, 0x40,
0x71, 0x3D, 0x00, 0x00, 0x14, 0x28, 0x3C, 0x50, 0x64, 0x78, 0x0C, 0x21, 0x35, 0x49, 0x03, 0x58,
0x4C, 0x71, 0x31, 0x1C, 0x6C, 0x18, 0x32, 0x4C, 0x71, 0x31, 0x00, 0x38, 0x6C, 0x18, 0x02, 0x63,
0x46, 0x19, 0x06, 0x63, 0x46, 0x01, 0x7C, 0x53, 0x00, 0x60, 0x18, 0x53, 0x37, 0x6C, 0x70, 0x0D,
0x03, 0x40, 0x31, 0x28, 0x60, 0x58, 0x0A, 0x1C, 0x01, 0x2F, 0x00, 0x03, 0x0C, 0x1B, 0x06, 0x2F,
0x00, 0x00, 0x0C, 0x1B, 0x06, 0x2F, 0x00, 0x00, 0x0C, 0x1B, 0x06, 0x60, 0x54, 0x0A, 0x05, 0x30,
0x7C, 0x47, 0x47, 0x01, 0x7C, 0x00, 0x0C, 0x03, 0x0C, 0x23, 0x00, 0x63, 0x00, 0x00, 0x63, 0x1E,
0x3C, 0x63, 0x18, 0x00, 0x00, 0x28, 0x50, 0x78, 0x20, 0x49, 0x71, 0x19, 0x42, 0x6A, 0x12, 0x07,
0xF7
];
// Show original data
DumpData(midiData, 16);
var headerLength = 7; // Bytes to skip
var resultData = new Array();
var decodedByteCount = 0; // Number of expanded bytes in result
var cumulator = 0;
var bitCount = 0;
for (var i = headerLength; // Skip header
i < midiData.length - 1; // Omit EOF
i++)
{
var rank = (i - headerLength) % 8; // We split the data in runs of 8 bytes
// We cumulate the bits of these runs (less the high bit) to make a big word of 56 bits
/*
cumulator |= midiData[i] << (7 * rank);
if (rank == 7) // End of the run
{
// Split the cumulator in 7 bytes
for (var j = 0; j < 7; j++)
{
var shift = 8 * j;
var byte = (cumulator & (0xFF << shift)) >> shift;
WScript.StdOut.Write(ByteToHex(byte) + ' ');
resultData[decodedByteCount++] = byte;
}
cumulator = 0; // Reset the buffer
}
*/
// Actually, we cannot do that, because JS' bit arithmetic seems to be limited to signed 32 bits!
// So I get the bytes out as soon as they are complete.
// Somehow, it is more elegant anyway (but reflects less the original algorithm).
cumulator |= midiData[i] << bitCount;
bitCount += 7;
//~ WScript.StdOut.Write((i - 7) + ':' + ByteToHex(midiData[i]) + ' (' + bitCount + ') ' + DecimalToHex(cumulator) + '\n');
if (bitCount >= 8)
{
var byte = cumulator & 0xFF;
bitCount -= 8;
cumulator >>= 8;
resultData[decodedByteCount++] = byte;
//~ WScript.StdOut.Write((i - 7) + ':' + ByteToHex(midiData[i]) + ' (' + bitCount + ') ' + DecimalToHex(cumulator) + ' > ' + ByteToHex(byte) + '\n');
}
}
DumpData(resultData, 14);
The utility routines:
function DumpData(data, lineLength)
{
WScript.StdOut.Write("Found " + data.length + " bytes\n");
var txt = '';
for (var i = 0; i < data.length; i++)
{
var rd = data[i];
if (rd > 31)
{
txt += String.fromCharCode(rd);
}
else
{
txt += '.';
}
WScript.StdOut.Write(ByteToHex(rd) + ' ');
if ((i+1) % lineLength == 0)
{
WScript.StdOut.Write(' ' + txt + '\n');
txt = '';
}
}
WScript.StdOut.Write(' ' + txt + '\n');
}
function NibbleToHex(halfByte)
{
return String.fromCharCode(halfByte < 10 ?
halfByte + 48 : // 0 to 9
halfByte + 55); // A to F
}
function ByteToHex(dec)
{
var h = (dec & 0xF0) >> 4;
var l = dec & 0x0F;
return NibbleToHex(h) + NibbleToHex(l);
}
function DecimalToHex(dec)
{
var result = '';
do
{
result = ByteToHex(dec & 0xFF) + result;
dec >>= 8;
} while (dec > 0);
return result;
}
Output:
Found 350 bytes
00 30 69 71 3A 05 C8 9E 4C A6 30 CA 78 54 .0iq:.ÈL¦0ÊxT
C1 C6 C8 98 F1 18 00 C7 C6 08 C6 C6 8C 61 ÁÆÈñ..ÇÆ.ÆÆa
6C 0C F0 A7 38 6F 2C 20 20 00 8C 01 60 0C l.ð§8o, ..`.
05 C6 2A 38 81 17 60 C0 D8 18 5E 00 00 63 .Æ*8.`ÀØ.^..c
63 78 01 00 8C 8D 01 4C 55 14 60 FC E3 31 cx...LU.`üã1
C0 07 30 06 8C 11 60 0C 00 8C 3D 80 F1 1E À.0..`..=ñ.
00 40 41 F1 A0 64 3C 23 54 4B 0E B0 D3 78 .#Añ d<#TK.°Óx
54 61 C6 C8 98 F1 18 00 C7 C6 08 C6 C6 8C TaÆÈñ..ÇÆ.ÆÆ
61 6C 0C F0 A7 38 6F 6C FA 04 00 8C 01 60 al.ð§8olú...`
0C 05 C6 2A 38 81 17 60 C0 D8 18 5E 00 00 ..Æ*8.`ÀØ.^..
63 63 78 01 00 8C 8D 01 4C 55 14 60 FC E3 ccx...LU.`üã
31 C0 07 30 06 8C 11 60 0C 00 8C 3D 80 31 1À.0..`..=1
1E 00 40 41 F1 A0 64 3C 23 54 4B 0E 30 5A ..#Añ d<#TK.0Z
95 54 C1 C6 C8 98 F1 18 00 C7 C6 08 C6 C6 TÁÆÈñ..ÇÆ.ÆÆ
8C 61 6C 0C F0 A7 38 6F 2C 20 20 00 8C 01 al.ð§8o, ..
60 0C 05 C6 2A 38 81 17 60 C0 D8 18 5E 00 `..Æ*8.`ÀØ.^.
00 63 63 78 01 00 8C 8D 01 4C 55 14 60 FC .ccx...LU.`ü
E3 31 C0 07 30 06 8C 11 60 0C 00 8C 3D 80 ã1À.0..`..=
F1 1E 00 40 41 F1 A0 64 3C 23 54 4B 0E B0 ñ..#Añ d<#TK.°
CC 78 8C C3 C6 C8 98 F1 18 00 C7 C6 08 C6 ÌxÃÆÈñ..ÇÆ.Æ
C6 8C 61 6C 0C F0 A7 00 30 66 7A 63 C3 1B Æal.ð§.0fzcÃ.
03 60 0C 05 C6 2A 38 81 17 60 C0 D8 18 5E .`..Æ*8.`ÀØ.^
00 00 63 63 78 01 00 8C 8D 01 4C 55 14 60 ..ccx...LU.`
FC E3 31 C0 07 30 06 8C 11 60 0C 00 8C 3D üã1À.0..`..=
BC 31 06 00 40 41 F1 A0 64 3C 23 54 4B 0E ¼1..#Añ d<#TK.
And just in case, the other unpacking algorithm:
// Here the 8 bits of 7 bytes of raw data are coded as 7 bytes of data stripped off of the high bit,
// while the stripped bits are grouped in the first byte of the data run.
// In other words, when we have a run of 8 bytes, the first one groups the high bits of the 7 next bytes.
// Information found at http://crystal.apana.org.au/ghansper/midi_introduction/file_dump.html
var headerLength = 7;
var resultData = new Array();
var decodedByteCount = 0; // Number of expanded bytes in result
var runCount = -1; // Number of runs in the encoded data
for (var i = headerLength; // Skip header
i < midiData.length - 1; // Omit EOF
i++)
{
var rank = (i - headerLength) % 8; // We split the data in runs of 8 bytes
if (rank == 0) // Start of the run
{
// Get the high bits
var highBits = midiData[i];
runCount++;
//~ WScript.StdOut.Write(runCount + ' > ' + (i - 7) + ' >> ' + ByteToHex(highBits) + '\n');
}
else
{
resultData[decodedByteCount++] = midiData[i] |
((highBits & (1 << (7 - rank))) << rank);
//~ WScript.StdOut.Write((i - 7) + ' >> ' + ByteToHex(midiData[i]) + ' > ' +
//~ ByteToHex(midiData[i] | ((highBits & (1 << (7 - rank))) << rank)) + '\n');
}
}
Thanks to your great work I came up with this as the pack algorithm.
It seems that the Alesis uses the same schema as the Moog Voyager.
packSysex : function(midiData) {
var header = [0xF0, 0x04, 0x01, 0x00, 0x03, 0x00]; //Voyager Single Preset Dump.
var resultData = new Array();
var packedByteCount = 0;
var bitCount = 0;
var thisByte;
var packedByte;
var nextByte = 0x0;
for (var i = 0; i <= midiData.length; i++)
{
thisByte = midiData[i];
packedByte = ((thisByte << bitCount) | nextByte) & 0x7F;
nextByte = midiData[i] >> (7-bitCount);
resultData[packedByteCount++] = packedByte;
bitCount++;
if(bitCount >= 7) {
bitCount = 0;
//Fill last byte
packedByte = nextByte & 0x7F;
resultData[packedByteCount++] = packedByte;
nextByte = 0x0;
}
}
resultData[packedByteCount++] = 0xF7;
resultData = header.concat(resultData);
return resultData;
},