I am looking for a way to retrieve the minCode, maxCode and valPtr from an arbitrary Huffman table.
For instance, the following is a Huffman DC table generated by JpegSnoop:
Destination ID = 0
Class = 0 (DC / Lossless Table)
Codes of length 01 bits (000 total):
Codes of length 02 bits (001 total): 00
Codes of length 03 bits (005 total): 01 02 03 04 05
Codes of length 04 bits (001 total): 06
Codes of length 05 bits (001 total): 07
Codes of length 06 bits (001 total): 08
Codes of length 07 bits (001 total): 09
Codes of length 08 bits (001 total): 0A
Codes of length 09 bits (001 total): 0B
Codes of length 10 bits (000 total):
Codes of length 11 bits (000 total):
Codes of length 12 bits (000 total):
Codes of length 13 bits (000 total):
Codes of length 14 bits (000 total):
Codes of length 15 bits (000 total):
Codes of length 16 bits (000 total):
Total number of codes: 012
And the following are its Mincode, MaxCode and valPtr respectively:
{ 0, 0, 2, 14, 30, 62, 126, 254, 510, 0, 0, 0, 0, 0, 0, 0 },//YDC
{ -1, 0, 6, 14, 30, 62, 126, 254, 510, -1, -1, -1, -1, -1, -1, -1 },//YDC
{ 0, 0, 1, 6, 7, 8, 9, 10, 11, 0, 0, 0, 0, 0, 0, 0 },//YDC
Now I'm really confused about how these values were derived.
I checked the itu-t81 file, but it was not very clear.
To generate the code bits, you start with all zero bits. Within each code length, increment the code like an integer for each symbol. When stepping up a code length, increment and then add a zero bit to the end.
So for your example code, we have each length, followed by the corresponding codes in binary:
2: 00
3: 010, 011, 100, 101, 110
4: 1110
5: 11110
6: 111110
7: 1111110
8: 11111110
9: 111111110
Converting those to the corresponding integer ranges for each bit length, we have:
2: 0..0
3: 2..6
4: 14..14
5: 30..30
6: 62..62
7: 126..126
8: 254..254
9: 510..510
You can see exactly those ranges in your MinCode and MaxCode vectors.
You also have a list of symbols that correspond to the codes. In this example, that list is simply:
00 01 02 03 04 05 06 07 08 09 0A 0B
(The particular values of the symbols are not relevant to the valPtr vector. Those could be anything.)
The codes are assigned to the symbols from shortest to longest, and within each length, in integer order. The valPtr vector is simply the index of the first symbol in that vector that corresponds to each bit length. To generate the vector, start at zero, and add the number of symbols of each code length to get the starting index for the next code length.
1: 0, 0 symbols
2: 0 + 0 = 0, 1 2-bit symbol
3: 0 + 1 = 1, 5 3-bit symbols
4: 1 + 5 = 6, 1 4-bit symbol
5: 6 + 1 = 7, 1 5-bit symbol
6: 7 + 1 = 8, 1 6-bit symbol
7: 8 + 1 = 9, 1 7-bit symbol
8: 9 + 1 = 10, 1 8-bit symbol
9: 10 + 1 = 11
The valPtr example vector are the numbers after the equal signs above.
Thanks, I have created a code that decodes the tables and returns the desired values. The code may be found on my GitHub Here.
Related
I have read the PNG specifications too much times and still confused how I should interpret the IDAT chunk. I have it decompressed using zlib and got all of the bytes that my IDAT chunk got.
I made an example image using krita. It's an 3x2 PNG image containing a different color every pixel.
See the 3 by 2 PNG image here
According to the PNG specification about filters it says that when the first byte of the IDAT chunk is 1 the filter method that have been applied is
Filtered(byte) = Original(byte) - Original(previous_byte)
With that formula in mind I decompressed my IDAT chunk (which was 29 bytes in length to store only 6 pixels). The first byte (which is byte number 0) contains the value 1. That is where the formula comes from.
Byte# Vaue
0 1
1 224
2 215
3 200
4 227
5 241
6 48
7 2
8 36
9 225
10 1
11 253
12 255
13 195
14 245
15 182
16 244
17 232
18 245
19 57
20 0
21 0
22 0
23 0
24 0
25 0
26 0
27 0
28 0
The first pixel is supposed to be RGB(224, 215, 200) which I reconstructed with a RGB to color converter. This seems pretty much the same color as the original pixel in the image. Here are my thoughts about all the color pixels.
Pixel 1: RGB(224, 215, 200) [read from byte 1, byte2 and byte3]
Pixel 2: RGB(195, 200, 248) [because byte 4:227 byte5:241 byte6:48]
Pixel 3: RGB(197, 236, 217) [because byte 7:2 byte8:36 byte9:225]
Pixel 4: RGB(198, 233, 217) [because byte10:1 byte11:253 byte12:255]
Pixel 5: RGB(137, 222, 142) [because byte13:195 byte14:245 byte15:182]
Pixel 6: RGB(107, 198, 131) [because byte16:244 byte17:232 byte18:245]
I have used the formula to get all the values from the pixels.
Reconstructing pixel 1, 2 and 3 looks pretty much the same, but pixel 4, 5 and 6 are not what I have expected. I think I am not reading the IDAT chunk the correct way. That could explain why there are 29 bytes for only 6 pixels RGB. I expected 19 bytes because 3 times 6 is 18 and 1 byte for the filtering method.
The IHDR says that the bit depth is 8 and the color type is 2. From the table in the specifications it says that each pixel is an R, G and B triple. Could someone point me to the right direction to read the IDAT chunk and explain it's length?
Your decompressed result length of 29 is not correct, which may have lead to your confusion.
Your image is 3x2 RGB pixels. That would be 3*3 * 2 = 18 bytes of data, plus 1 extra byte per row; a total of 20 bytes. Somehow you got an extra 9 dummy bytes, not part of the compressed data.
(I reconstructed your tiny image from the larger one and happily got the exact same numbers, else the explanation would necessarily be purely theoretical. For ease, I determined the offset of the zipped data with a hex viewer.)
>>> with open ('3x2b.png','rb') as f:
... result = f.seek (0x6a)
... data = f.read()
...
>>> d = zlib.decompress(data)
>>> print ([x for x in d])
[1, 224, 215, 200, 227, 241, 48, 2, 36, 225, 1, 253, 255, 195, 245, 182, 244, 232, 245, 57]
This 'unpacks' to the following two rows, with 3 RGB pixel values each:
filter RGB RGB RGB
1 (224,215,200) (227,241,48) (2,36,225)
1 (253,255,195) (245,182,244, (232,245,57)
All these values may be relative to an earlier result: the last complete row read before it, or the pixel to its left. For the first row, you must assume a row of all zeroes; the value "left" of the first pixel must be assumed to be 0 as well.
You see the two bytes marked 'filter'? That is where you went wrong. Each row has a filter byte of its own. You used the filter byte itself for the calculation of the second row.
Adding (the inverse of the "Sub" filter as indicated by the filter 1) yields in
; start of row 0, filter is 1 and 'initial pixel' is (0,0,0)
(224,215,200) (224+227,215+241,200+48)
=(195,200,248)
(195+2,200+36,248+225)
=(197,236,217)
; restart for row 1, filter is 1 again and start value (0,0,0):
(253,255,195) (253+245,255+182,195+244)
=(242,181,183)
(242+232,181+245,183+57)
=(218,170,240)
... exactly the colors I started out with.
This is Filter 1 ("Sub") and so uses the values to its left; for Filter 2 ("Up"), you need to use the corresponding byte in the previously decoded row, and for Average and Paeth, you need both.
I'm stuck trying to format a MIDI Sys Ex message that a device keeps rejecting as invalid. The problem is a section of the message that involves a type of data encoding described below.
According to the manual,
the device "will encode/interpret a consecutive group of 4-bytes"
Byte #0 - b31b30b29b28b27b26b25b24
Byte #1 - b23b22b21b20b19b18b17b16
Byte #2 - b15b14b13b12b11b10b09b08
Byte #3 - b07b06b05b04b03b02b01b00
as the following 5 consecutive SysEx bytes:"
Byte #0 - 0 b06b05b04b03b02b01b00
Byte #1 - 0 b13b12b11b10b09b08b07
Byte #2 - 0 b20b19b18b17b16b15b14
Byte #3 - 0 b27b26b25b24b23b22b21
Byte #4 - 0 0 0 0 b31b30b29b28
where "b" is the bit number. Notice the bit numbering has been flipped. Which way are you supposed to read the bits? MIDI data is, by convention, reverse bit ordered (MSB=7), if that helps. Also, the manual notes that "all data types are in Motorola big-endian byte order."
Here's a description of the message I'm trying to format correctly -
"A command will allow a consecutive group of one to four bytes to be edited. When 3 or less bytes are specified the device expects the Parameter Value field to be bit ordered as if it was performing a full 32-bit (4 byte) parameter change. For example, when editing a two byte parameter, Byte #0 will occupy the bit range of b24-b31, while Byte #1 will occupy bits b16-b23. The remaining bits (b00-b15) in the parameter value field should be set to zero."
bb Parameter Offset - 0 b06b05b04b03b02b01b00
bb Parameter Offset - 0 b13b12b11b10b09b08b07
bb Parameter Offset - 0 b20b19b18b17b16b15b14
bb Parameter Offset - 0 b27b26b25b24b23b22b21
bb Parameter Offset - 0 0 0 0 b31b30b29b28
0b Parameter Byte Size (1 to 4)
00
00
00
00
bb Parameter Value - 0 b06b05b04b03b02b01b00
bb Parameter Value - 0 b13b12b11b10b09b08b07
bb Parameter Value - 0 b20b19b18b17b16b15b14
bb Parameter Value - 0 b27b26b25b24b23b22b21
bb Parameter Value - 0 0 0 0 b31b30b29b28
So, when trying to enter offset values of 15H, 16H, 17H, and 18H, with respective values of let's say 00, 01, 02, 03 respectively, how would I encode those hex values, or do I even need to encode them? If I do need to, which direction do I write the bits so the binary values are correct?
When written, the order of bits in a byte is always big-endian, i.e., MSB first.
This can be confirmed by the fact that the MSB must be zero for data bytes.
Four bytes:
15h -> 00010101
16h -> 00010110
17h -> 00010111
18h -> 00011000
SysEx bytes:
0 0011000 -> 18h
0 0101110 -> 2eh
0 1011000 -> 58h
0 0101000 -> 28h
0000 0001 -> 01h
Four bytes:
01 -> 00000001
02 -> 00000010
03 -> 00000011
04 -> 00000100
SysEx bytes:
0 0000100 -> 04h
0 0000110 -> 06h
0 0001000 -> 08h
0 0001000 -> 08h
0000 0000 -> 00h
The basic approach to this conversion is to combine the four 8-bit values into a 32-bit value and then successively move the least significant seven bits into five bytes. Here's a sample program that does what you need.
#include <stdio.h>
#include <stdint.h>
void convert(uint8_t byte0, uint8_t byte1, uint8_t byte2, uint8_t byte3, uint8_t *outBytes) {
uint32_t inBytes = byte0 << 24 | byte1 << 16 | byte2 << 8 | byte3; //combine the input bytes into a single 32-bit value
for (int i = 0; i < 5; i++) {
outBytes[i] = inBytes & 0x7F; //Copy the least significant seven bits into the next byte in the output array
inBytes >>= 7; //Shift right to discard the seven bits that were copied
}
}
void printByteArray(uint8_t *byteArray) {
for (int i = 0; i < 5; i++) {
printf("Byte %d: %02xh\n", i, byteArray[i]);
}
printf("\n");
}
int main(int argc, char *argv[]) {
uint8_t sysExBytes[5]; //Five bytes to contain the converted SysExData
convert(0x15, 0x16, 0x17, 0x18, sysExBytes);
printByteArray(sysExBytes);
convert(0x00, 0x01, 0x02, 0x03, sysExBytes);
printByteArray(sysExBytes);
return 0;
}
Output:
Byte 0: 18h
Byte 1: 2eh
Byte 2: 58h
Byte 3: 28h
Byte 4: 01h
Byte 0: 03h
Byte 1: 04h
Byte 2: 04h
Byte 3: 00h
Byte 4: 00h
Your question is a bit confusing. 1 byte is 8 bits, therefore the following lines:
Byte - #0 b31b30b29b28b27b26b25b24
Byte - #1 b23b22b21b20b19b18b17b16
Byte - #2 b15b14b13b12b11b10b09b08
Byte - #3 b07b06b05b04b03b02b01b00
Do not make sense to me. The first line #0 has 8 bytes in it. I did some searching and this is a better explanation (http://www.music.mcgill.ca/~ich/classes/mumt306/midiformat.pdf). Taking the contents from page 2 and editing them for clarity.
Offset | Byte 0 | Byte 1 | Byte 2 | Byte 3
-------- bits | 24-31 | 16-23 | 8-15 | 0-7
00000000 | 00 | | |
00000040 | 40 | | |
0000007F | 7F | | |
00000080 | 81 | 00 | |
00002000 | C0 | 00 | |
00003FFF | FF | 7F | | <--- example
00004000 | 81 | 80 | 00 |
00100000 | C0 | 80 | 00 |
001FFFFF | FF | FF | 7F |
00200000 | 81 | 80 | 80 | 00
08000000 | C0 | 80 | 80 | 00
0FFFFFFF | FF | FF | FF | 7F
Example at offset 3FFF
Hex format 0xFF7F_0000 (32-bit number, unused bits are 0)
31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16
1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1
15 14 13 12 11 10 09 08 07 06 05 04 03 02 01 00
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Does this help?
I have been working on viterbi decoder in matlab2009 on simple 1/2 rate convolutional encoder.
Here is my code
trel = poly2trellis(3,[7 5]);
msg = [ 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1 ];
code = convenc(msg,trel);
% Traceback Length
tblen = 5;
ucode = real(awgn(1-2*code,tblen,'measured'));
dcd = vitdec(ucode,trel,tblen,'cont','unquant');
According to this input code
i am getting the code = 00 11 10 00 01 10 01 11 11 10 00 10 11 00 11
which is correct
but talking about the dcd which is output after viterbi decoder is coming incorrect
i.e 000000101110010. which is far different from my msg input.
guide me where i am going incorrect
The decoded output depends on the type of opmode Input you selected.
In case of cont, there is a delay in the output equal to tblen number of symbols whereas there are 'term' and trunc modes as well.
You can compare the initial msg(1,end-tblen) symbols with dcd(1,tblen+1:end). They are same!
You may check vitdec at Matlab help.
I have a (maybe simple) question about Matlab data import. I want to import a huge dataset (~1GB) which has a comma separated format like this:
08:05, 12, 33, 124, 13, 08:06, 22, 84, 12, 35, ..
Every 5th value is a timestamp. I want to import it with a fixed numbers of colums (5 colums), but there is no delimiter for the end of row. It should look like this in the end:
08:05 12 33 124 13
08:06 22 14 1 35
08:07 22 124 12 34
08:08 22 12 12 0
I thought about replacing every 5th comma by a subroutine, but it's too time consuming. Do you know a better solution? I'm hoping for a nice build-in function.
You can use fscanf and C-type format strings to accomplish this. For example:
fid=fopen('filename.txt');
A=reshape(fscanf(fid,'%d:%d, %d, %d, %d, %d, '),6,[])';
fclose(fid);
This stores your answer in a matrix A which will contain
A =
8 5 12 33 124 13
8 6 22 84 12 35
If you want to format this into a string or output file as you listed, you could use:
fprintf('%02d:%02d %-3d %-3d %-3d %-3d\n',A')
I'm fairly new the functional programming, so I'm going through some practice exercises. I want to write a function, given a matrix of unique naturals, let's say 5x5, return a collection of unique matrices of a smaller size, say 3x3, where the matrices must be intact, i.e. created from values that are adjacent in the original.
01 02 03 04 05
06 07 08 09 10
11 12 13 14 15
16 17 18 19 20
21 22 23 24 25
Simple. Just slide across, then down, one by one in groups of 3, to get something that looks like:
01 02 03 | 02 03 04 | 03 04 05 | 06 07 08
06 07 08 | 07 08 09 | 08 09 10 | 11 12 13
11 12 13 | 12 13 14 | 13 14 15 | 16 17 18
or, in Scala,
List(List(1, 2, 3), List(6, 7, 8), List(11, 12, 13))
List(List(2, 3, 4), List(7, 8, 9), List(12, 13, 14))
List(List(3, 4, 5), List(8, 9, 10), List(13, 14, 15))
List(List(6, 7, 8), List(11, 12, 13), List(16, 17, 18))
and so on and so on...
So I venture out with Scala (my language of choice because it allows me to evolve from imperative to functional, and I've spent the last few years in Java.
val array2D = "01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25".grouped(3).map(_.trim.toInt).grouped(5)
val sliced = array2D.map(row => row.sliding(3, 1).toList).sliding(3, 1).toList
Now I have a data structure I can work with, but I don't see a functional way. Sure I can traverse each piece of sliced, create a var matrix = new ListBuffer[Seq[Int]]() and imperatively create a bag of those and I'm done.
I want to find a functional, ideally point-free approach using Scala, but I'm stumped. There's got to be a way to zip with 3 or something like that... I've searched the ScalaDocs and can't seem to figure it out.
You got halfway there. In fact, I was having trouble figuring out how to do what you had done already. I broke up your code a bit to make it easier to follow. Also, I made array2D a List, so I could play with the code more easily. :-)
val input = "01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25"
val intArray = (input split " " map (_.toInt) toList)
val array2D = (intArray grouped 5 toList)
val sliced = array2D.map(row => row.sliding(3, 1).toList).sliding(3, 1).toList
Ok, so you have a bunch of lists, each one a bit like this:
List(List(List( 1, 2, 3), List( 2, 3, 4), List( 3, 4, 5)),
List(List( 6, 7, 8), List( 7, 8, 9), List( 8, 9, 10)),
List(List(11, 12, 13), List(12, 13, 14), List(13, 14, 15)))
And you want them like this:
List(List(List(1, 2, 3), List(6, 7, 8), List(11, 12, 13)),
List(List(2, 3, 4), List(7, 8, 9), List(12, 13, 14)),
List(List(3, 4, 5), List(8, 9, 10), List(13, 14, 15)))
Does that feel right to you? Each of the three sublists is a matrix on its own:
List(List(1, 2, 3), List(6, 7, 8), List(11, 12, 13))
is
01 02 03
06 07 08
11 12 13
So, basically, you want to transpose them. The next step, then, is:
val subMatrices = sliced map (_.transpose)
The type of that thing is List[List[List[Seq[Int]]]]. Let's consider that a bit... The 2D matrix is represented by a sequence of a sequence, so List[Seq[Int]] corresponds to a matrix. Let's say:
type Matrix = Seq[Seq[Int]]
val subMatrices: List[List[Matrix]] = sliced map (_.transpose)
But you want one one list of matrices, so you can flatten that:
type Matrix = Seq[Seq[Int]]
val subMatrices: List[Matrix] = (sliced map (_.transpose) flatten)
But, alas, a map plus a flatten is a flatMap:
type Matrix = Seq[Seq[Int]]
val subMatrices: List[Matrix] = sliced flatMap (_.transpose)
Now, you want the unique submatrices. That's simple enough: it's a set.
val uniqueSubMatrices = subMatrices.toSet
Or, if you wish to keep the result as a sequence,
val uniqueSubMatrices = subMatrices.distinct
And that's it. Full code just to illustrate:
type Matrix = Seq[Seq[Int]]
val input = "01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25"
val intArray = (input split " " map (_.toInt) toList)
val array2D: Matrix = (intArray grouped 5 toList)
val sliced: List[List[Matrix]] = (array2D map (row => row sliding 3 toList) sliding 3 toList)
val subMatrices: List[Matrix] = sliced flatMap (_.transpose)
val uniqueSubMatrices: Set[Matrix] = subMatrices.toSet
It could be written as a single expression, but unless you break it up into functions, it's going to be horrible to read. And you'd either have to use the forward pipe (|>, not in the standard library), or add these functions implicitly to the types they act on, or it will be difficult to read anyway.
Edit: Okay, I think I finally understand what you want. I'm going to show a way that works, not a way that is high-performance. (That's generally the mutable Java-like solution, but you already know how to do that.)
First, you really, really ought to do this with your own collections that work in 2D sensibly. Using a bunch of 1D collections to emulate 2D collections is going to lead to unnecessary confusion and complication. Don't do it. Really. It's a bad idea.
But, okay, let's do it anyway.
val big = (1 to 25).grouped(5).map(_.toList).toList
This is the whole matrix that you want. Next,
val smaller = (for (r <- big.sliding(3)) yield r.toList).toList
are the groups of rows that you want. Now, you should have been using a 2D data structure, because you want to do something that doesn't map well onto 1D operations. But:
val small = smaller.map(xss =>
Iterator.iterate(xss.map(_.sliding(3)))(identity).
takeWhile(_.forall(_.hasNext)).
map(_.map(_.next)).
toList
).toList
If you carefully pull this apart, you see that you're creating a bunch of iterators (xss.map(_.sliding(3))) and then iterating through them all in lock step by keeping hold of those same iterators step after step, stopping when at least one of them is empty, and mapping them onto their next values (which is how you walk forward with them).
Now that you've got the matrices you can store them however you want. Personally, I'd flatten the list:
val flattened = small.flatten
You wrote a structure that has the matrices side by side, which you can also do with some effort (again, because creating 2D operations out of 1D operations is not always straightforward):
val sidebyside = flattened.reduceRight((l,r) => (l,r).zipped.map(_ ::: _))
(note reduceRight to make this an O(n) operation instead of O(n^2)--joining to the end of long accumulating lists is a bad idea--but note also that with too many matrices this will probably overflow the stack).