Related
What I want to do is for a Brainfuck code to print out the ascii value of the input. For example, typing in an input of "a" will give an output of 97. The python equivalent of this is print(ord(input())). What I'm thinking is that once I get the input with the , command, I can split the input value's digits into separate cells, and then print each cell individually. What I mean by this is let's say you type in an input of a. The , command will store the ascii value of a in the first cell(cell 0), which is 97 in this case. Then I run some algorithm that will split the 97 into its individual digits. So, in this case, cell 1 will have a value of 0(because 97 has a hundred digit of 0), cell 2 will have a value of 9, and cell 3 will have a value of 7. Then we can add 48 to each of those cells(0 has an ascii value of 48) and print each cell individually, starting from cell 1(the hundreds place). The problem I'm facing is writing the digit separation algorithm. I can't seem to make it work. My idea is to subtract 100 from the original number until that number is less than 100 while keeping track of how many times 100 has been subtracted, then repeatedly subtract 10, and finally we are left with the ones place. But the problem with this idea is that I have no idea how to track if the number falls under 100 or 10. Any suggestions or ideas? Thanks for the help in advance.
What you are trying to implement is called "divmod". divmod is a function that divides two numbers (in your case positive integers) and stores the result and the remainder. Implementations for this in brainfuck exist: Divmod algorithm in brainfuck
Good luck!
I'm in a computer systems course and have been struggling, in part, with two's complement. I want to understand it, but everything I've read hasn't brought the picture together for me. I've read the Wikipedia article and various other articles, including my text book.
What is two's complement, how can we use it and how can it affect numbers during operations like casts (from signed to unsigned and vice versa), bit-wise operations and bit-shift operations?
Two's complement is a clever way of storing integers so that common math problems are very simple to implement.
To understand, you have to think of the numbers in binary.
It basically says,
for zero, use all 0's.
for positive integers, start counting up, with a maximum of 2(number of bits - 1)-1.
for negative integers, do exactly the same thing, but switch the role of 0's and 1's and count down (so instead of starting with 0000, start with 1111 - that's the "complement" part).
Let's try it with a mini-byte of 4 bits (we'll call it a nibble - 1/2 a byte).
0000 - zero
0001 - one
0010 - two
0011 - three
0100 to 0111 - four to seven
That's as far as we can go in positives. 23-1 = 7.
For negatives:
1111 - negative one
1110 - negative two
1101 - negative three
1100 to 1000 - negative four to negative eight
Note that you get one extra value for negatives (1000 = -8) that you don't for positives. This is because 0000 is used for zero. This can be considered as Number Line of computers.
Distinguishing between positive and negative numbers
Doing this, the first bit gets the role of the "sign" bit, as it can be used to distinguish between nonnegative and negative decimal values. If the most significant bit is 1, then the binary can be said to be negative, where as if the most significant bit (the leftmost) is 0, you can say the decimal value is nonnegative.
"Sign-magnitude" negative numbers just have the sign bit flipped of their positive counterparts, but this approach has to deal with interpreting 1000 (one 1 followed by all 0s) as "negative zero" which is confusing.
"Ones' complement" negative numbers are just the bit-complement of their positive counterparts, which also leads to a confusing "negative zero" with 1111 (all ones).
You will likely not have to deal with Ones' Complement or Sign-Magnitude integer representations unless you are working very close to the hardware.
I wonder if it could be explained any better than the Wikipedia article.
The basic problem that you are trying to solve with two's complement representation is the problem of storing negative integers.
First, consider an unsigned integer stored in 4 bits. You can have the following
0000 = 0
0001 = 1
0010 = 2
...
1111 = 15
These are unsigned because there is no indication of whether they are negative or positive.
Sign Magnitude and Excess Notation
To store negative numbers you can try a number of things. First, you can use sign magnitude notation which assigns the first bit as a sign bit to represent +/- and the remaining bits to represent the magnitude. So using 4 bits again and assuming that 1 means - and 0 means + then you have
0000 = +0
0001 = +1
0010 = +2
...
1000 = -0
1001 = -1
1111 = -7
So, you see the problem there? We have positive and negative 0. The bigger problem is adding and subtracting binary numbers. The circuits to add and subtract using sign magnitude will be very complex.
What is
0010
1001 +
----
?
Another system is excess notation. You can store negative numbers, you get rid of the two zeros problem but addition and subtraction remains difficult.
So along comes two's complement. Now you can store positive and negative integers and perform arithmetic with relative ease. There are a number of methods to convert a number into two's complement. Here's one.
Convert Decimal to Two's Complement
Convert the number to binary (ignore the sign for now)
e.g. 5 is 0101 and -5 is 0101
If the number is a positive number then you are done.
e.g. 5 is 0101 in binary using two's complement notation.
If the number is negative then
3.1 find the complement (invert 0's and 1's)
e.g. -5 is 0101 so finding the complement is 1010
3.2 Add 1 to the complement 1010 + 1 = 1011.
Therefore, -5 in two's complement is 1011.
So, what if you wanted to do 2 + (-3) in binary? 2 + (-3) is -1.
What would you have to do if you were using sign magnitude to add these numbers? 0010 + 1101 = ?
Using two's complement consider how easy it would be.
2 = 0010
-3 = 1101 +
-------------
-1 = 1111
Converting Two's Complement to Decimal
Converting 1111 to decimal:
The number starts with 1, so it's negative, so we find the complement of 1111, which is 0000.
Add 1 to 0000, and we obtain 0001.
Convert 0001 to decimal, which is 1.
Apply the sign = -1.
Tada!
Like most explanations I've seen, the ones above are clear about how to work with 2's complement, but don't really explain what they are mathematically. I'll try to do that, for integers at least, and I'll cover some background that's probably familiar first.
Recall how it works for decimal: 2345 is a way of writing 2 × 103 + 3 × 102 + 4 × 101 + 5 × 100.
In the same way, binary is a way of writing numbers using just 0 and 1 following the same general idea, but replacing those 10s above with 2s. Then in binary, 1111is a way of writing 1 × 23 + 1 × 22 + 1 × 21 + 1 × 20and if you work it out, that turns out to equal 15 (base 10). That's because it is 8+4+2+1 = 15.
This is all well and good for positive numbers. It even works for negative numbers if you're willing to just stick a minus sign in front of them, as humans do with decimal numbers. That can even be done in computers, sort of, but I haven't seen such a computer since the early 1970's. I'll leave the reasons for a different discussion.
For computers it turns out to be more efficient to use a complement representation for negative numbers. And here's something that is often overlooked. Complement notations involve some kind of reversal of the digits of the number, even the implied zeroes that come before a normal positive number. That's awkward, because the question arises: all of them? That could be an infinite number of digits to be considered.
Fortunately, computers don't represent infinities. Numbers are constrained to a particular length (or width, if you prefer). So let's return to positive binary numbers, but with a particular size. I'll use 8 digits ("bits") for these examples. So our binary number would really be 00001111or 0 × 27 + 0 × 26 + 0 × 25 + 0 × 24 + 1 × 23 + 1 × 22 + 1 × 21 + 1 × 20
To form the 2's complement negative, we first complement all the (binary) digits to form 11110000and add 1 to form 11110001but how are we to understand that to mean -15?
The answer is that we change the meaning of the high-order bit (the leftmost one). This bit will be a 1 for all negative numbers. The change will be to change the sign of its contribution to the value of the number it appears in. So now our 11110001 is understood to represent -1 × 27 + 1 × 26 + 1 × 25 + 1 × 24 + 0 × 23 + 0 × 22 + 0 × 21 + 1 × 20Notice that "-" in front of that expression? It means that the sign bit carries the weight -27, that is -128 (base 10). All the other positions retain the same weight they had in unsigned binary numbers.
Working out our -15, it is -128 + 64 + 32 + 16 + 1 Try it on your calculator. it's -15.
Of the three main ways that I've seen negative numbers represented in computers, 2's complement wins hands down for convenience in general use. It has an oddity, though. Since it's binary, there have to be an even number of possible bit combinations. Each positive number can be paired with its negative, but there's only one zero. Negating a zero gets you zero. So there's one more combination, the number with 1 in the sign bit and 0 everywhere else. The corresponding positive number would not fit in the number of bits being used.
What's even more odd about this number is that if you try to form its positive by complementing and adding one, you get the same negative number back. It seems natural that zero would do this, but this is unexpected and not at all the behavior we're used to because computers aside, we generally think of an unlimited supply of digits, not this fixed-length arithmetic.
This is like the tip of an iceberg of oddities. There's more lying in wait below the surface, but that's enough for this discussion. You could probably find more if you research "overflow" for fixed-point arithmetic. If you really want to get into it, you might also research "modular arithmetic".
2's complement is very useful for finding the value of a binary, however I thought of a much more concise way of solving such a problem(never seen anyone else publish it):
take a binary, for example: 1101 which is [assuming that space "1" is the sign] equal to -3.
using 2's complement we would do this...flip 1101 to 0010...add 0001 + 0010 ===> gives us 0011. 0011 in positive binary = 3. therefore 1101 = -3!
What I realized:
instead of all the flipping and adding, you can just do the basic method for solving for a positive binary(lets say 0101) is (23 * 0) + (22 * 1) + (21 * 0) + (20 * 1) = 5.
Do exactly the same concept with a negative!(with a small twist)
take 1101, for example:
for the first number instead of 23 * 1 = 8 , do -(23 * 1) = -8.
then continue as usual, doing -8 + (22 * 1) + (21 * 0) + (20 * 1) = -3
Imagine that you have a finite number of bits/trits/digits/whatever. You define 0 as all digits being 0, and count upwards naturally:
00
01
02
..
Eventually you will overflow.
98
99
00
We have two digits and can represent all numbers from 0 to 100. All those numbers are positive! Suppose we want to represent negative numbers too?
What we really have is a cycle. The number before 2 is 1. The number before 1 is 0. The number before 0 is... 99.
So, for simplicity, let's say that any number over 50 is negative. "0" through "49" represent 0 through 49. "99" is -1, "98" is -2, ... "50" is -50.
This representation is ten's complement. Computers typically use two's complement, which is the same except using bits instead of digits.
The nice thing about ten's complement is that addition just works. You do not need to do anything special to add positive and negative numbers!
I read a fantastic explanation on Reddit by jng, using the odometer as an analogy.
It is a useful convention. The same circuits and logic operations that
add / subtract positive numbers in binary still work on both positive
and negative numbers if using the convention, that's why it's so
useful and omnipresent.
Imagine the odometer of a car, it rolls around at (say) 99999. If you
increment 00000 you get 00001. If you decrement 00000, you get 99999
(due to the roll-around). If you add one back to 99999 it goes back to
00000. So it's useful to decide that 99999 represents -1. Likewise, it is very useful to decide that 99998 represents -2, and so on. You have
to stop somewhere, and also by convention, the top half of the numbers
are deemed to be negative (50000-99999), and the bottom half positive
just stand for themselves (00000-49999). As a result, the top digit
being 5-9 means the represented number is negative, and it being 0-4
means the represented is positive - exactly the same as the top bit
representing sign in a two's complement binary number.
Understanding this was hard for me too. Once I got it and went back to
re-read the books articles and explanations (there was no internet
back then), it turned out a lot of those describing it didn't really
understand it. I did write a book teaching assembly language after
that (which did sell quite well for 10 years).
Two complement is found out by adding one to 1'st complement of the given number.
Lets say we have to find out twos complement of 10101 then find its ones complement, that is, 01010 add 1 to this result, that is, 01010+1=01011, which is the final answer.
Lets get the answer 10 – 12 in binary form using 8 bits:
What we will really do is 10 + (-12)
We need to get the compliment part of 12 to subtract it from 10.
12 in binary is 00001100.
10 in binary is 00001010.
To get the compliment part of 12 we just reverse all the bits then add 1.
12 in binary reversed is 11110011. This is also the Inverse code (one's complement).
Now we need to add one, which is now 11110100.
So 11110100 is the compliment of 12! Easy when you think of it this way.
Now you can solve the above question of 10 - 12 in binary form.
00001010
11110100
-----------------
11111110
Looking at the two's complement system from a math point of view it really makes sense. In ten's complement, the idea is to essentially 'isolate' the difference.
Example: 63 - 24 = x
We add the complement of 24 which is really just (100 - 24). So really, all we are doing is adding 100 on both sides of the equation.
Now the equation is: 100 + 63 - 24 = x + 100, that is why we remove the 100 (or 10 or 1000 or whatever).
Due to the inconvenient situation of having to subtract one number from a long chain of zeroes, we use a 'diminished radix complement' system, in the decimal system, nine's complement.
When we are presented with a number subtracted from a big chain of nines, we just need to reverse the numbers.
Example: 99999 - 03275 = 96724
That is the reason, after nine's complement, we add 1. As you probably know from childhood math, 9 becomes 10 by 'stealing' 1. So basically it's just ten's complement that takes 1 from the difference.
In Binary, two's complement is equatable to ten's complement, while one's complement to nine's complement. The primary difference is that instead of trying to isolate the difference with powers of ten (adding 10, 100, etc. into the equation) we are trying to isolate the difference with powers of two.
It is for this reason that we invert the bits. Just like how our minuend is a chain of nines in decimal, our minuend is a chain of ones in binary.
Example: 111111 - 101001 = 010110
Because chains of ones are 1 below a nice power of two, they 'steal' 1 from the difference like nine's do in decimal.
When we are using negative binary number's, we are really just saying:
0000 - 0101 = x
1111 - 0101 = 1010
1111 + 0000 - 0101 = x + 1111
In order to 'isolate' x, we need to add 1 because 1111 is one away from 10000 and we remove the leading 1 because we just added it to the original difference.
1111 + 1 + 0000 - 0101 = x + 1111 + 1
10000 + 0000 - 0101 = x + 10000
Just remove 10000 from both sides to get x, it's basic algebra.
The word complement derives from completeness. In the decimal world the numerals 0 through 9 provide a complement (complete set) of numerals or numeric symbols to express all decimal numbers. In the binary world the numerals 0 and 1 provide a complement of numerals to express all binary numbers. In fact The symbols 0 and 1 must be used to represent everything (text, images, etc) as well as positive (0) and negative (1).
In our world the blank space to the left of number is considered as zero:
35=035=000000035.
In a computer storage location there is no blank space. All bits (binary digits) must be either 0 or 1. To efficiently use memory numbers may be stored as 8 bit, 16 bit, 32 bit, 64 bit, 128 bit representations. When a number that is stored as an 8 bit number is transferred to a 16 bit location the sign and magnitude (absolute value) must remain the same. Both 1's complement and 2's complement representations facilitate this.
As a noun:
Both 1's complement and 2's complement are binary representations of signed quantities where the most significant bit (the one on the left) is the sign bit. 0 is for positive and 1 is for negative.
2s complement does not mean negative. It means a signed quantity. As in decimal the magnitude is represented as the positive quantity. The structure uses sign extension to preserve the quantity when promoting to a register [] with more bits:
[0101]=[00101]=[00000000000101]=5 (base 10)
[1011]=[11011]=[11111111111011]=-5(base 10)
As a verb:
2's complement means to negate. It does not mean make negative. It means if negative make positive; if positive make negative. The magnitude is the absolute value:
if a >= 0 then |a| = a
if a < 0 then |a| = -a = 2scomplement of a
This ability allows efficient binary subtraction using negate then add.
a - b = a + (-b)
The official way to take the 1's complement is for each digit subtract its value from 1.
1'scomp(0101) = 1010.
This is the same as flipping or inverting each bit individually. This results in a negative zero which is not well loved so adding one to te 1's complement gets rid of the problem.
To negate or take the 2s complement first take the 1s complement then add 1.
Example 1 Example 2
0101 --original number 1101
1's comp 1010 0010
add 1 0001 0001
2's comp 1011 --negated number 0011
In the examples the negation works as well with sign extended numbers.
Adding:
1110 Carry 111110 Carry
0110 is the same as 000110
1111 111111
sum 0101 sum 000101
SUbtracting:
1110 Carry 00000 Carry
0110 is the same as 00110
-0111 +11001
---------- ----------
sum 0101 sum 11111
Notice that when working with 2's complement, blank space to the left of the number is filled with zeros for positive numbers butis filled with ones for negative numbers. The carry is always added and must be either a 1 or 0.
Cheers
2's complement is essentially a way of coming up with the additive inverse of a binary number. Ask yourself this: Given a number in binary form (present at a fixed length memory location), what bit pattern, when added to the original number (at the fixed length memory location), would make the result all zeros ? (at the same fixed length memory location). If we could come up with this bit pattern then that bit pattern would be the -ve representation (additive inverse) of the original number; as by definition adding a number to its additive inverse always results in zero. Example: take 5 which is 101 present inside a single 8 bit byte. Now the task is to come up with a bit pattern which when added to the given bit pattern (00000101) would result in all zeros at the memory location which is used to hold this 5 i.e. all 8 bits of the byte should be zero. To do that, start from the right most bit of 101 and for each individual bit, again ask the same question: What bit should I add to the current bit to make the result zero ? continue doing that taking in account the usual carry over. After we are done with the 3 right most places (the digits that define the original number without regard to the leading zeros) the last carry goes in the bit pattern of the additive inverse. Furthermore, since we are holding in the original number in a single 8 bit byte, all other leading bits in the additive inverse should also be 1's so that (and this is important) when the computer adds "the number" (represented using the 8 bit pattern) and its additive inverse using "that" storage type (a byte) the result in that byte would be all zeros.
1 1 1
----------
1 0 1
1 0 1 1 ---> additive inverse
---------
0 0 0
Many of the answers so far nicely explain why two's complement is used to represent negative numbers, but do not tell us what two's complement number is, particularly not why a '1' is added, and in fact often added in a wrong way.
The confusion comes from a poor understanding of the definition of a complement number. A complement is the missing part that would make something complete.
The radix complement of an n digit number x in radix b is, by definition, b^n-x.
In binary 4 is represented by 100, which has 3 digits (n=3) and a radix of 2 (b=2). So its radix complement is b^n-x = 2^3-4=8-4=4 (or 100 in binary).
However, in binary obtaining a radix's complement is not as easy as getting its diminished radix complement, which is defined as (b^n-1)-y, just 1 less than that of radix complement. To get a diminished radix complement, you simply flip all the digits.
100 -> 011 (diminished (one's) radix complement)
to obtain the radix (two's) complement, we simply add 1, as the definition defined.
011 +1 ->100 (two's complement).
Now with this new understanding, let's take a look of the example given by Vincent Ramdhanie (see above second response):
Converting 1111 to decimal:
The number starts with 1, so it's negative, so we find the complement of 1111, which is 0000.
Add 1 to 0000, and we obtain 0001.
Convert 0001 to decimal, which is 1.
Apply the sign = -1.
Tada!
Should be understood as:
The number starts with 1, so it's negative. So we know it is a two's complement of some value x. To find the x represented by its two's complement, we first need find its 1's complement.
two's complement of x: 1111
one's complement of x: 1111-1 ->1110;
x = 0001, (flip all digits)
Apply the sign -, and the answer =-x =-1.
I liked lavinio's answer, but shifting bits adds some complexity. Often there's a choice of moving bits while respecting the sign bit or while not respecting the sign bit. This is the choice between treating the numbers as signed (-8 to 7 for a nibble, -128 to 127 for bytes) or full-range unsigned numbers (0 to 15 for nibbles, 0 to 255 for bytes).
It is a clever means of encoding negative integers in such a way that approximately half of the combination of bits of a data type are reserved for negative integers, and the addition of most of the negative integers with their corresponding positive integers results in a carry overflow that leaves the result to be binary zero.
So, in 2's complement if one is 0x0001 then -1 is 0x1111, because that will result in a combined sum of 0x0000 (with an overflow of 1).
2’s Complements: When we add an extra one with the 1’s complements of a number we will get the 2’s complements. For example: 100101 it’s 1’s complement is 011010 and 2’s complement is 011010+1 = 011011 (By adding one with 1's complement) For more information
this article explain it graphically.
Two's complement is mainly used for the following reasons:
To avoid multiple representations of 0
To avoid keeping track of carry bit (as in one's complement) in case of an overflow.
Carrying out simple operations like addition and subtraction becomes easy.
Two's complement is one of the ways of expressing a negative number and most of the controllers and processors store a negative number in two's complement form.
In simple terms, two's complement is a way to store negative numbers in computer memory. Whereas positive numbers are stored as a normal binary number.
Let's consider this example,
The computer uses the binary number system to represent any number.
x = 5;
This is represented as 0101.
x = -5;
When the computer encounters the - sign, it computes its two's complement and stores it.
That is, 5 = 0101 and its two's complement is 1011.
The important rules the computer uses to process numbers are,
If the first bit is 1 then it must be a negative number.
If all the bits except first bit are 0 then it is a positive number, because there is no -0 in number system (1000 is not -0 instead it is positive 8).
If all the bits are 0 then it is 0.
Else it is a positive number.
To bitwise complement a number is to flip all the bits in it. To two’s complement it, we flip all the bits and add one.
Using 2’s complement representation for signed integers, we apply the 2’s complement operation to convert a positive number to its negative equivalent and vice versa. So using nibbles for an example, 0001 (1) becomes 1111 (-1) and applying the op again, returns to 0001.
The behaviour of the operation at zero is advantageous in giving a single representation for zero without special handling of positive and negative zeroes. 0000 complements to 1111, which when 1 is added. overflows to 0000, giving us one zero, rather than a positive and a negative one.
A key advantage of this representation is that the standard addition circuits for unsigned integers produce correct results when applied to them. For example adding 1 and -1 in nibbles: 0001 + 1111, the bits overflow out of the register, leaving behind 0000.
For a gentle introduction, the wonderful Computerphile have produced a video on the subject.
The question is 'What is “two's complement”?'
The simple answer for those wanting to understand it theoretically (and me seeking to complement the other more practical answers): 2's complement is the representation for negative integers in the dual system that does not require additional characters, such as + and -.
Two's complement of a given number is the number got by adding 1 with the ones' complement of the number.
Suppose, we have a binary number: 10111001101
Its 1's complement is: 01000110010
And its two's complement will be: 01000110011
Reference: Two's Complement (Thomas Finley)
I invert all the bits and add 1. Programmatically:
// In C++11
int _powers[] = {
1,
2,
4,
8,
16,
32,
64,
128
};
int value = 3;
int n_bits = 4;
int twos_complement = (value ^ ( _powers[n_bits]-1)) + 1;
You can also use an online calculator to calculate the two's complement binary representation of a decimal number: http://www.convertforfree.com/twos-complement-calculator/
The simplest answer:
1111 + 1 = (1)0000. So 1111 must be -1. Then -1 + 1 = 0.
It's perfect to understand these all for me.
I had previously wrote some code to split 3 columns into 4, however the code was very inefficient and time consuming. As I am working with millions of rows it wasn't suitable. (Below is my previous code)
tline = fgetl(fid);
ID=tline(1:4);
IDN = str2double(ID);
Day=tline(6:8);
DayN = str2double(Day);
HalfHour=tline(9:10);
HalfHourN = str2double(HalfHour);
Usage=tline(12:end);
UsageN = str2double(Usage);
There must be a more efficient and quicker way of doing this?
Going back to basics, I have produced a x by 3 matrix. but require an x by 4 matrix
To show what I am trying to do, examining one row -
I am trying to change
1001 36501 1005
to
1001 365 01 1005
Any help would be much appreciated!
Edit:
The second column I am trying to divide into two, is always composed of 5 characters. I am trying to get the first 3 characters into their own column, likewise for the remaining characters.
What might take time in your case is actually the use of the str2double function. It is known that this built-in function becomes very slow when the data set is large. You might try to get rid of it if possible.
you can use modulo
ans = (36501 - mod(36501,100))/100
This would give you 365
if you want the 1, it is mod(36501,100)
so this would effectively split your second column into 2 different numbers, you can then re name them etc.
hmmm on second thoughts, if all your numbers on your second column are 5 digits, this can be extremely efficient, since mod is computed in matlab by b = a - m.*floor(a./m);
check http://uk.mathworks.com/help/matlab/ref/mod.html it should work for vectors (i.e. your second column)
how to write a lisp program to convert given hexadecimal number into decimal. can somebody give me a clue.
thank you
I'm assuming its a homework problem so i'll give you a hint in the right direction.
Here is how to convert decimal to binary ->
Lets say you start with the number 9 in binary its 1001.
Start of by dividing 9 by 2. You get 4 with remainder 1. Save the remainder.
Now divide that 4 by 2 again, you get 2 with remainder 0. Save the remainder.
Divide that 2 again by 2, you get 1 with remainder 0. Save the remainder.
Divide that 1 by 2 and finally you get 0 with reaminder 1. Save the remainder.
If you read the saved remainders backwards you get 1001! The binary number you've been looking for. Best to push the remainders on the stack and pop them back out, that way they'll come out backwards.
It's already provided by Common Lisp.
The input is the string for the hex integer.
Then you parse the integer with radix 16
the result is the number
if you write the number with base 10 to an output stream, then you can get the number as a string in base 10
Let's say you have a List<List<Boolean>> and you want to encode that into binary form in the most compact way possible.
I don't care about read or write performance. I just want to use the minimal amount of space. Also, the example is in Java, but we are not limited to the Java system. The length of each "List" is unbounded. Therefore any solution that encodes the length of each list must in itself encode a variable length data type.
Related to this problem is encoding of variable length integers. You can think of each List<Boolean> as a variable length unsigned integer.
Please read the question carefully. We are not limited to the Java system.
EDIT
I don't understand why a lot of the answers talk about compression. I am not trying to do compression per se, but just encoding random sequence of bits down. Except each sequence of bits are of different lengths and order needs to be preserved.
You can think of this question in a different way. Lets say you have a list of arbitrary list of random unsigned integers (unbounded). How do you encode this list in a binary file?
Research
I did some reading and found what I really am looking for is Universal code
Result
I am going to use a variant of Elias Omega Coding described in the paper A new recursive universal code of the positive integers
I now understand how the smaller the representation of the smaller integers is a trade off with the larger integers. By simply choosing an Universal code with a "large" representation of the very first integer you save a lot of space in the long run when you need to encode the arbitrary large integers.
I am thinking of encoding a bit sequence like this:
head | value
------+------------------
00001 | 0110100111000011
Head has variable length. Its end is marked by the first occurrence of a 1. Count the number of zeroes in head. The length of the value field will be 2 ^ zeroes. Since the length of value is known, this encoding can be repeated. Since the size of head is log value, as the size of the encoded value increases, the overhead converges to 0%.
Addendum
If you want to fine tune the length of value more, you can add another field that stores the exact length of value. The length of the length field could be determined by the length of head. Here is an example with 9 bits.
head | length | value
------+--------+-----------
00001 | 1001 | 011011001
I don't know much about Java, so I guess my solution will HAVE to be general :)
1. Compact the lists
Since Booleans are inefficient, each List<Boolean> should be compacted into a List<Byte>, it's easy, just grab them 8 at a time.
The last "byte" may be incomplete, so you need to store how many bits have been encoded of course.
2. Serializing a list of elements
You have 2 ways to proceed: either you encode the number of items of the list, either you use a pattern to mark an end. I would recommend encoding the number of items, the pattern approach requires escaping and it's creepy, plus it's more difficult with packed bits.
To encode the length you can use a variable scheme: ie the number of bytes necessary to encode a length should be proportional to the length, one I already used. You can indicate how many bytes are used to encode the length itself by using a prefix on the first byte:
0... .... > this byte encodes the number of items (7 bits of effective)
10.. .... / .... .... > 2 bytes
110. .... / .... .... / .... .... > 3 bytes
It's quite space efficient, and decoding occurs on whole bytes, so not too difficult. One could remark it's very similar to the UTF8 scheme :)
3. Apply recursively
List< List< Boolean > > becomes [Length Item ... Item] where each Item is itself the representation of a List<Boolean>
4. Zip
I suppose there is a zlib library available for Java, or anything else like deflate or lcw. Pass it your buffer and make sure to precise you wish as much compression as possible, whatever the time it takes.
If there is any repetitive pattern (even ones you did not see) in your representation it should be able to compress it. Don't trust it dumbly though and DO check that the "compressed" form is lighter than the "uncompressed" one, it's not always the case.
5. Examples
Where one notices that keeping track of the edge of the lists is space consuming :)
// Tricky here, we indicate how many bits are used, but they are packed into bytes ;)
List<Boolean> list = [false,false,true,true,false,false,true,true]
encode(list) == [0x08, 0x33] // [00001000, 00110011] (2 bytes)
// Easier: the length actually indicates the number of elements
List<List<Boolean>> super = [list,list]
encode(super) == [0x02, 0x08, 0x33, 0x08, 0x33] // [00000010, ...] (5 bytes)
6. Space consumption
Suppose we have a List<Boolean> of n booleans, the space consumed to encode it is:
booleans = ceil( n / 8 )
To encode the number of bits (n), we need:
length = 1 for 0 <= n < 2^7 ~ 128
length = 2 for 2^7 <= n < 2^14 ~ 16384
length = 3 for 2^14 <= n < 2^21 ~ 2097152
...
length = ceil( log(n) / 7 ) # for n != 0 ;)
Thus to fully encode a list:
bytes =
if n == 0: 1
else : ceil( log(n) / 7 ) + ceil( n / 8 )
7. Small Lists
There is one corner case though: the low end of the spectrum (ie almost empty list).
For n == 1, bytes is evaluated to 2, which may indeed seem wasteful. I would not however try to guess what will happen once the compression kicks in.
You may wish though to pack even more. It's possible if we abandon the idea of preserving whole bytes...
Keep the length encoding as is (on whole bytes), but do not "pad" the List<Boolean>. A one element list becomes 0000 0001 x (9 bits)
Try to 'pack' the length encoding as well
The second point is more difficult, we are effectively down to a double length encoding:
Indicates how many bits encode the length
Actually encode the length on these bits
For example:
0 -> 0 0
1 -> 0 1
2 -> 10 10
3 -> 10 11
4 -> 110 100
5 -> 110 101
8 -> 1110 1000
16 -> 11110 10000 (=> 1 byte and 2 bits)
It works pretty well for very small lists, but quickly degenerate:
# Original scheme
length = ceil( ( log(n) / 7)
# New scheme
length = 2 * ceil( log(n) )
The breaking point ? 8
Yep, you read it right, it's only better for list with less than 8 elements... and only better by "bits".
n -> bits spared
[0,1] -> 6
[2,3] -> 4
[4,7] -> 2
[8,15] -> 0 # Turn point
[16,31] -> -2
[32,63] -> -4
[64,127] -> -6
[128,255] -> 0 # Interesting eh ? That's the whole byte effect!
And of course, once the compression kicks in, chances are it won't really matter.
I understand you may appreciate recursive's algorithm, but I would still advise to compute the figures of the actual space consumption or even better to actually test it with archiving applied on real test sets.
8. Recursive / Variable coding
I have read with interest TheDon's answer, and the link he submitted to Elias Omega Coding.
They are sound answers, in the theoretical domain. Unfortunately they are quite unpractical. The main issue is that they have extremely interesting asymptotic behaviors, but when do we actually need to encode a Gigabyte worth of data ? Rarely if ever.
A recent study of memory usage at work suggested that most containers were used for a dozen items (or a few dozens). Only in some very rare case do we reach the thousand. Of course for your particular problem the best way would be to actually examine your own data and see the distribution of values, but from experience I would say you cannot just concentrate on the high end of the spectrum, because your data lay in the low end.
An example of TheDon's algorithm. Say I have a list [0,1,0,1,0,1,0,1]
len('01010101') = 8 -> 1000
len('1000') = 4 -> 100
len('100') = 3 -> 11
len('11') = 2 -> 10
encode('01010101') = '10' '0' '11' '0' '100' '0' '1000' '1' '01010101'
len(encode('01010101')) = 2 + 1 + 2 + 1 + 3 + 1 + 4 + 1 + 8 = 23
Let's make a small table, with various 'tresholds' to stop the recursion. It represents the number of bits of overhead for various ranges of n.
threshold 2 3 4 5 My proposal
-----------------------------------------------
[0,3] -> 3 4 5 6 8
[4,7] -> 10 4 5 6 8
[8,15] -> 15 9 5 6 8
[16,31] -> 16 10 5 6 8
[32,63] -> 17 11 12 6 8
[64,127] -> 18 12 13 14 8
[128,255]-> 19 13 14 15 16
To be fair, I concentrated on the low end, and my proposal is suited for this task. I wanted to underline that it's not so clear cut though. Especially because near 1, the log function is almost linear, and thus the recursion loses its charm. The treshold helps tremendously and 3 seems to be a good candidate...
As for Elias omega coding, it's even worse. From the wikipedia article:
17 -> '10 100 10001 0'
That's it, a whooping 11 bits.
Moral: You cannot chose an encoding scheme without considering the data at hand.
So, unless your List<Boolean> have a length in the hundreds, don't bother and stick to my little proposal.
I'd use variable-length integers to encode how many bits there are to read. The MSB would indicate if the next byte is also part of the integer. For instance:
11000101 10010110 00100000
Would actually mean:
10001 01001011 00100000
Since the integer is continued 2 times.
These variable-length integers would tell how many bits there are to read. And there'd be another variable-length int at the beginning of all to tell how many bit sets there are to read.
From there on, supposing you don't want to use compression, the only way I can see to optimize it size-wise is to adapt it to your situation. If you often have larger bit sets, you might want for instance to use short integers instead of bytes for the variable-length integer encoding, making you potentially waste less bits in the encoding itself.
EDIT I don't think there exists a perfect way to achieve all you want, all at once. You can't create information out of nothing, and if you need variable-length integers, you obviously have to encode the integer length too. There is necessarily a tradeoff between space and information, but there is also minimal information that you can't cut out to use less space. No system where factors grow at different rates will ever scale perfectly. It's like trying to fit a straight line over a logarithmic curve. You can't do that. (And besides, that's pretty much exactly what you're trying to do here.)
You cannot encode the length of the variable-length integer outside of the integer and get unlimited-size variable integers at the same time, because that would require the length itself to be variable-length, and whatever algorithm you choose, it seems common sense to me that you'll be better off with just one variable-length integer instead of two or more of them.
So here is my other idea: in the integer "header", write one 1 for each byte the variable-length integer requires from there. The first 0 denotes the end of the "header" and the beginning of the integer itself.
I'm trying to grasp the exact equation to determine how many bits are required to store a given integer for the two ways I gave, but my logarithms are rusty, so I'll plot it down and edit this message later to include the results.
EDIT 2
Here are the equations:
Solution one, 7 bits per encoding bit (one full byte at a time):
y = 8 * ceil(log(x) / (7 * log(2)))
Solution one, 3 bits per encoding bit (one nibble at a time):
y = 4 * ceil(log(x) / (3 * log(2)))
Solution two, 1 byte per encoding bit plus separator:
y = 9 * ceil(log(x) / (8 * log(2))) + 1
Solution two, 1 nibble per encoding bit plus separator:
y = 5 * ceil(log(x) / (4 * log(2))) + 1
I suggest you take the time to plot them (best viewed with a logarithmic-linear coordinates system) to get the ideal solution for your case, because there is no perfect solution. In my opinion, the first solution has the most stable results.
I guess for "the most compact way possible" you'll want some compression, but Huffman Coding may not be the way to go as I think it works best with alphabets that have static per-symbol frequencies.
Check out Arithmetic Coding - it operates on bits and can adapt to a dynamic input probabilities. I also see that there is a BSD-licensed Java library that'll do it for you which seems to expect single bits as input.
I suppose for maximum compression you could concatenate each inner list (prefixed with its length) and run the coding algorithm again over the whole lot.
I don't see how encoding an arbitrary set of bits differ from compressing/encoding any other form of data. Note that you only impose a loose restriction on the bits you're encoding: namely, they are lists of lists of bits. With this small restriction, this list of bits becomes just data, arbitrary data, and that's what "normal" compression algorithms compress.
Of course, most compression algorithms work on the assumption that the input is repeated in some way in the future (or in the past), as in the LZxx family of compressor, or have a given frequency distribution for symbols.
Given your prerequisites and how compression algorithms work, I would advice doing the following:
Pack the bits of each list using the less possible number of bytes, using bytes as bitfields, encoding the length, etc.
Try huffman, arithmetic, LZxx, etc on the resulting stream of bytes.
One can argue that this is the pretty obvious and easiest way of doing this, and that this won't work as your sequence of bits have no known pattern. But the fact is that this is the best you can do in any scenario.
UNLESS, you know something from your data, or some transformation on those lists that make them raise a pattern of some kind. Take for example the coding of the DCT coefficients in JPEG encoding. The way of listing those coefficients (diagonal and in zig-zag) is made to favor a pattern in the output of the different coefficients for the transformation. This way, traditional compressions can be applied to the resulting data. If you know something of those lists of bits that allow you to re-arrange them in a more-compressible way (a way that shows some more structure), then you'll get compression.
I have a sneaking suspicion that you simply can't encode a truly random set of bits into a more compact form in the worst case. Any kind of RLE is going to inflate the set on just the wrong input even though it'll do well in the average and best cases. Any kind of periodic or content specific approximation is going to lose data.
As one of the other posters stated, you've got to know SOMETHING about the dataset to represent it in a more compact form and / or you've got to accept some loss to get it into a predictable form that can be more compactly expressed.
In my mind, this is an information-theoretic problem with the constraint of infinite information and zero loss. You can't represent the information in a different way and you can't approximate it as something more easily represented. Ergo, you need at least as much space as you have information and no less.
http://en.wikipedia.org/wiki/Information_theory
You could always cheat, I suppose, and manipulate the hardware to encode a discrete range of values on the media to tease out a few more "bits per bit" (think multiplexing). You'd spend more time encoding it and reading it though.
Practically, you could always try the "jiggle" effect where you encode the data multiple times in multiple ways (try interpreting as audio, video, 3d, periodic, sequential, key based, diffs, etc...) and in multiple page sizes and pick the best. You'd be pretty much guaranteed to have the best REASONABLE compression and your worst case would be no worse then your original data set.
Dunno if that would get you the theoretical best though.
Theoretical Limits
This is a difficult question to answer without knowing more about the data you intend to compress; the answer to your question could be different with different domains.
For example, from the Limitations section of the Wikipedia article on Lossless Compression:
Lossless data compression algorithms cannot guarantee compression for all input data sets. In other words, for any (lossless) data compression algorithm, there will be an input data set that does not get smaller when processed by the algorithm. This is easily proven with elementary mathematics using a counting argument. ...
Basically, since it's theoretically impossible to compress all possible input data losslessly, it's not even possible to answer your question effectively.
Practical compromise
Just use Huffman, DEFLATE, 7Z, or some ZIP-like off-the-shelf compression algorithm and enocde the bits as variable length byte arrays (or lists, or vectors, or whatever they are called in Java or whatever language you like). Of course, to read the bits back out may require a bit of decompression but that could be done behind the scenes. You can make a class which hides the internal implementation methods to return a list or array of booleans in some range of indices despite the fact that the data is stored internally in pack byte arrays. Updating the boolean at a give index or indices may be a problem but is by no means impossible.
List-of-Lists-of-Ints-Encoding:
When you come to the beginning of a list, write down the bits for ASCII '['. Then proceed into the list.
When you come to any arbitrary binary number, write down bits corresponding to the decimal representation of the number in ASCII. For example the number 100, write 0x31 0x30 0x30. Then write the bits corresponding to ASCII ','.
When you come to the end of a list, write down the bits for ']'. Then write ASCII ','.
This encoding will encode any arbitrarily-deep nesting of arbitrary-length lists of unbounded integers. If this encoding is not compact enough, follow it up with gzip to eliminate the redundancies in ASCII bit coding.
You could convert each List into a BitSet and then serialize the BitSet-s.
Well, first off you will want to pack those booleans together so that you are getting eight of them to a byte. C++'s standard bitset was designed for this purpose. You should probably be using it natively instead of vector, if you can.
After that, you could in theory compress it when you save to get the size even smaller. I'd advise against this unless your back is really up against the wall.
I say in theory because it depends a lot on your data. Without knowing anything about your data, I really can't say any more on this, as some algorithms work better than others on certian kinds of data. In fact, simple information theory tells us that in some cases any compression algorithm will produce output that takes up more space than you started with.
If your bitset is rather sparse (not a lot of 0's, or not a lot of 1's), or is streaky (long runs of the same value), then it is possible you could get big gains with compression. In almost every other circumstance it won't be worth the trouble. Even in that circumstance it may not be. Remember that any code you add will need to be debugged and maintained.
As you point out, there is no reason to store your boolean values using any more space than a single bit. If you combine that with some basic construct, such as each row begins with an integer coding the number of bits in that row, you'll be able to store a 2D table of any size where each entry in the row is a single bit.
However, this is not enough. A string of arbitrary 1's and 0's will look rather random, and any compression algorithm breaks down as the randomness of your data increases - so I would recommend a process like Burrows-Wheeler Block sorting to greatly increase the amount of repeated "words" or "blocks" in your data. Once that's complete a simple Huffman code or Lempel-Ziv algorithm should be able to compress your file quite nicely.
To allow the above method to work for unsigned integers, you would compress the integers using Delta Codes, then perform the block sorting and compression (a standard practice in Information Retrieval postings lists).
If I understood the question correctly, the bits are random, and we have a random-length list of independently random-length lists. Since there is nothing to deal with bytes, I will discuss this as a bit stream. Since files actually contain bytes, you will need to put pack eight bits for each byte and leave the 0..7 bits of the last byte unused.
The most efficient way of storing the boolean values is as-is. Just dump them into the bitstream as a simple array.
In the beginning of the bitstream you need to encode the array lengths. There are many ways to do it and you can save a few bits by choosing the most optimal for your arrays. For this you will probably want to use huffman coding with a fixed codebook so that commonly used and small values get the shortest sequences. If the list is very long, you probably won't care so much about the size of it getting encoded in a longer form that is.
A precise answer as to what the codebook (and thus the huffman code) is going to be cannot be given without more information about the expected list lengths.
If all the inner lists are of the same size (i.e. you have a 2D array), you only need the two dimensions, of course.
Deserializing: decode the lengths and allocate the structures, then read the bits one by one, assigning them to the structure in order.
#zneak's answer (beat me to it), but use huffman encoded integers, especially if some lengths are more likely.
Just to be self-contained: Encode the number of lists as a huffman encoded integer, then for each list, encode its bit length as a huffman encoded integer. The bits for each list follow with no intervening wasted bits.
If the order of the lists doesn't matter, sorting them by length would reduce the space needed, only the incremental length increase of each subsequent list need be encoded.
List-of-List-of-Ints-binary:
Start traversing the input list
For each sublist:
Output 0xFF 0xFE
For each item in the sublist:
Output the item as a stream of bits, LSB first.
If the pattern 0xFF appears anywhere in the stream,
replace it with 0xFF 0xFD in the output.
Output 0xFF 0xFC
Decoding:
If the stream has ended then end any previous list and end reading.
Read bits from input stream. If pattern 0xFF is encountered, read the next 8 bits.
If they are 0xFE, end any previous list and begin a new one.
If they are 0xFD, assume that the value 0xFF has been read (discard the 0xFD)
If they are 0xFC, end any current integer at the bit before the pattern, and begin reading a new one at the bit after the 0xFC.
Otherwise indicate error.
If I understand correctly our data structure is ( 1 2 ( 33483 7 ) 373404 9 ( 337652222 37333788 ) )
Format like so:
byte 255 - escape code
byte 254 - begin block
byte 253 - list separator
byte 252 - end block
So we have:
struct {
int nmem; /* Won't overflow -- out of memory first */
int kind; /* 0 = number, 1 = recurse */
void *data; /* points to array of bytes for kind 0, array of bigdat for kind 1 */
} bigdat;
int serialize(FILE *f, struct bigdat *op) {
int i;
if (op->kind) {
unsigned char *num = (char *)op->data;
for (i = 0; i < op->nmem; i++) {
if (num[i] >= 252)
fputs(255, f);
fputs(num[i], f);
}
} else {
struct bigdat *blocks = (struct bigdat *)op->data
fputs(254, f);
for (i = 0; i < op->nmem; i++) {
if (i) fputs(253, f);
serialize(f, blocks[i]);
}
fputs(252, f);
}
There is a law about numeric digit distribution that says for sets of sets of arbitrary unsigned integers, the higher the byte value the less it happens so put special codes at the end.
Not encoding length in front of each takes up far less room, but makes deserialize a difficult exercise.
This question has a certain induction feel to it. You want a function: (bool list list) -> (bool list) such that an inverse function (bool list) -> (bool list list) generates the same original structure, and the length of the encoded bool list is minimal, without imposing restrictions on the input structure. Since this question is so abstract, I'm thinking these lists could be mind bogglingly large - 10^50 maybe, or 10^2000, or they can be very small, like 10^0. Also, there can be a large number of lists, again 10^50 or just 1. So the algorithm needs to adapt to these widely different inputs.
I'm thinking that we can encode the length of each list as a (bool list), and add one extra bool to indicate whether the next sequence is another (now larger) length or the real bitstream.
let encode2d(list1d::Bs) = encode1d(length(list1d), true) # list1d # encode2d(Bs)
encode2d(nil) = nil
let encode1d(1, nextIsValue) = true :: nextIsValue :: []
encode1d(len, nextIsValue) =
let bitList = toBoolList(len) # [nextIsValue] in
encode1d(length(bitList), false) # bitList
let decode2d(bits) =
let (list1d, rest) = decode1d(bits, 1) in
list1d :: decode2d(rest)
let decode1d(bits, n) =
let length = fromBoolList(take(n, bits)) in
let nextIsValue :: bits' = skip(n, bits) in
if nextIsValue then bits' else decode1d(bits', length)
assumed library functions
-------------------------
toBoolList : int -> bool list
this function takes an integer and produces the boolean list representation
of the bits. All leading zeroes are removed, except for input '0'
fromBoolList : bool list -> int
the inverse of toBoolList
take : int * a' list -> a' list
returns the first count elements of the list
skip : int * a' list -> a' list
returns the remainder of the list after removing the first count elements
The overhead is per individual bool list. For an empty list, the overhead is 2 extra list elements. For 10^2000 bools, the overhead would be 6645 + 14 + 5 + 4 + 3 + 2 = 6673 extra list elements.