I am trying to iterate a linked list in BPF, can I use bpf_loop for this?, if so how?
Tried using bpf_loops, but it required a fixed number to bound the loops, what way I can do that?
These are the comments for bpf_loop.
bpf_loop
For nr_loops, call callback_fn function
with callback_ctx as the context parameter.
The callback_fn should be a static function and
the callback_ctx should be a pointer to the stack.
The flags is used to control certain aspects of the helper.
Currently, the flags must be 0. Currently, nr_loops is
limited to 1 << 23 (~8 million) loops.
long (*callback_fn)(u32 index, void *ctx);
where index is the current index in the loop. The index
is zero-indexed.
If callback_fn returns 0, the helper will continue to the next
loop. If return value is 1, the helper will skip the rest of
the loops and return. Other return values are not used now,
and will be rejected by the verifier.
Returns
The number of loops performed, -EINVAL for invalid flags,
-E2BIG if nr_loops exceeds the maximum number of loops.
You can pass the linked list via the callback_ctx which is shared between iterations.
You can hardcode the number of loops to the max of 1 << 23 (~8 million) (perhaps less, it depends on how the verifier imposes penalties).
If you can advance the linked list you do so and perform whatever logic you want.
If you are done or at the end of the linked list you return 1 to
break out of the loop.
If you expect to need more than 8M iterations, you can at least detect when you are not done by looking at the return value (perhaps running a second loop, I don't know if that is allowed or not)
I was able to confirm from the documentation that bpf_map_update_elem is an atomic operation if done on HASH_MAPs. Source (https://man7.org/linux/man-pages/man2/bpf.2.html). [Cite: map_update_elem() replaces existing elements atomically]
My question is 2 folds.
What if the element does not exist, is the map_update_elem still atomic?
Is the XDP operation bpf_map_delete_elem thread safe from User space program?
The map is a HASH_MAP.
Atomic ops, race conditions and thread safety are sort of complex in eBPF, so I will make a broad answer since it is hard to judge from your question what your goals are.
Yes, both the bpf_map_update_elem command via the syscall and the helper function update the maps 'atmomically', which in this case means that if we go from value 'A' to value 'B' that the program always sees either 'A' or 'B' not some combination of the two(first bytes of 'B' and last bytes of 'A' for example). This is true for all map types. This holds true for all map modifying syscall commands(including bpf_map_delete_elem).
This however doesn't make race conditions impossible since the value of the map may have changed between a map_lookup_elem and the moment you update it.
What is also good to keep in mind is that the map_lookup_elem syscall command(userspace) works differently from the helper function(kernelspace). The syscall will always return a copy of the data which isn't mutable. But the helper function will return a pointer to the location in kernel memory where the map value is stored, and you can directly update the map value this way without using the map_update_elem helper. That is why you often see hash maps used like:
value = bpf_map_lookup_elem(&hash_map, &key);
if (value) {
__sync_fetch_and_add(&value->packets, 1);
__sync_fetch_and_add(&value->bytes, skb->len);
} else {
struct pair val = {1, skb->len};
bpf_map_update_elem(&hash_map, &key, &val, BPF_ANY);
}
Note that in this example, __sync_fetch_and_add is used to update parts of the map value. We need to do this since updating it like value->packets++; or value->packets += 1 would result in a race condition. The __sync_fetch_and_add emits a atomic CPU instruction which in this case fetches, adds and writes back all in one instruction.
Also, in this example, the two struct fields are atomically updated, but not together, it is still possible for the packets to have incremented but bytes not yet. If you want to avoid this you need to use a spinlock(using the bpf_spin_lock and bpf_spin_unlock helpers).
Another way to sidestep the issue entirely is to use the _PER_CPU variants of maps, where you trade-off congestion/speed and memory use.
I've tried searching and found nothing (don't know what these would be called if they were things already, so searching is kind of hard), so forgive me if this is dumb or already answered somewhere. For the sake of argument lets says I'm using bcrypt or something of that reputation/quality when I say I'm hashing something.
First, is there are reason that your hashing algorithm cannot vary with the password or it's intermediate hashes?
public static byte[] myHash(byte[] input, byte[] saltA, byte[] saltB) {
return input[0] % 2 == 0
? bcrypt(bcrypt(input, saltA), saltB)
: bcrypt(bcrypt(input, saltB), saltA);
}
I feel like this doesn't use much CPU - it's just two iterations of bcrypt, and I've suggestions for 10+ iterations in other security discussions - but let's say bcrypt has discovered to be fully reversible if you knew the salt and the hash, one unhashing would now necessitate unhashing it twice - once with saltA, then with saltB and once vice versa, giving you two candidate passwords, one of which is a decoy with a 50% false positive rate (that is, it rehashes to the correct hash because it's first bit is correctly even or odd), requiring heuristics or human eyes to correctly identify the real one, so you've at least doubled the computing resources needed and possibly required human intervention. But we can do better:
public static byte[] myBetterHash(byte[] input, byte[] saltA, byte[] saltB) {
byte[] curr = input;
for (int i = 0; i < 5; i++) {
switch(input[i] % 3) {
case 0: curr = bcrypt(bcrypt(bcrypt(curr, input), saltA), saltB); break;
case 1: curr = bcrypt(bcrypt(bcrypt(curr, saltB), input), saltA); break;
case 2: curr = bcrypt(bcrypt(bcrypt(curr, saltA), saltB), input); break;
}
}
return input;
}
Now there are 3 unhashes per iteration over the 5 iterations, yielding 243 candidate passwords, and probably dozens of false positives to eliminate, but even if not, then had to do 243 times the unhashing work they would have if you had just done it. Also, the inclusion of the input again as a salt in subsequent hashes makes it impossible to actually do the unhashing, plus it requires the attacker to hold onto a little extra memory. That said, my last idea is as follows:
public static byte[] myBestHash(byte[] input, byte[] saltA, byte[] saltB) {
byte[] curr = input;
byte[][] arr = new byte[16][]
for (int i = 0; i < 16; i++) {
arr[i] = curr;
switch(curr[0] % 4) {
case 0: curr = bcrypt(curr, saltA); break;
case 1: curr = bcrypt(curr, saltB); break;
case 2: curr = bcrypt(curr, arr[input[i] % i]); break;
case 3: curr = bcrypt(bcrypt(curr, saltA), saltB); break;
}
}
return input;
}
Now the attacker has to deal with an immense number of potential unhashings (3^16 = over 4 million), each of which has to be verified with the above memory intensive (it holds onto 16 intermediate hashes and there's no way to optimize that out).
Second, I feel like the memory intensiveness of that final example paired with the branching salts and maybe even the fact that one of the branches calls bcrypt twice instead of once might, in some combination, make things harder to brute force with graphics cards by making the tack at hand ill-suited to them or making the process waste more I/O than normal. If nothing else, extending this approach beyond 16 iterations will continue to bloat the RAM usage, making it harder to parallelize. Imagine if 256 iterations were used and space for 1024 intermediate hashes had to be held onto for every hash that got leaked in an attack - if the intermediate hashes themselves are, say 1024 bits (= 128 bytes), that's 32kB of wasted memory for every iteration of the brute force attack, which isn't much, but it will definitely add up to at least slightly slower iterations for a brute force attacker and fewer iterations in parallel (due to the extra memory - though 32kB isn't much against a modern password cracking rig, that's 32GB written to memory for a million guesses, and if nothing else that should slow things down a little extra).
So, am I onto something, or is this completely stupid?
This more properly belongs on crypto, but here's my $0.02:
Your calculations are slightly off and depend on some assumptions you make but which don't necessarily hold.
Now the idea of extra iterations isn't new - PBKDF2 uses iterations in a very similar way and bcrypt uses iterations internally already.
The "multi salt" idea gains you little in reality, since the salt is usually stored alongside the plaintext password in plaintext anyways.
This is one of those things where you think "well, if 1 function call and 1 salt are good, then 50 iterations and 2 salts are better. This couldn't possibly hurt!" But this isn't how cryptography works and this sort of thing could hurt although it doesn't in this case; it just wastes resources.
Please don't do this sort of thing blindly. If you want to increase resistance to brute force dictionary attacks against hashes, adjust the bcrypt difficulty factor. Similarly, choose a long and unique salt and let bcrypt do it's thing.
A) The adversary is obviously prepared to compute bcrypt() as most often used, for very low cost. B) Maybe also prepared to run N simple iterations, with some cheat, cheaper than we would expect. C) Very likely not able to reverse the hash function.
In any case the "outer" procedure using bcrypt() should be extremely stupidly written to lower the strength. Branching makes it harder for the adversary to use it's standard crack-rigs: certainly for A) and B), and maybe also for C) - but in case of C) there are big problems anyway.
Combine different types of cryptographic hashing functions, apply branching and unless something trivial mistake is applied it won't be weaker; but most likely harder for the adversary to apply the standard crack-rigs. If the base hash you're building on is fundamentally flawed (many assumes if it was the case they would know it: but that's just naive) the branching might not solve it. But likely make it harder to crack than easier. Wasting resources is good in this case: branching and different procedure than the "stupidly just apply it multiple times" will waste more adversary resources than honest resources: so good.
Consider a lookup function with the following signature, which needs to return an integer for a given string key:
int GetValue(string key) { ... }
Consider furthermore that the key-value mappings, numbering N, are known in advance when the source code for function is being written, e.g.:
// N=3
{ "foo", 1 },
{ "bar", 42 },
{ "bazz", 314159 }
So a valid (but not perfect!) implementation for the function for the input above would be:
int GetValue(string key)
{
switch (key)
{
case "foo": return 1;
case "bar": return 42;
case "bazz": return 314159;
}
// Doesn't matter what we do here, control will never come to this point
throw new Exception();
}
It is also known in advance exactly how many times (C>=1) the function will be called at run-time for every given key. For example:
C["foo"] = 1;
C["bar"] = 1;
C["bazz"] = 2;
The order of such calls is not known, however. E.g. the above could describe the following sequence of calls at run-time:
GetValue("foo");
GetValue("bazz");
GetValue("bar");
GetValue("bazz");
or any other sequence, provided the call counts match.
There is also a restriction M, specified in whatever units is most convenient, defining the upper memory bound of any lookup tables and other helper structures that can be used by the GetValue (the structures are initialized in advance; that initialization is not counted against the complexity of the function). For example, M=100 chars, or M=256 sizeof(object reference).
The question is, how to write the body of GetValue such that it is as fast as possible - in other words, the aggregate time of all GetValue calls (note that we know the total count, per everything above) is minimal, for given N, C and M?
The algorithm may require a reasonable minimal value for M, e.g. M >= char.MaxValue. It may also require that M be aligned to some reasonable boundary - for example, that it may only be a power of two. It may also require that M must be a function of N of a certain kind (for example, it may allow valid M=N, or M=2N, ...; or valid M=N, or M=N^2, ...; etc).
The algorithm can be expressed in any suitable language or other form. For runtime performance constrains for generated code, assume that the generated code for GetValue will be in C#, VB or Java (really, any language will do, so long as strings are treated as immutable arrays of characters - i.e. O(1) length and O(1) indexing, and no other data computed for them in advance). Also, to simplify this a bit, answers which assume that C=1 for all keys are considered valid, though those answers which cover the more general case are preferred.
Some musings on possible approaches
The obvious first answer to the above is using a perfect hash, but generic approaches to finding one seem to be imperfect. For example, one can easily generate a table for a minimal perfect hash using Pearson hashing for the sample data above, but then the input key would have to be hashed for every call to GetValue, and Pearson hash necessarily scans the entire input string. But all sample keys actually differ in their third character, so only that can be used as the input for the hash instead of the entire string. Furthermore, if M is required to be at least char.MaxValue, then the third character itself becomes a perfect hash.
For a different set of keys this may no longer be true, but it may still be possible to reduce the amount of characters considered before the precise answer can be given. Furthermore, in some cases where a minimal perfect hash would require inspecting the entire string, it may be possible to reduce the lookup to a subset, or otherwise make it faster (e.g. a less complex hashing function?) by making the hash non-minimal (i.e. M > N) - effectively sacrificing space for the sake of speed.
It may also be that traditional hashing is not such a good idea to begin with, and it's easier to structure the body of GetValue as a series of conditionals, arranged such that the first checks for the "most variable" character (the one that varies across most keys), with further nested checks as needed to determine the correct answer. Note that "variance" here can be influenced by the number of times each key is going to be looked up (C). Furthermore, it is not always readily obvious what the best structure of branches should be - it may be, for example, that the "most variable" character only lets you distinguish 10 keys out of 100, but for the remaining 90 that one extra check is unnecessary to distinguish between them, and on average (considering C) there are more checks per key than in a different solution which does not start with the "most variable" character. The goal then is to determine the perfect sequence of checks.
You could use the Boyer search, but I think that the Trie would be a much more effiecent method. You can modify the Trie to collapse the words as you make the hit count for a key zero, thus reducing the number of searches you would have to do the farther down the line you get. The biggest benefit you would get is that you are doing array lookups for the indexes, which is much faster than a comparison.
You've talked about a memory limitation when it comes to precomputation - is there also a time limitation?
I would consider a trie, but one where you didn't necessarily start with the first character. Instead, find the index which will cut down the search space most, and consider that first. So in your sample case ("foo", "bar", "bazz") you'd take the third character, which would immediately tell you which string it was. (If we know we'll always be given one of the input words, we can return as soon as we've found a unique potential match.)
Now assuming that there isn't a single index which will get you down to a unique string, you need to determine the character to look at after that. In theory you precompute the trie to work out for each branch what the optimal character to look at next is (e.g. "if the third character was 'a', we need to look at the second character next; if it was 'o' we need to look at the first character next) but that potentially takes a lot more time and space. On the other hand, it could save a lot of time - because having gone down one character, each of the branches may have an index to pick which will uniquely identify the final string, but be a different index each time. The amount of space required by this approach would depend on how similar the strings were, and might be hard to predict in advance. It would be nice to be able to dynamically do this for all the trie nodes you can, but then when you find you're running out of construction space, determine a single order for "everything under this node". (So you don't end up storing a "next character index" on each node underneath that node, just the single sequence.) Let me know if this isn't clear, and I can try to elaborate...
How you represent the trie will depend on the range of input characters. If they're all in the range 'a'-'z' then a simple array would be incredibly fast to navigate, and reasonably efficient for trie nodes where there are possibilities for most of the available options. Later on, when there are only two or three possible branches, that becomes wasteful in memory. I would suggest a polymorphic Trie node class, such that you can build the most appropriate type of node depending on how many sub-branches there are.
None of this performs any culling - it's not clear how much can be achieved by culling quickly. One situation where I can see it helping is when the number of branches from one trie node drops to 1 (because of the removal of a branch which is exhausted), that branch can be eliminated completely. Over time this could make a big difference, and shouldn't be too hard to compute. Basically as you build the trie you can predict how many times each branch will be taken, and as you navigate the trie you can subtract one from that count per branch when you navigate it.
That's all I've come up with so far, and it's not exactly a full implementation - but I hope it helps...
Is a binary search of the table really so awful? I would take the list of potential strings and "minimize" them, the sort them, and finally do a binary search upon the block of them.
By minimize I mean reducing them to the minimum they need to be, kind of a custom stemming.
For example if you had the strings: "alfred", "bob", "bill", "joe", I'd knock them down to "a", "bi", "bo", "j".
Then put those in to a contiguous block of memory, for example:
char *table = "a\0bi\0bo\0j\0"; // last 0 is really redundant..but
char *keys[4];
keys[0] = table;
keys[1] = table + 2;
keys[2] = table + 5;
keys[3] = table + 8;
Ideally the compiler would do all this for you if you simply go:
keys[0] = "a";
keys[1] = "bi";
keys[2] = "bo";
keys[3] = "j";
But I can't say if that's true or not.
Now you can bsearch that table, and the keys are as short as possible. If you hit the end of the key, you match. If not, then follow the standard bsearch algorithm.
The goal is to get all of the data close together and keep the code itty bitty so that it all fits in to the CPU cache. You can process the key from the program directly, no pre-processing or adding anything up.
For a reasonably large number of keys that are reasonably distributed, I think this would be quite fast. It really depends on the number of strings involved. For smaller numbers, the overhead of computing hash values etc is more than search something like this. For larger values, it's worth it. Just what those number are all depends on the algorithms etc.
This, however, is likely the smallest solution in terms of memory, if that's important.
This also has the benefit of simplicity.
Addenda:
You don't have any specifications on the inputs beyond 'strings'. There's also no discussion about how many strings you expect to use, their length, their commonality or their frequency of use. These can perhaps all be derived from the "source", but not planned upon by the algorithm designer. You're asking for an algorithm that creates something like this:
inline int GetValue(char *key) {
return 1234;
}
For a small program that happens to use only one key all the time, all the way up to something that creates a perfect hash algorithm for millions of strings. That's a pretty tall order.
Any design going after "squeezing every single bit of performance possible" needs to know more about the inputs than "any and all strings". That problem space is simply too large if you want it the fastest possible for any condition.
An algorithm that handles strings with extremely long identical prefixes might be quite different than one that works on completely random strings. The algorithm could say "if the key starts with "a", skip the next 100 chars, since they're all a's".
But if these strings are sourced by human beings, and they're using long strings of the same letters, and not going insane trying to maintain that data, then when they complain that the algorithm is performing badly, you reply that "you're doing silly things, don't do that". But we don't know the source of these strings either.
So, you need to pick a problem space to target the algorithm. We have all sorts of algorithms that ostensibly do the same thing because they address different constraints and work better in different situations.
Hashing is expensive, laying out hashmaps is expensive. If there's not enough data involved, there are better techniques than hashing. If you have large memory budget, you could make an enormous state machine, based upon N states per node (N being your character set size -- which you don't specify -- BAUDOT? 7-bit ASCII? UTF-32?). That will run very quickly, unless the amount of memory consumed by the states smashes the CPU cache or squeezes out other things.
You could possibly generate code for all of this, but you may run in to code size limits (you don't say what language either -- Java has a 64K method byte code limit for example).
But you don't specify any of these constraints. So, it's kind of hard to get the most performant solution for your needs.
What you want is a look-up table of look-up tables.
If memory cost is not an issue you can go all out.
const int POSSIBLE_CHARCODES = 256; //256 for ascii //65536 for unicode 16bit
struct LutMap {
int value;
LutMap[POSSIBLE_CHARCODES] next;
}
int GetValue(string key) {
LutMap root = Global.AlreadyCreatedLutMap;
for(int x=0; x<key.length; x++) {
int c = key.charCodeAt(x);
if(root.next[c] == null) {
return root.value;
}
root = root.next[c];
}
}
I reckon that it's all about finding the right hash function. As long as you know what the key-value relationship is in advance, you can do an analysis to try and find a hash function to meet your requrements. Taking the example you've provided, treat the input strings as binary integers:
foo = 0x666F6F (hex value)
bar = 0x626172
bazz = 0x62617A7A
The last column present in all of them is different in each. Analyse further:
foo = 0xF = 1111
bar = 0x2 = 0010
bazz = 0xA = 1010
Bit-shift to the right twice, discarding overflow, you get a distinct value for each of them:
foo = 0011
bar = 0000
bazz = 0010
Bit-shift to the right twice again, adding the overflow to a new buffer:
foo = 0010
bar = 0000
bazz = 0001
You can use those to query a static 3-entry lookup table. I reckon this highly personal hash function would take 9 very basic operations to get the nibble (2), bit-shift (2), bit-shift and add (4) and query (1), and a lot of these operations can be compressed further through clever assembly usage. This might well be faster than taking run-time infomation into account.
Have you looked at TCB . Perhaps the algorithm used there can be used to retrieve your values. It sounds a lot like the problem you are trying to solve. And from experience I can say tcb is one of the fastest key store lookups I have used. It is a constant lookup time, regardless of the number of keys stored.
Consider using Knuth–Morris–Pratt algorithm.
Pre-process given map to a large string like below
String string = "{foo:1}{bar:42}{bazz:314159}";
int length = string.length();
According KMP preprocessing time for the string will take O(length).
For searching with any word/key will take O(w) complexity, where w is length of the word/key.
You will be needed to make 2 modification to KMP algorithm:
key should be appear ordered in the joined string
instead of returning true/false it should parse the number and return it
Wish it can give a good hints.
Here's a feasible approach to determine the smallest subset of chars to target for your hash routine:
let:
k be the amount of distinct chars across all your keywords
c be the max keyword length
n be the number of keywords
in your example (padded shorter keywords w/spaces):
"foo "
"bar "
"bazz"
k = 7 (f,o,b,a,r,z, ), c = 4, n = 3
We can use this to compute a lower bound for our search. We need at least log_k(n) chars to uniquely identify a keyword, if log_k(n) >= c then you'll need to use the whole keyword and there's no reason to proceed.
Next, eliminate one column at a time and check if there are still n distinct values remaining. Use the distinct chars in each column as a heuristic to optimize our search:
2 2 3 2
f o o .
b a r .
b a z z
Eliminate columns with the lowest distinct chars first. If you have <= log_k(n) columns remaining you can stop. Optionally you could randomize a bit and eliminate the 2nd lowest distinct col or try to recover if the eliminated col results in less than n distinct words. This algorithm is roughly O(n!) depending on how much you try to recover. It's not guaranteed to find an optimal solution but it's a good tradeoff.
Once you have your subset of chars, proceed with the usual routines for generating a perfect hash. The result should be an optimal perfect hash.
The online documentation says
Hash[expr]
gives an integer hash code for the expression expr.
Hash[expr,"type"]
gives an integer hash code of the specified type for expr.
It also gives "possible hash code types":
"Adler32" Adler 32-bit cyclic redundancy check
"CRC32" 32-bit cyclic redundancy check
"MD2" 128-bit MD2 code
"MD5" 128-bit MD5 code
"SHA" 160-bit SHA-1 code
"SHA256" 256-bit SHA code
"SHA384" 384-bit SHA code
"SHA512" 512-bit SHA code
Yet none of these correspond to the default returned by Hash[expr].
So my questions are:
What method does the default Hash use?
Are there any other hash codes built in?
The default hash algorithm is, more-or-less, a basic 32-bit hash function applied to the underlying expression representation, but the exact code is a proprietary component of the Mathematica kernel. It's subject to (and has) change between Mathematica versions, and lacks a number of desirable cryptographic properties, so I personally recommend you use MD5 or one of the SHA variants for any serious application where security matters. The built-in hash is intended for typical data structure use (e.g. in a hash table).
The named hash algorithms you list from the documentation are the only ones currently available. Are you looking for a different one in particular?
I've been doing some reverse engeneering on 32 and 64 bit Windows version of Mathematica 10.4 and that's what I found:
32 BIT
It uses a Fowler–Noll–Vo hash function (FNV-1, with multiplication before) with 16777619 as FNV prime and 84696351 as offset basis. This function is applied on Murmur3-32 hashed value of the address of expression's data (MMA uses a pointer in order to keep one instance of each data). The address is eventually resolved to the value - for simple machine integers the value is immediate, for others is a bit trickier. The Murmur3-32 implementing function contains in fact an additional parameter (defaulted with 4, special case making behaving as in Wikipedia) which selects how many bits to choose from the expression struct in input. Since a normal expression is internally represented as an array of pointers, one can take the first, the second etc.. by repeatedly adding 4 (bytes = 32 bit) to the base pointer of the expression. So, passing 8 to the function will give the second pointer, 12 the third and so on. Since internal structs (big integers, machine integers, machine reals, big reals and so on) have different member variables (e.g. a machine integer has only a pointer to int, a complex 2 pointers to numbers etc..), for each expression struct there is a "wrapper" that combine its internal members in one single 32-bit hash (basically with FNV-1 rounds). The simplest expression to hash is an integer.
The murmur3_32() function has 1131470165 as seed, n=0 and other params as in Wikipedia.
So we have:
hash_of_number = 16777619 * (84696351 ^ murmur3_32( &number ))
with " ^ " meaning XOR.
I really didn't try it - pointers are encoded using WINAPI EncodePointer(), so they can't be exploited at runtime. (May be worth running in Linux under Wine with a modified version of EncodePonter?)
64 BIT
It uses a FNV-1 64 bit hash function with 0xAF63BD4C8601B7DF as offset basis and 0x100000001B3 as FNV prime, along with a SIP64-24 hash (here's the reference code) with the first 64 bit of 0x0AE3F68FE7126BBF76F98EF7F39DE1521 as k0 and the last 64 bit as k1. The function is applied to the base pointer of the expression and resolved internally. As in 32-bit's murmur3, there is an additional parameter (defaulted to 8) to select how many pointers to choose from the input expression struct. For each expression type there is a wrapper to condensate struct members into a single hash by means of FNV-1 64 bit rounds.
For a machine integer, we have:
hash_number_64bit = 0x100000001B3 * (0xAF63BD4C8601B7DF ^ SIP64_24( &number ))
Again, I didn't really try it. Could anyone try?
Not for the faint-hearted
If you take a look at their notes on internal implementation, they say that "Each expression contains a special form of hash code that is used both in pattern matching and evaluation."
The hash code they're referring to is the one generated by these functions - at some point in the normal expression wrapper function there's an assignment that puts the computed hash inside the expression struct itself.
It would certainly be cool to understand HOW they can make use of these hashes for pattern matching purpose. So I had a try running through the bigInteger wrapper to see what happens - that's the simplest compound expression.
It begins checking something that returns 1 - dunno what.
So it executes
var1 = 16777619 * (67918732 ^ hashMachineInteger(1));
with hashMachineInteger() is what we said before - including values.
Then it reads the length in bytes of the bigInt from the struct (bignum_length) and runs
result = 16777619 * (v10 ^ murmur3_32(v6, 4 * v4));
Note that murmur3_32() is called if 4 * bignum_length is greater than 8 (may be related to the max value of machine integers $MaxMachineNumber 2^32^32 and by converse to what a bigInt is supposed to be).
So, the final code is
if (bignum_length > 8){
result = 16777619 * (16777619 * (67918732 ^ ( 16777619 * (84696351 ^ murmur3_32( 1, 4 )))) ^ murmur3_32( &bignum, 4 * bignum_length ));
}
I've made some hypoteses on the properties of this construction. The presence of many XORs and the fact that 16777619 + 67918732 = 84696351 may make one think that some sort of cyclic structure is exploited to check patterns - i.e. subtracting the offset and dividing by the prime, or something like that. The software Cassandra uses the Murmur hash algorithm for token generation - see these images for what I mean with "cyclic structure". Maybe various primes are used for each expression - must still check.
Hope it helps
It seems that Hash calls the internal Data`HashCode function, then divides it by 2, takes the first 20 digits of N[..] and then the IntegerPart, plus one, that is:
IntegerPart[N[Data`HashCode[expr]/2, 20]] + 1