XOR Key normalizing and hashing

Comparing XOR keys can be a bit of a pain, typically one would expect attackers to use the same shellcode to extract embedded executable code, giving the XOR key another similarity to compare when looking at different attacks.

One of the things we came up with early on is a key sum, simply counting the ascii values:

for ($i = 0; $i < strlen($key) - 1; $i+=2) {
$sum += hexdec($key[$i].$key[$i+1]);
}

This can be helpful with shorter keys, however given a 256 byte key counting up from 00 or down from FF can have the same key sum = 32640. This is where another solution is needed - a key hash. Keys can be found anywhere in a document, so hashing a key needs an extra step - normalizing the key. We decided to go with finding the highest values as the starting point - so the key would start with FF if present, if two or more FF appear, the next value is compared to get the highest multi-byte value. Once we have the normalized key, we take a simply md5:

Given this key:

080706050403020100fffefdfcfbfaf9f8f7f6f5f4f3f2f1f0efeeedecebeae9e8e7e6e5e4e3e2e1e0dfdedddcdbdad9d8d7d6d5d4d3d2d1d0cfcecdcccbcac9c8c7c6c5c4c3c2c1c0bfbebdbcbbbab9b8b7b6b5b4b3b2b1b0afaeadacabaaa9a8a7a6a5a4a3a2a1a09f9e9d9c9b9a999897969594939291908f8e8d8c8b8a898887868584838281807f7e7d7c7b7a797877767574737271706f6e6d6c6b6a696867666564636261605f5e5d5c5b5a595857565554535251504f4e4d4c4b4a494847464544434241403f3e3d3c3b3a393837363534333231302f2e2d2c2b2a292827262524232221201f1e1d1c1b1a191817161514131211100f0e0d0c0b0a09

When normalized becomes:

fffefdfcfbfaf9f8f7f6f5f4f3f2f1f0efeeedecebeae9e8e7e6e5e4e3e2e1e0dfdedddcdbdad9d8d7d6d5d4d3d2d1d0cfcecdcccbcac9c8c7c6c5c4c3c2c1c0bfbebdbcbbbab9b8b7b6b5b4b3b2b1b0afaeadacabaaa9a8a7a6a5a4a3a2a1a09f9e9d9c9b9a999897969594939291908f8e8d8c8b8a898887868584838281807f7e7d7c7b7a797877767574737271706f6e6d6c6b6a696867666564636261605f5e5d5c5b5a595857565554535251504f4e4d4c4b4a494847464544434241403f3e3d3c3b3a393837363534333231302f2e2d2c2b2a292827262524232221201f1e1d1c1b1a191817161514131211100f0e0d0c0b0a09080706050403020100

And the hash of the above is c9b69399459095f1b991eb1997a4d066.

Given this key:

4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfe0e1e2e3

Will be normalized to:

ff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfe0e1e2e034e5e6e7e8e9eaebecedeee

And hashed as 9b7e14df2ffe3c32566f3fff8481c6f2.

Normalize XOR key source code:

function normalizeKey($key) {
$values = hex2str($key).hex2str($key);
$size= strlen($values) / 2;
$high = chr(0x00);
$highest = '';
$highestLoc = 0;
for ($j = 0; $j < $size; $j++) {
for ($i = 0; $i < $size; $i++) {
if (strlen($highest) > 0) {
$check = substr($values,$i,strlen($highest));
//echo "Compare [".dechex(ord($highest))."] and [".dechex(ord($check))."]\n";
if ($highest == $check) {
//echo "Found partial [".dechex(ord($high))."]\n";
$pos = $i+strlen($highest);
if ($values[$pos] > $high) {
//echo strhex($highest)." ".strlen($highest)."\n";
$highestLoc = $i-1;
$high = $values[$pos];
//echo "found highest at $highestLoc [".strhex($highest).dechex(ord($high))."]\n";
}
}
} else {
if ($values[$i] > $high) {
$highestLoc = $i-1;
$high = $values[$i];
//echo "found highest at $highestLoc [".dechex(ord($high))."]\n";

}
}

}
$highest .= $high;
$high = chr(0x00);

$search = '';
for ($l = 0; $l < strlen($highest); $l++) {
$search .= "\x".dechex(ord($highest[$l]));
}

if (preg_match_all("/$search/s", $values, $matches, PREG_OFFSET_CAPTURE)) {
if (count($matches[0]) <= 2) {
break;

}
}

}

$new = '';
for($i = $highestLoc+1; $i < $highestLoc+$size+1; $i++) {
$new .= $values[$i];
}

return strhex($new);
}

Now you can search on ViCheck for similar attacks by searching either the XOR Key sum, or key hash by clicking the "more" link on the report page on ViCheck.ca.