Add basic entropy calculation

Author: malvidin

README.md

@@ -51,92 +51,104 @@ The above example can be explained as:
 - Pass the output of `hex` to `emit` with the argument `'decrypted'`, creating a `decrypted` field
 
 ## Functions
-`btoa()`
-Encodes input to a Base64 string.
+### `btoa()`
+- Encodes input to a Base64 string.
 
 `b64(), atob()`
-Decodes a Base64 encoded string.
+- Decodes a Base64 encoded string.
 
 `b32()`
-Decodes a Base32 encoded string.
+- Decodes a Base32 encoded string.
 
 `b58()`
-Decodes a Base58 encoded string.
+- Decodes a Base58 encoded string.
 
 `rotx(count)`
-Implements Caesarian shift. The count argument specifies the amount to shift and must be an integer.
+- Implements Caesarian shift. The count argument specifies the amount to shift and must be an integer.
 
 `rol(count)`
-Implements rotate-on-left to each character within the string using an 8 bit boundary. The count argument specifies the amount to rotate and must be an integer.
+- Implements rotate-on-left to each character within the string using an 8 bit boundary. The count argument specifies the amount to rotate and must be an integer.
 
 `ror(count)`
-Implements rotate-on-right to each character within the string using an 8 bit boundary. The count argument specifies the amount to rotate and must be an integer.
+- Implements rotate-on-right to each character within the string using an 8 bit boundary. 
+- The count argument specifies the amount to rotate and must be an integer.
 
 `xor(key)`
-Implements basic XOR cipher against the field with the supplied key. The key can be provided as a string or integer.
+- Implements basic XOR cipher against the field with the supplied key. 
+- The key can be provided as a string or integer.
 
 `rc4('key')`
-Implements the RC4 cipher against the field with the supplied key. The key provided must be a string.
+- Implements the RC4 cipher against the field with the supplied key. 
+- The key provided must be a string.
 
 `hex()`
-Transforms input into its hexadecimal representation.
+- Transforms input into its hexadecimal representation.
 
 `unhex()`
-Transforms hexadecimal input into its byte form.
+- Transforms hexadecimal input into its byte form.
 
 `save('name')`
-Saves the current state to memory as name.
+- Saves the current state to memory as name.
 
 `load('name')`
-Recalls the previously saved state name from memory.
+- Recalls the previously saved state name from memory.
 
 `ascii()`
-Transforms input into ASCII output. Non-printable characters will be replaced with a period.
+- Transforms input into ASCII output. Non-printable characters will be replaced with a period.
 
 `emit('name')`
-Outputs the current state as UTF-8 to the field name.
+- Outputs the current state as UTF-8 to the field name.
 
 `substr(offset, count)`
-Returns a substring of the input, starting at the index offset with the number of characters count. Set the count to `'null'` to return from the start offset to the end of the input.
+- Returns a substring of the input, starting at the index offset with the number of characters count. 
+- Set the count to `'null'` to return from the start offset to the end of the input.
 
 `slice(start, end)`
-Returns a slice of the input, starting at start offset to the end offset. Set the end to `'null'` to go to the end of the input.
+- Returns a slice of the input, starting at start offset to the end offset. 
+- Set the end to `'null'` to go to the end of the input.
 
 `decode('codec')`
-Returns a decoded version of the input based on the codec, python codec list is available on https://docs.python.org/3/library/codecs.html#standard-encodings
+- Returns a decoded version of the input based on the codec.
+- Python codec list is available on https://docs.python.org/3/library/codecs.html#standard-encodings
 
 `escape`
-Returns a string where control characters, \, and non-ASCII characters are backslash escaped (e.g. `\x0a`, `\\`, `\x80`).
+- Returns a string where control characters, \, and non-ASCII characters are backslash escaped (e.g. `\x0a`, `\\`, `\x80`).
 
 `unescape`
-Returns a string run through python unicode_escape (i.e. return the unicode point(s)). Reverses `escape`. Also unescapes Unicode codepoints (`\uxxxx` or `\Uxxxxxxxx`), which `escape` does not produce.
+- Returns a string run through python unicode_escape (i.e. return the unicode point(s)). Reverses `escape`. 
+- Also unescapes Unicode codepoints (`\uxxxx` or `\Uxxxxxxxx`), which `escape` does not produce.
 
 `htmlescape`
-Returns a string with `&`, `<`, and `>` XML escaped like `&amp;`.
+- Returns a string with `&`, `<`, and `>` XML escaped like `&amp;`.
 
 `htmlunescape`
-Returns a string with HTML references like `&gt;` and `&#62;` unescaped to `>`.
+- Returns a string with HTML references like `&gt;` and `&#62;` unescaped to `>`.
 
 `tr('from', 'to')`
-Takes an argument to translate "from" and an argument of characters to translate "to" and then returns a result with the result (similar to `tr` in Unix).
+- Takes an argument to translate "from" and an argument of characters to translate "to" and then returns a result with the result (similar to `tr` in Unix).
 
 `rev()`
-Returns the input in reverse order.
+- Returns the input in reverse order.
 
 `find('subseq', start)`
-Returns the index of a subsequence "subseq" starting at index "start", or `-1` if the subsequence is not found.
+- Returns the index of a subsequence "subseq" starting at index "start", or `-1` if the subsequence is not found.
 
 `b32re()`
-Returns a reverse-endian base32 decoded string, as used in the SunBurst DGA.
+- Returns a reverse-endian base32 decoded string, as used in the SunBurst DGA.
 
 `b64re()`
-Returns a reverse-endian base64 decoded string.
+- Returns a reverse-endian base64 decoded string.
 
 `zlib_inflate()`
-Returns zlib.decompress() inflated bytes. Default window size of -15 (raw inflate) is used if a wbits value is not provided.
+- Returns zlib.decompress() inflated bytes. 
+- Default window size of -15 (raw inflate) is used if a wbits value is not provided.
 
 `zlib_deflate()`
-Returns zlib.compress() deflated bytes. Default level of -1 (currently 6) and window size of -15 (raw deflate) if values are not provided.
+- Returns zlib.compress() deflated bytes. 
+- Default level of -1 (currently 6) and window size of -15 (raw deflate) if values are not provided.
+
+`entropy()`
+- Returns base2 entropy of input. The maximum entropy for Unicode strings can be greater than 8.
 
 _Note: you must use **single quotes** around the strings._
 
@@ -152,6 +164,7 @@ Unicode values can be expressed as `\u0000` or `\U00000000`
 Quotation marks (double quotes) **cannot** be used.
 
 `"This is not a valid string"`
+
 ## Integers
 Integers can be specified numerically or as hexadecimal representations by prefixing values with a 0x.
 
@@ -234,6 +247,7 @@ Steven (malvidin on github)
 ## 2.4.1
 - Added support for null argument padding, so `find('decrypt2')` is equivalent to `find('decrypt2', 0)`
 - Added zlib_deflate for internal validation of zlib_inflate, which can also be used for information analysis
+- Add basic entropy calculation
 
 ## 2.4.0
 Merged pull request from Steven (malvidin on github)

bin/lib/decryptlib.py

@@ -4,9 +4,11 @@
 import base64
 import binascii
 import itertools
+import math
 import string
 import tokenize
 import zlib
+from collections import Counter
 from html import escape as html_escape, unescape as html_unescape
 from io import StringIO
 
@@ -494,6 +496,13 @@ def fn_zlib_deflate(data, args):
         raise Exception(f"{fn_name}(): {exc}\n" + zlib.compressobj.__doc__)
 
 
+@numargs(0)
+def fn_entropy(data, args):
+    data_len = len(data)
+    counter = Counter(data)
+    return abs(sum(c/data_len * math.log2(c/data_len) for c in counter.values()))
+
+
 def get_args(g):
     global g_record
     global g_register
@@ -655,5 +664,8 @@ def parse_statement(s):
         elif cmd == "zlib_deflate":
             yield fn_zlib_deflate, get_args(g)
 
+        elif cmd == "entropy":
+            yield fn_entropy, get_args(g)
+
         else:
             raise Exception(f"'{cmd}' is not a recognized command")

default/searchbnf.conf

@@ -43,5 +43,5 @@ usage = public
 #tags = searchcommands_app
 #
 [decrypt-options]
-syntax = field=<string> atob | b64 | btoa | b32 | b58 | hex | unhex | rol(<int>) | ror(<int>) | rotx('<string>') | xor('<string>') | rc4('<string>') | emit('<string>') | load('<string>') | save('<string>') | substr(<int>, <int>) | slice(<int>, <int>) | ascii | decode('<string>') | escape | unescape | tr('<string>', '<string>') | rev | find('<string>', <int>) | b32re | b64re | zlib_inflate(<int>)| zlib_deflate(<int>, <int>)
+syntax = field=<string> atob | b64 | btoa | b32 | b58 | hex | unhex | rol(<int>) | ror(<int>) | rotx('<string>') | xor('<string>') | rc4('<string>') | emit('<string>') | load('<string>') | save('<string>') | substr(<int>, <int>) | slice(<int>, <int>) | ascii | decode('<string>') | escape | unescape | tr('<string>', '<string>') | rev | find('<string>', <int>) | b32re | b64re | zlib_inflate(<int>)| zlib_deflate(<int>, <int>) | entropy()
 description = Pass the field name to work with, then the command or command(s) to be used, an emit() option can be passed to choose the field to return, defaults to the field name "decrypted"