Traditional signatured-based Malicious Software (malware) detection systems, which are based on files stored on disk, nowadays are often called to be obsolete. The vast number of different possibilities to obfuscate a malicious executable are a huge problem for these detection systems. The level of obfuscation can be differed by the constant evolving generations of improved techniques. Most of the obfuscation techniques hide their constant code until its final execution. In this final step the constant code of a malicious software is present in the memory.
To monitor malicious behavior and to overcome most of the obfuscation techniques, the examination of memory seems plausible. A non-intrusive and lightweight method was described earlier [41], which is based on the monitoring of executable memory regions in real-time. The idea of monitoring malicious behavior by customizing the memory manager of an operating system has several benefits. Especially the possibility to examine malware in an unpacked and unencrypted manner.
With the development of several independent algorithms and the improvement of these algorithms, the application of Approximate Matching algorithms is getting more and more suitable for different areas [6]. The application of Similarity or Fuzzy Hashing algorithms to the field of fragment detection in memory was already mentioned in some publications. The main goal of this thesis is the examination, ifthe adaptation of existing Fuzzy Hashing algorithms leads to new signatures.
The thesis introduces several problems and considerations, which have to be respected before the application of a hash function on memory fragments. First, an extended literature review was done to identify other scientific sources, which try to apply hash functions on memory. In the main chapter of this work an approach is introduced to hash memory fragments and extract features in a reliable manner.This work introduces a twofold approach, which tries to respect the mutable parts of an executable in memory and to overcome several obfuscation techniques.
The performance is tested by the implementation of a memory acquiring debugger plugin. An additional script was implemented to simulate assembly obfuscation.