XOR Encryption: Breaking Down Encryption and Evading Signatures

Shellcode Encryption with XOR: Protecting Your Payload

In the previous post on shellcode injection, we learned how to create and inject machine code directly into memory. But there’s a serious problem we can’t ignore: if you leave your shellcode unencrypted, antivirus will detect it in seconds.

Antivirus and EDR (Endpoint Detection and Response) systems scan binaries looking for patterns of known shellcode. It’s like leaving a sign that says “Malicious code here!”. To evade this detection, we need to obfuscate our payload, and XOR is the simplest, fastest, and most effective tool to do it.

In this complete tutorial, you won’t just learn how to implement XOR step by step, but you’ll understand the mathematics behind the operation, explore three different variants (from the most basic to the most robust), and discover exactly why it works… and also its limitations.

What is XOR? The Gateway to Simple Cryptography

Before implementing, we need to understand what XOR really is and why it works so well for obfuscating shellcode.

The XOR Operation (Exclusive OR)

XOR is a fundamental logical operation in computing. If you’ve never worked with bits, don’t worry: it’s very simple. XOR compares two bits and returns 1 if they are different, and 0 if they are equal.

XOR Operation Table

A	B	A XOR B
0	0	0
0	1	1
1	0	1
1	1	0

In C code, XOR is represented with the ^ operator:

unsigned char result = 0xAB ^ 0x42;  // Example: 171 XOR 66 = 213

The Fundamental Property: Reversibility

The most important characteristic of XOR is its reversibility. This is what makes it perfect for obfuscation: if you apply XOR twice with the same key, you get the original value back.

Mathematically:

A XOR B = C
C XOR B = A

Practical Example:

unsigned char original_data = 0x72;  // 114 in decimal (letter 'r')
unsigned char key = 0x33;             // 51 in decimal

// Encrypt
unsigned char encrypted = original_data ^ key;  // 0x72 ^ 0x33 = 0x41 (65)

// Decrypt (the SAME operation!)
unsigned char decrypted = encrypted ^ key;     // 0x41 ^ 0x33 = 0x72 (114)
// Result: decrypted == original_data ✓

Why is this important? Because we can use exactly the same function for both encrypting and decrypting. You don’t need different logic for each one. Apply XOR with the key and you’re done.

Why XOR for Shellcode?

XOR is the star of shellcode obfuscation. Not because it’s the best encryption (spoiler: it isn’t), but because it’s practically perfect for this specific use:

Advantages:

• ⚡ Extreme speed - One operation per byte, with no perceptible latency
• 📦 No dependencies - You don’t need complex cryptographic libraries
• 🔄 Bidirectional - Use the same function for both encrypting and decrypting
• 📝 Minimalist code - Just a few lines of C
• 🎭 Breaks signatures - Transforms recognizable data into “random” noise

Limitations are important:

• ❌ Not secure against experts - If someone dedicated wants to break it, they will
• ❌ Vulnerable if repeated - If the attacker sees multiple examples of the same shellcode, they can find patterns

The Mathematics Behind XOR: Understanding Obfuscation

How XOR Transforms Bytes

This is where the magic happens. When you XOR a byte, each one of its bits is compared with the corresponding bit of the key. Let’s see a real example:

Original shellcode:  0x72    (114 in decimal)
Key:                 0x33    (51 in decimal)
                     ----
Result:              0x41    (65 in decimal)

In binary:

01110010  (0x72)
00110011  (0x33)
--------
01000001  (0x41)  ← This is completely different

The important thing is that this process repeats for each byte of the shellcode. A 1000-byte shellcode is completely transformed into 1000 “noisy” bytes that don’t look like code.

Why Does it Avoid Detection?

Antivirus has gigantic databases with “signatures” of known shellcode. They search for patterns like: \xfc\x48\x83\xe4\xf0...

But when you encrypt with XOR:

• The original \xfc\x48\x83\xe4... becomes \xaf\x7b\xb0\xd7... (completely different)
• The signature doesn’t match
• Antivirus sees “random” data and lets it through

The problem is that when the code executes in memory, it has to decrypt itself, and at that moment it’s exposed. That’s why it’s critical to decrypt it as late as possible during execution.

Variant 1: XOR with Single Key (Simple)

This is the most basic form: you take a single byte (for example: 0x42) and use it as the key to encrypt each byte of the shellcode. It’s simple, straightforward, and it works. It’s perfect for learning the concept.

However, there’s a serious security problem: if someone wants to break your encryption, they only have to try 256 possible combinations (0x00 to 0xFF). On a modern computer, this takes milliseconds. So it’s easy to understand and fast to execute, but terrible for real security.

Basic Implementation

/**
 * Encrypts/Decrypts a shellcode using XOR with a single-byte key
 *
 * Parameters:
 *  - pShellcode: Pointer to buffer containing the shellcode
 *  - sShellcodeSize: Size in bytes of the shellcode
 *  - bKey: A byte that acts as the key
 */
VOID XorByOneKey(IN PBYTE pShellcode, IN SIZE_T sShellcodeSize, IN BYTE bKey) {
    for (size_t i = 0; i < sShellcodeSize; i++) {
        pShellcode[i] = pShellcode[i] ^ bKey;
    }
}

Security Analysis

Advantages: Ultra simple to implement (only 3 lines of code) and runs at lightning speed.

Serious Problems: There are only 256 possible keys. If your shellcode contains repeated bytes (like multiple zeros), these are always encrypted the same way, leaving clues. Someone with basic tools can break this in seconds by testing all 256 combinations automatically. Don’t use this in production if you expect to resist dedicated analysis.

Variant 2: Dynamic XOR (i + key)

Here we level up. Instead of using the same byte as the key for everything, we change the key based on the byte position.

For example, if your base key is 0x42:

• Byte 0 is encrypted with: 0x42 + 0 = 0x42
• Byte 1 is encrypted with: 0x42 + 1 = 0x43
• Byte 2 is encrypted with: 0x42 + 2 = 0x44
• …and so on

This is much more secure than variant 1. The keyspace is now 256 × shellcode_size. If your shellcode is 1000 bytes, you have 256,000 possible combinations instead of just 256. Still, it’s not impenetrable if someone knows part of the original shellcode.

Improved Implementation

/**
 * Encrypts/Decrypts using dynamic XOR
 * The key changes for each byte: (bKey + i) where i is the index
 *
 * Parameters:
 *  - pShellcode: Pointer to buffer
 *  - sShellcodeSize: Size of shellcode
 *  - bKey: Base key
 */
VOID XorByIndexKey(IN PBYTE pShellcode, IN SIZE_T sShellcodeSize, IN BYTE bKey) {
    for (size_t i = 0; i < sShellcodeSize; i++) {
        pShellcode[i] = pShellcode[i] ^ (bKey + i);
    }
}

Security Analysis

Better: Each byte uses a different key, making frequency analysis more complicated.

Still vulnerable: If someone has an example of the original shellcode, they can deduce the base key. Also, if they see multiple examples of the same shellcode encrypted with this method, the patterns will be identical every time, exposing that it’s the same payload.

Variant 3: XOR with Multiple Keys (Rotor)

This is the one you should use if you want security. Instead of a single key or an index-based key, you use an array of multiple bytes that acts as a “rotor”.

Imagine your key is: [0x12, 0x34, 0x56, 0x78, 0x9A, ...] (say, 32 bytes)

The rotor cycles like this:

• Byte 0 is encrypted with: key[0] = 0x12
• Byte 1 is encrypted with: key[1] = 0x34
• Byte 2 is encrypted with: key[2] = 0x56
• …
• Byte 31 is encrypted with: key[31] = 0x??
• Byte 32 is encrypted with: key[0] = 0x12 (the rotor returns to the beginning!)

This exponentially multiplies the keyspace: 256^32 combinations. Good luck brute-forcing that. Plus, it’s much more realistic cryptographically (although it’s still XOR, not a “serious” algorithm like AES).

Advanced Implementation

/**
 * Encrypts/Decrypts using XOR with key array (rotor)
 * The key repeats cyclically through the shellcode
 *
 * Parameters:
 *  - pShellcode: Pointer to shellcode buffer
 *  - sShellcodeSize: Size of shellcode
 *  - pKeyBuffer: Pointer to array of bytes containing the key
 *  - sKeySize: Size of the key array
 */
VOID XorByMultipleKeys(IN PBYTE pShellcode, IN SIZE_T sShellcodeSize,
                       IN PBYTE pKeyBuffer, IN SIZE_T sKeySize) {
    for (size_t i = 0, j = 0; i < sShellcodeSize; i++) {
        pShellcode[i] = pShellcode[i] ^ pKeyBuffer[j];

        // Rotate the key index
        j = (j + 1) % sKeySize;  // or simply: if (++j >= sKeySize) j = 0;
    }
}

Security Analysis

The good: The keyspace is massive (256^32 if you use 32 bytes). Frequency analysis is much more complicated because patterns are distributed across multiple key bytes. It’s professional and works well in practice.

The limitations: It’s still XOR, not a “serious” algorithm like AES. If someone manages to recover part of the key, the rest of the shellcode falls. Also, you need to store the key array somewhere, and you must do it safely.

Complete Implementation: Step by Step

Next, we’ll create a program that demonstrates XOR encryption and decryption interactively. The program takes a shellcode, displays it in its original form, encrypts it with an 8-byte key, and then decrypts it to verify that XOR’s reversibility works correctly.

Step 1: XOR Function (The Foundation)

// Encrypts or decrypts a buffer with XOR
void XorEncryptDecrypt(unsigned char *data, int size,
                       unsigned char *key, int key_size) {
    for (int i = 0, j = 0; i < size; i++) {
        data[i] = data[i] ^ key[j];
        j = (j + 1) % key_size;  // Rotate the key
    }
}

Step 2: Complete Main Program

#include <windows.h>
#include <stdio.h>
#include <string.h>

#define XOR_KEY_SIZE 32

int main() {
    unsigned char shellcode[] = { 0xfc, 0x48, 0x83, 0xe4, 0xf0, 0xe8, 0xc0, 0x00 };
    unsigned char encrypted[8] = {0};
    unsigned char xor_key[8] = { 0x12, 0x34, 0x56, 0x78, 0x9A, 0xBC, 0xDE, 0xF0 };
    int size = 8;

    printf("ORIGINAL:   ");
    for (int i = 0; i < size; i++) printf("%02x ", shellcode[i]);
    printf("\nPress ENTER...\n");
    getchar();

    for (int i = 0; i < size; i++) encrypted[i] = shellcode[i];

    XorEncryptDecrypt(encrypted, size, xor_key, 8);
    printf("\nENCRYPTED:  ");
    for (int i = 0; i < size; i++) printf("%02x ", encrypted[i]);
    printf("\nPress ENTER...\n");
    getchar();

    XorEncryptDecrypt(encrypted, size, xor_key, 8);
    printf("\nDECRYPTED:  ");
    for (int i = 0; i < size; i++) printf("%02x ", encrypted[i]);
    printf("\n");

    return 0;
}

Advanced Obfuscation: Complementary Techniques

XOR is effective on its own, but it’s even more powerful when combined with other defensive techniques.

Technique 1: Layered Encryption

Instead of a single round of XOR, apply XOR twice with different keys. First encrypt with key 1, then encrypt the result with key 2:

// Encrypt in two layers
XorEncryptDecrypt(pShellcode, sShellcodeSize, key1, key1Size);
XorEncryptDecrypt(pShellcode, sShellcodeSize, key2, key2Size);

// Decrypt in reverse order
XorEncryptDecrypt(pShellcode, sShellcodeSize, key2, key2Size);
XorEncryptDecrypt(pShellcode, sShellcodeSize, key1, key1Size);

Why? It makes analysis exponentially harder. Someone who wants to break it must discover both keys, not just one.

Technique 2: Hide the Key

Storing the key in plain text in the binary is the weak point. Better alternatives:

// Option 1: Encrypted key in the binary
BYTE encrypted_key[] = { 0x45, 0x67, 0x89... };
// Decrypt it first with another technique

// Option 2: Generate the key at runtime
BYTE key[32];
GenerateKeyFromEnvironment(key);
// Based on information from the current system

// Option 3: Read the key from the registry
GetKeyFromRegistry(HKEY_LOCAL_MACHINE, "Software\\...", key);

This way, the analyzer cannot simply read the key from the executable.

Limitations of XOR: Understanding the Realities

Being honest: XOR has limits, and it’s important to know them.

1. XOR is Not “Serious” Encryption

XOR was used in Enigma machines in World War II. Today it’s considered a cryptographic toy. Don’t use it for:

• Protecting sensitive data against professional adversaries
• Meeting banking or government standards
• Anything requiring real confidentiality

But for shellcode obfuscation? It’s more than enough.

2. Shellcode is Always Exposed

Here’s the uncomfortable truth: No matter how you hide it, the shellcode MUST be decrypted in memory to execute. It’s a fundamental problem, not a limitation of XOR.

The key is to delay that moment as long as possible during execution.

3. EDR Remains Vigilant

XOR disguises the code, but EDR (especially modern EDR) can detect:

• The decryption pattern - The XOR loop has a recognizable “signature”
• Suspicious APIs - VirtualAlloc + VirtualProtect + CreateThread together = malicious activity
• Post-execution behavior - What the code does once executed

Practical Analysis: Detectability

In the real world, EDRs can detect your code for three main reasons:

1. The XOR pattern is recognizable - The loop for(i=0; i<size; i++) buf[i] ^= key[j] has a characteristic “signature”. Decryption patterns are well known.

2. The combination of APIs is suspicious - VirtualAlloc → VirtualProtect → CreateThread is a classic code injection pattern. EDR specifically searches for this sequence.

3. What it does afterwards matters more - EDR may be less interested in how the code is loaded and more in WHAT it does. If your payload opens cmd.exe or exfiltrates data, that gets detected with behavioral analysis.

Realistic Conclusion: XOR adds a layer of obfuscation and evades signature-based detections, but it’s not a silver bullet. It’s one piece of a bigger puzzle.

What We’ve Learned

We’ve covered quite a bit of ground:

1. The mathematics of XOR - Bits, reversible operations, why it evades signatures
2. Three variants - From ultra-simple to professional with rotor
3. Why each one is better (or worse) - The real trade-offs
4. Complete and functional code - From theory to practice
5. Advanced techniques - Layered encryption and key obfuscation
6. The hard truth - What works and what doesn’t in the real world

Final Reflection

XOR is your first defensive shield against static detections. Antivirus looking for signatures won’t see you. EDR is another story - that requires more sophisticated techniques like anti-analysis, injection without suspicious APIs, or simply “normal” behavior.

But here’s the important part: There is no perfect obfuscation. Any code that executes must be in plain text at some point. The realistic goal is not “undetectable” (spoiler: it doesn’t exist). The goal is to make the cost of analysis so high that it’s not worth the effort.

Combine XOR with:

• Layered encryption (multiple rounds)
• Key obfuscation
• Less obvious APIs
• Behavior that looks legitimate