/ctf-reverse

CTF Reverse - Anti-Analysis Techniques & Bypasses

Comprehensive reference for anti-debugging, anti-VM, anti-DBI, and integrity-check techniques encountered in CTF challenges, with practical bypasses.

Table of Contents

• Linux Anti-Debug (Advanced)
  • ptrace-Based
  • /proc Filesystem Checks
  • Timing-Based Detection
  • Signal-Based Anti-Debug
  • Syscall-Level Evasion
  • Trap-Flag Self-Check with cmovz Patcher (Hack.lu 2018)
  • SIGFPE Handler for mprotect Code Mutation (Hack.lu 2018)
• Windows Anti-Debug (Advanced)
  • PEB (Process Environment Block) Checks
  • NtQueryInformationProcess
  • Heap Flags
  • TLS Callbacks
  • Hardware Breakpoint Detection
  • Software Breakpoint Detection (INT3 Scanning)
  • Exception-Based Anti-Debug
  • NtSetInformationThread (Thread Hiding)
• Anti-VM / Anti-Sandbox
  • CPUID Hypervisor Bit
  • MAC Address / Hardware Fingerprinting
  • Timing-Based VM Detection
  • File / Registry Artifacts
  • Resource Checks (CPU Count, RAM, Disk)
• Anti-DBI (Dynamic Binary Instrumentation)
  • Frida Detection
  • Pin/DynamoRIO Detection
• Code Integrity / Self-Hashing
• Anti-Disassembly Techniques
  • Opaque Predicates
  • Junk Bytes / Overlapping Instructions
  • Jump-in-the-Middle
  • Function Chunking / Scattered Code
  • Control Flow Flattening (Advanced)
  • Mixed Boolean-Arithmetic (MBA) Identification & Simplification
• Comprehensive Bypass Strategies

For CTF writeup techniques (SIGILL handler, SIGFPE strace side-channel, instruction trace inversion, call-less function chaining, parent-patched child binary dump), see anti-analysis-ctf.md.

• Universal Bypass Checklist
• Layered Anti-Debug (Real-World Pattern)
• Quick Reference: Check to Bypass

Linux Anti-Debug (Advanced)

ptrace-Based

Self-ptrace (most common):

if (ptrace(PTRACE_TRACEME, 0, 0, 0) == -1) exit(1); // Already traced = debugger attached

Bypasses:

# 1. LD_PRELOAD (see patterns.md for full hook)
LD_PRELOAD=./hook.so ./binary

# 2. Patch with pwntools
python3 -c "
from pwn import *
elf = ELF('./binary', checksec=False)
elf.asm(elf.symbols.ptrace, 'xor eax, eax; ret')
elf.save('patched')
"

# 3. GDB: catch the syscall
gdb ./binary
(gdb) catch syscall ptrace
(gdb) run
# When it stops at ptrace:
(gdb) set $rax = 0
(gdb) continue

# 4. Kernel config (requires root)
echo 0 > /proc/sys/kernel/yama/ptrace_scope

Double-ptrace pattern:

// Fork child to ptrace parent — blocks all other debuggers
pid_t child = fork();
if (child == 0) {
    ptrace(PTRACE_ATTACH, getppid(), 0, 0);
    // Child sits in waitpid loop, keeping parent traced
} else {
    // Parent continues with real logic
}

Bypass: Kill the watchdog child process, then attach debugger.

/proc Filesystem Checks

// TracerPid check
FILE *f = fopen("/proc/self/status", "r");
// Looks for "TracerPid:\t0" — non-zero means debugger

// /proc/self/exe link check (some debuggers change this)
readlink("/proc/self/exe", buf, sizeof(buf));

// /proc/self/maps — check for debugger libraries
grep("frida", "/proc/self/maps");

Bypasses:

# 1. LD_PRELOAD fopen/fread to fake /proc contents
# 2. Mount namespace isolation
unshare -m bash -c 'mount --bind /dev/null /proc/self/status && ./binary'

# 3. GDB: set breakpoint at fopen, change filename argument
(gdb) b fopen
(gdb) run
(gdb) set {char[20]} $rdi = "/dev/null"
(gdb) continue

Timing-Based Detection

// rdtsc (CPU timestamp counter)
uint64_t start = __rdtsc();
// ... code ...
uint64_t delta = __rdtsc() - start;
if (delta > THRESHOLD) exit(1);  // too slow = debugger

// clock_gettime
struct timespec ts1, ts2;
clock_gettime(CLOCK_MONOTONIC, &ts1);
// ... code ...
clock_gettime(CLOCK_MONOTONIC, &ts2);

// gettimeofday
struct timeval tv1, tv2;
gettimeofday(&tv1, NULL);

Bypasses:

# 1. Frida hook (see tools-dynamic.md for clock_gettime hook)

# 2. GDB: skip rdtsc by patching with constant
(gdb) set {unsigned char[2]} 0x401234 = {0x90, 0x90}  # NOP the rdtsc

# 3. Pin tool to fix TSC reads
# 4. faketime library
LD_PRELOAD=/usr/lib/faketime/libfaketime.so.1 FAKETIME="2024-01-01" ./binary

Signal-Based Anti-Debug

// SIGTRAP handler — INT3 under debugger is caught by debugger, not handler
signal(SIGTRAP, handler);
__asm__("int3");
// If handler runs: no debugger. If debugger catches: debugged.

// SIGALRM timeout — kill self if analysis takes too long
signal(SIGALRM, kill_handler);
alarm(5);

// SIGSEGV handler that does real work (see patterns.md for MBA pattern)
signal(SIGSEGV, real_logic_handler);
*(int*)0 = 0;  // deliberate crash → handler runs real code

Bypasses:

# GDB: pass signals to program instead of handling them
(gdb) handle SIGTRAP nostop pass
(gdb) handle SIGALRM ignore
(gdb) handle SIGSEGV nostop pass

# For alarm-based: patch alarm() to return immediately

Syscall-Level Evasion

// Direct syscall instead of libc — bypasses LD_PRELOAD hooks
long ret;
asm volatile("syscall" : "=a"(ret) : "a"(101), "D"(0), "S"(0), "d"(0), "r"(0));
// Syscall 101 = ptrace on x86_64

Bypass: Must patch the binary itself or use ptrace to intercept at syscall level.

# GDB: catch syscall
(gdb) catch syscall 101
(gdb) commands
> set $rax = 0
> continue
> end

Windows Anti-Debug (Advanced)

PEB (Process Environment Block) Checks

// BeingDebugged flag (offset 0x2 in PEB)
bool debugged = NtCurrentPeb()->BeingDebugged;

// NtGlobalFlag (offset 0x68/0xBC in PEB)
// When debugger: FLG_HEAP_ENABLE_TAIL_CHECK | FLG_HEAP_ENABLE_FREE_CHECK | FLG_HEAP_VALIDATE_PARAMETERS = 0x70
DWORD flags = *(DWORD*)((BYTE*)NtCurrentPeb() + 0xBC); // 64-bit offset
if (flags & 0x70) exit(1);

Bypass (x64dbg):

# ScyllaHide plugin auto-patches PEB fields
# Manual: dump PEB, zero BeingDebugged and NtGlobalFlag

NtQueryInformationProcess

// ProcessDebugPort (0x7)
DWORD_PTR debugPort = 0;
NtQueryInformationProcess(GetCurrentProcess(), 7, &debugPort, sizeof(debugPort), NULL);
if (debugPort != 0) exit(1);

// ProcessDebugObjectHandle (0x1E)
HANDLE debugObj = NULL;
NTSTATUS status = NtQueryInformationProcess(GetCurrentProcess(), 0x1E, &debugObj, sizeof(debugObj), NULL);
if (status == 0) exit(1); // STATUS_SUCCESS means debugger present

// ProcessDebugFlags (0x1F) — returns inverse: 0 = debugger present
DWORD noDebug = 0;
NtQueryInformationProcess(GetCurrentProcess(), 0x1F, &noDebug, sizeof(noDebug), NULL);
if (noDebug == 0) exit(1);

Bypass: Hook NtQueryInformationProcess to return fake values, or use ScyllaHide.

Heap Flags

// Process heap has debug flags when debugger attached
PHEAP heap = (PHEAP)GetProcessHeap();
// Flags at offset 0x70 (64-bit): should be HEAP_GROWABLE (0x2)
// ForceFlags at offset 0x74: should be 0
if (heap->Flags != 0x2 || heap->ForceFlags != 0) exit(1);

TLS Callbacks

Key technique: TLS (Thread Local Storage) callbacks execute BEFORE main() / entry point.

// Registered in PE header's TLS directory
void NTAPI TlsCallback(PVOID DllHandle, DWORD Reason, PVOID Reserved) {
    if (Reason == DLL_PROCESS_ATTACH) {
        if (IsDebuggerPresent()) {
            ExitProcess(1);  // Kills process before main runs
        }
    }
}

#pragma comment(linker, "/INCLUDE:_tls_used")
#pragma data_seg(".CRT$XLB")
PIMAGE_TLS_CALLBACK callbacks[] = { TlsCallback, NULL };

Detection in IDA/Ghidra: Check PE TLS Directory → AddressOfCallBacks. Functions listed there run before EP.

Bypass: Set breakpoint on TLS callback in x64dbg (Options → Events → TLS Callbacks), or patch the TLS directory entry.

Hardware Breakpoint Detection

// Read debug registers via GetThreadContext
CONTEXT ctx;
ctx.ContextFlags = CONTEXT_DEBUG_REGISTERS;
GetThreadContext(GetCurrentThread(), &ctx);
if (ctx.Dr0 || ctx.Dr1 || ctx.Dr2 || ctx.Dr3) exit(1);

// Also via exception handler: deliberate exception, check DR regs in handler

Bypass:

# x64dbg: use software breakpoints instead, or hook GetThreadContext
# Frida: hook GetThreadContext to zero DR registers

Software Breakpoint Detection (INT3 Scanning)

// CRC / hash check over code section
unsigned char *code = (unsigned char*)function_addr;
uint32_t checksum = 0;
for (int i = 0; i < code_size; i++) {
    checksum += code[i];
    if (code[i] == 0xCC) exit(1);  // INT3 = software breakpoint
}
if (checksum != EXPECTED_CHECKSUM) exit(1);

Bypass: Use hardware breakpoints (DR0-DR3) instead of software breakpoints. Or hook the scanning function.

Exception-Based Anti-Debug

// UnhandledExceptionFilter — under debugger, filter is NOT called
SetUnhandledExceptionFilter(handler);
RaiseException(EXCEPTION_ACCESS_VIOLATION, 0, 0, NULL);
// If handler runs: no debugger
// If debugger catches: debugger present

// INT 2D — debugger single-step anomaly
__asm { int 2dh }  // Debugger silently consumes the exception
// If execution continues: debugger present

NtSetInformationThread (Thread Hiding)

// Hide thread from debugger — stops all debug events
typedef NTSTATUS(NTAPI *pNtSIT)(HANDLE, ULONG, PVOID, ULONG);
pNtSIT NtSIT = (pNtSIT)GetProcAddress(GetModuleHandle("ntdll"), "NtSetInformationThread");
NtSIT(GetCurrentThread(), 0x11 /*ThreadHideFromDebugger*/, NULL, 0);
// After this, debugger won't see breakpoints or exceptions from this thread

Bypass: Hook NtSetInformationThread to ignore class 0x11, or patch the call.

Anti-VM / Anti-Sandbox

CPUID Hypervisor Bit

int regs[4];
__cpuid(regs, 1);
if (regs[2] & (1 << 31)) {  // ECX bit 31 = hypervisor present
    exit(1);
}

// Hypervisor brand string
__cpuid(regs, 0x40000000);
char brand[13] = {0};
memcpy(brand, &regs[1], 12);
// "VMwareVMware", "Microsoft Hv", "KVMKVMKVM", "XenVMMXenVMM"

Bypass: Patch cpuid results or use LD_PRELOAD to hook wrapper functions.

MAC Address / Hardware Fingerprinting

Known VM MAC prefixes:
  VMware:     00:0C:29, 00:50:56
  VirtualBox: 08:00:27
  Hyper-V:    00:15:5D
  Parallels:  00:1C:42
  QEMU:       52:54:00

Timing-Based VM Detection

// VM exits on privileged instructions are measurably slower
uint64_t start = __rdtsc();
__cpuid(regs, 0);  // Forces VM exit
uint64_t delta = __rdtsc() - start;
if (delta > 500) { /* likely VM */ }

File / Registry Artifacts

Files: C:\Windows\System32\drivers\vm*.sys, vbox*.dll, VBoxService.exe
Registry: HKLM\SOFTWARE\VMware, Inc.\VMware Tools
Services: VMTools, VBoxService
Processes: vmtoolsd.exe, VBoxTray.exe, qemu-ga.exe
Linux: /sys/class/dmi/id/product_name contains "VirtualBox"|"VMware"
       dmesg | grep -i "hypervisor detected"

Resource Checks (CPU Count, RAM, Disk)

// Sandboxes typically have minimal resources
SYSTEM_INFO si;
GetSystemInfo(&si);
if (si.dwNumberOfProcessors < 2) exit(1);

MEMORYSTATUSEX ms;
ms.dwLength = sizeof(ms);
GlobalMemoryStatusEx(&ms);
if (ms.ullTotalPhys < 2ULL * 1024 * 1024 * 1024) exit(1); // < 2GB RAM

// Disk size check (< 60GB = sandbox)
GetDiskFreeSpaceEx("C:\\", NULL, &total, NULL);

Bypass: Use a VM configured with adequate resources (4+ CPUs, 8GB+ RAM, 100GB+ disk).

Anti-DBI (Dynamic Binary Instrumentation)

Frida Detection

// 1. Check /proc/self/maps for frida-agent
FILE *f = fopen("/proc/self/maps", "r");
while (fgets(line, sizeof(line), f)) {
    if (strstr(line, "frida") || strstr(line, "gadget")) exit(1);
}

// 2. Check for Frida's default port (27042)
int sock = socket(AF_INET, SOCK_STREAM, 0);
struct sockaddr_in addr = {.sin_family=AF_INET, .sin_port=htons(27042), .sin_addr.s_addr=inet_addr("127.0.0.1")};
if (connect(sock, (struct sockaddr*)&addr, sizeof(addr)) == 0) exit(1);

// 3. Check for inline hooks (function prologue modification)
// Compare first bytes of libc functions against expected values
unsigned char *strcmp_bytes = (unsigned char *)strcmp;
if (strcmp_bytes[0] == 0xE9 || strcmp_bytes[0] == 0xFF) exit(1); // JMP = hooked

// 4. Thread name check
// Frida creates threads with names like "gmain", "gdbus", "frida-*"
DIR *dir = opendir("/proc/self/task");
while ((entry = readdir(dir))) {
    char comm_path[256];
    snprintf(comm_path, sizeof(comm_path), "/proc/self/task/%s/comm", entry->d_name);
    // Read comm and check for "gmain", "gdbus"
}

// 5. Named pipe detection (Windows)
// Frida creates \\.\pipe\frida-* named pipes

Frida bypass of Frida detection:

// Hook the detection functions themselves
Interceptor.attach(Module.findExportByName(null, "strstr"), {
    onEnter(args) {
        this.haystack = Memory.readUtf8String(args[0]);
        this.needle = Memory.readUtf8String(args[1]);
    },
    onLeave(retval) {
        if (this.needle && (this.needle.includes("frida") || this.needle.includes("gadget"))) {
            retval.replace(ptr(0)); // Not found
        }
    }
});

// Early Frida load (before anti-DBI runs)
// Use frida-gadget as early-init shared library

Pin/DynamoRIO Detection

// Check for instrumentation libraries in /proc/self/maps
// Pin: "pin-", "pinbin", "pinatrace"
// DynamoRIO: "dynamorio", "drcov", "drrun"

// Instruction count timing — DBI adds overhead
// Execute known instruction sequence, compare execution time

Code Integrity / Self-Hashing

// CRC32 over .text section
uint32_t crc = compute_crc32(text_start, text_size);
if (crc != EXPECTED_CRC) exit(1);  // Code was modified (breakpoints, patches)

// MD5/SHA256 of function bodies
unsigned char hash[32];
SHA256(function_addr, function_size, hash);
if (memcmp(hash, expected_hash, 32) != 0) exit(1);

Bypasses:

1. Hardware breakpoints (don't modify code, DR0-DR3)
2. Patch the comparison to always succeed
3. Hook the hash function to return expected value
4. Emulate instead of debug (Unicorn/Qiling — no code modification)
5. Snapshot + restore: dump memory before and after, diff to find checks

Self-checksumming in loops:

// Continuous integrity check in separate thread
void *watchdog(void *arg) {
    while (1) {
        if (compute_crc32(text_start, text_end - text_start) != saved_crc) {
            memset(flag_buffer, 0, flag_len);  // Destroy flag
            exit(1);
        }
        usleep(100000);
    }
}

Bypass: Kill the watchdog thread or patch its sleep to infinite.

Anti-Disassembly Techniques

Opaque Predicates

; Condition that always evaluates the same way but looks data-dependent
mov eax, [some_memory]
imul eax, eax          ; x^2
and eax, 1             ; x^2 mod 2 is always 0 for any x
jnz fake_branch        ; Never taken, but disassembler doesn't know
; real code here

Identification: Z3/SMT can prove branch is always/never taken.

Junk Bytes / Overlapping Instructions

jmp real_code
db 0xE8           ; Looks like start of CALL to linear disassembler
real_code:
mov eax, 1        ; Real code — disassembler may misalign here

Fix: Switch to graph-mode disassembly (Ghidra/IDA handle this well). Manual: undefine and re-analyze from correct offset.

Jump-in-the-Middle

; Jumps into the middle of a multi-byte instruction
eb 01          ; jmp +1 (skip next byte)
e8             ; fake CALL opcode — disassembler tries to decode as call
90             ; real: NOP (landed here from jmp)

Function Chunking / Scattered Code

Functions split into non-contiguous chunks connected by unconditional jumps. Defeats linear function boundary detection.

Tool: IDA's "Append function tail" or Ghidra's "Create function" at each chunk.

Control Flow Flattening (Advanced)

Beyond basic switch-case (see patterns.md): modern OLLVM variants use:

• Bogus control flow: Fake branches with opaque predicates
• Instruction substitution: a + b → a - (-b), a ^ b → (a | b) & ~(a & b)
• String encryption: Strings decrypted at runtime, cleared after use

Deobfuscation tools:

• D-810 (IDA plugin): Pattern-based deobfuscation, MBA simplification
• GOOMBA (Ghidra): Automated deobfuscation for OLLVM
• Miasm: Symbolic execution for deobfuscation
• Arybo / SiMBA: MBA expression simplification

# D-810: install in IDA plugins directory, Edit → Plugins → D-810
# Simplifies MBA expressions: (a | b) & ~(a & b) → a ^ b
# Removes opaque predicates via pattern matching

Mixed Boolean-Arithmetic (MBA) Identification & Simplification

# Common MBA patterns and their simplified forms:
# (x & y) + (x | y) == x + y
# (x ^ y) + 2*(x & y) == x + y
# (x | y) - (x & ~y) == y
# ~(~x & ~y) == x | y (De Morgan's)
# (x | y) & ~(x & y) == x ^ y

# SiMBA tool for automated simplification:
# pip install simba-simplifier
from simba import simplify_mba
expr = "(a | b) + (a & b) - (~a & b)"
print(simplify_mba(expr))  # → a

See anti-analysis-ctf.md for CTF writeup techniques: SIGILL handler for mode switching (Hack.lu 2015), SIGFPE strace side-channel (PlaidCTF 2017), instruction trace inversion (MeePwn 2017), call-less function chaining (THC 2018), and parent-patched child binary dump via process_vm_writev (Google CTF Quals 2018).

Comprehensive Bypass Strategies

Universal Bypass Checklist

1. Identify all anti-analysis checks — search for: ptrace, IsDebuggerPresent, rdtsc, cpuid, NtQuery, GetTickCount, CheckRemoteDebuggerPresent, /proc/self, SIGTRAP, alarm
2. Static patching — NOP/patch checks with pwntools or Ghidra before running
3. LD_PRELOAD (Linux) — hook libc functions returning fake values
4. ScyllaHide (Windows x64dbg) — patches PEB, hooks NT functions automatically
5. Emulation (Unicorn/Qiling) — no debugger artifacts to detect
6. Kernel-level bypass — modify /proc/sys/kernel/yama/ptrace_scope, use prctl

Layered Anti-Debug (Real-World Pattern)

Many CTF challenges stack multiple checks:

1. TLS callback → IsDebuggerPresent (before main)
2. main() → ptrace(TRACEME)
3. Watchdog thread → timing check + /proc scan
4. Code section → self-CRC32 integrity
5. Signal handler → real logic in SIGSEGV handler

Approach: Identify ALL checks before patching. Patch or hook each one systematically. Run under emulator if too many to patch individually.

Quick Reference: Check to Bypass

Trap-Flag Self-Check with cmovz Patcher (Hack.lu 2018)

Pattern: The binary checks EFLAGS by doing pushf; pop edx; and edx, 0x100 (isolating the single-step Trap Flag) and using the result inside a cmovz so the correct instruction is overwritten only when the TF bit is clear. Single-stepping in gdb leaves TF set, the cmovz never fires, and the program silently runs the wrong code path without crashing.

check_debugger:
    pushf
    pop   edx
    and   edx, 0x100          ; Trap Flag only
    test  edx, edx
    cmovz eax, ebx            ; overwrite `eax` only when NOT single-stepping
    mov   [rip+target], eax

Bypass with hardware breakpoints:

(gdb) hbreak *0x56557267       # hardware BP, no INT3, no TF side effect
(gdb) run
(gdb) # inspect EAX at the hbreak — the patched value is now written

Key insight: pushf; pop reg; and reg, 0x100 is the cleanest way to check TF without triggering a trap. Single-stepping changes the visible EFLAGS and can also perturb the instruction pipeline, so software breakpoints plus stepi both poison the check. Hardware breakpoints (hbreak) run the instruction normally and halt afterwards, so the check fires in "not debugging" mode. Apply the same fix to any anti-debug that reads EFLAGS, RFLAGS, or DR6.

References: Hack.lu CTF 2018 — Forgetful Commander, writeup 11858

SIGFPE Handler for mprotect Code Mutation (Hack.lu 2018)

Pattern: The binary installs a custom SIGFPE handler with sys_sigaction and arranges for an arithmetic instruction to trap (e.g., div by zero). The handler runs with kernel-delivered context, calls mprotect on the .text page to make it writable, and mutates code that would otherwise stay constant. Standard static analysis misses the mutation because the FPE never fires in the normal path, and standard dynamic analysis misses it because most debuggers intercept SIGFPE before delivery.

// Handler installed at startup
void on_fpe(int sig, siginfo_t *info, void *uap) {
    ucontext_t *ctx = uap;
    void *page = (void *)((uintptr_t)ctx->uc_mcontext.gregs[REG_RIP] & ~0xfff);
    mprotect(page, 0x1000, PROT_READ | PROT_WRITE | PROT_EXEC);
    // Patch a constant used by the next check
    *((uint32_t *)(page + 0x42)) = 0xDEADBEEF;
}

Bypass:

# Let SIGFPE reach the program, not the debugger
gdb ./challenge
(gdb) handle SIGFPE nostop noprint pass
(gdb) break *on_fpe
(gdb) run

Key insight: Signal handlers that mprotect + mutate code are cross-delimited in a way decompilers cannot model. SIGFPE is particularly effective because it is rarely raised during normal execution, so the mutation stays dormant until the attacker hits the crafted input. When you see sys_sigaction(SIGFPE,...) or signal(SIGFPE,...) in a binary that also calls mprotect, trace the handler with strace -e signal=SIGFPE and annotate the mutated region by diffing the .text pages before and after the first FPE.

References: Hack.lu CTF 2018 — Cheat Console, writeup 11868