CTF Reverse - Emulation and Side-Channel Tooling
Emulation frameworks (Qiling, Triton) and side-channel measurement tools (Intel Pin, LD_PRELOAD hooks) for CTF challenges where anti-debug, self-modifying code, or cross-architecture targets make plain GDB/Frida impractical.
For core dynamic analysis tools (Frida, angr, lldb, x64dbg), see tools-dynamic.md.
Table of Contents
- Qiling Framework (Cross-Platform Emulation)
- Triton (Dynamic Symbolic Execution)
- Intel Pin Instruction-Counting Side Channel (Hackover CTF 2015)
- Opcode-Only Trace Reconstruction (0CTF 2016)
- LD_PRELOAD time() Freeze for Deterministic Analysis (EKOPARTY 2017)
Qiling Framework (Cross-Platform Emulation)
Qiling emulates binaries with OS-level support (syscalls, filesystem, registry). Built on Unicorn but adds the OS layer that Unicorn lacks.
Qiling Installation
pip install qiling
# Download rootfs for target OS:
git clone https://github.com/qilingframework/rootfsBasic Usage
from qiling import Qiling
from qiling.const import QL_VERBOSE
# Linux ELF emulation
ql = Qiling(["./binary", "arg1"], "rootfs/x8664_linux",
verbose=QL_VERBOSE.DEFAULT)
ql.run()
# Windows PE emulation (no Windows needed!)
ql = Qiling(["rootfs/x86_windows/bin/binary.exe"], "rootfs/x86_windows")
ql.run()
# ARM/MIPS emulation (IoT firmware)
ql = Qiling(["rootfs/arm_linux/bin/binary"], "rootfs/arm_linux")
ql.run()Anti-Debug Bypass via Emulation
from qiling import Qiling
ql = Qiling(["./binary"], "rootfs/x8664_linux")
# Hook ptrace syscall — return 0 (success)
def hook_ptrace(ql, ptrace_request, pid, addr, data):
ql.log.info("ptrace bypassed")
return 0
ql.os.set_syscall("ptrace", hook_ptrace)
# Hook specific address (e.g., anti-VM check)
def skip_check(ql):
ql.arch.regs.rax = 0 # Force success
ql.log.info(f"Skipped check at {ql.arch.regs.rip:#x}")
ql.hook_address(skip_check, 0x401234)
ql.run()Input Fuzzing with Qiling
# Emulate binary with different inputs to find flag
import string
from qiling import Qiling
def test_input(candidate):
ql = Qiling(["./binary"], "rootfs/x8664_linux",
verbose=QL_VERBOSE.DISABLED, stdin=candidate.encode())
ql.run()
return ql.os.stdout.read()
for ch in string.printable:
output = test_input("flag{" + ch)
if b"Correct" in output:
print(f"Found: {ch}")Advantages over GDB/Frida:
- No debugger artifacts (bypasses all anti-debug by default)
- Cross-platform without hardware (ARM, MIPS, RISC-V on x86 host)
- Scriptable with Python (faster iteration than GDB)
- Snapshot/restore for brute-forcing
Key insight: Qiling emulates the entire OS layer (syscalls, filesystem, registry), not just the CPU. This means anti-debug checks like ptrace(TRACEME) naturally return success without patching, and you can analyze ARM/MIPS binaries on an x86 host without QEMU or real hardware.
When to use: Foreign architecture binaries, IoT firmware, heavy anti-debug, automated testing of many inputs.
Triton (Dynamic Symbolic Execution)
See tools-advanced.md for full Triton reference. Quick usage:
from triton import *
ctx = TritonContext(ARCH.X86_64)
# Symbolize input buffer
for i in range(32):
ctx.symbolizeMemory(MemoryAccess(0x600000 + i, CPUSIZE.BYTE), f"flag_{i}")
# Process instructions and collect constraints
# At comparison point, solve for flag
model = ctx.getModel(ctx.getPathConstraintsAst())
flag = ''.join(chr(v.getValue()) for _, v in sorted(model.items()))Key insight: Triton excels at single-path DSE (Dynamic Symbolic Execution) where angr's path explosion is a problem. Feed it a concrete execution trace, symbolize specific inputs, and solve for constraints at comparison points. Faster than angr for linear code paths with known execution flow.
Best for: Single-path symbolic execution, deobfuscation, taint analysis. Faster than angr for linear code paths.
Intel Pin Instruction-Counting Side Channel (Hackover CTF 2015)
Pattern: Brute-force input character-by-character against a binary using Intel Pin's inscount0 tool. Each correct character causes deeper execution (more instructions) in the comparison logic.
import string
from subprocess import Popen, PIPE
pin = './pin'
tool = './source/tools/ManualExamples/obj-intel64/inscount0.so'
binary = './target'
key = ''
while True:
best_count, best_char = 0, ''
for c in string.printable:
cmd = [pin, '-injection', 'child', '-t', tool, '--', binary]
p = Popen(cmd, stdout=PIPE, stdin=PIPE, stderr=PIPE)
p.communicate((key + c + '\n').encode())
with open('inscount.out') as f:
count = int(f.read().split()[-1])
if count > best_count:
best_count, best_char = count, c
key += best_char
print(f"Found: {key}")Key insight: Movfuscated binaries (compiled with movfuscator) expand every instruction into sequences of mov operations, making static analysis impractical. However, character-by-character comparison still creates measurable instruction count differences. Pin's inscount0.so counts total executed instructions — the correct character at each position causes ~1000+ more instructions (proceeding further in the comparison). Also works for obfuscated binaries with sequential input checks.
Intel Pin Instruction Counting with Genetic Algorithm (hxp CTF 2017)
For self-modifying code that decrypts the next chunk only after each character check passes, standard character-by-character Pin counting fails because the search space is too large and characters may interact. Use a genetic algorithm instead to explore the input space more efficiently.
import subprocess
import random
import string
PIN_PATH = '/tmp/pin-3.5/pin'
TOOL_PATH = 'source/tools/ManualExamples/obj-intel64/inscount0.so'
def fitness(candidate):
"""Run binary under Pin and return instruction count as fitness."""
proc = subprocess.Popen(
[PIN_PATH, '-t', TOOL_PATH, '--', './binary'],
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = proc.communicate(candidate.encode())
# inscount0 writes count to stderr or inscount.out
try:
with open('inscount.out') as f:
return int(f.read().split()[-1])
except:
return 0
def mutate(individual, rate=0.1):
"""Randomly mutate characters in the individual."""
result = list(individual)
for i in range(len(result)):
if random.random() < rate:
result[i] = random.choice(string.printable[:62])
return result
# Genetic algorithm parameters
FLAG_LEN = 40
POP_SIZE = 100
SURVIVORS = 20
# Initialize random population
population = [random.choices(string.printable[:62], k=FLAG_LEN) for _ in range(POP_SIZE)]
for generation in range(10000):
# Score each individual by instruction count
scored = [(fitness(''.join(p)), p) for p in population]
scored.sort(reverse=True)
best_score, best_individual = scored[0]
print(f"Gen {generation}: {best_score} {''.join(best_individual)}")
# Keep top survivors, mutate to refill population
survivors = [s[1] for s in scored[:SURVIVORS]]
population = survivors + [mutate(random.choice(survivors)) for _ in range(POP_SIZE - SURVIVORS)]Modified Pin for Go binaries (table-lookup flag checking):
When standard inscount fails because counter increments don't correlate with correctness (e.g., table-lookup comparison), modify Pin's icount tool to only count executions at the success-branch address. Brute-force character-by-character with this targeted counter:
// Modified inscount0.cpp — count only executions of a specific address
static ADDRINT target_addr = 0x401234; // success-branch address
static UINT64 target_count = 0;
VOID CountAtTarget(ADDRINT ip) {
if (ip == target_addr) target_count++;
}
VOID Instruction(INS ins, VOID *v) {
INS_InsertCall(ins, IPOINT_BEFORE, (AFUNPTR)CountAtTarget,
IARG_INST_PTR, IARG_END);
}Key insight: When each correct character unlocks a new code section (self-modifying or multi-stage decryption), instruction count increases monotonically with correctness. A genetic algorithm explores the input space more efficiently than character-by-character brute-force because it can discover multiple correct characters simultaneously. Converges in approximately 30 minutes for 40-character flags. For table-lookup comparisons where total instruction count doesn't correlate, target a specific branch address instead.
References: hxp CTF 2017
Opcode-Only Trace Reconstruction (0CTF 2016)
Given an execution trace with only opcodes (no register/memory values), reconstruct the program: sort/dedup trace by address, split into basic blocks, annotate functions. Sorting algorithms are particularly vulnerable -- branch decisions leak element ordering.
Approach:
- Sort trace entries by address, deduplicate to recover code layout
- Identify basic block boundaries (jumps, calls, returns)
- Map branch taken/not-taken decisions from trace order
- For sorting algorithms, partition comparisons reveal relative ordering of all input elements
Key insight: Execution traces without data values still leak information through branch decisions. Quicksort partition comparisons reveal which element is greater/lesser at each step, enabling full recovery of the sorted input from branch direction alone.
LD_PRELOAD time() Freeze for Deterministic Analysis (EKOPARTY 2017)
Override time() via LD_PRELOAD to return a constant value, freezing any timestamp-seeded PRNG. Once the binary's cipher becomes deterministic, brute-force each output byte without understanding the VM or cipher internals.
// freeze_time.c — compile: gcc -shared -fPIC -o freeze.so freeze_time.c
#include <time.h>
time_t time(time_t *t) {
if (t) *t = 1234567890;
return 1234567890;
}# Build and use:
gcc -shared -fPIC -o freeze.so freeze_time.c
LD_PRELOAD=./freeze.so ./binary
# Byte-at-a-time oracle: run with frozen time, try each candidate byte,
# observe output — correct byte produces expected output character.
for byte in $(seq 0 255); do
output=$(echo -n "$(printf '\x%02x' $byte)" | LD_PRELOAD=./freeze.so ./binary)
# Check output against known/expected
doneIf srand() or rand() is also involved, override rand() too:
int rand(void) { return 42; }Key insight: LD_PRELOAD function interception freezes non-determinism sources (time, rand). Once deterministic, even complex VMs become tractable byte-at-a-time oracles.
References: EKOPARTY CTF 2017
LD_PRELOAD memcmp Side-Channel for Byte-by-Byte Bruteforce (Blaze CTF 2018)
Pattern: Replace memcmp with an LD_PRELOAD library that returns the number of matching bytes instead of the standard -1/0/1 result. This converts any memcmp-based validation into a byte-by-byte oracle. Automate with GDB Python scripting to bruteforce each character position.
// memcmp_hook.c - compile: gcc -shared -fPIC -o hook.so memcmp_hook.c
int memcmp(const char *s1, const char *s2, int n) {
int cnt = 0;
for (int i = 0; i < n; ++i) {
if (s1[i] == s2[i]) cnt++;
else break;
}
return cnt;
}# Use with GDB: LD_PRELOAD=./hook.so gdb ./binary
# Set breakpoint after memcmp, read return value to count matching bytes
# Iterate characters at each position to find the one that increases countKey insight: Replacing memcmp via LD_PRELOAD to return match count converts any comparison-based validation into a byte-by-byte oracle. Combined with GDB scripting, this automates bruteforce of password/flag checks without reversing the validation algorithm.
Detection: Binary uses memcmp or strcmp for flag validation (visible in ltrace output or import table). The comparison function is called with user input and a computed/stored expected value.
References: Blaze CTF 2018