/ctf-ai-ml

CTF AI/ML - Model Attacks

Techniques for attacking ML models directly: weight manipulation, model inversion, encoder collision, LoRA adapter exploitation, model extraction, and membership inference. For adversarial example generation and data poisoning, see adversarial-ml.md. For LLM-specific attacks, see llm-attacks.md.

Table of Contents

• ML Model Weight Perturbation Negation (DiceCTF 2026)
• ML Model Inversion via Gradient Descent (BSidesSF 2025)
• Neural Network Encoder Collision (RootAccess2026)
• LoRA Adapter Weight Merging (ApoorvCTF 2026)
• Model Extraction via Query API
• Membership Inference Attack

ML Model Weight Perturbation Negation (DiceCTF 2026)

Pattern: A GPT-2 model has been fine-tuned to suppress a specific behavior (e.g., generating a flag). The challenge provides both the original base model and the fine-tuned (suppressed) model. By computing the weight delta and negating it, you reverse the suppression into amplification.

The key mathematical insight: if W_chal = W_orig + delta where delta was learned to suppress output, then W_recovered = W_orig - delta = 2*W_orig - W_chal amplifies the original behavior instead.

import torch
from transformers import GPT2LMHeadModel, GPT2Tokenizer

# Load both models
original = GPT2LMHeadModel.from_pretrained("gpt2")
challenge = GPT2LMHeadModel.from_pretrained("./challenge_model")

# Compute negated weights: W_recovered = 2*W_orig - W_chal
recovered = GPT2LMHeadModel.from_pretrained("gpt2")
orig_sd = original.state_dict()
chal_sd = challenge.state_dict()
rec_sd = recovered.state_dict()

for key in orig_sd:
    rec_sd[key] = 2 * orig_sd[key] - chal_sd[key]

recovered.load_state_dict(rec_sd)

# Generate with recovered model
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token

prompt = "The flag is"
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
    output = recovered.generate(
        **inputs,
        max_new_tokens=100,
        temperature=0.7,
        do_sample=True,
        num_return_sequences=5,
    )

for seq in output:
    print(tokenizer.decode(seq, skip_special_tokens=True))

Key insight: Fine-tuning adds a delta to weights. Negating that delta reverses suppression into amplification. Check which layers differ most with (orig_sd[k] - chal_sd[k]).abs().max() to confirm fine-tuning targeted specific layers. If only certain layers changed, the delta is sparse and negation is even more effective.

Variations:

• Partial negation with scaling: Sometimes W_orig + alpha * (W_orig - W_chal) with alpha > 1 works better than pure negation. Try alpha values from 1.0 to 3.0.
• Layer-selective negation: Only negate layers that show significant delta (threshold by L2 norm of difference).
• LoRA-aware negation: If fine-tuning used LoRA, the delta is low-rank. Extract and negate only the LoRA component.

ML Model Inversion via Gradient Descent (BSidesSF 2025)

Pattern: Given a trained model and a target output (e.g., a specific class label or embedding vector), recover the input by optimizing a random input tensor using gradient descent to minimize the distance between the model's output and the target.

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import transforms
from PIL import Image

# Load the challenge model
# NOTE (PyTorch 2.6+): torch.load defaults to weights_only=True, which blocks
# full pickled models. CTF model files need weights_only=False (only load
# files you trust — untrusted pickles can execute arbitrary code).
model = torch.load("challenge_model.pt", map_location="cpu", weights_only=False)
model.eval()

# Target: the output we want to invert (e.g., a specific embedding or class)
target_output = torch.load("target_embedding.pt", weights_only=False)  # shape depends on model

# Initialize random input (e.g., 3x224x224 image)
input_tensor = torch.randn(1, 3, 224, 224, requires_grad=True)

optimizer = optim.Adam([input_tensor], lr=0.01)
mse_loss = nn.MSELoss()

for step in range(2000):
    optimizer.zero_grad()
    output = model(input_tensor)
    loss = mse_loss(output, target_output)

    # Optional: add total variation regularization for smoother images
    tv_loss = (
        torch.sum(torch.abs(input_tensor[:, :, :, :-1] - input_tensor[:, :, :, 1:])) +
        torch.sum(torch.abs(input_tensor[:, :, :-1, :] - input_tensor[:, :, 1:, :]))
    )
    total_loss = loss + 1e-4 * tv_loss

    total_loss.backward()
    optimizer.step()

    # Clamp to valid image range
    with torch.no_grad():
        input_tensor.clamp_(0, 1)

    if step % 200 == 0:
        print(f"Step {step}: loss={loss.item():.6f}")

# Save recovered image
recovered = input_tensor.squeeze(0).detach()
img = transforms.ToPILImage()(recovered)
img.save("recovered_input.png")
print("Recovered input saved to recovered_input.png")

Key insight: Neural networks are differentiable, so you can backpropagate through them to optimize the input. Total variation regularization produces more natural-looking images. If the model has batch normalization, set it to eval mode (model.eval()) to use running statistics rather than batch statistics.

Variations:

• Feature visualization: Maximize a specific neuron's activation instead of matching a target output.
• Deep Dream style: Use layer activations as optimization targets for artistic reconstruction.
• Gradient-free inversion: If gradients are unavailable (black-box), use CMA-ES or other evolutionary strategies.

Neural Network Encoder Collision (RootAccess2026)

Pattern: Given a neural network encoder, find two distinct inputs that produce identical (or nearly identical) output embeddings. This exploits the dimensionality reduction inherent in encoders — the mapping from high-dimensional input to lower-dimensional embedding space is not injective.

import torch
import torch.nn as nn
import torch.optim as optim

# Load the encoder model (weights_only=False required on PyTorch 2.6+ for pickled models)
encoder = torch.load("encoder.pt", map_location="cpu", weights_only=False)
encoder.eval()

# Initialize two random inputs
input_a = torch.randn(1, 3, 64, 64, requires_grad=True)
input_b = torch.randn(1, 3, 64, 64, requires_grad=True)

optimizer = optim.Adam([input_a, input_b], lr=0.005)

for step in range(5000):
    optimizer.zero_grad()

    emb_a = encoder(input_a)
    emb_b = encoder(input_b)

    # Minimize distance between embeddings
    collision_loss = nn.MSELoss()(emb_a, emb_b)

    # Maximize distance between inputs (so they are distinct)
    input_diff = nn.MSELoss()(input_a, input_b)
    diversity_loss = -input_diff  # negative because we want to maximize

    # Regularize to valid range
    range_penalty = (
        torch.relu(-input_a).sum() + torch.relu(input_a - 1).sum() +
        torch.relu(-input_b).sum() + torch.relu(input_b - 1).sum()
    )

    loss = collision_loss + 0.1 * diversity_loss + 0.01 * range_penalty
    loss.backward()
    optimizer.step()

    with torch.no_grad():
        input_a.clamp_(0, 1)
        input_b.clamp_(0, 1)

    if step % 500 == 0:
        dist = (emb_a - emb_b).norm().item()
        inp_dist = (input_a - input_b).norm().item()
        print(f"Step {step}: emb_dist={dist:.8f}, input_dist={inp_dist:.4f}")

# Verify collision
with torch.no_grad():
    final_a = encoder(input_a)
    final_b = encoder(input_b)
    print(f"Final embedding distance: {(final_a - final_b).norm().item():.10f}")
    print(f"Final input distance: {(input_a - input_b).norm().item():.4f}")
    print(f"Embeddings equal: {torch.allclose(final_a, final_b, atol=1e-6)}")

Key insight: Encoders compress information, so collisions must exist by the pigeonhole principle. The optimization simultaneously minimizes embedding distance while maximizing input distance. Starting with very different random initializations helps avoid trivial solutions.

Variations:

• Targeted collision: Force both inputs to map to a specific target embedding.
• Near-collision with hamming constraint: Find inputs that differ in only a few pixels but produce identical embeddings.
• Hash-like collision: If the encoder output is discretized (e.g., binary hash), collision search is easier via relaxation.

LoRA Adapter Weight Merging (ApoorvCTF 2026)

Pattern: A LoRA (Low-Rank Adaptation) adapter is provided alongside a base model. The adapter encodes hidden information in its low-rank weight matrices. Merging the adapter into the base model and generating output (or visualizing weight patterns) reveals the flag.

LoRA modifies weights as: W_merged = W_base + alpha * (B @ A) where A and B are the low-rank matrices and alpha is the scaling factor.

import torch
from safetensors import safe_open
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load base model
base_model = AutoModelForCausalLM.from_pretrained("gpt2")
tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token

# Inspect LoRA adapter structure
adapter = safe_open("adapter_model.safetensors", framework="pt")
print("LoRA keys:", list(adapter.keys()))
# Typical keys: base_model.model.transformer.h.0.attn.c_attn.lora_A.weight
#               base_model.model.transformer.h.0.attn.c_attn.lora_B.weight

# Manual merge: for each LoRA pair, compute W_merged = W_base + alpha * (B @ A)
alpha = 1.0  # Check adapter_config.json for lora_alpha and r values
# Effective alpha = lora_alpha / r

lora_a_keys = [k for k in adapter.keys() if "lora_A" in k]
lora_b_keys = [k for k in adapter.keys() if "lora_B" in k]

base_sd = base_model.state_dict()

for a_key in lora_a_keys:
    b_key = a_key.replace("lora_A", "lora_B")
    # Map LoRA key back to base model key
    # e.g., "base_model.model.transformer.h.0.attn.c_attn.lora_A.weight"
    #     -> "transformer.h.0.attn.c_attn.weight"
    base_key = a_key.replace("base_model.model.", "").replace(".lora_A.weight", ".weight")

    A = adapter.get_tensor(a_key)  # shape: (r, in_features)
    B = adapter.get_tensor(b_key)  # shape: (out_features, r)

    delta = alpha * (B @ A)  # shape: (out_features, in_features)

    if base_key in base_sd:
        base_sd[base_key] = base_sd[base_key] + delta
        print(f"Merged {base_key}: delta norm = {delta.norm():.4f}")

base_model.load_state_dict(base_sd)

# Generate with merged model
prompt = "The secret is"
inputs = tokenizer(prompt, return_tensors="pt")
with torch.no_grad():
    output = base_model.generate(
        **inputs,
        max_new_tokens=100,
        temperature=0.7,
        do_sample=True,
    )
print(tokenizer.decode(output[0], skip_special_tokens=True))

Alternative: using PEFT library for automatic merging:

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("gpt2")
model = PeftModel.from_pretrained(base, "./lora_adapter_dir")
model = model.merge_and_unload()  # Merge LoRA into base weights

tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token
inputs = tokenizer("The flag is", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Key insight: LoRA adapters modify only a small subset of weights via low-rank matrices. The adapter_config.json file contains lora_alpha, r (rank), and target_modules which tell you exactly which layers were modified. Sometimes the hidden content is not in the model's text output but in the weight matrices themselves — try visualizing B @ A as an image.

Variations:

• Weight visualization: Reshape B @ A delta matrices and render as images; flags may be encoded visually in the weight pattern.
• Multi-adapter stacking: Multiple LoRA adapters applied sequentially; merge in correct order.
• Quantized adapters: QLoRA uses 4-bit quantization; dequantize before merging.

Model Extraction via Query API

Pattern: A challenge exposes a model via an API endpoint. By sending carefully crafted inputs and observing outputs (predictions, confidence scores, logits), you can reconstruct the model's parameters or decision boundary. This is especially effective against simple models (linear, decision tree, small neural networks).

import numpy as np
import requests
from sklearn.linear_model import LogisticRegression

API_URL = "http://challenge:8080/predict"

def query_model(x):
    """Send input to model API and get prediction/confidence."""
    resp = requests.post(API_URL, json={"input": x.tolist()})
    return resp.json()  # e.g., {"class": 1, "confidence": 0.87}

# Strategy 1: Decision boundary mapping for 2D models
# Sample a grid of points to map the decision boundary
xs = np.linspace(-5, 5, 100)
ys = np.linspace(-5, 5, 100)
X_grid = np.array([[x, y] for x in xs for y in ys])
labels = []
confidences = []

for point in X_grid:
    result = query_model(point)
    labels.append(result["class"])
    confidences.append(result["confidence"])

labels = np.array(labels)
confidences = np.array(confidences)

# Fit a surrogate model to the extracted labels
surrogate = LogisticRegression()
surrogate.fit(X_grid, labels)
print(f"Extracted weights: {surrogate.coef_}")
print(f"Extracted bias: {surrogate.intercept_}")

# Strategy 2: Exact weight extraction for linear models
# For a linear model f(x) = sigmoid(w*x + b), query with basis vectors
dim = 10  # input dimensionality
weights = np.zeros(dim)
# Query with zero vector to get bias term
base_result = query_model(np.zeros(dim))
base_logit = np.log(base_result["confidence"] / (1 - base_result["confidence"]))

for i in range(dim):
    e_i = np.zeros(dim)
    e_i[i] = 1.0
    result = query_model(e_i)
    logit = np.log(result["confidence"] / (1 - result["confidence"]))
    weights[i] = logit - base_logit

print(f"Extracted weights: {weights}")
print(f"Extracted bias: {base_logit}")

Key insight: Linear models can be extracted exactly with dim + 1 queries (one per basis vector plus zero vector). For neural networks, use model distillation: train a student network on input-output pairs from the API. Decision trees can be extracted by probing decision boundaries with binary search.

Variations:

• Logit extraction: If API returns only class labels (not confidence), use inputs near decision boundaries with binary search.
• Functionally equivalent extraction: Train a neural network on API responses; fidelity > 99% is often achievable with 10K-100K queries.
• Side-channel extraction: Timing differences in API responses can leak model architecture (deeper = slower).

Membership Inference Attack

Pattern: Determine whether a specific data sample was part of the model's training set. Training data members typically produce higher confidence predictions and lower loss values than non-members, because the model has memorized them to some degree.

import torch
import torch.nn.functional as F
import numpy as np
from sklearn.metrics import roc_auc_score

# Load challenge model (weights_only=False required on PyTorch 2.6+ for pickled models)
model = torch.load("target_model.pt", map_location="cpu", weights_only=False)
model.eval()

def get_prediction_metrics(model, x, true_label):
    """Compute metrics that distinguish members from non-members."""
    with torch.no_grad():
        logits = model(x.unsqueeze(0))
        probs = F.softmax(logits, dim=1)
        confidence = probs[0, true_label].item()
        loss = F.cross_entropy(logits, torch.tensor([true_label])).item()
        entropy = -(probs * torch.log(probs + 1e-10)).sum().item()
    return {
        "confidence": confidence,
        "loss": loss,
        "entropy": entropy,
        "top1_margin": (probs.max() - probs.topk(2).values[0, 1]).item(),
    }

# Method 1: Simple threshold attack
# Members typically have higher confidence and lower loss
def threshold_attack(metrics, threshold=0.9):
    """Predict membership based on confidence threshold."""
    return metrics["confidence"] > threshold

# Method 2: Shadow model attack (more sophisticated)
# Train shadow models on known in/out splits to learn the membership signal
def shadow_model_attack(target_model, candidate_samples, candidate_labels):
    """Use multiple metrics to predict membership."""
    results = []
    for x, y in zip(candidate_samples, candidate_labels):
        m = get_prediction_metrics(target_model, x, y)
        # High confidence + low loss + low entropy = likely member
        score = m["confidence"] - 0.5 * m["entropy"]
        results.append({
            "sample_label": y,
            "member_score": score,
            **m,
        })

    # Sort by membership likelihood
    results.sort(key=lambda r: r["member_score"], reverse=True)
    return results

# Example usage
candidate = torch.randn(3, 224, 224)  # single candidate image
label = 5  # true class
metrics = get_prediction_metrics(model, candidate, label)
print(f"Confidence: {metrics['confidence']:.4f}")
print(f"Loss: {metrics['loss']:.4f}")
print(f"Entropy: {metrics['entropy']:.4f}")
print(f"Likely member: {threshold_attack(metrics)}")

Key insight: Models overfit to training data, producing measurably different behavior on seen vs. unseen samples. The gap between training and test confidence is the core signal. More sophisticated attacks train a binary classifier on (metrics, member/non-member) pairs from shadow models trained on similar data distributions.

Variations:

• Label-only attack: When only the predicted class is returned (no confidence), use perturbation sensitivity: members are more robust to small perturbations.
• Augmentation attack: Apply data augmentations; members maintain consistent predictions across augmentations.
• LiRA (Likelihood Ratio Attack): Train multiple shadow models with/without the target sample; compare loss distributions.