/bright-data-best-practices

Web Unlocker API Reference

Table of Contents

• Overview
• Authentication
• REST API (Recommended)
• Proxy Interface
• All Request Parameters
• Special Headers
• Response Structure
• Async Requests
• Output Formats
• Features
• Billing Model
• Best Practices
• Anti-Patterns

Overview

Web Unlocker is Bright Data's HTTP-based unlocking proxy. It automatically selects the best proxy network, handles anti-bot protections, solves CAPTCHAs, and retries failed attempts. Use it for non-interactive data extraction where you need HTML, JSON, markdown, or screenshots of web pages via a simple HTTP request.

When to use Web Unlocker vs other APIs:

Authentication

Get your API key from the Bright Data Control Panel → Account Settings, or from your zone's Overview tab.

# Environment variable (recommended)
export BRIGHTDATA_API_KEY="your-api-key"
export BRIGHTDATA_UNLOCKER_ZONE="your-zone-name"

REST API (Recommended)

Endpoint: POST https://api.brightdata.com/request

Header: Authorization: Bearer YOUR_API_KEY

Minimal Request

import requests

response = requests.post(
    "https://api.brightdata.com/request",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={
        "zone": "YOUR_ZONE_NAME",
        "url": "https://example.com",
        "format": "raw"
    }
)
print(response.text)

const response = await fetch("https://api.brightdata.com/request", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${API_KEY}`,
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    zone: "YOUR_ZONE_NAME",
    url: "https://example.com",
    format: "raw"
  })
});
const html = await response.text();

curl -H "Authorization: Bearer $BRIGHTDATA_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"zone":"'"$BRIGHTDATA_UNLOCKER_ZONE"'","url":"https://example.com","format":"raw"}' \
     https://api.brightdata.com/request

Proxy Interface

Alternative to the REST API — route requests through a proxy endpoint.

• Host: brd.superproxy.io
• Port: 33335
• Credentials: brd-customer-{CUSTOMER_ID}-zone-{ZONE_NAME}:{ZONE_PASSWORD}

import requests

proxies = {
    "http": "http://brd-customer-CUSTOMER_ID-zone-ZONE_NAME:PASSWORD@brd.superproxy.io:33335",
    "https": "http://brd-customer-CUSTOMER_ID-zone-ZONE_NAME:PASSWORD@brd.superproxy.io:33335"
}
# Install Bright Data's SSL certificate to avoid SSL errors
response = requests.get("https://example.com", proxies=proxies, verify="/path/to/brightdata-cert.crt")

Special proxy username flags (append to username string):

• -country-XX → Geo-target to country (e.g., -country-us)
• -ua-mobile → Use mobile user agent
• -debug-full → Enable debug header in response

All Request Parameters

REST API Body Parameters

Special Headers

Used with the proxy interface. Pass in proxy username or as custom request headers.

Proxy username flags (appended to username):

• -ua-mobile → Uses mobile user agents instead of desktop
• -debug-full → Adds x-brd-debug response header with: request ID, traffic metrics, billing status, destination IP, headers used, peer info, render status

Response Structure

Synchronous (format: "json")

{
  "status": 200,
  "headers": {
    "content-type": "text/html",
    "...": "..."
  },
  "body": "<html>...</html>"
}

Synchronous (format: "raw")

Returns the raw HTML/content as a string directly.

Error Codes

Async Requests

Setup

Enable in Control Panel: Your Zone → Advanced Options → Toggle "Asynchronous requests" ON.

Flow

Step 1: Submit request

response = requests.post(
    "https://api.brightdata.com/unblocker/req",
    params={"zone": "YOUR_ZONE_NAME"},
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={"url": "https://example.com"}
)
response_id = response.json()["response_id"]

Step 2: Poll for result

import time

while True:
    result = requests.get(
        "https://api.brightdata.com/unblocker/get_result",
        params={"response_id": response_id},
        headers={"Authorization": f"Bearer {API_KEY}"}
    )
    if result.status_code == 200:
        print(result.json()["body"])
        break
    time.sleep(5)

Async Key Facts

• Responses typically complete within 5 minutes, up to 8 hours during peak
• Results stored for 48 hours
• Better for slow-responding sites or large-scale URL processing
• Use webhooks for production: set webhook URL in zone settings to receive push notifications

Output Formats

HTML (default)

Raw HTML string. Use format: "raw".

Markdown

Clean markdown extracted from the page. Best for LLM consumption.

REST API:

{ "zone": "...", "url": "...", "format": "raw", "data_format": "markdown" }

Proxy: Add header: x-unblock-data-format: markdown

Screenshot

PNG screenshot of the rendered page. Useful for debugging and visual verification.

REST API:

{ "zone": "...", "url": "...", "format": "raw", "data_format": "screenshot" }

Features

CAPTCHA Solving

Enabled by default. Automatically solves CAPTCHAs encountered during requests. Can be disabled in Advanced Settings for a lightweight solution when CAPTCHA solving isn't needed.

Premium Domains

Certain high-difficulty websites require additional resources. Enable "Premium Domains" in zone settings during creation. Only requests targeting premium-classified domains are billed at the higher rate.

Geolocation Targeting

Auto-selects optimal IP location. Override with country parameter (REST API) or -country-XX (proxy). Country-level targeting only (not city-level—use Browser API for that).

Mobile User Agent

Append -ua-mobile to proxy username, or use via proxy username flag, to use mobile-specific user agents.

Auto-Throttling

System automatically adjusts based on success rates:

• Default threshold: 70% success rate
• Automatically applies better-performing configurations
• When custom headers/cookies are enabled: customizable threshold

Debug Header

Add -debug-full to proxy username to receive x-brd-debug response header containing: request ID, traffic metrics, billing status, destination IP, headers used, peer info, render status.

Success Rate API

Query domain-specific success rates for the past 7 days:

GET https://api.brightdata.com/unblocker/success_rate/?zone=YOUR_ZONE&domain=example.com
Authorization: Bearer YOUR_API_KEY

Custom Headers & Cookies (Advanced)

Override automatically-generated headers and cookies.

When to use: When you need to pass specific session cookies, authentication headers, or custom values to reach a particular site version.

How to enable: Control Panel → Your Zone → Advanced Options → Toggle "Custom Headers & Cookies" ON.

Critical billing note: Enabling custom headers/cookies means you are billed for 100% of requests (both successful and failed), because you are taking control of request parameters. Standard mode only bills for successes.

Restrictions:

• Custom values must be from a compliance-pre-approved list
• Unlisted values require approval from the compliance team
• Cannot pass authentication/login credentials

Billing Model

Monitor usage via the "Traffic" column in My Proxies (CPM = cost per 1,000 successful requests).

Best Practices

1. Start with direct API endpoint targeting

Many sites expose clean API endpoints. Try hitting the API directly first — it often succeeds without extra config and is cheaper.

# Try the API endpoint first
response = requests.post(
    "https://api.brightdata.com/request",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={"zone": ZONE, "url": "https://site.com/api/products.json", "format": "raw"}
)

2. Fall back to main webpage if API fails

If the direct API endpoint fails, scrape the primary webpage instead.

3. Use Browser API for complex JS-heavy scenarios

When you need to execute JavaScript, click elements, or interact with the page — don't try to force Web Unlocker. Use Browser API.

4. Use x-unblock-expect to prevent partial loads

For pages that load content progressively, specify an expected element to ensure the content you need is present.

{
  "zone": "...",
  "url": "https://example.com/products",
  "format": "raw",
  "headers": { "x-unblock-expect": ".product-list" }
}

5. Use markdown format for LLM pipelines

When feeding scraped content to an LLM, use data_format: "markdown" to get clean, structured text without HTML noise.

6. Use async for bulk/large-scale processing

Async improves stability for slow sites and enables processing large batches of URLs without blocking.

7. Use geolocation for region-restricted content

Pass country parameter when you need region-specific content (prices, availability, localized pages).

{ "zone": "...", "url": "https://example.com", "format": "raw", "country": "de" }

8. Monitor success rates before custom header mode

Check your domain success rates via the API before enabling custom headers (which changes billing to 100%).

Anti-Patterns

DO NOT use Web Unlocker with browser automation libraries.

• Puppeteer, Playwright, Selenium → use Browser API instead
• Chrome, Firefox, Edge → use Bright Data proxy networks (Residential, ISP, etc.)
• Anti-detect browsers (Adspower, Multilogin) → use proxy networks

Web Unlocker is optimized for singular HTTP requests, not browser sessions.

DO NOT enable custom headers unless necessary. Enabling custom headers changes billing from success-only to 100% of all requests.

DO NOT ignore the x-unblock-expect header for paginated/dynamic content. Without it, you may receive partial pages before JavaScript has populated content.