All skills
inference-shell avatar

/agent-browser

@b576e8e

Browser automation for AI agents via inference.sh. Navigate web pages, interact with elements using @e refs, take screenshots, record video. Capabilities: web scraping, form filling, clicking, typing, drag-drop, file upload, JavaScript execution. Use for: web automation, data extraction, testing, agent browsing, research. Triggers: browser, web automation, scrape, navigate, click, fill form, screenshot, browse web, playwright, headless browser, web agent, surf internet, record video

Use this Skill: https://skilld.dev/gh/inference-shell/skills/agent-browser

This session only. Nothing lands on disk.

referencesauthentication.md

≈1.9k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Authentication Patterns

Login flows, OAuth, 2FA, and authenticated browsing.

Related: session-management.md for session details, SKILL.md for quick start.

Contents

Basic Login Flow

Standard username/password login:

#!/bin/bash

# Start session
SESSION=$(belt app run agent-browser --function open --session new --input '{
  "url": "https://app.example.com/login"
}' | jq -r '.session_id')

# Get form elements
# Expected: @e1 [input type="email"], @e2 [input type="password"], @e3 [button] "Sign In"

# Fill credentials
belt app run agent-browser --function interact --session $SESSION --input '{
  "action": "fill", "ref": "@e1", "text": "user@example.com"
}'

belt app run agent-browser --function interact --session $SESSION --input '{
  "action": "fill", "ref": "@e2", "text": "'"$PASSWORD"'"
}'

# Submit
belt app run agent-browser --function interact --session $SESSION --input '{
  "action": "click", "ref": "@e3"
}'

# Wait for redirect
belt app run agent-browser --function interact --session $SESSION --input '{
  "action": "wait", "wait_ms": 2000
}'

# Verify login succeeded
RESULT=$(belt app run agent-browser --function snapshot --session $SESSION --input '{}')
URL=$(echo $RESULT | jq -r '.url')

if [[ "$URL" == *"/login"* ]]; then
  echo "Login failed - still on login page"
  exit 1
fi

echo "Login successful"
# Continue with authenticated actions...

OAuth / SSO Flows

For OAuth redirects (Google, GitHub, etc.):

#!/bin/bash

SESSION=$(belt app run agent-browser --function open --session new --input '{
  "url": "https://app.example.com/auth/google"
}' | jq -r '.session_id')

# Wait for redirect to Google
belt app run agent-browser --function interact --session $SESSION --input '{
  "action": "wait", "wait_ms": 3000
}'

# Snapshot to see Google login form
RESULT=$(belt app run agent-browser --function snapshot --session $SESSION --input '{}')
echo $RESULT | jq '.elements_text'

# Fill Google email
belt app run agent-browser --function interact --session $SESSION --input '{
  "action": "fill", "ref": "@e1", "text": "user@gmail.com"
}'

# Click Next
belt app run agent-browser --function interact --session $SESSION --input '{
  "action": "click", "ref": "@e2"
}'

# Wait and snapshot for password field
belt app run agent-browser --function interact --session $SESSION --input '{
  "action": "wait", "wait_ms": 2000
}'
RESULT=$(belt app run agent-browser --function snapshot --session $SESSION --input '{}')

# Fill password
belt app run agent-browser --function interact --session $SESSION --input '{
  "action": "fill", "ref": "@e1", "text": "'"$GOOGLE_PASSWORD"'"
}'

# Click Sign in
belt app run agent-browser --function interact --session $SESSION --input '{
  "action": "click", "ref": "@e2"
}'

# Wait for redirect back to app
belt app run agent-browser --function interact --session $SESSION --input '{
  "action": "wait", "wait_ms": 5000
}'

# Verify we're back on the app
RESULT=$(belt app run agent-browser --function snapshot --session $SESSION --input '{}')
URL=$(echo $RESULT | jq -r '.url')
echo "Final URL: $URL"

Two-Factor Authentication

For 2FA, you may need human intervention or TOTP generation:

With TOTP Code

# After password, check for 2FA prompt
RESULT=$(belt app run agent-browser --function snapshot --session $SESSION --input '{}')
ELEMENTS=$(echo $RESULT | jq -r '.elements_text')

if echo "$ELEMENTS" | grep -qi "verification\|2fa\|authenticator"; then
  # Generate TOTP code (requires oathtool)
  TOTP_CODE=$(oathtool --totp -b "$TOTP_SECRET")

  # Fill 2FA code
  belt app run agent-browser --function interact --session $SESSION --input '{
    "action": "fill", "ref": "@e1", "text": "'"$TOTP_CODE"'"
  }'

  # Submit
  belt app run agent-browser --function interact --session $SESSION --input '{
    "action": "click", "ref": "@e2"
  }'
fi

With Manual Intervention

For SMS or hardware token 2FA:

# Record video so user can see the 2FA prompt
SESSION=$(belt app run agent-browser --function open --session new --input '{
  "url": "https://app.example.com/login",
  "record_video": true
}' | jq -r '.session_id')

# ... login flow ...

# At 2FA step, prompt user
echo "2FA code sent. Enter the code:"
read -r CODE

belt app run agent-browser --function interact --session $SESSION --input '{
  "action": "fill", "ref": "@e1", "text": "'"$CODE"'"
}'

Session Reuse Patterns

Since sessions maintain cookies, you can reuse authenticated sessions:

#!/bin/bash
# login-and-work.sh

# Login once
login() {
  SESSION=$(belt app run agent-browser --function open --session new --input '{
    "url": "https://app.example.com/login"
  }' | jq -r '.session_id')

  # ... login steps ...

  echo $SESSION
}

# Do work with authenticated session
do_work() {
  local SESSION=$1

  # Navigate to protected page
  belt app run agent-browser --function interact --session $SESSION --input '{
    "action": "goto", "url": "https://app.example.com/dashboard"
  }'

  # Extract data
  belt app run agent-browser --function snapshot --session $SESSION --input '{}'
}

# Main
SESSION=$(login)
do_work $SESSION

# Don't close if you want to reuse!
# belt app run agent-browser --function close --session $SESSION --input '{}'

Cookie Extraction

Extract cookies for use in other tools:

# Get cookies via JavaScript
RESULT=$(belt app run agent-browser --function execute --session $SESSION --input '{
  "code": "document.cookie"
}')
COOKIES=$(echo $RESULT | jq -r '.result')
echo "Cookies: $COOKIES"

# Get all cookies including httpOnly (more complete)
RESULT=$(belt app run agent-browser --function execute --session $SESSION --input '{
  "code": "JSON.stringify(performance.getEntriesByType(\"resource\").map(r => r.name))"
}')

Security Best Practices

1. Never Hardcode Credentials

# Good: Use environment variables
'{"action": "fill", "ref": "@e2", "text": "'"$PASSWORD"'"}'

# Bad: Hardcoded
'{"action": "fill", "ref": "@e2", "text": "mypassword123"}'

2. Use Secure Environment Variables

# Set securely
export PASSWORD=$(cat /path/to/secure/password)

# Or use a secrets manager
export PASSWORD=$(vault read -field=password secret/app)

3. Don't Log Sensitive Data

# Good: Redact sensitive info
echo "Logging in as $USERNAME"

# Bad: Logging passwords
echo "Password: $PASSWORD"  # Never do this!

4. Close Sessions After Use

# Always clean up
trap 'belt app run agent-browser --function close --session $SESSION --input "{}" 2>/dev/null' EXIT

5. Use Video Recording for Debugging Only

Video may capture sensitive information:

# Only enable when debugging
if [ "$DEBUG" = "true" ]; then
  RECORD_VIDEO="true"
else
  RECORD_VIDEO="false"
fi

6. Verify Login Success

Always confirm authentication worked:

# Check URL changed from login page
URL=$(echo $RESULT | jq -r '.url')
if [[ "$URL" == *"/login"* ]] || [[ "$URL" == *"/signin"* ]]; then
  echo "ERROR: Login failed"
  exit 1
fi

# Or check for specific element on authenticated page
ELEMENTS=$(echo $RESULT | jq -r '.elements_text')
if ! echo "$ELEMENTS" | grep -q "Logout\|Dashboard\|Welcome"; then
  echo "ERROR: Not authenticated"
  exit 1
fi

Source: SKILL.md on GitHub

1 alert13d5 checks · Risk SAFE
  • Gen Agent Trust Hub13d

    The skill provides powerful browser automation capabilities including JavaScript execution and file uploads. While these are intended features, they introduce risks such as indirect prompt injection from malicious websites and the potential for session data exfiltration via scripts.

  • Socket13d

    2 alerts: gptAnomaly, gptSecurity

  • Snyk13d

    Risk: MEDIUM · 1 issue

  • Runlayer6mo

    7/10 files flagged

  • ZeroLeaks5mo

    Score: 93/100 · 2 sections analyzed

Signed by skilld at b576e8e. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub last week.

Activeupdated 5 months ago
What it can do
Runs commands
All 1 allowed tools
Bash(belt *)
  • browser-automation
  • playwright
  • web-scraping
  • form-filling
  • screenshot
  • video-recording
  • javascript-execution
  • inference-sh
  • headless-browser

README badge

README badge for inference-shell/skills/agent-browser

Browser automation using Playwright with inference.sh, controlled via the `belt` CLI. Navigate pages, interact with elements using @e refs, take screenshots, record video, and run JavaScript. Suitable for web scraping, form filling, clicking, typing, drag-drop, file upload, and automated testing workflows.

Generated from the current SKILL.md.

What browser engine does this use?
Playwright. The skill wraps Playwright under the hood with a simplified @e ref system for element interaction.
Do I need to install anything besides the skill?
Yes, you need the belt CLI installed. The skill documentation includes install instructions.
Can I record video of browser sessions?
Yes. Pass record_video: true when opening a session, and the video file is returned when you close the session.
How do I interact with page elements?
Use the @e refs returned by open and snapshot calls. Pass the ref to interact actions like click, fill, drag, upload, or execute JavaScript.
Do element refs stay valid after navigation?
No. Refs are invalidated after navigation, form submission, or dynamic content loading. Always call snapshot after these events to get fresh refs.

Generated from the current SKILL.md. These answers refresh after source changes.