Local Agent Builder eval setup

Required LangTracer access

Generate a key on the LangTracer API keys page. Add it to .env.eval at the repository root:

LANGTRACER_URL=https://lang-tracer.n8n-maintenance.workers.dev
LANGTRACER_API_KEY=<generated-key>

Do not paste the key into chat or put it in a command. The main skill checks that both variables exist before it creates an eval.

Remaining environment

Create .env.eval at the repository root. Use one Instance AI model provider and one sandbox provider.

N8N_INSTANCE_AI_MODEL=anthropic/claude-sonnet-4-6
N8N_INSTANCE_AI_MODEL_API_KEY=<model-key>
ANTHROPIC_API_KEY=<same-model-key>
N8N_AI_ASSISTANT_BASE_URL=

N8N_INSTANCE_AI_SANDBOX_ENABLED=true
N8N_INSTANCE_AI_SANDBOX_PROVIDER=daytona
DAYTONA_API_URL=https://app.daytona.io/api
DAYTONA_API_KEY=<daytona-key>

N8N_EVAL_EMAIL=nathan@n8n.io
N8N_EVAL_PASSWORD=PlaywrightTest123

Use the n8n sandbox service instead of Daytona when it is available. Set N8N_INSTANCE_AI_SANDBOX_PROVIDER=n8n-sandbox, N8N_SANDBOX_SERVICE_URL, and, when required, N8N_SANDBOX_SERVICE_API_KEY.

LangSmith is optional. Unset LANGSMITH_API_KEY and LANGSMITH_TRACING for a throwaway direct run. This keeps the run out of the shared LangSmith dataset.

Build after code changes

From the repository root:

(
  pnpm --filter n8n... build > build.log 2>&1
  build_status=$?
  tail -n 20 build.log
  exit "$build_status"
)

Restart the instance after each build. A running process keeps the old code in memory.

Start an isolated instance

Create a temporary user directory:

mktemp -d /tmp/n8n-agent-eval.XXXXXX

Use the printed path as N8N_USER_FOLDER. Start the built CLI from packages/cli:

N8N_PORT=5680 \
N8N_RUNNERS_BROKER_PORT=5681 \
N8N_USER_FOLDER=<printed-temp-directory> \
E2E_TESTS=true \
N8N_ENABLED_MODULES=instance-ai,agents \
N8N_AI_ENABLED=true \
pnpm exec dotenvx run -f ../../.env.eval -- pnpm start

Use different ports when 5680 or 5681 is busy.

Seed the owner

Run this once after the isolated instance starts:

curl -sf -X POST http://localhost:5680/rest/e2e/reset \
  -H 'Content-Type: application/json' \
  -d '{"owner":{"email":"nathan@n8n.io","password":"PlaywrightTest123","firstName":"Eval","lastName":"Owner"},"admin":{"email":"admin@n8n.io","password":"PlaywrightTest123","firstName":"Admin","lastName":"User"},"members":[],"chat":{"email":"chat@n8n.io","password":"PlaywrightTest123","firstName":"Chat","lastName":"User"}}'

Run a local disk case

From packages/@n8n/instance-ai:

pnpm exec dotenvx run -f ../../../.env.eval -- \
  env -u LANGSMITH_API_KEY -u LANGSMITH_TRACING \
  pnpm eval:instance-ai \
  --base-url http://localhost:5680 \
  --filter <slug> \
  --tier agents \
  --concurrency 1 \
  --verbose

The run writes eval-results.json, an HTML report, and a PR-comment preview. The warnings about disabled run-debug capture are benign.

/create-agent-builder-eval

local-setup.md