Index of every page on this site: llms.txt
All skills
comet-ml avatar
by Cometcomet-ml/opik23k stars
1,964
Run it yourself

Python SDK patterns for Opik. Use when working in sdks/python, on SDK APIs, integrations, or message processing.

Use this Skill: https://skilld.dev/gh/comet-ml/opik/python-sdk

Nothing lands on disk. Nothing to clean up.

Fork this Skill

Edit a local copy. It keeps the author and licence.

testing.md

≈1.4k tokens on demand. Your agent reads this file only when SKILL.md points to it.

Python SDK Testing Patterns

Test Naming Convention

# Pattern: test_WHAT__CASE_DESCRIPTION__EXPECTED_RESULT
def test_tracked_function__error_inside_inner_function__caught_in_top_level_span():
    pass

# Happy path: test_WHAT__happyflow
def test_optimization_lifecycle__happyflow():
    pass

Using fake_backend (Integration Tests)

For testing integrations that create traces/spans:

from tests.testlib import TraceModel, SpanModel, ANY_BUT_NONE, assert_equal
from opik.decorator import tracker

def test_track__one_nested_function__happyflow(fake_backend):
    @tracker.track
    def f_inner(x):
        return "inner-output"

    @tracker.track
    def f_outer(x):
        f_inner("inner-input")
        return "outer-output"

    f_outer("outer-input")
    tracker.flush_tracker()

    EXPECTED_TRACE_TREE = TraceModel(
        id=ANY_BUT_NONE,
        name="f_outer",
        input={"x": "outer-input"},
        output={"output": "outer-output"},
        start_time=ANY_BUT_NONE,
        end_time=ANY_BUT_NONE,
        spans=[
            SpanModel(
                id=ANY_BUT_NONE,
                name="f_outer",
                input={"x": "outer-input"},
                output={"output": "outer-output"},
                spans=[
                    SpanModel(
                        id=ANY_BUT_NONE,
                        name="f_inner",
                        input={"x": "inner-input"},
                        output={"output": "inner-output"},
                        spans=[],
                    )
                ],
            )
        ],
    )

    assert len(fake_backend.trace_trees) == 1
    assert_equal(EXPECTED_TRACE_TREE, fake_backend.trace_trees[0])

Using Verifiers (E2E Tests)

For actual API call tests:

from tests.e2e import verifiers

def test_trace_creation__e2e__happyflow(opik_client: opik.Opik):
    trace = opik_client.trace(
        name="test-trace",
        input={"query": "test"},
        output={"result": "success"}
    )

    verifiers.verify_trace(
        opik_client=opik_client,
        trace_id=trace.id,
        name="test-trace",
        input={"query": "test"},
        output={"result": "success"},
    )

Testlib Utilities

from tests.testlib import ANY_BUT_NONE, ANY_STRING, assert_equal

# ANY_BUT_NONE - matches any value that is not None
# ANY_STRING - matches any string
# assert_equal - deep comparison with ANY_* support

Parameterized Tests

@pytest.mark.parametrize(
    "text,expected_sentiment",
    [
        ("I love this product!", "positive"),
        ("This is terrible.", "negative"),
        ("The sky is blue.", "neutral"),
    ],
)
def test_sentiment_classification(text, expected_sentiment):
    metric = Sentiment()
    result = metric.score(text)
    assert expected_sentiment in result.reason

Key Rules

  • Always test public API only
  • Use fake_backend for integration tests
  • Use verifiers for E2E tests
  • Study existing similar tests before adding new ones

Running E2E Tests Locally

Pick the backend setup that matches what you're doing. Both need a few env vars the CI workflow sets implicitly.

Option A — CI-equivalent (recommended for full-suite runs)

# backend in docker, matches GitHub Actions:
TOGGLE_RUNNERS_ENABLED=true ./opik.sh --backend

# then run the suite:
cd sdks/python
OPIK_URL_OVERRIDE=http://localhost:5173/api/ \
  venv/bin/pytest tests/e2e/ \
    --ignore=tests/e2e/test_guardrails.py \
    -vv --durations=20

Option B — dev-runner (iterating on backend code)

The native Java backend inherits your shell env, so you must export MinIO credentials and the runners flag before starting it — otherwise attachment and runner tests fail on environmental grounds, not real regressions.

export AWS_ACCESS_KEY_ID=THAAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=LESlrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
export TOGGLE_RUNNERS_ENABLED=true
./scripts/dev-runner.sh --restart

cd sdks/python
OPIK_URL_OVERRIDE=http://localhost:8080/ \
  venv/bin/pytest tests/e2e/ \
    --ignore=tests/e2e/test_guardrails.py \
    -vv --durations=20

The AWS_* values are MinIO's root-user/root-password (from deployment/docker-compose/docker-compose.yaml). They are the canonical "EXAMPLE" strings — no real AWS account is involved; the Java backend's S3 client reuses the standard AWS env-var names when pointed at MinIO via S3_URL.

Why the gotchas

  • TOGGLE_RUNNERS_ENABLED defaults to false in docker-compose. Without it, 8 tests in tests/e2e/runner/ error out on setup because the runners feature is disabled in the backend.
  • --ignore=tests/e2e/test_guardrails.py — the guardrails Python service isn't part of the default compose stack. CI ignores these explicitly (python_sdk_e2e_tests.yml:90).
  • MinIO creds — only relevant in Option B (dev-runner). Option A's backend container already has them baked in.

Debugging a failing e2e test

# Capture SDK DEBUG logs to a file:
OPIK_FILE_LOGGING_LEVEL=DEBUG OPIK_LOGGING_FILE=/tmp/opik-sdk.log \
  venv/bin/pytest tests/e2e/test_tracing.py::test_name -vv

# And the backend-side error is in:
docker logs opik-backend-1   # Option A
tail -f /tmp/opik-opik-backend.log   # Option B (dev-runner)

Source: SKILL.md on GitHub

No rule matched.

skilld matched fixed text patterns in SKILL.md and file names. Patterns miss obfuscated code.

skilld run checks every file with the same patterns. It asks for approval before it loads a Skill with a behavior marked Needs approval.

No alerts8mo4 checks · Risk SAFE
  • Gen Agent Trust Hub8mo

    The skill provides comprehensive documentation and a refactor assistant for the Opik Python SDK by Comet ML. It is safe for its intended development use, though the refactoring assistant's file-editing capabilities introduce a surface for indirect prompt injection when processing untrusted code.

  • Socket8mo

    No alerts

  • Snyk8mo

    Risk: LOW · No issues

  • Runlayer8mo

    5 files scanned · No issues

Signed by skilld at 9b3d88e. This ties the file your Agent reads to that commit on GitHub. It does not review the instructions.

Last checked against GitHub yesterday.

Activeupdated 10 months ago

README badge

README badge for comet-ml/opik/python-sdk