/beacon

Platform Engineering Observability

IDP observability architecture, Backstage SLO integration, Service Catalog, Golden Path design, Developer Experience metrics, Platform Maturity Model

1. IDP Observability Architecture (Paved Road Pattern)

An Internal Developer Platform (IDP) should provide "paved roads" — standardized, opinionated observability paths that developers follow by default.

Three-Layer Architecture

Layer 3: PLATFORM PORTAL (Backstage / custom IDP UI)
  ├── Service health scores (SLO scorecards)
  ├── Ownership mapping (who owns what)
  ├── Tech radar (deprecation signals)
  └── Self-service SLO creation wizard

Layer 2: OBSERVABILITY PLATFORM
  ├── OTel Collector fleet (standardized pipelines)
  ├── Prometheus + Thanos / Grafana Mimir (metrics)
  ├── Grafana Weave / Jaeger (traces)
  ├── Loki / OpenSearch (logs)
  └── Alertmanager → PagerDuty / Opsgenie

Layer 1: GOLDEN PATH INSTRUMENTATION
  ├── OTel SDK auto-instrumentation (via init containers or sidecars)
  ├── Standard Helm chart with pre-wired telemetry
  └── Service template with instrumentation included

Paved Road Principles

2. Backstage + SLO Integration

ServiceEntity SLO Scorecard

Extend the Backstage catalog-info.yaml to embed SLO metadata:

# catalog-info.yaml
apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: payment-service
  annotations:
    # SLO scorecard annotation
    beacon.io/slo-scorecard: |
      slos:
        - name: availability
          target: 0.999
          current: 0.9994
          status: healthy
          error_budget_remaining: 0.62
        - name: latency-p95
          target: 0.95
          threshold_ms: 200
          current: 0.972
          status: healthy
          error_budget_remaining: 0.44
    # Link to Grafana SLO dashboard
    grafana/dashboard-url: "https://grafana.internal/d/slo-payment-service"
spec:
  type: service
  lifecycle: production
  owner: team-payments

Backstage SLO Plugin Integration

// SLO data provider for Backstage plugin
export const sloDataProvider: SLODataProvider = {
  async getSLOStatus(entityRef: string): Promise<SLOStatus[]> {
    const response = await fetch(
      `${config.backendUrl}/api/slos/${entityRef}`
    );
    return response.json();
  },
};

3. Service Catalog → SLO Auto-generation

Metadata → SLO Definition Mapping

When a new service is registered in the Service Catalog, SLOs can be auto-generated based on metadata:

# SLO generation rules (stored in platform config)
slo_templates:
  - condition:
      type: service
      lifecycle: production
      tier: critical
    generate:
      - name: availability
        sli_type: availability
        target: 0.999
        window: 30d
      - name: latency-p95
        sli_type: latency
        target: 0.95
        threshold_ms: 200
        window: 30d

  - condition:
      type: service
      lifecycle: production
      tier: standard
    generate:
      - name: availability
        sli_type: availability
        target: 0.995
        window: 30d

GitOps SLO Generation Workflow

Service registered in Catalog
         ↓
Catalog webhook triggers SLO generator
         ↓
Generator reads service tier from metadata
         ↓
Renders SLO YAML from template
         ↓
Opens PR to infrastructure/slos/ repo
         ↓
PR review → merge → ArgoCD applies SLO CRDs
         ↓
Grafana SLO dashboard auto-provisioned

4. Golden Path Design

A Golden Path provides a complete, pre-configured observability setup for new services.

Golden Path Components

Golden Path OTel Init Container (Kubernetes)

# Helm chart snippet — auto-inject OTel agent
initContainers:
  - name: otel-agent-init
    image: ghcr.io/platform/otel-agent-init:latest
    env:
      - name: OTEL_SERVICE_NAME
        valueFrom:
          fieldRef:
            fieldPath: metadata.labels['app.kubernetes.io/name']
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: "http://otel-collector.observability.svc:4317"
    volumeMounts:
      - name: otel-agent
        mountPath: /otel-agent

Standard SLO Configuration Template

# templates/slo.yaml — included in Golden Path
apiVersion: sloth.slok.dev/v1
kind: PrometheusServiceLevel
metadata:
  name: {{ .Values.serviceName }}-slos
  namespace: {{ .Release.Namespace }}
spec:
  service: {{ .Values.serviceName }}
  slos:
    - name: availability
      objective: {{ .Values.slo.availability | default 99.5 }}
      description: "{{ .Values.serviceName }} availability SLO"
      sli:
        events:
          error_query: |
            sum(rate(http_requests_total{
              job="{{ .Values.serviceName }}",
              status=~"5.."
            }[{{"{{"}}window{{"}}"}}]))
          total_query: |
            sum(rate(http_requests_total{
              job="{{ .Values.serviceName }}"
            }[{{"{{"}}window{{"}}"}}]))
      alerting:
        name: {{ .Values.serviceName | title }}AvailabilitySLO
        page_alert:
          labels:
            severity: critical
        ticket_alert:
          labels:
            severity: warning

5. Developer Experience Metrics

Track the health of the platform itself through DX (Developer Experience) metrics.

Core DX Metrics

DORA Metrics Integration

The platform should surface DORA metrics per team/service:

# Deployment frequency (deploys/day per team)
increase(deployments_total[1d])

# Change failure rate
sum(rate(deployments_total{status="failed"}[7d]))
/ sum(rate(deployments_total[7d]))

# Mean time to restore (MTTR)
avg_over_time(incident_duration_seconds[30d])

6. Platform Maturity Model (L1–L4)

Assess and plan platform observability evolution across four maturity levels.

Maturity Assessment Checklist

L1 → L2 (Structured)

• SLOs defined for all production services (availability + latency minimum)
• Error budgets calculated and visible
• Standard dashboard template adopted by > 50% of teams
• Alert runbooks exist for all P1 alerts
• On-call rotation established with defined escalation path

L2 → L3 (Proactive)

• Golden Path adopted by > 80% of new services
• SLO auto-generation from Service Catalog live
• Error budget policy drives sprint prioritization
• DX metrics tracked (TTD, onboarding time, adoption rate)
• Platform team has its own SLOs

L3 → L4 (Adaptive)

• Anomaly detection on key SLIs (ML-based or statistical)
• Predictive capacity models running in production
• Auto-remediation runbooks executing for known failure patterns (→ Mend)
• DX feedback loop: survey → roadmap → measure improvement
• Observability cost optimization automated (adaptive sampling)

7. IDP SLOs (Platform Eating Its Own Cooking)

The platform team must define and maintain SLOs for the IDP itself.

Source: Backstage Annotations · DORA Research · Team Topologies — Platform Teams · Humanitec: IDP Maturity Model