/typescript-e2e-testing

Kafka E2E Testing Knowledge

Overview

E2E testing with Kafka requires a fundamentally different approach than other infrastructure. Common patterns like fromBeginning: true fail to provide test isolation.

Why Kafka Testing is Different

Common Approaches That FAIL

Key Insight: auto.offset.reset: 'earliest' or fromBeginning: true does NOT provide test isolation because:

1. Reads ALL historical messages including from previous test runs
2. Multiple tests share the same topic - messages accumulate
3. Consumer group offset tracking causes inconsistent behavior

The Proven Solution: Pre-Subscription + Buffer Clearing

┌─────────────────────────────────────────────────────────────────────────┐
│                        E2E Test Architecture                             │
├─────────────────────────────────────────────────────────────────────────┤
│  ┌──────────────┐         ┌──────────────┐         ┌──────────────┐    │
│  │   Test       │         │   NestJS     │         │    Output    │    │
│  │   Helper     │         │   App        │         │    Topic     │    │
│  │  (Producer)  │         │  (Consumer)  │         │   Listener   │    │
│  └──────┬───────┘         └──────┬───────┘         └──────┬───────┘    │
│         │                        │                        │             │
│         │ 1. Publish             │ 2. Consume            │ 3. Produce  │
│         ▼                        ▼                        ▼             │
│  ┌─────────────────────────────────────────────────────────────────┐   │
│  │                         KAFKA BROKER                             │   │
│  │  ┌─────────────┐                          ┌─────────────┐       │   │
│  │  │ Input Topic │  ───────────────────▶    │ Output Topic│       │   │
│  │  └─────────────┘                          └─────────────┘       │   │
│  └─────────────────────────────────────────────────────────────────┘   │
│                                            │                            │
│                                            │ 4. Collect in Buffer      │
│                                            ▼                            │
│  ┌──────────────────────────────────────────────────────────────────┐  │
│  │                    MESSAGE BUFFER (In-Memory)                     │  │
│  │    [msg1, msg2, ...]  ◀── Clear between tests                     │  │
│  └──────────────────────────────────────────────────────────────────┘  │
│                                            │                            │
│                                            │ 5. Poll & Assert           │
│                                            ▼                            │
│  ┌──────────────────────────────────────────────────────────────────┐  │
│  │                      TEST ASSERTIONS                              │  │
│  └──────────────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────────┘

How It Works

1. Test helper subscribes BEFORE the application starts
2. Messages accumulate in memory buffers per topic
3. Tests poll for expected messages with timeout
4. Buffers are cleared between tests (NOT topics)

Kafka-Specific Timing

Admin API for Test Synchronization

CRITICAL: Use Admin API to determine when tests are ready to proceed, instead of blind waits.

Why Admin API?

Key Admin API Methods for E2E Tests

Consumer Group States

PreparingRebalance (1) → CompletingRebalance (2) → Stable (3)
          ↑                                            │
          └──────────── (new subscription) ────────────┘

Note: @confluentinc/kafka-javascript (node-rdkafka) uses numeric states, KafkaJS uses strings.

Usage Pattern

// After starting microservices or subscribing to topics
const groupId = client.getConsumerGroupId();
await kafkaHelper.waitForConsumerGroupStable(groupId, 15000);
// NOW safe to publish test messages

Broker Options for E2E Tests

Test Category Organization

test/
├── e2e/
│   ├── 01-happy-path.e2e-spec.ts      # Basic message flow
│   ├── 02-error-handling.e2e-spec.ts  # Malformed messages
│   ├── 03-performance.e2e-spec.ts     # Throughput, latency
│   └── 04-data-integrity.e2e-spec.ts  # Zero data loss
├── fixtures/events/
│   ├── valid-events.ts
│   └── malformed-events.ts
└── helpers/
    ├── kafka-helper.ts                # KafkaTestHelper
    └── assertion-helpers.ts

Cross-File Test Isolation

The librdkafka State Corruption Problem

When running multiple E2E test files in a single Jest session, tests that connect to invalid brokers can corrupt librdkafka's internal state. This corruption:

1. Persists within the Jest process - Cannot be cleaned up by broker health checks
2. Affects subsequent test files - Tests that passed individually now fail
3. Manifests as timeouts - Connection-related failures in previously working tests

Symptoms

Solution: Custom Jest Test Sequencer

Control test file execution order to ensure tests with invalid broker connections run LAST:

// test/e2e/test-sequencer.js
import Sequencer from '@jest/test-sequencer';

const TEST_ORDER = [
  // Clean broker operations first
  'standard-client.e2e-spec.ts',
  // Complex integrations middle
  'request-response.e2e-spec.ts',
  // Invalid broker tests LAST
  'error-handling.e2e-spec.ts',
];

export default class CustomSequencer extends Sequencer.default {
  sort(tests) {
    const orderMap = new Map(TEST_ORDER.map((name, index) => [name, index]));
    return [...tests].sort((a, b) => {
      const aOrder = orderMap.get(a.path.split('/').pop()) ?? 999;
      const bOrder = orderMap.get(b.path.split('/').pop()) ?? 999;
      return aOrder - bOrder;
    });
  }
}

See isolation.md for detailed implementation.

Broker Health Checks

When to Use Broker Health Checks

Key Methods

// Quick health check (lightweight)
const healthy = await kafkaHelper.isBrokerHealthy(5000);

// Wait for broker with polling
await kafkaHelper.waitForBrokerReady(30000);

// Comprehensive health verification
await kafkaHelper.ensureBrokerHealthy(60000);

See test-helper.md for implementation details.

Kafka State Accumulation

The Problem

Even with proper cleanup, Kafka state accumulates across test runs within the same Jest process:

Impact on Tests

Solution: Kafka Container Restart

For complex tests (request-response, high-concurrency), restart the Kafka Docker container in beforeAll:

async function restartKafkaContainer(): Promise<void> {
  await execAsync('docker restart kafka-e2e');
  await new Promise(r => setTimeout(r, 20000)); // Wait for full initialization
}

beforeAll(async () => {
  await restartKafkaContainer();
  // ... rest of setup
}, 120000);

See isolation.md for detailed implementation and when to use this pattern.

Key Files in This Section