Scaling Decision Framework
Overview
Decision matrix for choosing the right scaling action based on observed bottleneck. For initial KPU sizing methodology and CloudWatch trend signals, see resource-optimization.md. This guide is for in-flight scaling decisions on running applications.
Decision Matrix
What is the bottleneck?
│
├─► CPU > 70% sustained or backpressure > 500ms/s
│ → INCREASE Parallelism (adds KPUs, restart required)
│
├─► OOM errors or heapMemoryUtilization > 85%
│ → DECREASE ParallelismPerKPU (more memory per subtask)
│ → Same total parallelism, more KPUs, lower throughput-per-dollar
│
├─► Variable/spiky load + can tolerate restart-on-scale
│ → ENABLE AutoScalingEnabled=true
│ → MSF reacts on CPU only; 5-15 min reaction; restart per event
│
└─► CPU < 30% sustained, idle > 500ms/s, no backpressure
→ DECREASE Parallelism (over-provisioned)
→ Validate with 60+ minute trend before reducingScaling Impact Table
| Change | KPU Effect | Memory Per Subtask | Cost Impact |
|---|---|---|---|
| Parallelism 4 → 8, PPK=1 | 4 → 8 KPU | 4 GB (unchanged) | 2× |
| Parallelism 4 → 8, PPK=2 | 2 → 4 KPU | 2 GB (unchanged) | 2× |
| PPK 2 → 1, Parallelism=8 | 4 → 8 KPU | 2 → 4 GB | 2× |
| PPK 1 → 2, Parallelism=8 | 8 → 4 KPU | 4 → 2 GB | 0.5× |
| Parallelism 8 → 4, PPK=1 | 8 → 4 KPU | 4 GB (unchanged) | 0.5× |
KPU formula: KPU = Parallelism / ParallelismPerKPU. Add 1 orchestration KPU for total billed.
Pre-Scaling Validation
Always validate the bottleneck against trends, not point-in-time values, before scaling. Pull the last 6 hours of:
containerCPUUtilization— Maximum (hottest container)backPressuredTimeMsPerSecond— Average and MaximumidleTimeMsPerSecond— Average and MinimumheapMemoryUtilization— MaximumlastCheckpointDuration— trend (rising = pressure)
If signals conflict (high CPU but also high idle, or backpressure with low CPU), the bottleneck is downstream — investigate the operator graph via Flink Dashboard before scaling. Adding KPUs to a sink-bound or skew-bound job wastes money. See first-fault-isolation.md for live diagnosis via the Dashboard.
AutoScalingEnabled Behavior
MSF built-in autoscaling is CPU-only with fixed thresholds:
- Scale up:
containerCPUUtilization > 75%for 15 consecutive 1-min datapoints → doublesCurrentParallelism - Scale down:
containerCPUUtilization < 10%for 360 consecutive 1-min datapoints (6h) → halvesCurrentParallelism, never below configuredParallelism - Each event triggers a full restart (10–30s downtime). Backpressure, lag, and memory pressure do not trigger autoscaling.
- During scaling: status is
AUTOSCALING. Onlystop-application --forceis valid.
For backpressure-driven, lag-driven, or memory-driven autoscaling: disable AutoScalingEnabled and implement custom scaling via CloudWatch alarms → Lambda → update-application.
Stateful vs Stateless ParallelismPerKPU
| Workload | ParallelismPerKPU | Why |
|---|---|---|
| Stateful (keyed windows, joins, large RocksDB state) | 1 | Each subtask needs full 4 GB; sharing causes RocksDB contention and OOM |
| Stateless transforms (map, filter, simple routing) | 2 | Half memory per subtask is fine; doubles compute density per KPU |
| I/O-blocking (async lookups, slow sinks) | 2–4 | Subtasks spend most time blocked; pack more per KPU to fill CPU |
| Source operators matched to shards/partitions | Match to source count | Use setParallelism() on the source only |
Scaling Guardrails
- ❌ Cannot scale during a transitional state (STARTING, UPDATING, STOPPING, AUTOSCALING)
- ❌ Cannot scale during another in-flight update — sequence operations
- ⚠️ Every scaling change triggers a restart with 10–30s downtime (varies with state size and snapshot recency)
- ⚠️ Take a snapshot before scaling for rollback safety
- ⚠️ Scaling beyond 64 KPU requires a Service Quotas increase (default limit)
- ⚠️ Setting Parallelism > source partition/shard count creates idle subtasks — wasted KPUs
Anti-Patterns
| Anti-Pattern | Why It Fails |
|---|---|
| Scaling up to fix checkpoint failures | Checkpoint failures are usually serialization, S3 perms, or alignment timeout — not capacity |
| Scaling up to fix data skew | Hot keys still go to the same subtask. Fix with rebalance() or pre-splitting keys |
| Scaling up to fix slow sink | Sink is the bottleneck regardless of KPU count. Optimize sink (batching, async I/O) or switch sink |
| Scaling during active restart loop | Restart loops persist until root cause is fixed; new KPUs join the loop |
| Reducing PPK and Parallelism in separate updates | Each triggers a restart; two sequential updates = double downtime. Combine in a single update-application call |
| Scaling on a single high CPU spike | Spikes from checkpointing or GC are normal; require sustained 30+ minute trend |
References
- MSF Auto-Scaling Behavior
- resource-optimization.md — KPU sizing methodology and trend-based signals
- first-fault-isolation.md — diagnose before scaling