Chapter 6: Auto-scaling and Zero-scaling

การออกแบบระบบให้สามารถปรับเพิ่มจำนวน container (Scale-out) ได้อัตโนมัติตามปริมาณ Request

Auto-scaling Strategies

Request-based Scaling

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: llm-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llm-serving
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: External
    external:
      metric:
        name: requests_per_second
      target:
        type: Value
        value: 100

GPU Utilization Scaling

metrics:
- type: Resource
  resource:
    name: nvidia.com/gpu
    target:
      type: Utilization
      averageUtilization: 70

Zero-scaling (Scale-to-Zero)

ลดขนาดจนถึงศูนย์เมื่อไม่มีการใช้งาน:

• Cost saving - No idle GPU costs
• Cold start penalty - First request latency
• Best for: Sporadic traffic patterns

Implementation Pattern

Traffic Spike -> KPA (Knative Pod Autoscaler)
               -> Scale from 0 to N pods
               -> Route traffic
               -> Monitor metrics
               -> Scale down when idle