Skip to main content

Scaling

One stateless Deployment. Scaling is a replica count.

Autoscaling​

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 5
  targetCPUUtilizationPercentage: 75

CPU is the right signal here. Analysis is CPU-bound: scoring and the Monte Carlo simulation both burn cycles, and analyze_text runs in a threadpool so the event loop stays free.

warning

SESSION_SECRET must be set in the application secret before you run more than one replica. Without it, an SSO callback landing on a different pod than the one that started the flow will fail, and the symptom is an intermittent login loop that looks like an identity provider problem.

Resources​

resources:
  requests:
    cpu: 250m
    memory: 512Mi
  limits:
    cpu: "1"
    memory: 1Gi

Raise SIMULATION_SAMPLES and CPU per analysis rises with it. If you have tuned that up, raise the CPU limit too or analyses queue behind the throttle.

Probes are tuned for long requests​

The liveness probe is deliberately slack: 15 seconds by 6 failures, giving 90 seconds of headroom. An analysis holds a worker thread for seconds at a time, and a busy pod should never be restarted mid-request.

Startup allows roughly 150 seconds, covering CSV parse and database connect on a cold start.

Do not tighten these to match a house standard without measuring. The defaults exist because the workload is not a typical web service.

Timeouts upstream​

Scaling out does not help if the load balancer gives up first.

  • nginx: proxy-read-timeout: "600" and proxy-send-timeout: "600"
  • GKE: BackendConfig with timeoutSec: 600. The GCLB default of 30 seconds is shorter than a long analysis, and past it the load balancer returns an HTML 502 the app never sees
  • ALB: idle_timeout.timeout_seconds=600

Body size matters too. Transcripts and audio go well past the nginx 1MB default, which returns its own HTML 413.

The database​

Each replica holds a connection pool. Beyond roughly five replicas against a default max_connections of 100, either raise it or add PgBouncer in transaction pooling mode.

Rendering latency​

If OPENAI_API_KEY is set, each turn includes a network call to OpenAI. That latency is outside your control and outside your cluster. If tail latency matters more than prose, turning rendering off removes it entirely. See Rendering.

Last updated on