Cloud Cost Optimization: From Runaway Bills to Unit Economics

Cloud cost is the engineering problem that compounds invisibly. When you're building, the AWS bill is someone else's problem — finance deals with it. When you're scaling, costs grow linearly with usage and nobody flags the slope. The bill triples when a batch job that ran once a week starts running hourly, when a microservices refactor doubles inter-AZ traffic, or when a new ML training pipeline launches 200 GPUs and never terminates them because the job succeeded but the cleanup script had a bug.

The typical trajectory: a startup reaches $50K/month and the CTO sees it for the first time. By $200K/month, they hire a platform engineer to "optimize infrastructure." By $500K/month, the board is asking about unit economics and cost-per-user. Netflix famously moved aggressively to spot instances and saved over $60M/year (Netflix Tech Blog, 2015: "Netflix's Approach to the Cloud") — but they started that work only after cost became a board-level concern.

Cloud cost optimization is not a one-time project. It is a continuous engineering discipline with three distinct phases:

1. Discovery: understand what you are spending and why (cost allocation, tagging)
2. Rightsizing: eliminate wasted capacity before buying commitments
3. Commitment purchasing: lock in discounts only after usage patterns stabilize

The biggest mistake engineers make is purchasing Reserved Instances before rightsizing. You lock in a discount on a waste pattern and then can't change it for a year.

AWS EC2 pricing is structured as a discount ladder from most expensive to cheapest. Understanding the mechanics — not just the names — is what lets you architect cost-efficiently.

On-demand: Full list price. No commitment. Appropriate only for truly unpredictable, short-lived workloads where you cannot forecast demand. For any stable workload, on-demand is paying a 60-100% premium over what you should be paying.

Savings Plans (preferred over RIs for most workloads): You commit to a minimum $/hour spend across any EC2 instance in any region. Compute Savings Plans give you 66% savings vs. on-demand; EC2 Instance Savings Plans give 72% savings but lock you to a specific instance family. The key advantage over RIs: if you rightsize from m5.2xlarge to m5.xlarge mid-year, your savings plan still applies. With an RI, you are committed to the m5.2xlarge until expiry.

Reserved Instances (RIs): Commit to a specific instance type, AZ, and region. Standard 1-year RI: ~40% savings. Standard 3-year RI: ~60-70% savings. Convertible RI: ~45% savings but allows changing instance type mid-term. Use RIs only for infrastructure that will not change — RDS databases, ElasticSearch clusters, core Kubernetes nodes.

Spot/preemptible instances: AWS Spot gives 70-90% savings but can reclaim with 2-minute notice. GCP preemptible instances give 60-80% savings with 30-second notice. Spot instances are not risky for the right workloads — they are the standard choice for ML training, batch ETL, CI/CD runners, and stateless web tier scaling. The non-obvious insight: spot interruption rates are usually 1-5% for most instance types in most AZs. For a job that runs 4 hours, the expected interruption probability is low — especially if you use Spot Fleet with diversification across multiple instance types and AZs.

Savings Plans + Spot is the production pattern: Use Savings Plans to cover your baseline 24/7 load at ~66% discount. Run all batch and training workloads on spot. Run scale-out traffic spikes on spot or on-demand (autoscaling handles the delta). Netflix runs ~70% of its compute on spot (Netflix Tech Blog, 2015); typical ML training clusters should target 70% spot + 30% on-demand for fault tolerance (AWS re:Invent 2022, "Cost Optimization at Scale").

AWS internal data and third-party studies consistently find that the average EC2 instance runs at 10-20% CPU utilization (Gartner, 2021: "Optimizing Cloud Infrastructure Costs"; AWS Compute Optimizer documentation). This means 80% of the compute you are paying for produces no value. Rightsizing is the single highest-ROI cost action — and it must happen before you purchase any commitments.

How to rightsize correctly:

Step 1: Gather real utilization data. Use AWS Compute Optimizer (free, analyzes 14 days of CloudWatch metrics) or your APM tool (Datadog, New Relic, Prometheus). Look at CPU, memory, network I/O, and disk I/O. The recommendation engine will flag instances where observed maximum usage is well below instance capacity.

Step 2: Use p50 for batch, p99 for online. This is the insight most engineers miss. For batch jobs — ML training, data pipelines, ETL — the job controls its own resource consumption and you can tune parallelism. Size the instance so that p50 job utilization fits comfortably. For online services, you need headroom for traffic spikes, so size for p99 with 20-30% buffer.

Step 3: Test before committing. Change one service at a time. Run the smaller instance for 2 weeks and monitor latency, error rates, and queue depth. A batch job that previously ran in 2 hours may run in 2.5 hours on a smaller instance — if that's acceptable, the savings outweigh the tradeoff.

Step 4: Memory is often the binding constraint, not CPU. Java services, ML inference servers, and in-memory databases frequently pin on memory while CPU is idle. A c5 instance (compute-optimized) costs 20% less than m5 but has no memory advantage. If your service is memory-bound, moving to r5 (memory-optimized) can let you run half as many instances.

Lyft's ~20% cost reduction through rightsizing took 6 weeks of analysis and 3 months of careful instance size changes across ~50 services (Lyft Engineering Blog, 2019: "Optimizing Cloud Costs at Lyft"). The key: they used actual observed utilization from Prometheus, not instance-launch-time estimates.

S3 storage costs seem small in isolation — $0.023/GB/month is less than 3 cents. But production systems accumulate data at scale: 100TB of logs costs $2,300/month. With proper lifecycle policies, the same 100TB costs ~$400/month. The engineering effort is a one-time Terraform config.

The S3 storage class hierarchy (AWS pricing, us-east-1 — aws.amazon.com/s3/pricing):

• S3 Standard: $0.023/GB/month. No retrieval fee. Use for frequently accessed data (last 30 days of logs, model serving artifacts, active feature stores).
• S3 Intelligent-Tiering: $0.023/GB/month (frequent tier) + $0.0025/1,000 objects/month monitoring fee. Automatically moves objects between Standard and Infrequent Access based on access patterns. Best for buckets where you cannot predict access patterns — e.g., ML experiment artifacts where some experiments get revisited and others never do.
• S3 Infrequent Access (S3-IA): $0.0125/GB/month storage + $0.01/GB retrieval. 46% cheaper than Standard for storage, but adds retrieval cost. Break-even: if you retrieve the data less than once per month, S3-IA is cheaper. Use for data accessed monthly or less: compliance archives, old feature engineering outputs, historical datasets.
• S3 Glacier Instant Retrieval: $0.004/GB/month + $0.03/GB retrieval. 83% cheaper than Standard. Retrieval in milliseconds. Use for data accessed a few times per year: model training checkpoints from completed projects, old experiment logs, regulatory archives.
• S3 Glacier Deep Archive: $0.00099/GB/month. 95% cheaper than Standard. 12-hour retrieval time. Use for cold data that is kept for compliance only and may never need to be accessed: audit logs beyond 1 year, raw data for lineage purposes.

The production lifecycle policy pattern:

0-30 days:   S3 Standard         ($0.023/GB)
30-90 days:  S3-IA               ($0.0125/GB, save 46%)
90-365 days: Glacier Instant      ($0.004/GB, save 83%)
365+ days:   Glacier Deep Archive ($0.001/GB, save 96%) or delete

For most log pipelines, deleting after 1 year is the right choice — storage of year-old logs rarely justifies the cost vs. the probability of access.

Egress costs are the most consistently underestimated line item in cloud bills. Engineers design architectures for latency and throughput; they don't consider that data movement has a cost. At 10TB/day of inter-AZ traffic between microservices — a moderate number for a mid-sized product — you are paying $30,000/month just for internal traffic that could be eliminated by co-location.

The egress cost hierarchy (AWS, approximate):

• Intra-AZ (same Availability Zone): Free. Data transfers between EC2 instances in the same AZ, or from EC2 to S3 in the same region via VPC endpoint, costs nothing.
• Inter-AZ (different AZ, same region): $0.01/GB each way ($0.02/GB round-trip). This is the most common surprise. A microservices architecture with 20 services making cross-AZ calls at 100MB/request at 1,000 RPS generates enormous inter-AZ charges that no one budgeted.
• Cross-region: $0.02-0.09/GB depending on regions. US to EU is $0.02/GB; US to South America is $0.09/GB. Multi-region architectures must account for this in their replication cost models.
• Internet egress (EC2/S3 to internet): $0.09/GB for first 10TB/month, $0.085/GB for 10-50TB, $0.07/GB for 50-150TB, $0.05/GB above 150TB. If you serve media from S3 directly, you pay $0.09/GB. CloudFront charges $0.0085/GB for high-volume transfers — a 10x reduction for CDN-eligible content.
• NAT Gateway: $0.045/GB processed. Outbound internet traffic from private subnets through NAT Gateway pays both the NAT processing fee and the EC2 internet egress fee. S3 and DynamoDB traffic from private subnets should go through VPC endpoints (free) to avoid NAT Gateway charges entirely.

The architectural fix: the non-obvious insight is that AZ affinity for services is a cost decision, not just a latency decision. If Service A calls Service B 10 times per request, and they are in different AZs, you pay $0.02/GB for that traffic. At scale, this becomes a dedicated cost-optimization project. Karpenter's topology-aware scheduling can automatically place pods that communicate frequently into the same AZ, eliminating the inter-AZ charge.

Kubernetes introduces a specific class of cost inefficiency: wasted reserved capacity from over-provisioned resource requests. When a pod requests 4 CPU but uses 0.5 CPU, the scheduler treats those 3.5 CPU as occupied. The node is at "full capacity" but physically underutilized. You pay for an extra node when existing nodes have plenty of available physical capacity.

The three autoscaling primitives:

HPA (Horizontal Pod Autoscaler): Scales the number of pod replicas based on observed metrics (CPU utilization, custom metrics via KEDA, Prometheus). HPA is the primary cost-control tool for online services — it scales down during off-peak hours. A web service that needs 20 pods at peak and 3 pods at night saves 85% of compute during the 14 hours it runs at low traffic.

VPA (Vertical Pod Autoscaler): Analyzes historical resource usage and updates pod resource requests to match actual consumption. VPA fixes the over-provisioning problem — it will lower a pod's CPU request from 4 CPU to 0.8 CPU based on observed usage. Warning: VPA requires pod restarts to apply changes and is incompatible with HPA on the same CPU metric. Production pattern: use VPA in recommendation-only mode to surface rightsizing opportunities, then apply them manually.

Karpenter (next-generation node provisioner): Karpenter replaces the Kubernetes Cluster Autoscaler. The key differences: Cluster Autoscaler can only add nodes from pre-configured node groups; Karpenter provisions any node type that fits pending pods. Karpenter intelligently chooses spot instances when workloads can tolerate interruption (controlled via pod annotations), right-sizes nodes to the pod requirements (no wasted capacity on oversized nodes), and consolidates workloads by evicting pods onto fewer nodes and terminating underutilized nodes. In production deployments, Karpenter typically reduces node count by 20-40% compared to Cluster Autoscaler through better bin packing.

The bin packing insight: bin packing efficiency depends entirely on accurate pod resource requests. If every pod requests 2x more than it uses, bin packing is impossible — the scheduler sees nodes as full when they have physical headroom. The tooling stack for accurate requests: VPA recommendation mode → Goldilocks (open-source VPA dashboard) → manual review → apply and monitor.

FinOps (Financial Operations) is the organizational discipline of making cloud costs visible, attributable, and actionable at the team level. The core insight: engineers don't optimize what they can't see. When the AWS bill is a single number reviewed by the CFO once a month, no individual team has a reason to care about cost. When each team sees their own weekly spend in their sprint review, cost optimization becomes part of engineering culture.

Cost allocation tags — the foundation: Every cloud resource must be tagged with at minimum: team, service, environment (prod/staging/dev), cost-center. Use AWS Organizations Service Control Policies (SCPs) to block resource creation without mandatory tags. Without tagging enforcement, you will have 30-40% of your bill as "untagged" with no attribution within 6 months. AWS Cost Explorer and tools like Infracost, CloudHealth, or Kubecost can then generate per-team spend reports from tag data.

Showback vs. Chargeback:

• Showback: teams receive visibility into their cloud spend but are not billed for it internally. Low friction, good starting point.
• Chargeback: teams are actually charged against their cost center or team budget for their cloud usage. Higher friction to implement (requires finance integration) but produces dramatically stronger cost optimization behavior. When an ML team's training runs come out of their headcount budget equivalent, they start thinking carefully about spot vs. on-demand and whether to keep 200 GPUs running overnight.

Unit economics — the maturity signal: Absolute spend is a vanity metric when the company is growing. A company with $500K/month AWS spend and 10M users is more cost-efficient than one with $100K/month AWS spend and 50K users. Track:

• Cost per request (total infra cost / request volume) — tracks infrastructure efficiency as load scales
• Cost per active user per month (total infra cost / MAU) — tracks product cost efficiency
• Cost per GB processed (for data pipelines) — tracks pipeline efficiency

Set budget alerts at 10-20% over the 30-day rolling baseline. Wire them to PagerDuty or Slack, not just email — an unexpected 40% spend spike at 2am from a runaway training job should wake someone up, not appear in Monday's report.