How to Deploy Llama 3 Self-Hosted on AWS: Full Cost Breakdown (2026)

Meta released the Llama 3.1 family in July 2024 in three sizes: 8B, 70B, and 405B parameters. Each targets a different cost-quality tier, and each has hard GPU VRAM requirements that determine which EC2 instances can serve it. The official model card on HuggingFace documents the context window (128k tokens for all three variants) and supported quantization formats.

The Llama 3.1 8B model in full FP16 precision requires approximately 16 GB of VRAM, which means a single A10G (24 GB on g5.xlarge) can serve it with headroom for KV cache. In practice most teams run 8B in Q4_K_M quantization (about 4.5 GB), which allows 4–5 concurrent request contexts at once on a single A10G and costs only $1.006/hr on-demand — making it the cheapest production-grade open inference setup on AWS.

The Llama 3.1 70B model requires approximately 140 GB in BF16, which means you need either 4× A10G (96 GB, only feasible with Q4 quantization at ~35–38 GB) or 2× A100 80GB for FP16. The practical sweet spot is the g5.12xlarge at $5.672/hr running 70B in Q4_K_M — good quality, usable throughput (~80–120 tokens/second at batch 8), and roughly 5× cheaper than a p4d.24xlarge.

The Llama 3.1 405B model is in a class of its own. Full BF16 requires ~810 GB of VRAM — 8× A100 80GB (640 GB) is not enough for FP16, so you need either Q4 quantization on a p4d.24xlarge or a full p5.48xlarge (8× H100 SXM, 640 GB) running BF16 with tensor parallelism. The p5.48xlarge runs $98.32/hr on-demand. At that price, the break-even against GPT-5 Pro API calls is genuinely high — only teams processing millions of tokens per hour reach it.

Quantization compresses model weights from 16-bit floats to 4-bit or 8-bit integers, reducing VRAM requirements roughly 2–4× at the cost of some quality. For Llama 3.1 specifically, the community benchmarks on HuggingFace Open LLM Leaderboard show that Q4_K_M retains approximately 99% of BF16 performance on reasoning and coding tasks for the 70B model, making it the default recommendation for cost-sensitive deployments.

Practical VRAM requirements after quantization: Llama 3.1 8B in Q4_K_M ≈ 4.5 GB; 8B in Q8 ≈ 8.5 GB; 70B in Q4_K_M ≈ 38 GB; 70B in Q5_K_M ≈ 48 GB; 70B in Q8 ≈ 70 GB; 405B in Q4_K_M ≈ 220 GB; 405B in Q8 ≈ 430 GB. These figures come from GGUF format measurements — GPTQ and AWQ formats produce similar VRAM footprints but differ in inference framework compatibility.

For AWS specifically, Q4_K_M on the 70B model opens up the g5.12xlarge (4× A10G, 96 GB total) at $5.672/hr — a configuration that fits comfortably with room for a 32k-token KV cache. Moving to Q5_K_M on that same instance is tight (48 GB model + KV cache overhead approaches 96 GB limit) and requires reducing max context length or batch size. For production use with predictable latency, the g5.12xlarge running 70B Q4_K_M is the most common cost-optimized setup.

AWS Inferentia2 (inf2 instances) is a purpose-built AI inference chip that is consistently 40–70% cheaper per token than equivalent GPU instances for compiled inference workloads. The catch: you need to compile your model to the Neuron SDK format using AWS Neuron, which adds a one-time setup cost of roughly 2–4 hours and some operational complexity. The AWS Neuron documentation covers the compilation workflow for HuggingFace Transformers models.

The inf2.xlarge at $0.758/hr runs Llama 3.1 8B compiled at roughly 30–50 tokens/second — cheaper than a g5.xlarge ($1.006/hr) for equivalent throughput. The inf2.48xlarge at $12.98/hr runs Llama 3.1 70B at 100–160 tokens/second, compared to a g5.12xlarge at $5.672/hr for 80–120 tokens/second. The inf2.48xlarge is more expensive per-hour but delivers better throughput, so the cost-per-token at full utilization is comparable.

The Inferentia2 path makes most sense when you have a stable, high-volume workload where the one-time compilation cost is amortized over millions of requests. For experimental or low-volume use, g5 instances are more flexible because they run vLLM or llama.cpp without any compilation step, and you can switch models in minutes. For the 70B model in production at scale, benchmark both paths and pick based on your actual request patterns.

To calculate break-even, you need to compare the fixed cost of running an instance 24/7 against what you would pay per-token on the leading APIs. As of June 2026: GPT-5 standard is priced at approximately $2.50/1M input tokens and $10.00/1M output tokens. Claude Opus 4.x (Anthropic) is approximately $15.00/1M input and $75.00/1M output. Gemini 2.5 Pro (Google) is approximately $1.25/1M input and $10.00/1M output for prompts under 200k tokens. These figures are sourced from each provider's live pricing page; see our full model cost comparison for the complete table.

For Llama 3.1 70B on a g5.12xlarge at $5.672/hr running 24/7: monthly fixed cost = $5.672 × 24 × 30 = approximately $4,084/month. At 100 tokens/second average throughput and a 1:2 input-to-output token ratio, you generate roughly 8.64 billion tokens per month at full utilization. Compared to GPT-5 standard (blended ~$4.17/1M tokens at 1:2 ratio): 8.64B tokens × $4.17/1M = $36,046/month in API fees. That is a 88% cost reduction at full utilization. The break-even point — where API fees equal the instance cost — is approximately 980M tokens/month, or about 378 tokens/second sustained throughput.

For the 8B model on a g5.xlarge at $1.006/hr: monthly cost = $724. At 50 tokens/second throughput (utilization 100%), you generate ~129 billion tokens/month — but realistically utilization is 15–40%, so effective monthly token output is 19–52 billion tokens. The break-even against GPT-5 nano (approximately $0.10/1M input, $0.40/1M output, blended ~$0.167/1M) is very high — you would need ~4.3 billion tokens/month just to break even, which requires 57% sustained utilization. The 8B self-hosted path only wins against API at very high volume or against premium model tiers.

The comparison against Claude Opus 4.x is more favorable for self-hosting: at $15/1M input and $75/1M output (blended ~$30/1M at 1:2 ratio), a g5.12xlarge running 70B breaks even at just ~136M tokens/month — roughly 52 tokens/second sustained. Any team currently spending >$2,000/month on Claude Opus 4.x should run this math seriously. The quality gap between Llama 3.1 70B and Claude Opus 4.x is real (Opus 4.x leads on complex reasoning and writing), but for structured extraction, classification, and retrieval-augmented generation tasks, 70B Q4 is a genuine Opus alternative at 95% cost reduction.

The inference server is as important as the instance type. The three dominant open-source options in 2026 are vLLM, llama.cpp (with its server mode), and HuggingFace Text Generation Inference (TGI). Each has different strengths for the AWS deployment context.

vLLM is the production standard for high-throughput GPU inference. It implements PagedAttention — continuous batching that fills GPU memory more efficiently than static batching — and achieves the highest tokens/second numbers in most benchmarks. For g5 and p4d/p5 instances, vLLM is the default choice. The vLLM documentation covers multi-GPU tensor parallelism setup for the 70B and 405B models.

llama.cpp excels at memory-constrained setups and runs on CPU (slowly) or CUDA with quantized models. Its GGUF format supports the widest range of quantization levels and runs reliably on a g5.xlarge for 8B inference. Throughput is 20–40% lower than vLLM for the same hardware, but operational simplicity is significantly higher — a single binary, no Python dependency chain, and compatible with the llama-server HTTP API that mimics OpenAI's endpoint format.

TGI (Text Generation Inference) from HuggingFace sits between the two: better tooling for model loading from HuggingFace Hub, native support for the Messages API format, and built-in OpenTelemetry instrumentation. For teams already invested in the HuggingFace ecosystem, TGI reduces setup friction. For pure throughput at scale, vLLM typically wins by 15–30% on GPU hardware. The choice between them rarely changes the break-even math meaningfully — focus on the instance type and model quantization level first.

The break-even math above used on-demand pricing, which is the worst-case scenario. AWS Reserved Instances (1-year, no upfront) typically reduce GPU instance costs by 30–40%. A g5.12xlarge reserved 1-year no-upfront drops from $5.672/hr to approximately $3.60/hr — reducing the monthly fixed cost from $4,084 to $2,592. That drops the break-even token volume from 980M to 620M tokens/month.

Spot Instances reduce costs further — typically 60–75% off on-demand — but GPU instances like g5 and p4d have variable spot availability and can be interrupted with 2 minutes notice. For inference workloads, spot interruptions are disruptive but manageable if you architect for it: use a load balancer across multiple spot instances, keep request queues in SQS, and have on-demand fallback for critical requests. Teams running pure batch inference (content generation, nightly processing) are good candidates for spot. Real-time user-facing inference should use Reserved or on-demand with spot as overflow.

AWS also offers Compute Savings Plans (more flexible than Reserved Instances — they apply to any EC2 instance family) and EC2 Instance Savings Plans (specific to a family like g5). For stable, predictable workloads, a 1-year EC2 Instance Savings Plan on g5 delivers the best discount rate. Run the AWS Savings Plans calculator with your target instance to model the exact savings before committing.

EC2 compute is the largest line item, but it is not the only one. Llama 3.1 70B in Q4_K_M format is approximately 38 GB, and the 405B in Q4 is approximately 220 GB. Storing model weights on EBS gp3 volumes costs $0.08/GB/month — $3/month for the 8B model, $38/month for the 70B, $220/month for the 405B. These are small relative to compute but worth accounting for.

Data transfer costs are relevant if your inference server is called from outside AWS or across regions. Within the same AWS region, data transfer between EC2 instances is free. Outbound to the internet is $0.09/GB for the first 10 TB/month. For typical inference workloads with short outputs (500–2000 tokens per request), outbound data volume is modest — a million 1000-token responses is roughly 4 GB, or $0.36 in egress fees. This becomes material only at very high request volumes.

Hidden operational costs are the ones most teams underestimate. Running your own inference stack requires someone to monitor GPU utilization, handle model loading errors, manage CUDA driver updates, and rotate instances when spot capacity is reclaimed. Conservatively, this is 2–5 hours/week of DevOps time at whatever your engineering hourly cost is. At $100/hr fully-loaded, that is $800–2,000/month of implicit cost that does not appear on the AWS bill. Factor this into your break-even calculation honestly. If you are a solo founder or small team, this hidden cost often reverses the case for self-hosting below $3k/month API spend.

Llama 3.1 405B is competitive with GPT-5 standard on several benchmarks (Meta's technical report reports MMLU of 88.6% and HumanEval of 89%), making it the only open model that genuinely competes with frontier closed models on reasoning-heavy tasks. But the infrastructure cost is severe: the minimum viable setup is a p4d.24xlarge ($32.77/hr) running Q4 quantization, or a p5.48xlarge ($98.32/hr) for full BF16.

At $32.77/hr on-demand, the p4d.24xlarge running 405B Q4 costs $23,594/month to run 24/7. That is only cheaper than the Claude Opus 4.x API if you are processing more than approximately 787M tokens/month at the Opus blended rate (~$30/1M). For comparison, a Claude Opus 4.x heavy user at $20,000/month in API fees is processing roughly 667M tokens/month — just below the break-even. At $25,000+/month in Opus spend, self-hosted 405B starts winning on pure cost.

The more realistic 405B use case is not cost arbitrage but capability availability: teams that need frontier-level reasoning but cannot send data to external APIs due to compliance constraints (HIPAA, SOC 2 Type II, GDPR data residency). A p4d.24xlarge fully inside your VPC, with no data leaving AWS, solves that problem in a way no API can. For data-sovereignty requirements, the 405B infrastructure cost is often justified on compliance grounds alone, not on raw cost-per-token. See our guide to AI compliance requirements in 2026 for the full compliance stack discussion.

First, request a g5.12xlarge quota increase in your AWS account if you have not already — new accounts default to 0 vCPU quota for GPU instance families. Submit the increase request via Service Quotas → EC2 → Running On-Demand G instances. Approval typically takes 1–24 hours. Use Deep Learning AMI (Amazon Linux 2, CUDA 12.x) as the base image — it ships with NVIDIA drivers and the CUDA toolkit pre-installed, saving 30–60 minutes of setup.

Install vLLM: `pip install vllm==0.5.4`. Download the model from HuggingFace — you will need a HuggingFace token and Meta's gated model access (request at Meta's form). Then launch: `python -m vllm.entrypoints.openai.api_server --model meta-llama/Meta-Llama-3.1-70B-Instruct --quantization awq --tensor-parallel-size 4 --max-model-len 32768 --port 8000`. The `--tensor-parallel-size 4` flag distributes the model across all 4 A10G GPUs; `--quantization awq` uses AWQ 4-bit quantization.

Once the server starts (model loading takes 3–8 minutes on first load), the endpoint is OpenAI-compatible at `http://localhost:8000/v1`. Any code using the OpenAI Python SDK works by changing `base_url` to your instance's internal IP and `model` to `meta-llama/Meta-Llama-3.1-70B-Instruct`. For production, put an Application Load Balancer in front, terminate TLS at the ALB, and use an EC2 Auto Scaling Group for fault tolerance. The full production setup is beyond this guide's scope but AWS documentation on ALB with EC2 covers the networking configuration.

The break-even numbers above assume 100% utilization, which you will not achieve in practice. Real production inference workloads typically run at 20–60% utilization averaged across the day — lower at night, spiky during business hours. Effective monthly token output at 30% average utilization is 30% of the 24/7 maximum, which raises the effective cost-per-token by 3.3× compared to full utilization. This is why the self-host math often looks great on a whiteboard and then disappoints in production: you pay for the instance whether it is busy or not, but API costs scale with actual usage.

The strongest case for self-hosting is high, stable, predictable volume — nightly batch processing, continuous document extraction pipelines, or real-time applications with consistent load. The weakest case is spiky, low-volume, or experimental workloads where you might run 10 requests one hour and 10,000 the next. For spiky workloads, API wins on cost; self-hosting wins only if you can keep GPU utilization above ~50% consistently. For a fuller treatment of this tradeoff, see our post on self-host vs API cost breakeven analysis.

Our recommendation for 2026: start with APIs until your monthly spend on a single model exceeds $3,000–5,000/month for a stable workload. Below that, the DevOps overhead and utilization mismatch make self-hosting net-negative on cost. Above $5k/month on Claude Opus 4.x or above $15k/month on GPT-5 standard, run the utilization-adjusted break-even with your actual request patterns. If you cross break-even, start with a single g5.12xlarge running 70B Q4 — it is the best cost-per-capability starting point in the Llama 3.1 lineup. Scale horizontally (more instances behind a load balancer) before moving to more expensive instance types. And before making any model selection decision, use our AI Prompt Cost Calculator to model the exact API vs self-host cost gap for your token volume.

One more consideration: Llama 3.1 is not the only open model worth evaluating. The open-model landscape in 2026 includes Llama 4, Mistral Large, and several fine-tuned variants that may outperform base Llama 3.1 on specific tasks. The instance types and cost math in this guide apply to any model that fits the same VRAM constraints — the framework transfers directly. For a broader comparison of open vs closed models for production workloads, see our choose LLM for production guide and the LLM cost engineering deep dive.

Self-hosted inference has no per-token billing, but it has idle-time billing — you pay the full instance cost even when the GPU is at 0% utilization. The most important cost control mechanism is automatic scale-to-zero or schedule-based instance stopping for non-production environments. Use AWS EC2 Instance Scheduler or Lambda-based stop/start scripts triggered by CloudWatch alarms on GPU utilization. If your instance sits idle for more than 30 minutes, stopping it (not terminating — stopping preserves the EBS volume with model weights) saves the full compute cost.

GPU utilization is the key metric to watch. CloudWatch does not natively expose GPU metrics — you need to install the NVIDIA DCGM exporter or CloudWatch Agent with GPU plugin to get per-GPU utilization, memory usage, and temperature. Set alarms at 90% utilization to trigger auto-scaling or alert for manual scaling. Set alarms at below 5% utilization for more than 60 minutes to trigger scale-down. This simple alerting setup recovers a significant fraction of idle compute spend.

For tracking cost-per-token on a self-hosted setup (which matters for chargebacks, product pricing, and ROI analysis), instrument your inference server with request logging: capture input token count, output token count, latency, and instance-hour cost at time of request. With this data you can calculate effective cost-per-token and compare directly against what you would have paid on API. Most teams skip this instrumentation and later cannot demonstrate the ROI of their self-hosted investment. Build it from day one.