A100 40GB
Same compute as 80GB variant but half the VRAM. Lower cost, suitable for models up to ~30B params.
A100 40GB Overview
The A100 40GB is the smaller-memory version of NVIDIA’s Ampere workhorse, retaining the GA100 Tensor Core design and 312 TFLOPS of FP16/BF16 throughput. It offers 40GB of HBM2e, 1,555 GB/s of bandwidth, and NVLink 3.0 support, making it a mature option for research and production workloads that do not need 80GB per card.
Forty gigabytes can accommodate roughly a 20B-parameter model at FP16 before allowing for activations and cache, while 1,555 GB/s gives it far more inference bandwidth than GDDR-based cards in its price band. NVLink 3.0 can connect cards for distributed training or model parallelism, though a single card cannot make a larger model fit by itself.
The A100 40GB is well suited to mid-size training, established CUDA pipelines, and inference where Ampere software maturity matters. It lacks Hopper’s FP8 Transformer Engine, and its lower bandwidth and capacity make it less compelling for large-context or very large-model serving.
Memory
Compute Performance
Hardware Specifications
Chip
Processors
Clocks
Memory
Power
Relative Performance
Relative to highest-spec GPU in database
Limitations
Live Cloud PricingOn-demand hourly rates
Compare A100 40GB vs…
Use Case Guidance
LLM Model Size Guidance
Estimates only. Actual capacity depends on context length, KV cache, and framework overhead.
Related Guides
LLM APIs Running on This GPU Class
Providers that serve frontier LLM inference on Ampere-class hardware.
A100 40GB vs Alternatives — Spec Comparison
| Spec | A100 40GB this | A100 80GB | A100 SXM4 80GB | A100 PCIe 80GB |
|---|---|---|---|---|
| VRAM | 40GB HBM2e | 80GB HBM2e▲ | 80GB HBM2e▲ | 80GB HBM2e▲ |
| Memory Bandwidth | 1555 GB/s | 2000 GB/s▲ | 2000 GB/s▲ | 1935 GB/s |
| FP16 TFLOPS | 312▲ | 312▲ | 312▲ | 312▲ |
| BF16 TFLOPS | 312▲ | 312▲ | 312▲ | 312▲ |
| FP8 TFLOPS | — | — | — | — |
| INT8 TOPS | 624▲ | 624▲ | 624▲ | 624▲ |
| TDP | 300W | 400W▲ | 400W▲ | 300W |
| Process Node | TSMC 7nm▲ | TSMC 7nm▲ | TSMC 7nm▲ | TSMC 7nm▲ |
| Architecture | GA100▲ | GA100▲ | GA100▲ | GA100▲ |
| Release Year | 2020 | 2020 | 2020 | 2021▲ |
| Max model (FP16) | ~20B params | ~40B params▲ | ~40B params▲ | ~40B params▲ |
| Max model (INT4) | ~80B params | ~160B params▲ | ~160B params▲ | ~160B params▲ |
Related GPUs
Frequently Asked Questions
How much VRAM does the A100 40GB have?
The A100 40GB has 40GB of HBM2e memory with 1555 GB/s bandwidth. This enables running models up to approximately 80B parameters at INT4 precision, 40B at INT8, or 20B at FP16.
What is the FP16 performance of the A100 40GB?
The A100 40GB delivers 312 TFLOPS of FP16 performance and 312 TFLOPS BF16. INT8 throughput is 624 TOPS. For transformer inference, memory bandwidth (1555 GB/s) is often the binding constraint rather than raw TFLOPS.
What is the A100 40GB best used for?
The A100 40GB is best suited for: ML training, Mid-size model inference, Research. Same compute as 80GB variant but half the VRAM. Lower cost, suitable for models up to ~30B params.
What interconnect does the A100 40GB use?
The A100 40GB uses NVLink 3.0 / PCIe 4.0 with 600 GB/s NVLink bandwidth for multi-GPU configurations. NVLink enables near-linear tensor-parallel scaling across multiple cards for models that exceed single-card VRAM.
What LLM model sizes can the A100 40GB run?
With 40GB of HBM2e, the A100 40GB can run models up to approximately 20B parameters at FP16 (2 bytes/param), 40B at INT8 (1 byte/param), or 80B at INT4/GGUF (0.5 bytes/param). These are estimates — actual capacity depends on context length, KV cache size, and framework overhead. Longer context windows require more KV cache memory, reducing the effective model size that fits.
How does the A100 40GB compare to the A100 for LLM inference?
The A100 40GB has 312 TFLOPS FP16 vs the A100 80GB's 312 TFLOPS, and 1555 GB/s memory bandwidth vs the A100's 2,039 GB/s. For memory-bound autoregressive LLM inference, bandwidth is the primary determinant of tokens-per-second. The A100's higher bandwidth gives it a throughput advantage for large model inference, despite the A100 40GB's lower cost.
What is the power consumption of the A100 40GB?
The A100 40GB has a TDP (Thermal Design Power) of 300W. This is the maximum sustained power draw under full load. For data center deployments, total rack power consumption is typically 1.2–1.5× the GPU TDP when accounting for CPU, memory, networking, and cooling overhead. At 300W, the A100 40GB is in the mid-range tier — compatible with standard data center power infrastructure.
Ready to rent?
Compare A100 40GB prices across 102+ providers
Live on-demand & spot rates · monthly cost estimates · availability status