Cloud GPUs are sold in three ways. On-demand capacity is billed by the hour with no commitment, but the newest GPUs are often unavailable when you need them. Short-term reservations, such as AWS Capacity Blocks or Google Cloud’s flex-start VMs, guarantee capacity for days or weeks. Committed contracts reserve GPUs for a year or more at a lower rate. Most of the H100, H200, B200, and B300 capacity in the market is sold through the last two, which is why on-demand pricing pages only show part of the picture.
This post compares the three models, shows what on-demand GPUs cost across providers, and lists what to check before you sign.
On-demand: flexible, when you can get it
On-demand GPUs are the easiest to start with: create an instance, pay by the hour, delete it when the job is done. Prices vary widely between providers for the same GPU. These are public on-demand list prices per GPU per hour from providers’ own pricing pages at the time of writing, with AWS prices for US East:
| GPU | Hyperscalers | Specialized GPU clouds |
|---|---|---|
| H100 80 GB | AWS p5 $6.88, Google Cloud a3 $11.06 | Denvr $2.45, Hyperstack $3.20, Crusoe $3.90, Lambda from $3.99, Verda $4.89 |
| H200 141 GB | AWS p5en $7.91, Azure ND H200 v5 $10.60 | Crusoe $4.29, Nebius $5.40, Together AI $5.99, CoreWeave $6.31 |
| B200 180 GB | AWS p6-b200 $14.24, Google Cloud a4 $8.06 with flex-start | Hyperstack $6.00, Lambda from $6.69, Together AI $8.19, CoreWeave $8.60 |
| B300 288 GB | AWS p6-b300 $17.80 | FPT $6.99, Hyperstack $7.40, Civo $7.49, Crusoe $8.96, Nebius $9.50 |
In Europe, OVHcloud lists H100 at €2.80 and H200 at €5.25 per GPU per hour, and Scaleway lists H100 at €3.17 and B300 at €7.50. Prices exclude VAT, change often, and differ by region. For the newest GPUs, see where to rent NVIDIA B300. The full list, with interconnects and locations, is on the GPU Kubernetes page.
The catch is availability. A price on a pricing page does not mean capacity in a region today, and the newest rack-scale systems such as GB200 and GB300 NVL72 are sold almost only through reservations. On-demand works best for development, inference that scales with traffic, and bursts on top of a reserved base.
Short-term reservations: capacity for days or weeks
Hyperscalers sell guaranteed GPU capacity for fixed windows, aimed at training runs and fine-tuning jobs that need many GPUs for a known period.
AWS Capacity Blocks for ML reserve GPU instances such as p5, p5en, p6-b200, and p6-b300 for a future window. You can book up to eight weeks ahead, with up to 64 instances per block and 256 across blocks. The price depends on supply and demand at the time of purchase, is paid up front, and does not change afterward. Blocks cannot be canceled, and Savings Plans and Reserved Instance discounts do not apply. Instances in a block are placed close together in EC2 UltraClusters for low-latency networking.
Google Cloud flex-start VMs, part of Dynamic Workload Scheduler, queue your request until capacity is available and then run for up to seven days. You pay only while the VMs run, not while the request waits, at a discount to on-demand.
Many specialized GPU clouds also sell reserved clusters with InfiniBand networking, quoted on request.
Committed contracts: the base of the market
For steady workloads, one-year to three-year reservations are where most capacity and the lowest rates are. GPU clouds, hyperscalers, and marketplaces quote these individually. Some marketplaces start reserved terms at 12 months. The rate is lower than on-demand, but the commitment is real: you pay for the GPUs whether or not your workloads use them, and the hardware generation is fixed for the term.
A committed contract makes sense when you know your baseline: the GPUs that run training and inference around the clock. Committing beyond that baseline turns into paying for idle capacity.
Choosing a buying model
| Workload | Buying model |
|---|---|
| Experiments, notebooks, development | On-demand |
| Inference that follows traffic | Reserved base, on-demand for peaks |
| A training run of days or weeks | Capacity blocks, flex-start, or a short GPU cloud reservation |
| Continuous training and production inference | Committed contract for the baseline |
| Interruptible batch jobs with checkpoints | Spot or preemptible capacity where available |
Most teams end up with a mix: committed capacity for the baseline, short reservations for large runs, and on-demand for everything that comes and goes.
What to compare besides the hourly price
The per-GPU price is the easiest number to compare and rarely the one that decides the deal. Before signing, check:
- GPU memory and interconnect. NVLink inside the node matters for large models, and InfiniBand or RoCE between nodes matters for multi-node training. A cheaper cluster without a fast east-west network can cost more per finished training run.
- Per GPU or per instance. Some providers price per GPU, others per 8-GPU node, and some include CPU, memory, and local NVMe while others charge separately.
- Location and data residency. Training data under GDPR or customer contracts may need to stay in the EU, which narrows the list quickly.
- Storage and egress. Moving datasets and checkpoints in and out of a provider can add up to a meaningful share of the bill.
- Lead time and terms. When the capacity is delivered, whether payment is up front, and what happens if hardware fails during the term.
- How the nodes reach your platform. Instances in a cloud account, bare-metal servers handed over with SSH access, or a managed service each need a different integration.
Running on capacity from several providers
The best price for each part of your workload is rarely at one provider. A training reservation at a GPU cloud, on-demand inference GPUs in a hyperscaler region close to users, and a few servers in your own data center are a common combination, and without a common platform each becomes its own environment.
Cloudfleet runs them as one Kubernetes cluster. Reserved nodes join the cluster as soon as the provider hands them over, through the Cloudfleet API, Terraform, or a single CLI command, and node auto-provisioning adds on-demand GPUs from your AWS and Google Cloud accounts when demand spikes. Workloads choose a GPU with labels such as cfke.io/accelerator-name, as the GPU workloads documentation describes. You contract and pay providers directly, with no markup on compute.
On the buying side, the Cloudfleet team works closely with most infrastructure providers and helps you source GPU capacity: one request with your GPU model, count, interconnect, term, and location, and concrete on-demand, reserved, or committed offers in return. For the difference between that and a GPU marketplace, see GPU marketplace vs managed GPU Kubernetes.
Frequently asked questions
What is the cheapest cloud GPU?
It depends on the GPU and the term. For H100, public on-demand list prices range from about $2.45 to $11 per GPU per hour across providers, with specialized GPU clouds usually below the hyperscalers. Reserved and committed contracts are lower still, but are quoted individually.
Is there still a GPU shortage?
For the newest GPUs, yes. B200, B300, and rack-scale GB200 and GB300 systems are mostly sold through reservations and committed contracts, and on-demand capacity is often unavailable in popular regions. Inference GPUs such as L4 and L40S are listed on demand at most hyperscalers and European clouds.
Should I buy GPUs on demand or reserved?
Buy reserved or committed capacity for the GPUs you use around the clock, and on-demand for everything above that baseline. Use short-term reservations such as AWS Capacity Blocks or Google Cloud flex-start for training runs with a known duration.
How do I rent a GPU cluster?
Specialized GPU clouds and hyperscalers rent clusters of 8-GPU nodes with InfiniBand or RoCE networking, on demand for some GPUs and as reservations for most. To run workloads on the cluster, join the nodes to a Kubernetes platform such as Cloudfleet, which handles drivers, scheduling, and scaling.
