A&A INSIGHTS
When AI SaaS Costs Spike with Usage: Separate Caching from Rate Limits
Caching reuses identical requests to cut duplicates, while rate limiting caps total requests per window. Cloudflare AI Gateway separates these two cost factors into distinct features.
日本語で読む
THE STARTING POINT
Caching reduces duplicate calls while rate limiting caps request volume. Cloudflare AI Gateway caches identical requests and also enforces rate limits per time window. These are two separate cost controls, configured independently.
There Are Two Different Cost Drivers
A&A perspective
AI SaaS costs that grow unpredictably usually come from two different sources. One is duplicate requests for the same prompt causing repeated charges. The other is total request volume increasing as usage grows. This separation of decisions is what A&A recommends. Cloudflare AI Gateway keeps these two factors under separate controls. Splitting the decisions makes cost explainable.
From the sources
Caching serves responses from Cloudflare cache for identical requests, reducing duplicate calls to the provider and saving cost. Caching supports only text and image responses and applies only to identical requests.
From the sources
Rate limiting controls traffic to your application, preventing expensive bills and suspicious activity. Rate limits are defined as request counts within a time frame.
Caching Handles Repeated Identical Requests
A&A perspective
Caching decides whether a request can be reused rather than re-paid. AI Gateway hashes provider, endpoint, model, auth header, and full request body by default, caching only identical requests. This is not automatic cost reduction. Intentionally setting shared cache keys lets you reduce the number of reusable common questions.
From the sources
Caching is disabled by default. You enable it via dashboard or API and configure TTL, cache key, and skip behavior per request. The cf-aig-cache-status header returns HIT or MISS, letting you confirm whether a response was cached.
Hypothetical example
For a FAQ bot, setting a cache on 'I want to cancel my order' means that prompt is paid once. Requests containing unique customer IDs should skip caching to avoid incorrect responses. This avoids shared caching that ignores per-customer data.
Rate Limiting Bounds Total Request Volume
A&A perspective
Rate limiting is about bounding total volume, not reusing responses. AI Gateway lets you define max requests per time window. You can choose fixed or sliding window techniques. This prevents unexpected bills from traffic surges.
From the sources
When requests exceed the rate, the server returns a 429 Too Many Requests status code. This lets the client know the request was rejected due to rate limiting.
Hypothetical example
Setting a limit of 1000 requests per hour per free-tier user prevents billing surprises from traffic surges. However, if per-request pricing is high, total cost is still unpredictable.
These Two Controls Are Independent Decisions
A&A perspective
Caching and rate limiting are not alternative approaches to the same problem. Caching decides request reuse while rate limiting bounds total volume. They are configured independently. Separating the decisions makes cost explainable.
From the sources
Cache in AI Gateway is volatile; simultaneous identical requests may not be served from cache. This means caching effectiveness is not guaranteed during concurrent access.
From the sources
Rate limiting controls the number of requests in a specific time frame. This makes total request volume predictable.
Hypothetical example
A support chat might cache common questions while rate-limiting total requests per user, making predictable cost control possible.
Configure Both to Make Costs Predictable
A&A perspective
Caching reduces eligible repeated requests and rate limiting bounds total volume. Both together make cost predictable; either alone is incomplete. These are design choices, not guarantees.
From the sources
Caching supports only text and image responses and applies only to identical requests. This means small prompt variations greatly affect cache hit rates.
Hypothetical example
To keep monthly cost fixed, cache FAQ responses and rate-limit per-user request volume. This helps suppress rising costs.
What These Controls Do Not Guarantee
A&A perspective
Shared caching that ignores per-customer data or update frequency is inappropriate. Rate limiting alone does not guarantee a spend ceiling. These are design choices, not guarantees.
From the sources
Cache in AI Gateway is volatile; simultaneous identical requests may not be served from cache.
From the sources
Rate limiting controls the number of requests in a specific time frame. This makes total request volume predictable.
Hypothetical example
Even with a rate limit of 10,000 requests/hour, high per-request pricing means total cost is still unpredictable. Cache effectiveness also depends on key design.
Caching and rate limiting address different cost drivers. Caching reduces duplicate eligible requests while rate limiting bounds total requests per window. Neither alone yields predictable cost, and shared caching that ignores customer data or high per-request pricing does not guarantee a spend ceiling.
Sources & editorial note
Primary pages read for this article. Publication dates below belong to the sources; access dates record our research.
- Caching
Cloudflare · 2026-08-27
Accessed 2026-09-27 - Rate limiting
Cloudflare · 2026-06-05
Accessed 2026-09-27
AI-assisted editorial production
A&A uses AI for research, writing, translation and editorial checks. Source facts, our analysis and hypothetical examples are labeled separately.
Editorial check: 2026-09-27