Cloudflare AI Search Commentary: Why RAG apps should design index limits, crawling, and charging boundaries before prompts

Publication date: 2026-05-21 | Category: Development information

1) One-line problem definition

Key line: The cause of RAG app failure is often not the model, but the slow setting of storage, indexing, crawling, and charging boundaries for the search infrastructure.

Cloudflare AI Search bundles document storage, vector indexing, and web crawling into one product, allowing developers to quickly create search-based AI capabilities. Instances created after April 16, 2026 include built-in storage, built-in vector index, and web crawling, and starting June 3, 2026, existing instances will also be automatically migrated to managed infrastructure.

This article is a practical guide for developers and small teams creating Workers, RAG, document search, customer support chatbot, and in-house knowledge search. The scope includes adoption decisions, limits, migration, cost boundaries, and operational checklists for Cloudflare AI Search. On the contrary, this is not an article that unconditionally recommends a specific vendor or provides a deep mathematical explanation of the vector search algorithm.

2) Conclusion first

Key line: Cloudflare AI Search is advantageous for quick RAG construction, but teams that require large volumes and sophisticated search tuning should review limits and independence first.

A good fit for the team: Workers-based app, document-based customer support, search based on website crawling, fast RAG PoC with a small team
Teams to watch out for: Organizations where tens of millions of documents, complex ranking tuning, independent vector DB operation, and multi-cloud data sovereignty are key
My judgment: The essence of AI Search is not “the function of providing vector DB for free” but Folding the operational boundaries of the search app into a managed form within the Cloudflare account. Select.

The Free plan has explicit limits of 100 instances, 100,000 files per instance, 20,000 queries per month, and 500 pages crawled per day. The Paid Workers plan expands to 5,000 instances, 1M files or 500K hybrid search, and unlimited queries and crawls. This number is not a marketing figure, but rather an architectural design baseline.

3) Core structure decomposition

Key line: AI Search absorbs part of the structure that directly linked R2, Vectorize, Workers AI, AI Gateway, and Browser Run into the product boundary.

3-1. Data source layer

The first layer of the search app is the raw data. This includes document files, websites, R2 buckets, and internal APIs. The new instance of AI Search offers built-in storage, so files you upload directly will go into the product's internal storage. However, for instances that used the existing R2 as a data source, the R2 data source itself will be maintained even after migration.

3-2. Index hierarchy

The second layer is the vector index. A vector index is a repository that converts document fragments into numeric coordinates to quickly find similar meanings. Past instances used the Vectorize database within your account, but after the managed migration on June 3, 2026, they will move to a managed vector database within your AI Search instance.

3-3. Crawling layer

The third layer is the web crawler. You may need a feature like Browser Run to read dynamic JavaScript pages. In the new structure, existing crawled pages from website data sources will be moved to built-in storage, and future crawl results will also be stored there. According to the Cloudflare document, Browser Run usage that occurs in AI Search will no longer be billed separately.

3-4. Model·Gateway layer

The fourth layer is Workers AI, which is used for embedding, query rewriting, and answer generation, and AI Gateway for usage control. Importantly, these will remain separate services after the AI Search managed migration. In other words, just because “AI Search is a free beta” does not mean that all model usage and gateway costs disappear.

4) Explanation of design intent

Key line: Cloudflare's direction is to reduce the time developers spend assembling search infrastructure, and instead let them operate by account, limit, and billing boundaries.

When creating a RAG directly, storage, document parser, chunking, embedding, vector DB, search API, re-ranking, prompting, and monitoring are usually attached separately. This approach offers a high degree of freedom but a large footprint for small teams. AI Search bundles much of this storage, indexing, and crawling into a managed product.

An alternative is to give up. You have less freedom to have fine-grained control over Vectorize yourself, tweak detailed ranking strategies in an external vector database, or keep your search pipeline completely vendor-independent. In particular, the limit of 5 custom metadata fields per instance and 500 characters of text metadata value has practical implications for document filtering design.

My interpretation is this. AI Search's competitors are not simply Pinecone or Qdrant. The real competitor is “the way developers hand-weave R2, Vectorize, Workers AI, Browser Run, and AI Gateway each time.” For teams already within the Cloudflare ecosystem, a managed perimeter makes you productive. Conversely, this boundary may be limiting for teams where search itself is the core product.

5) Evidence and comparison

Key line: The selection criteria is not “can it do vector search”, but Limits, operational responsibility, cost estimation, search tuning Degree of freedom.

Approach	Strengths	Limit	Recommendation status
Cloudflare AI Search	Storage, vector index, and web crawling can be started in a managed manner, easy to connect with the Workers app	Must be designed within product limits, such as 5 metadata fields and 4MB file size	Cloudflare-based RAG, document search, fast customer support chatbot
Direct assembly: R2 + Vectorize + Workers AI + Gateway	You can control each service settings and costs in more detail	Pipeline assembly, failure response, and migration responsibility remains with the development team	Teams needing to customize their search pipeline
External vector DB: Qdrant, Pinecone, pgvector, etc.	Search tuning, data sovereignty, multi-cloud options are wide	Network, authentication, and operation costs are added when connecting to Cloudflare Workers	Organizations where high-volume searches, complex filtering, and independent infrastructure are important

The numbers in the official document are important. Your new Free instance can support up to 100,000 files, 20,000 queries per month, and 500 pages crawled per day. Paid Workers plan expands to 1M files or 500K hybrid search, unlimited queries and crawls. The maximum file size for all plans is 4MB, and there are 5 custom metadata fields.

Another basis is Cloudflare TypeScript SDK v6.0.0. In the April 30, 2026 release, AI Search was added as a new top-level resource called client.aiSearch, with 46 methods handling instances, namespaces, tokens, and items. This is a sign that AI Search is expanding to the developer API surface rather than just being an experimental dashboard feature.

6) Actual operation flow / step-by-step execution method

Key one line: AI Search introduction requires calculating document size, filter criteria, query volume, and crawling scope first before creating an instance.

Categorize the documents you are searching for.
Divide into FAQs, product documents, blogs, internal manuals, policy documents, and check the average and maximum size of each file. Files larger than 4MB require splitting or a separate pipeline.
Select 5 metadata first.
Example: product, locale, doc_type, updated_at, access_level. Assuming you can keep adding filter conditions later will make your design shaky.
Calculate your crawling budget.
If you are on the Free plan, you should split the initial index and update cycles within the limit of 500 pages per day. If you have a 5,000-page site, you should consider taking at least 10 days or a paid plan instead of finishing it all at once.
Estimate your query volume on a monthly basis.
20,000 queries per month for the Free plan is an average of about 666 queries per day. Internal testing, user searches, bot traffic, and retry requests should all be included in the calculation.
Separate the cost of answer generation.
Apart from the AI Search open beta limit, Workers AI and AI Gateway usage are subject to separate billing and policies. Separate search calls and create calls in logs.
Check existing instances before and after migration.
Managed migration will take place for up to 3 days starting June 3, 2026. Check whether there is a built-in storage label, whether the Vectorize DB is moved, whether the R2 data source is maintained, and whether separate Browser Run charges are stopped.
Create a search quality test set.
Set 30 representative questions, expected documents, allowed answers, and prohibited answers, and revalidate them every time the index changes.

#Example: Design memo before introducing AI Search
ai_search_plan:
max_file_size: "Split into 4 MB or less"
  metadata_fields:
    - product
    - locale
    - doc_type
    - updated_at
    - access_level
  free_plan_budget:
    monthly_queries: 20000
    daily_crawled_pages: 500
  separate_costs:
    - Workers AI
    - AI Gateway
  quality_gate:
    source_required: true
    stale_document_days: 30
    no_answer_when_source_missing: true

7) Mistakes/Pitfalls

Key one-liners: AI Search is easy to get started with, but if you see limits and costs too late, you'll have to redesign it during operation.

Mistake 1: Mistaking free beta as completely free
Prevention: Manage AI Search limits, Workers AI charging, and AI Gateway charging in separate tables.
Recovery: Search in logs Separate requests and model creation requests and set up monthly cost alerts.
Mistake 2: Assuming you can attach a lot of metadata
Prevention: Choose only the filters you absolutely need within the limit of 5 custom metadata fields.
Recovery: Combine fields that can be combined, such as doc_type, and separate detailed filters into application-level permission logic.
Mistake 3: Viewing web crawling only as site-wide collection
Prevention: Check sitemap, robots policy, dynamic pages, duplicate URLs, canonical URLs first.
Recovery: Important Prioritize documents first, exclude old pages and tag pages.
Mistake 4: Ignoring Migration of Existing Instances
Prevention: Vectorize/R2 resources associated with Built-in Storage label before June 3, 2026. Take note.
Recovery: Verify with sample queries and billing items that your search results, data sources, and cost items have changed as expected after the migration.
Mistake 5: Viewing search failures only as a prompt issue
Prevention: Record the documents retrieved, score, and document update date for each answer.
Recovery: Missing indexes for incorrect answers before tuning the model; Check for outdated documents, incorrect metadata, and missing over 4MB.

8) Strengths and limitations

Key line: Cloudflare AI Search's strength is operational simplification, but its limitations are search pipeline independence and detailed tuning.

Strengths

Quickly start storing, indexing, and crawling within your Cloudflare account.
Easy to connect with Workers-based apps, and TypeScript SDK can also handle AI Search resources directly.
Reduce footprint by migrating Vectorize DB and web crawl storage from existing instances to managed infrastructure.

Limit

The 4 MB file, 5 metadata, and 500 character text metadata limitations impact information structure design.
There is a free limit during the open beta period, but Cloudflare says it will announce prices at least 30 days before the start of billing, so long-term costs are not yet fixed.
Workers AI and AI Gateway are separate services, so answer generation costs and policies must be managed separately.

Counterexample: If search quality itself is the core competitiveness of the product, it may be better to use an independent vector DB, self-reranking, and evaluation pipeline rather than AI Search. On the other hand, if your goal is to “add a document-based answer function reliably and quickly,” AI Search is a good starting point.

9) Points to study more deeply

Key one-liners: Your next lesson should start with search quality assessment, metadata design, and cost observations rather than prompts.

Cloudflare AI Search’s built-in storage and built-in vector index structure
To what extent are Workers AI, AI Gateway, and Browser Run separated from AI Search
Search quality assessment: representative questions, expected documents, missing source rate, recency SLA
Document chunking strategy: 4MB file limit, long document division, title/section preservation method
Authority filter design: How to express access control within 5 metadata fields
Resource client.aiSearch in TypeScript SDK v6 and breaking changes

10) Execution Checklist + Author’s Perspective

Key line: Before introducing AI Search, the “stopping standard” of the search app must be determined before the “operating standard”.

Is there an original that exceeds the maximum 4MB limit per document?
Have you determined the five metadata necessary for filtering?
Does the Free plan exceed the monthly crawl limit of 20,000 queries and 500 pages per day?
Can Workers AI and AI Gateway costs be viewed separately from AI Search limits?
If it is an existing instance, did you record the resource status before and after the managed transfer on June 3, 2026?
Are the source documents, document update dates, and access rights included in the search results?
Is there a gate that stops distribution when an answer without a source, an old document, or an unauthorized document is exposed?

Definition of Done: If 30 representative questions find expected documents, 0 unsourced answers, and query volume, crawl volume, and Workers AI costs are visible separately on the monthly dashboard, you've passed the first operational criteria.

My recommendation: For teams already using Cloudflare, get started quickly with AI Search, but create a limit table and cost table together from day one. The demonstration of “Search is possible” is easy. In real-world operations, you need to be able to explain who can retrieve which documents, which documents are missing, and where the costs are incurred. If the search scale is large or ranking is the product competitiveness, it is better to use AI Search only as a PoC baseline and review independent search infrastructure in parallel.

Cloudflare AI Search Commentary: Why RAG apps should design index limits, crawling, and charging boundaries before prompts

Cloudflare AI Search Commentary: Why RAG apps should design index limits, crawling, and charging boundaries before prompts

1) One-line problem definition

2) Conclusion first