Stop Building Vector Pipelines for Documents That Never Get Read

Most enterprise RAG architectures follow a standard assumption: before an LLM can answer a question about your documents, you must build an end-to-end ingestion pipeline. You chunk the text, call embedding APIs across millions of tokens, provision a dedicated vector database cluster, and maintain sync jobs to keep embeddings fresh.

In practice, enterprise search access patterns follow a steep power-law distribution. Compounded by storage audits showing that over 85% of corporate unstructured data is dormant or 'dark', internal repository telemetry consistently reveals that fewer than 10% of ingested enterprise documents are ever queried.

Pre-indexing entire document stores creates substantial upfront compute expense for data that sits idle. Worse, vector architectures carry an ongoing maintenance liability: the embedding model lifecycle. When cloud providers deprecate an embedding model (such as transitioning from text-embedding-gecko to text-embedding-004 or text-embedding-005), embedding spaces cannot be mixed. Upgrading requires a full re-indexing pass across the entire corpus, re-parsing and re-embedding dormant files simply to maintain index compatibility.

To address these inefficiencies, we benchmarked Just-In-Time RAG (JIT-RAG): an architectural strategy where document parsing and indexing are deferred until query time, or bypassed entirely using frontier long-context models.

We evaluated four production architectures across 25 golden evaluation test cases on a complex, table-dense 50-page enterprise annual report using Gemini 3.8 Flash and DeepEval. The empirical findings reveal clear operational boundaries for when to index, when to parse locally, and when to stream raw documents directly.

---

1. The Four Contenders: Architectural Blueprint

JIT-RAG treats retrieval as a spectrum rather than a single fixed pipeline. Depending on document volume, visual layout complexity, and query frequency, different pipelines offer distinct operational trade-offs.

The architectural blueprint below outlines data flow, stage responsibilities, and benchmark characteristics across all four architectures:

Architecture Overview:

1. Method 1: BigQuery Native (Zero-Copy)
   • Ingestion: Google Cloud Document AI Layout Parser extracts structured text and table hierarchies.
   • Embeddings & Search: In-warehouse ML.GENERATE_EMBEDDING using text-embedding-005, indexed with ScaNN and queried via SQL VECTOR_SEARCH.
   • Synthesis: Gemini 3.8 Flash (gemini-3.8-flash).
   • Zero Client Overhead: Parsing and vector retrieval run entirely inside Google Cloud's analytical warehouse, removing the need for external vector database infrastructure.
   • Upcoming Native SQL Parser: Google recently announced native document parsing via AI.PARSE_DOCUMENT (in upcoming preview). This allows SQL queries to parse PDFs directly from Object Tables without separate Document AI orchestration. We will publish a follow-up benchmark comparing AI.PARSE_DOCUMENT once public preview access is available.

-- BigQuery Native SQL Pipeline Reference (Object Tables + Vector Search)
CREATE OR REPLACE TABLE `rag_dataset.document_embeddings` AS
SELECT 
  uri,
  chunk_content,
  ml_generate_embedding_result AS embedding
FROM ML.GENERATE_EMBEDDING(
  MODEL `rag_dataset.embedding_model`,
  (SELECT uri, chunk_content FROM `rag_dataset.parsed_chunks`),
  STRUCT('RETRIEVAL_DOCUMENT' AS task_type)
);

-- Retrieve top chunks directly in SQL
SELECT base.uri, base.chunk_content, distance
FROM VECTOR_SEARCH(
  TABLE `rag_dataset.document_embeddings`,
  'embedding',
  (SELECT ml_generate_embedding_result AS query_embedding 
   FROM ML.GENERATE_EMBEDDING(
     MODEL `rag_dataset.embedding_model`,
     (SELECT 'Q4 operating margin breakdown' AS content),
     STRUCT('RETRIEVAL_QUERY' AS task_type)
   )),
  top_k => 5,
  distance_type => 'COSINE'
);

2. Method 2: Multimodal Gemini Flash Markdown

   • Ingestion: Renders PDF pages into images and passes them to gemini-3.5-flash-lite vision OCR to reconstruct clean Markdown tables and multi-column text.
   • Embeddings & Search: In-memory cosine index using text-embedding-004.
   • Synthesis: Gemini 3.8 Flash (gemini-3.8-flash).
   • Visual Fidelity: Suitable for unstructured filings where native text extractors fail on nested tables or visual callouts.

3. Method 3a: PyMuPDF4LLM (Lightweight Local CPU)

   • Ingestion: Fast local CPU layout extraction via pymupdf4llm on commodity hardware, incurring $0.00 in cloud parsing fees.
   • Embeddings & Search: In-memory top-K cosine retrieval with text-embedding-004.
   • Synthesis: Gemini 3.8 Flash (gemini-3.8-flash).
   • Characteristics: Sub-2-second ingestion for standard digital-native documents without scanned elements.

4. Method 3b: Docling Local CPU (IBM Research TableFormer)

   • Ingestion: Local CPU neural layout analysis using IBM Research's Docling and TableFormer models, also incurring $0.00 in cloud parsing fees.
   • Embeddings & Search: In-memory top-K cosine retrieval with text-embedding-004.
   • Synthesis: Gemini 3.8 Flash (gemini-3.8-flash).
   • Characteristics: Reconstructs table grids and multi-column layouts locally on CPU without sending images to cloud vision APIs.

5. Method 4: Direct GCS Long-Context Streaming

   • Ingestion: Zero ETL. The raw PDF is uploaded to Google Cloud Storage (gs://...) and passed directly into Gemini's 1M-token context window.
   • Embeddings & Search: No embeddings. No vector database.
   • Synthesis: Gemini 3.8 Flash (gemini-3.8-flash).
   • Characteristics: Immediately queryable upon upload with zero preprocessing delay.

---

2. Ingestion & Query Economics

We benchmarked each pipeline on a 50-page enterprise annual report (approx. 26,000 tokens) containing multi-column financial statements, balance sheets, and notes. Tests were run with Gemini 3.8 Flash and evaluated across 25 golden question-answer pairs using DeepEval.

Key Economic Takeaways

The benchmark reveals a fundamental economic trade-off between upfront ingestion latency and recurring per-query serving cost across all four architectures:

• Method 1: BigQuery Native (Document AI + Vector Search): Ingestion takes 1.69s per page (approx. 0.6 pages/sec) at approx. $0.0276 per page (approx. 2.8¢ / page, or $1.38 total for 50 pages), driven primarily by Document AI table parsing rates ($25 to $30 per 1,000 pages). At query time, top retrieved chunks yield 4.75s P50 query latency and $0.000060 per query (approx. $0.000001 per page), operating completely serverless inside SQL without external vector database clusters.
• Method 2: Multimodal Gemini Flash Markdown: Ingestion requires 1.84s per page (approx. 0.5 pages/sec) at approx. $0.00016 per page (approx. 0.016¢ / page, or $0.0081 total) using vision tokens to preserve unstructured graphics and tables. At query time, responses return in 3.01s P50 query latency at $0.000079 per query (approx. $0.000002 per page).
• Method 3a: PyMuPDF4LLM (Lightweight Local CPU): Ingestion is the fastest of any indexed method at 0.14s per page (approx. 7.3 pages/sec) and cheapest at $0.000012 per page (approx. 0.001¢ / page, or $0.0006 total for embeddings), completely eliminating cloud OCR fees. At query time, responses return in 3.2s P50 query latency at $0.000079 per query (approx. $0.000002 per page).
• Method 3b: Docling CPU (IBM Research TableFormer): Ingestion takes 5.05s per page (approx. 0.2 pages/sec) at $0.000024 per page (approx. 0.002¢ / page, or $0.0012 total) to reconstruct complex tables locally on CPU. At query time, responses return in 3.60s P50 query latency at $0.000079 per query (approx. $0.000002 per page).
• Method 4: Direct GCS Long-Context Streaming: Ingestion requires 0.00s latency (Instant) at $0.00 per page, completely eliminating ingestion ETL pipelines. However, querying re-reads all 50 pages (approx. 26,000 tokens) on every inquiry, resulting in a higher 37.14s P50 query latency and $0.001995 per query (approx. $0.000040 per page per turn, 25x higher serving cost than chunked methods).

Notable Callout: The 2-Query Break-Even Threshold
Because local CPU ingestion costs under 0.002¢ per page and completes in seconds, building an on-demand chunked index amortizes after just two user questions on the same document. For interactive or multi-turn sessions, JIT chunking reduces recurring query costs by 25x to 33x while maintaining sub-5-second P50 query latencies.

Notable Callout: Zero Cold-Start vs. Long-Context Tax
Method 4 is unmatched for ad-hoc, one-off executive queries where zero pipeline ETL and immediate time-to-first-answer are paramount. However, re-reading entire documents across repeated queries or standard multi-turn chat sessions (without paying dedicated hourly context-cache storage fees) incurs an ongoing "long-context tax" in both elevated latency (37s P50) and token expense ($0.002 per turn).

Notable Callout: BigQuery Serverless Simplicity
Document AI accounts for over 95% of BigQuery Native's upfront ingestion expense ($1.3812 total), yet it provides an integrated "all-in-SQL" architecture that keeps data within enterprise boundaries. Forthcoming native SQL document parsing via AI.PARSE_DOCUMENT is expected to reduce both ingestion costs and pipeline orchestration overhead.

---

3. Retrieval Quality Benchmarks (DeepEval)

We evaluated answer quality across 25 golden test cases using DeepEval, testing Faithfulness, Answer Relevancy, Contextual Recall, and Hallucination against an enterprise baseline threshold of 0.70.

Quality Analysis:

1. Direct Context Injection (Method 4) scored 1.00 on Faithfulness: Because the entire document remains accessible within the prompt, there are no chunk-boundary truncation errors. Traditional chunking occasionally splits financial disclosures or footnotes across boundaries, causing the model to miss qualifying text. Method 4 avoids this issue entirely.

2. Local CPU Comparison: PyMuPDF4LLM vs. Docling TableFormer: PyMuPDF4LLM is fast (1.24s), but scored 0.58 on Faithfulness and 0.62 on Recall because borderless tables caused numbers to merge into adjacent prose text.

   Docling resolved this on CPU: TableFormer recognized table boundaries and multi-column headers locally, raising faithfulness to 0.77 and recall to 0.92, closely matching cloud vision models while avoiding cloud parsing charges.

3. Multimodal Vision OCR (Method 2): gemini-3.5-flash-lite vision extraction handled non-standard graphic layouts and charts effectively, achieving 0.94 contextual recall and 0.79 faithfulness.

---

4. Attentional Distribution: The "Lost in the Middle" Effect

To determine whether fact placement influenced retrieval accuracy, we stratified test cases across three document regions:

• Early Section (Primacy): Pages 1–15 (9 Test Cases)
• Middle Section (Valley): Pages 16–35 (8 Test Cases)
• Late Section (Recency): Pages 36–50 (8 Test Cases)

Attentional Behavior:

• Chunked RAG (Methods 1, 2, 3a, 3b) is position-invariant: Vector search retrieves relevant text chunks independently of where they appear in the original PDF. A chunk on page 28 has the same retrieval probability as one on page 2.
• Direct Long-Context displays modest attentional dilution: Recall dropped by 14.5% when target facts were located strictly within the middle 20 pages.

---

5. The Large Context Window Paradox: Why Full-Document Injection Has Limits

With frontier models like Gemini 3.8 Flash supporting a 1M-token context window, a common proposal is to bypass chunking and embeddings entirely by passing the full document directly to the model.

While Method 4 (Direct GCS Streaming) is practical for ad-hoc, low-frequency queries, relying on full-context injection across enterprise systems introduces clear operational trade-offs:

1. The Compounding Token Tax (Linear Cost Escalation)

Context stuffing turns every query into a full-document re-read. When a PDF spans 300 pages (approx. 150,000 tokens), sending 100 queries processes 15 million input tokens. At sustained volume, operational costs scale linearly with query traffic.

Embedding-based RAG shifts this cost curve. You parse and embed the 150,000 tokens once during ingestion. Subsequent queries inject only the top 3 to 5 relevant chunks (approximately 1,500 tokens) into the synthesis prompt. Marginal query costs remain stable at fractions of a cent, regardless of whether the source catalog contains 50 pages or 5,000 pages.

2. Pre-Fill Latency and Time-to-First-Token

Pre-fill latency is a major operational bottleneck for direct context injection. Before a transformer generates its first output token, the underlying accelerators must compute multi-head attention across every token in the prompt. Under concurrent user traffic, this compute demand leads to queueing and elevated tail latency. In our tests, Method 4 reached a 50.31-second P90 latency on 50-page documents under load.

Chunked pipelines (Methods 2, 3a, and 3b) consistently returned answers in 2.8 to 3.8 seconds (P50). Because the model evaluates attention over a compact 1,500-token context, pre-fill completes quickly. In multi-turn chat or agentic workflows that require sequential reasoning steps, 30-to-50-second turn latencies degrade the user experience, whereas sub-4-second responses maintain interactive conversational flow.

3. Real-World Semantic Distractors vs. Synthetic Benchmarks

Frontier model benchmarks frequently demonstrate 99%+ recall on synthetic "Needle in a Haystack" evaluations. However, enterprise documents differ substantially from synthetic tests. Synthetic needles are typically obvious outliers (such as inserting an arbitrary code phrase into unrelated text). Real enterprise filings contain semantic distractors: multiple quarters with identical line items ("Operating Income", "Capital Expenditures"), amendment revisions, and standard boilerplate.

When attention is spread across 200,000 tokens of repetitive text, attentional dilution can occur. In our tests, fact recall in the middle 20 pages decreased by 14.5% under direct long-context injection. Pre-filtering with vector embeddings removes irrelevant sections before the synthesis stage begins.

4. Repository-Scale Discovery

While a 1M-token window accommodates several large reports (approx. 750,000 words), corporate document repositories contain thousands of files across cloud drives and content systems. Passing tens of millions of tokens into an active context window is impractical. An indexing layer, whether vector search, BM25, or hybrid routing, remains necessary for cross-document discovery.

5. Context Caching Considerations

Prompt caching can reduce repeated token input costs, but involves operational trade-offs. Cloud providers charge ongoing storage rates to retain KV caches in accelerator memory. For sporadic query patterns, cache maintenance fees can offset embedding savings. Additionally, caches depend on exact prefix matching; modifying system instructions, user role permissions, or tool definitions invalidates the cache, requiring a full pre-fill.

6. Auditability and Compliance Provenance

In regulated sectors such as finance, healthcare, and legal compliance, generated answers must be auditable. Chunked RAG provides explicit chunk identifiers and source spans that compliance teams can verify. Full-context injection generates soft references across extensive text, making specific validation more challenging.

---

6. The Break-Even Analysis: Economic Crossover Points

On a per-page basis, Method 4 incurs $0.00 in upfront ingestion cost, but adds approximately $0.00004 per page on every query turn ($0.001995 per 50-page turn). Chunked methods require an upfront ingestion investment (ranging from $0.000012/page on CPU to $0.0276/page via Document AI), but lock in stable query serving costs of under $0.00008 per query regardless of how large the catalog grows.

Crossover Guidelines:

1. Method 4 vs. Local CPU (Methods 3a & 3b): 1 to 2 queries. Because local CPU ingestion costs only approx. $0.000012 to $0.000024 per page (and takes just 0.14s per page for PyMuPDF4LLM), indexing amortizes almost instantly. If a user asks two or more questions, local CPU indexing pays for itself immediately.
2. Method 4 vs. Vision Markdown (Method 2): 4 queries. At approx. $0.00016 per page for vision OCR, chunked indexing becomes more cost-effective than long-context streaming by query 5.
3. Method 4 vs. BigQuery Native: approx. 714 queries. Document AI parsing fees (approx. $0.0276 per page) require higher query volume to break even against long-context streaming. Once BigQuery SQL-native document parsing becomes available, this break-even threshold is expected to drop significantly.

---

7. The Operational Cost of Model Deprecation

Standard RAG evaluations focus on initial ingestion and query inference costs, often overlooking lifecycle maintenance expenses.

Unlike software libraries with backwards-compatible interfaces, vector spaces cannot be mixed across model generations. An embedding generated with text-embedding-005 cannot be compared against text-embedding-004 vectors; cosine similarities across different models are mathematically invalid.

When cloud providers retire embedding model versions (typically every 12 to 24 months), organizations cannot perform incremental migrations. They must re-read, re-chunk, and re-embed their entire document repository.

For an enterprise maintaining 500,000 PDF documents (10,000,000 pages), retiring an embedding model triggers vastly different migration costs across architectures. Method 1 (BigQuery Native + Document AI) faces the heaviest burden, requiring $276,000 in layout parsing fees and approx. 4,700 hours of compute. Method 2 (Multimodal Flash Vision MD) reduces this re-indexing liability to $1,600 across 5,100 vision inference hours. Lightweight local CPU parsing via Method 3a (PyMuPDF4LLM) completes the entire 10M-page re-indexing run for just $120 in embedding API calls and approx. 390 CPU hours. In stark contrast, Method 4 (Direct GCS Long-Context) incurs exactly $0.00 in re-indexing costs and 0.00s in downtime, upgrading model generations simply by updating a single model parameter in code. In typical enterprise repositories where over 90% of archived documents are never queried again, pre-indexing at catalog scale creates massive wasted compute.

Lifecycle Comparison:

With Method 4 (Direct GCS Long-Context), transitioning an application to a new model requires updating a single model identifier:

# Upgrading model versions in direct long-context mode
response = client.models.generate_content(
    model="gemini-3.8-flash",
    contents=[gcs_part, prompt]
)

No re-embedding pipeline, vector database migration, or index re-clustering is required.

---

8. The Adaptive JIT-RAG Decision Framework

Rather than standardizing on a single retrieval pattern across all workloads, production systems benefit from dynamic routing based on document attributes and expected access patterns:

---

9. Architectural Recommendations

Based on empirical benchmark results across latency, cost, and retrieval accuracy, we recommend the following routing principles:

1. Ad-Hoc Executive Queries (< 2 queries): Use Direct GCS Long-Context (Method 4). Requiring $0.00 upfront per page and 0.00s ingestion latency, bypassing vectorization provides immediate responses with 1.00 faithfulness and zero infrastructure overhead for one-off analyses.
2. Interactive Multi-Turn Sessions (> 2 queries): Use lightweight local CPU parsing (Method 3a at 0.14s/page latency and approx. $0.000012/page for clean digital text, or Method 3b at 5.05s/page and approx. $0.000024/page for table-heavy layouts). Building an in-memory index after the second query locks in 25x lower per-query costs and sub-4-second response times.
3. Complex Visual Documents: Use Gemini Flash Vision OCR (Method 2 at 1.84s/page latency and approx. $0.00016/page) when layout structure and embedded graphics are critical to answering domain questions.
4. Turnkey Enterprise Data Lake Option: BigQuery Native (ML.GENERATE_EMBEDDING and VECTOR_SEARCH) is best evaluated as an integrated option rather than mandatory guidance. While Document AI parsing introduces higher upfront ingestion costs (approx. 2.8¢ per page and 1.69s/page latency), it provides a native, serverless "one-solution-fits-all" architecture. For organizations whose documents already reside in Google Cloud Storage and whose teams operate primarily in SQL, it eliminates the operational burden of external vector database clusters, keeps data inside enterprise governance perimeters, and requires zero client-side infrastructure.
5. Cold Archive Strategy: Avoid pre-indexing enterprise archives with low access probability. Deferring extraction and embedding until access time eliminates unnecessary processing and protects against embedding model deprecation cycles.

---

Resources & Repository

The complete benchmark harness, 25 golden evaluation test cases, routing logic, and reproducible DeepEval scripts are available on GitHub:

• Repository: rubans/jit-rag-poc