In this article
The million-token race has stopped being a capability contest and become a make-or-buy calculation. Claude Haiku 5.5 lists a 1M-token context at $0.10 per million input up to 100K prompt tokens and $0.50 above that; OrcaCyber Zero 1.5 sells security-tuned 1M context at $3.00 input; Gemini 4 Argon’s $2.00 introductory input is a pricing anchor, not yet an API deployment option. At the same time, pplx-embed-v2-late provides a shared-space 0.6B query encoder and 9B indexer. The practical conclusion is now a token-efficiency repricing problem: for bounded, stable material, read the whole document; for large, frequently updated corpora, still index and retrieve. Which side wins is arithmetic, not capability.
Whole-document prompting crossed the cost threshold for bounded material
Haiku 5.5’s above-100K input tier is $0.50 per million tokens, so a 400K-token document costs $0.20 per pass before output; a 100K-token pass costs $0.01 at the base input price. OrcaRouter’s calculator estimates a 10M-token monthly workload at 70/30 input/output at $43.50 at list rates, versus $550.00 on Claude Mythos Preview. For a security incident corpus, a specific repository snapshot, or a bounded contract set, that is a single API call with no embedding pipeline, no vector database, no re-indexing. Argon’s cached input tokens at 95% off input make repeated reads of the same long artifact even more plausible for continuous analysis. The boundary condition is boundedness: whole-document prompting works when the corpus fits under the window and is stable enough that re-reading the whole thing is not waste.
Retrieval still owns large, changing corpora because indexing cost is paid once
pplx-embed-v2-late is not a single-vector compressor. It keeps one 128-dimensional vector per token and scores with MaxSim: each query token can match its strongest document token rather than collapsing a document into one embedding. This is constrained inference, not raw model scale: the 0.6B encoder needs about 1.2 GB of bf16 weight memory, while the 9B indexer needs 16–18 GB, so the sensible topology is offline 9B indexing plus live 0.6B query encoding. On ViDoRe v3 image retrieval, querying a 9B-encoded index with the 0.6B encoder scores 63.5%, above the 0.6B symmetric baseline of 62.3%. The main trade is storage, because index size grows linearly with document length. But for a large corpus, high query volume, or frequent edits, the fixed offline indexing cost is amortized; whole-document prompting would re-upload the full corpus on every query. Updates become an offline re-indexing task rather than a repeated per-query token cost. That is why retrieval remains the economics winner at scale before generation quality enters the calculation.
The break-even framework: corpus size, query volume, update frequency
The decision reduces to three variables: corpus size, query volume, and update frequency.
| Decision variable | Whole-document prompting | RAG with pplx-embed-v2-late | Break-even lever |
|---|---|---|---|
| Corpus size | Cost per query = input price × document tokens; bounded by window | Index cost once; query cost tied to retrieved passage count | Larger corpus favors RAG |
| Update frequency | Every new version re-read in full at each query | 9B indexing runs offline; live query uses 0.6B encoder | Frequent updates favor RAG |
| Query volume | Full document tokens paid each query | Index amortized over many queries | High query volume favors RAG |
| Cost evidence | Haiku 5.5: $0.10 / $0.50 input tiers; OrcaCyber: $3.00/$7.50; Argon intro $2.00 input | MIT weights; 0.6B ≈1.2 GB, 9B ≈16–18 GB; 63.5% asymmetric ViDoRe v3 | Fixed index vs per-query token spend |
The strongest case against this framework is that cheap million-token windows make RAG overhead unnecessary for many workloads. If Haiku 5.5 can read 1M tokens for $0.50 at long-prompt rates, why maintain embeddings, vector storage, and an indexing pipeline? The answer is repeat-query economics: whole-document prompting pays the full corpus token cost on every query, while retrieval pays the fixed index cost once and then only retrieved tokens per query. For a large corpus with many queries, the second curve crosses below the first. For frequent updates, re-indexing is offline rather than repeated document reads. The framework breaks exactly where the cost curve flips: if a corpus stays small, query volume stays low, and the material rarely changes, fixed indexing cost can exceed repeated whole-document reads, and RAG becomes overhead.
What would have to be true for this argument to be wrong: context tokens would need to become cheap enough and 1M-token latency/output fast enough that reading the full corpus per query is always below index storage and query-hop costs; or retrieval indexing would need to stay expensive enough that no realistic query volume amortizes it. Today Haiku’s above-100K tier is still $0.50 per million input, and pplx’s open-weights model is not yet an API, so teams carry GPU and storage costs locally. The framework stands until those costs converge. This is pipeline architecture, not better models, applied to retrieval economics.
Frequently asked questions
What does Claude Haiku 5.5 cost for long-context input?
Anthropic lists $0.10 per million input tokens for prompts up to 100,000 tokens and $0.50 per million above that threshold. Output costs $0.50 per million below 100K and $2.50 above it, with a 1M-token context window.
How does pplx-embed-v2-late improve retrieval at scale?
It is an MIT-licensed ColBERT-style pair: a 0.6B query encoder and a 9B indexer sharing one embedding space. It stores a 128-dimensional vector per token and uses MaxSim scoring, so query tokens can match strong document tokens instead of relying on a single document vector.
When should I use whole-document prompting instead of RAG?
Use whole-document prompting for bounded, stable material that fits near the context window and is queried infrequently. At Haiku 5.5's $0.10–$0.50 input tiers, a single full read can be cheaper than building and maintaining an index.
What is the OrcaCyber Zero 1.5 cost advantage?
OrcaRouter lists Zero 1.5 at $3.00 input and $7.50 output per million tokens, with a 1M-token context. Its calculator estimates a 10M monthly workload at $43.50 at list rates versus $550.00 on Anthropic's Claude Mythos Preview.
Is Gemini 4 Argon available as an API for production use?
No. At launch it is rolling out to higher-tier Pro and Ultra subscribers in the Gemini app, AI mode in Search, and Ultra subscribers on Antigravity. It is not open to API developers, though it lists $2.00 introductory input pricing.
What does the 63.5% pplx-embed-v2-late result mean?
It means an offline 9B-built index can be queried by the smaller 0.6B model and still outperform the all-0.6B setup on ViDoRe v3 image retrieval. The reported 63.5% is above the 62.3% symmetric baseline.
Related Reading
Sentence Transformers v6.0 Adds ColBERT Training in 14.5 Hours on One GPU
Sentence Transformers v6.0 adds MultiVectorEncoder with full ColBERT-style training. A single RTX 3090 run beats every zero-shot retriever by 0.06 NDCG@10.
Row-Level RAG Chunks Cut Table Context by 7.7x
Serialising each table row as its own chunk returns 122 chars instead of 943 on a targeted lookup — a 7.7x context reduction with no parser changes.
Build a Hybrid Sparse-Dense RAG Search Pipeline in Milvus
Configure Milvus with BGE-M3 dense and sparse embeddings, dual indices, and Reciprocal Rank Fusion for resilient lexical and semantic retrieval.