RAG-as-a-service: benchmarking built-in search vs Pinecone + OpenAI embeddings

Every team building an agent eventually copy-pastes the same six-step pipeline:

1. Chunk the docs.
2. Embed each chunk.
3. Push embeddings into a vector DB.
4. Embed the query at request time.
5. Top-K lookup.
6. Re-rank, format, cite.

That stack is fine. It is also a year of fixed costs (vector DB tier, embedding spend, re-index pipeline) for a product that just wants "answer questions about my docs."

Every more.md profile ships a /search endpoint. We measured it against the canonical pipeline on a 350-page knowledge base.

The harness

• Corpus: 347 pages, ~1.1M tokens, mixed Markdown + tables.
• Queries: 200 hand-labelled with one or more correct citations.
• Metric: MRR@5 (mean reciprocal rank of first correct hit), end-to-end latency p50 / p95, $ per 1k queries.
• Stacks compared:
  • more.md: GET /u/<entity>/search?q=... (no setup; cold cache).
  • Pinecone + OpenAI: text-embedding-3-small, Pinecone serverless, chunk size 512, overlap 64, re-rank with a small cross-encoder.

Numbers

Stack	MRR@5	p50	p95	$/1k queries	Setup time
more.md (built-in)	0.71	64ms	142ms	$0.00*	0 minutes
Pinecone + OpenAI embeddings	0.74	88ms	210ms	$0.62	1–2 days

*more.md /search is included in the hosted plan and free at small volumes; the underlying compute lives in your tier price.

The 0.03 MRR gap is real and explained by the absence of a re-rank step in the W1 implementation. We are wiring a small cross-encoder into the W2 release; preliminary numbers close the gap.

Why this works at all

more.md indexes every page on publish. The index is a tuned BM25 + sparse overlay over chunked Markdown, with the citation always being the canonical <entity>/<page>#L<line> URL. No vector DB, no embedding spend, no re- index job.

For most "answer questions about my docs" workloads, the right stack is zero infrastructure. For long-tail queries that need semantic retrieval, you can layer your own vector store on top of the same chunked Markdown. /search returns the chunks too.

When you still want vectors

• Multi-modal corpora (images, audio).
• Cross-corpus federation (your docs + a partner's docs).
• Heavy semantic-similarity workloads (deduplication, clustering).

For everything else, start with GET /search?q=... and only reach for a vector DB when the metric proves you need one.