DuskByte

When not to use a vector database

November 16, 20268 min read

Vector search gets recommended far more often than it is needed. This article sets out five situations where an index is the wrong answer, the row count where that changes, and a four step ladder that runs from database search to a dedicated vector service.

The short answer

Below roughly 50,000 rows, do not add a vector index. Scan every row and compare. It is fast enough, it is always right, and it gives you nothing to tune.

Three questions decide the rest:

  1. How many rows will you search, and how fast do they change?
  2. Does the query carry a meaning, or an exact token such as a SKU or an error code?
  3. Can you prove, with a number, that the results got better?

If you cannot answer the third one, an index is not the first thing to build.

Why the question keeps coming up

The reasons teams reach for a vector database are sound ones.

Keyword search misses paraphrase. A customer types "warm coat for a wet commute" and the catalogue says "waterproof insulated parka". Not one word matches. The customer leaves. Embeddings solve that, and the demo is convincing, because the demo is built on the half of the problem that vector search is good at.

The other half arrives later. Identifiers. Permissions. Freshness. And the question of whether anyone can tell that the new search is better than the old one. That half is where the cost lands, and it is the half worth thinking about before you choose a database.

Case 1: the corpus is small

Under roughly 50,000 rows, an exact scan is fast enough, and it returns the right answer every time.

A scan is one multiply and one add per dimension per row. The work is linear and it spreads across cores. Published figures for the SIFT-1M benchmark, one million vectors on a Xeon, show what an index buys at that size:

IndexQueries a secondRecall
Flat, exact scan16100%
IVFabout 90096%
HNSWabout 3,00095%

That is the argument for indexing a million rows. It is not the argument for indexing fifty thousand. On those same figures a flat scan of a million vectors takes about 62 milliseconds. Twenty times fewer rows is about 3 milliseconds, on one machine, with nothing lost.

What you give up by indexing early:

Case 2: the query is a token, not a meaning

If your users search for part numbers, SKUs, invoice references or error codes, a vector index on its own will make search worse.

Embeddings put similar meanings near each other. An exact string carries no meaning to place. ERR_5521 and ERR_5512 land close together, and the wrong one comes back first. The system looks confident while it is wrong, which is harder to debug than an empty result.

This is why the production pattern in retail and support is hybrid: dense retrieval for meaning, BM25 style keyword matching for tokens, and the two fused at query time.

There is a simpler version of this advice. Read 200 real queries from your search log first. If most of them are identifiers, you do not need the dense half at all. Postgres full text search, or a well built index on the identifier column, will beat embeddings on accuracy and on running cost. We would rather say that at the start than build the wrong thing well.

Case 3: the data changes faster than the index

A vector index is a derived copy of your data. Every write has to reach it, and some index types have to be rebuilt before the copy is useful again.

HNSW takes inserts one at a time, which is part of why it became the default. Deletes are the awkward case: most engines mark a row as deleted and only reclaim the space on a rebuild. Cluster based indexes such as IVF need a training pass over a sample before they work well, so a corpus that turns over quickly needs that pass repeated.

One question settles it. How long may a user wait between a change in the source system and that change being searchable?

None of that rules out an index. It does mean the index stops being a feature and becomes infrastructure, with the staffing that implies.

Case 4: nobody can tell whether it is working

Without labelled questions you cannot tell an improvement from a regression, and every change turns into an argument about taste.

This is the most common reason a search project stalls, and it is not a speed problem. In office hours with 79 practitioners running vector search, relevance of results was reported as the biggest blocker to production, ahead of cost, latency and throughput combined.

The fix is small and dull:

  1. Take 30 real questions from your search log.
  2. For each one, write down the text that a correct answer must contain.
  3. Run all 30 on every change to chunking, model or index settings.
  4. Fail the build when the score drops.

Thirty questions take an afternoon. They are the difference between tuning and guessing. If nobody in the business will own them, that is useful information: the project is not ready, and an index will not make it ready.

Case 5: the content is the problem

Retrieval finds what exists. It cannot find what nobody wrote down.

Three versions of this turn up again and again. The answer lives in one person's head. The answer lives in a document that was correct two years ago. The answer is split across a table in a PDF and a paragraph on a wiki, and neither half makes sense alone.

The third one is worth a demonstration. In a small test we ran while writing this, one question failed, and the index was not at fault. The chunker had put the name of a setting in one chunk and the explanation of it in the next. Changing one chunking parameter, with the index and the model untouched, moved recall at 3 from 0.80 to 1.00. That was a seven chunk test corpus, not a benchmark, but the failure shape is the ordinary one.

Practitioners put roughly 80% of RAG failures in the ingestion and chunking layer rather than in the model. That figure is widely repeated and carries no published method, so treat it as a direction rather than a measurement. The direction matches what we see.

Audit the content before you buy an index. It is the less interesting half of the work and it decides more of the result.

The ladder we recommend

Start on the step that matches the problem you have today. Move up only when a measurement tells you to.

  1. Database search. Postgres full text search, or whatever your database already gives you. Handles identifiers, filters and most find-the-thing queries. No new service, no new copy of the data.
  2. Exact vector scan. Embed the rows, store the vectors in a column, compare the query against all of them. Perfect recall, nothing to tune, and it tells you whether meaning based search helps at all before you commit to infrastructure.
  3. An approximate index inside the database you already run. pgvector with HNSW. The common starting settings are m = 16 and ef_construction = 64.
  4. A dedicated vector service. Only when scale, filtering or operational load make step 3 the limit.

What moves you up a step:

MoveTrigger
1 to 2Real queries in the log miss because of paraphrase, not typos
2 to 3More than about 50,000 rows, or p95 search latency over budget
3 to 4Index memory, filtered search or reindex time becomes the constraint

Plenty of businesses belong on step 1 or step 2 and are being quoted for step 4.

When a vector index does earn its place

Four conditions. When all four hold, build it.

That describes a lot of serious systems, and we build them. The point of this article is not that vector search is overrated. It is that the index is the last decision, not the first.

Once all four hold, the interesting questions start. Which index, and under what constraint. How much memory the graph will hold at your row count. How to reindex a live system without stopping it. Those are the next articles in this series.

How we would scope this

Assessment before execution. On a search or retrieval brief, that means four things before anyone chooses a database.

  1. Read the query log. Count how many queries are identifiers and how many are sentences.
  2. Count the rows, and the growth rate, and ask how fast they change.
  3. Check that the content holding the answers exists, is current, and survives chunking.
  4. Write the 30 questions, and agree who owns them.

Then build the smallest step on the ladder that passes those tests, put it behind a measurement, and move up when the measurement says to. Changes go out in phases, each one rollback safe, each release tested before it ships.

If you are weighing a search or retrieval build on a platform that already carries revenue, our architecture review covers exactly this ground.

Want to talk through your own project?

Book a call. You'll talk to the person who'd actually architect it, not an account manager.