Three questions to answer before adding RAG
Most RAG failures start in the documents, not in the retrieval.
Contents
Adding retrieval makes it feel like the model knows your data. In practice you get stuck in three places.
1. Can a person answer it by reading that document?
If a human cannot find the answer, retrieval will not find it either. RAG improves retrieval quality; it does not write your documents.
2. Does your chunking match the document structure?
Splitting by character count cuts tables in half, and half a table helps no question. Split on headings, and keep a note of which section each chunk came from.
3. Which is cheaper — a wrong answer or "I don't know"?
Without this answer you cannot set a retrieval threshold. For customer-facing replies, "I don't know" is cheap. For internal exploration, it is not. The answer can differ per screen inside one product.
Pick the vector store after those three. Reversing the order means choosing a tool first and then trimming requirements down to what that tool can do.