Chunking

Embedding models have token limits (typically 512 to 8,192 tokens), and even when they don't, embedding a whole 200-page PDF as one vector loses all locality — the vector becomes an average of everything and matches nothing well. So you chunk: split the document into smaller units, embed each one, and index them separately. This unsexy step quietly decides whether your RAG system works. Common strategies, roughly in order of sophistication: fixed-size chunking (every 500 tokens, with maybe 50-token overlap to avoid cutting mid-thought) is the default and surprisingly hard to beat. Sentence- or paragraph-boundary chunking respects natural breaks and keeps ideas intact. Recursive character splitting (LangChain's classic) tries paragraph, then sentence, then word, falling back as needed. Semantic chunking uses an embedding model to find topic shifts and splits there. Document-aware chunking respects structure — headings, tables, code blocks, list items. The hard problem is context loss. A chunk in the middle of a contract that says "the party shall pay $50,000 within 30 days" has no idea who "the party" is or what contract it's from. Solutions in 2026: prepend section headers and document titles to each chunk, store metadata (doc_id, section, page) for filtering and citation, generate "contextual chunks" where an LLM rewrites each chunk with a one-sentence summary of where it came from, or use late chunking where you embed the whole doc first and pool slices. There's no universally right chunk size. ~200 tokens is good for precise factoid retrieval (Q&A over a wiki), ~1,000 tokens is better for summarization or contextual reasoning, and code/legal often want larger units that respect structure. Test on your eval set, not vibes.