# Why fixed-size chunking destroys tables in your documents, and how to chunk them correctly > Source: https://netray.co/troubleshoot/chunking-splitting-tables-badly > Publisher: Netray (netray.co), AI and integration engineering for ERP and manufacturing systems. > Section: On-Prem AI Troubleshooting ## Short answer Generic character-count or token-count chunking treats a document as an undifferentiated stream of text, so it routinely cuts a table's header row away from its data rows, leaving a retrieved chunk full of numbers with no column labels to explain what they mean. The fix is structure-aware chunking that detects table boundaries, keeps the header attached to every chunk of that table's rows, and never splits mid-row. ## Error Title Chunking splits tables and headers, destroying their meaning ## Error Signature RAG answers are wrong for anything in a table even though the table is in the source document ## Alternate Signatures - chunker splits a table row from its header - retrieved chunk has numbers with no column labels - table data returns garbled or headerless results in RAG ## Category rag-retrieval ## Stack - langchain - vector-db ## Affects Any RAG pipeline ingesting documents with tables, specs sheets, price lists, or structured data, regardless of chunking library used. ## Quick Fix ### Fastest path to correct table chunking **Steps:** - Switch from naive character or token-count splitting to a structure-aware chunker that detects markdown tables, HTML tables, or PDF table regions as distinct units. - For every chunk derived from a table, repeat the header row or a compact column-label summary at the top of the chunk, even if that means some duplication across chunks. - Never split a table mid-row; if a table must be split across multiple chunks, split between row groups and repeat the header in each resulting chunk. - Keep section headings attached to the content beneath them so a chunk of body text always retains its parent heading for context. - Re-chunk and re-index any previously ingested documents that contain tables using the new structure-aware logic; old chunks cannot be patched in place. ## Symptoms - Questions about specific table values get wrong or made-up answers even though the correct table is in the corpus - Retrieved chunks contain rows of numbers or short codes with no visible column headers or labels - A chunker's output for a table looks like a fragment of one row plus a fragment of the next, cut at an arbitrary character count - Tables that fit entirely within one chunk answer correctly, but larger tables split across chunks fail - Answers about a document section are correct for prose paragraphs but consistently wrong for anything under a heading followed by a table ## Causes ### Chunking splits purely on character or token count with no awareness of document structure **Likelihood:** most-common **Explanation:** A naive splitter has no concept of what a table is. It will cut through a table exactly where the count threshold falls, which is essentially random with respect to row and header boundaries, frequently separating column labels from the data rows that give them meaning. **Fix:** Use a document parser that identifies table regions (markdown table syntax, HTML table tags, or a PDF layout model's table detection) as atomic or semi-atomic units, and chunk around them rather than through them. ### Table headers are not repeated when a table is split across multiple chunks **Likelihood:** common **Explanation:** Even structure-aware chunkers can produce multiple chunks from one large table if it exceeds the target chunk size. If the header row is only included in the first chunk, every subsequent chunk contains data rows with no column labels, making them uninterpretable in isolation to both a retriever and the generating LLM. **Fix:** Explicitly repeat the header row, or a short synthesized column-label line, at the top of every chunk derived from the same table, accepting the modest token duplication cost as necessary overhead. ### Section headings are stripped or separated from the body content during chunking **Likelihood:** common **Explanation:** Splitters that operate on raw extracted text without heading awareness can place a heading at the end of one chunk and the actual content beneath it in the next chunk, so neither chunk alone conveys what the content is about, which hurts both embedding relevance and answer quality. **Fix:** Use a heading-aware or markdown-aware splitter that keeps each heading attached to the paragraphs and tables beneath it within the same chunk, or at minimum repeats the heading text at the top of every chunk under that section. ### PDF-to-text extraction flattens tables into unstructured text before chunking ever sees table boundaries **Likelihood:** occasional **Explanation:** If the ingestion pipeline extracts raw text from a PDF without preserving table structure, the chunker downstream has no signal left to detect a table at all; it just sees text with irregular spacing, and structure-aware chunking has nothing to key off. **Fix:** Use a PDF extraction tool or layout model that outputs structured table data (as markdown tables or a row/column data structure) rather than plain flattened text, so table structure survives into the chunking stage. ## Diagnostics ### Manually inspect chunks generated from a document section containing a table **Command:** python -c "for c in chunk_document('doc_with_table.pdf'): print(c)" **Interpretation:** If any printed chunk contains numeric table rows with no visible header text, this confirms the chunker is splitting tables without preserving headers. ### Check whether the PDF extraction step preserves table structure at all **Command:** python -c "print(extract_text('doc_with_table.pdf')[table_start:table_end])" **Interpretation:** If the extracted text shows misaligned or run-together numbers with no clear row/column delimiters, the problem starts at extraction, before chunking even runs; fix extraction first. ### Test retrieval and generation quality on a set of table-specific questions **Command:** python eval_table_qa.py --questions table_questions.jsonl **Interpretation:** A large accuracy gap between prose questions and table questions confirms the issue is table handling specifically, not general retrieval quality, and justifies prioritizing structure-aware chunking over other fixes. ## Prevention - Choose a document parser and chunker combination that explicitly supports table detection before building the rest of the pipeline around it. - Always repeat table headers in every chunk derived from that table, and treat this as a hard rule in the chunking code, not an edge case. - Test chunking quality specifically on documents containing tables as part of routine ingestion pipeline validation. - Keep headings attached to their content through the entire pipeline, from extraction through chunking through indexing. - Re-validate chunking output any time the source document format changes, such as a new PDF template or export tool. ## When To Escalate If your corpus includes complex or nested tables, engineering drawings with embedded data tables, or scanned documents where table structure has to be reconstructed with a vision model rather than parsed directly, structure-aware chunking becomes a genuinely hard document-understanding problem rather than a chunker configuration change, and is worth scoping as its own workstream before it blocks the rest of the RAG rollout. ## Faq ### Does duplicating table headers in every chunk waste too many tokens? **Answer:** The overhead is usually small, typically one short row of column labels per chunk, and it is far cheaper than the cost of wrong answers on table-derived questions. For very wide tables, synthesize a compact column-label summary rather than repeating the full header verbatim if token budget is tight. ### Should tables be chunked separately from surrounding prose? **Answer:** Generally yes. Treating a table as its own chunk rather than merging it with surrounding paragraph text usually improves embedding quality, since the vector isn't diluted by unrelated prose, and improves answer quality, since the model gets a clean, self-contained table to reason over. ### Can I fix this without re-ingesting my whole document set? **Answer:** No. Chunking happens at ingestion time, so any documents already chunked with the old, structure-unaware method have corrupted table chunks baked into the vector index. You need to re-chunk and re-embed those documents, though you can usually detect which documents contain tables programmatically to avoid re-processing everything. ### What is the best chunk size for a table-heavy document? **Answer:** Size chunks around whole tables or logical row groups rather than a fixed token count; a small table might fit in one chunk with room for surrounding context, while a large table should be split at row-group boundaries with the header repeated, rather than forcing every chunk to hit the same token target. ## Search terms chunking splits tables, RAG table extraction, structure aware chunking, table header chunk, PDF table RAG, markdown table chunking, chunking strategy for tables, RAG document structure --- Talk to an engineer: https://netray.co/contact?intent=erp_ai | Machine catalog: https://netray.co/agent/catalog.json