Chunking and Parsing

Chunking and parsing configuration

When documents are ingested into an In-House knowledge base, they are split into chunks before being embedded and stored in the vector index. Chunking configuration directly affects retrieval quality: how much context each result carries, how precisely the search can target specific passages, and how much overlap exists between adjacent results.

Chunking Parameters

Chunking is configured at knowledge base creation time and applies to all documents ingested into that knowledge base.

Max Tokens per Chunk

The maximum number of tokens allowed in a single chunk. Range is 100 tokens to the upper limit set by the embedding model.

  • Smaller chunks (300–700 tokens) yield more focused retrieval. Queries return the specific passage most relevant to the question, with less surrounding noise.

  • Larger chunks (1000–2000 tokens) preserve more context per result. Useful when answers require understanding a broader passage, such as multi-step procedures or policy sections.

The form auto-fills max tokens to the selected embedding model’s limit when you choose a model. You can reduce it manually.

Overlap Percentage

The percentage of a chunk’s content that is repeated at the start of the next chunk. Range is 5–50%.

Overlap prevents information at a chunk boundary from being split across two chunks and missed during retrieval. For example, if a sentence spans the end of chunk A and the start of chunk B, with overlap both chunks will contain that sentence, ensuring it is retrieved regardless of which chunk is ranked higher.

  • Lower overlap (5–10%) reduces storage and indexing cost.

  • Higher overlap (20–40%) reduces the risk of boundary information being missed, at the cost of some redundancy.

Embedding Model

The embedding model is set at creation time and cannot be changed. Each model has a maximum token limit that caps the Max Tokens per Chunk value. Switching models requires creating a new knowledge base, since existing embeddings are incompatible with a different model’s vector space.

Parsing Strategy

The parsing strategy (set at creation) determines how documents are processed before chunking:

With LLM

An LLM parses the document content before chunking. This produces higher-quality text extraction from complex layouts — PDFs with multi-column text, documents containing tables, scanned images with embedded text, and presentations with mixed content. LLM parsing adds ingestion time and cost.

Without LLM

Plain text extraction is used. Faster and less expensive. Works well for clean, text-based files such as Markdown, plain-text PDFs, and Word documents with standard formatting.

The parsing strategy cannot be changed after a knowledge base is created.

Chunking Strategies (Bedrock)

For Bedrock knowledge bases, the chunking strategy is configured as part of the data source setup and offers more options:

  • Fixed Size — chunks are split at a fixed token count with a configurable overlap.

  • Hierarchical — documents are split into parent and child chunks. Parent chunks provide broader context; child chunks are used for retrieval. Useful for complex documents where both context and precision matter.

  • Semantic — chunks are split at semantic boundaries (paragraph and sentence breaks) rather than at fixed token counts. Results in more natural chunks.

Best Practices

  • Run a retrieval test after ingestion (see Retrieval Testing) to verify that your chunking settings produce the expected results before connecting the knowledge base to an agent.

  • If retrieval is returning off-topic passages, reduce Max Tokens per Chunk and re-ingest.

  • If answers are incomplete because relevant context is spread across multiple results, increase Max Tokens per Chunk or increase Overlap Percentage.

  • For technical documentation with dense cross-references, hierarchical chunking (Bedrock) often outperforms fixed-size chunking.