How Does Document Chunking Work? How Does Search Happen When a Query Arrives?

Step-by-Step Explanation

Introduction — Why Should You Learn RAG?

Nowadays, many systems such as AI chatbots, document search systems, and company AI assistants are built using RAG (Retrieval-Augmented Generation).

If you are:

• Learning Data Science
• Interested in AI/LLM Engineering
• Planning to build chatbots or document search systems

then understanding RAG is an extremely important skill.

In this blog, we will explain RAG step by step in a way that even a complete beginner can understand.

What Is RAG? — Simple Definition

RAG = Retrieval + Generation

This means:

• Retrieval → Search for relevant information from documents
• Generation → Use an LLM to generate an answer

In simple words:

RAG is a system that searches for the correct information from documents before generating an answer.

Instead of allowing an AI model to simply guess an answer, RAG provides the model with relevant information from trusted documents.

Understanding RAG with a Real-Life Example

Imagine that a company has:

• 1,000 PDF files
• Company policies
• Research notes
• Internal documents

Now, a user asks:

"How many days of sick leave are allowed?"

Instead of allowing the AI to guess the answer, the system:

1. Searches for documents related to sick leave
2. Finds the relevant section
3. Reads the relevant information
4. Generates an answer based on that information

This entire process is called RAG.

Main Components of a RAG System

A typical RAG system contains these five main components:

1. Documents
2. Chunking
3. Embeddings
4. Vector Database
5. LLM

Now, let's understand each step.

Step 1 — Document Loading

First, we need a document.

For example:

• Leave Policy
• Sick Leave Policy
• Work From Home Policy

A document can contain a large amount of text.

For example:

Leave Policy

Employees can take annual leave according to company policy.

Sick Leave

Employees can take sick leave when they are ill.

Work From Home

Employees can work from home after receiving approval.

The Problem

Searching through a very large document directly can be inefficient.

Therefore, we move to the next step:

Chunking

Step 2 — Chunking — One of the Most Important Concepts

Chunking means splitting a large document into smaller pieces called chunks.

Original Document

Section 1 → Leave Policy

Section 2 → Sick Leave

Section 3 → Work From Home

After chunking:

Chunk 1

Leave Policy

Employees can take 20 days of annual leave.

Chunk 2

Sick Leave

Employees can take sick leave when they are ill.

Chunk 3

Work From Home

Employees can work from home when approved.

Why Do We Create Chunks?

Chunking helps because:

• Search becomes more efficient
• The system can find only the relevant part
• Less unnecessary information is sent to the LLM
• Retrieval accuracy can improve
• Large documents become easier to process

What Is a Typical Chunk Size?

A common starting point is:

• 300–500 words per chunk
• 50–100 words of overlap

For example:

Chunk 1 → Words 1–500
Chunk 2 → Words 450–950
Chunk 3 → Words 900–1400

Why Is Overlap Used?

Suppose an important sentence starts at the end of one chunk and continues into the next chunk.

Without overlap, the meaning may be separated.

Overlap helps preserve context and reduces the chance of losing important information during chunking.

Step 3 — Embeddings — Converting Text into Numbers

Computers cannot directly understand the meaning of text in the same way humans do.

Therefore, text is converted into a numerical representation called a vector.

For example:

Text

Sick leave is allowed when an employee is ill.

Vector

[0.78, 0.22, 0.19, 0.55, ...]

This process is called Embedding.

The vector represents the semantic meaning of the text.

Texts with similar meanings tend to have similar vector representations.

For example:

"How many sick days are available?"

and

"What is the sick leave allowance?"

use different words, but their meanings are similar.

Their embeddings should therefore be close to each other in vector space.

Step 4 — Vector Database — Storing the Embeddings

After converting all document chunks into vectors, we store those vectors in a Vector Database.

Popular vector databases include:

• FAISS
• Pinecone
• Chroma
• Weaviate

The vector database allows the system to efficiently search for chunks that are semantically similar to a user's query.

Step 5 — What Happens When a Query Arrives?

Now, suppose the user asks:

"How many days of sick leave are allowed?"

The RAG system performs several steps.

Query Processing — Step by Step

Step 1 — Convert the Query into a Vector

The user's query:

How many days of sick leave are allowed?

is converted into a vector:

Query → Query Vector

The same embedding model is generally used to represent both the document chunks and the user query.

Step 2 — Perform Vector Search

The system compares the query vector with the vectors stored in the vector database.

For example:

Chunk 1 → Annual Leave Policy
Chunk 2 → Sick Leave Policy        ← Relevant Match
Chunk 3 → Work From Home Policy

The system retrieves the most relevant chunks.

This process is called Retrieval.

Step 3 — Send the Relevant Chunks to the LLM

The system sends the user's question together with the relevant retrieved context.

Question

How many days of sick leave are allowed?

Retrieved Context

Employees are entitled to 10 days of sick leave per year.

The LLM now has the relevant information needed to answer the question.

Step 4 — Generate the Final Answer

The LLM generates an answer based on the retrieved context:

Employees are entitled to 10 days of sick leave per year.

This is the Generation part of RAG.

A Simple Real-Life Analogy

Imagine that you are inside a large library.

A user asks:

"Where can I find a Python book?"

You do not read every book in the entire library.

Instead, you:

1. Search the library index
2. Find the relevant section
3. Locate the Python books
4. Open the relevant book
5. Provide the answer

The analogy is:

Library          = Vector Database
Search           = Retrieval
Relevant Book    = Retrieved Context
Answer Generation = LLM

That is the basic idea behind RAG.

Simple RAG Pipeline Summary

The complete RAG workflow looks like this:

1. Load documents
        ↓
2. Split documents into chunks
        ↓
3. Convert chunks into embeddings
        ↓
4. Store embeddings in a vector database
        ↓
5. Receive a user query
        ↓
6. Convert the query into an embedding
        ↓
7. Search for similar/relevant chunks
        ↓
8. Send the retrieved chunks to the LLM
        ↓
9. Generate the final answer