What I Learned From Building My First RAG Application - Custom AI Chatbot

Building an AI application taught me that getting a model to generate an answer is one thing. Helping it find the right information before answering is a different challenge altogether.

When I started exploring AI applications, I was curious about how to make language models more useful in real-world situations.

AI models can answer questions remarkably well, but what happens when you want an AI assistant to answer using information from a specific website, a collection of documents, or a business's own knowledge base?

And I explored Retrieval-Augmented Generation (RAG).

It helped me understand that creating a useful AI assistant involves much more than connecting a language model to an API. Here are some of the most important lessons I learned.

1. An AI Model Doesn't Automatically Know Your Business

One of the first things I had to understand was the difference between general knowledge and application-specific knowledge.

A language model may know a great deal about a subject, but that doesn't mean it knows the latest details about a particular business, its services, its policies, or the content published on its website.

For example, imagine asking an AI assistant:

“Which services does this wellness centre offer, and what does each service involve?”

A general-purpose language model might generate a plausible answer based on its existing knowledge. However, that answer might not reflect the actual services offered by the business.

I wanted to explore an approach where the assistant could use the business's own information instead of relying entirely on what the model already knew.

This is where RAG becomes useful.

Rather than retraining a language model every time the knowledge source changes, a RAG application can retrieve relevant information from an external knowledge base and provide it to the model as context.

My biggest takeaway: An AI application becomes much more useful when it has access to the right information for the task it needs to perform.

2. Breaking Text Into Meaningful Chunks Matters

Large documents need to be divided into smaller sections, commonly called chunks, so that relevant information can be retrieved efficiently.

But splitting text isn't as simple as choosing a character limit. If a paragraph is divided incorrectly, important context can be lost.

Finding the right balance between chunk size and context is essential for effective retrieval.

3. Embeddings Help AI Find Relevant Information

I explored sentence-transformer embeddings to understand how text can be converted into numerical representations that capture semantic meaning.

This allows a retrieval system to find relevant content even when a user's question uses different words from the original document.

For example, a user asking about muscle tension might find relevant information on a website that describes treatments for muscular discomfort.

This is one of the ways semantic search can go beyond exact keyword matching.

4. A Vector Database Is Only One Piece of the Puzzle

I explored ChromaDB to store and retrieve text embeddings.

The basic RAG workflow looks like this:

User Question → Embeddings → Retrieval → Relevant Context → Language Model → Answer

Each component has a specific role. However, simply connecting these components doesn't guarantee accurate answers.

The system must retrieve the right information and provide sufficient context for the language model to generate a useful response.

5. RAG Doesn't Eliminate Hallucinations

One important lesson was that RAG can help ground AI responses in external information, but it doesn't guarantee correctness.

The system might retrieve irrelevant content, misunderstand the context, or generate information that isn't supported by the source.

This is why testing retrieval quality, providing clear instructions, and handling unanswered questions appropriately are so important.

Sometimes, the best response is to acknowledge that the available information isn't sufficient.

6. Building an AI Application Goes Beyond the Model

My exploration involved technologies such as Python, LangChain, ChromaDB, sentence-transformer embeddings, FastAPI, Next.js, and TypeScript.

Each technology contributes to a different part of the application, from processing information and retrieving relevant content to connecting the backend with the user interface.

I learned to look at AI development as a complete system rather than simply a model integration exercise.

Testing, error handling, performance, and user experience all matter.

Final Thoughts

The Real Learning Was in Connecting the Pieces in the workflow.

Building my first RAG application was an opportunity to explore how several technologies and concepts work together to solve a practical problem.

I learned that collecting information, preparing text, generating embeddings, retrieving relevant content, and producing an answer are distinct steps. Each introduces its own challenges, and each can affect the quality of the final result.

More importantly, I began to appreciate the difference between an AI demonstration that works with a few sample questions and an application designed to be genuinely useful to people.

Not every website needs an AI chatbot, but when users need to navigate large amounts of information, RAG can offer a practical way to make that knowledge more accessible.

For me, this project was an important step in exploring how AI engineering and web development can come together to build applications that solve real problems.