Fully Local RAG

pip install sentence-transformers chromadb ollama

Step 2: Local Embeddings

from sentence_transformers import SentenceTransformer

# Load the model locally
# This downloads the model to your machine once, then loads from disk
print("Loading embedding model...")
embed_model = SentenceTransformer('all-MiniLM-L6-v2')

text = "The cat sits outside"
embedding = embed_model.encode(text)

print(f"Model loaded successfully.")
print(f"Embedding dimension: {len(embedding)}")
print(f"First 5 numbers: {embedding[:5]}")

Step 3: Local Vector Store (ChromaDB)

Now we need a place to store these embeddings. We will use ChromaDB. We will configure it to save data to a folder on your computer so it persists even if you close the script.

import chromadb

# Initialize the client with a persistent path
# This creates a folder named 'local_rag_db' in your project directory
chroma_client = chromadb.PersistentClient(path="./local_rag_db")

# Create (or get) a collection
# A collection is like a table in SQL
collection = chroma_client.get_or_create_collection(name="private_documents")

print("ChromaDB initialized locally.")

Step 4: Ingesting Private Data

Let's add some "secret" data that we pretend cannot leave the building. We will manually embed the documents using our local model and then push them into Chroma.

# Our "Secret" documents
documents = [
    "Project Apollo: The launch code is 8842. Do not share.",
    "Project Zeus: The meeting is moved to Room 404.",
    "HR Policy: Lunch is free on Fridays only."
]

ids = ["doc1", "doc2", "doc3"]

# 1. Create embeddings locally
print("Embedding documents...")
embeddings = embed_model.encode(documents)

# 2. Add to ChromaDB
# We store the embedding AND the original text
collection.add(
    embeddings=embeddings,
    documents=documents,
    ids=ids
)

print(f"Added {len(documents)} confidential documents to local DB.")

Step 5: The Retrieval

Now, let's ask a question. We need to:

query = "What is the launch code for Apollo?"

# 1. Embed the query
query_embedding = embed_model.encode([query])

# 2. Query the database
results = collection.query(
    query_embeddings=query_embedding,
    n_results=1 # We just want the best match
)

retrieved_doc = results['documents'][0][0]
print(f"Query: {query}")
print(f"Retrieved Context: {retrieved_doc}")

Step 6: The Generation (Connecting Ollama)

Finally, we connect the retrieved context to our local LLM (Ollama). This replaces the call to GPT-4.

import ollama

def local_rag_chat(question):
    print(f"\nProcessing: {question}")
    
    # 1. Retrieve
    q_embed = embed_model.encode([question])
    results = collection.query(
        query_embeddings=q_embed,
        n_results=1
    )
    
    # Check if we found anything
    if not results['documents'][0]:
        return "I couldn't find any information on that."
        
    context = results['documents'][0][0]
    print(f"Found context: {context}")
    
    # 2. Construct Prompt
    prompt = f"""
    You are a helpful assistant. Answer the question based ONLY on the provided context.
    
    Context: {context}
    
    Question: {question}
    """
    
    # 3. Generate with Ollama
    # ensure you have run 'ollama pull llama3' in your terminal previously
    response = ollama.chat(model='llama3', messages=[
        {'role': 'user', 'content': prompt},
    ])
    
    return response['message']['content']

# Test it out
answer = local_rag_chat("What is the launch code?")
print("-" * 50)
print("FINAL ANSWER:")
print(answer)

You should see the system find the "Project Apollo" document and then Llama 3 should tell you the code is 8842. If you unplug your internet cable right now and run this block again, it will still work perfectly.

---

Now You Try

You have a working local RAG pipeline. Now, let's expand its capabilities.

In a real company, you might want to search only "HR" documents.

Currently, if the distance between the query and the document is huge (meaning they aren't related), the system still returns a document.

* Add logic: If the distance is greater than a certain threshold (e.g., 1.5), do not send the context to Ollama. Instead, return "No relevant documents found locally."

---

Challenge Project: The "Local vs. Cloud" Showdown

Your stakeholders are skeptical. They think local AI is "too dumb" compared to OpenAI. You need to create a comparison tool to prove the tradeoffs.

QUESTION 1: Who founded the company?
------------------------------------------------
LOCAL (Llama3):
Answer: The company was founded by...
Time: 4.2 seconds

CLOUD (GPT-4o):
Answer: According to the documents, the founders were...
Time: 1.1 seconds
------------------------------------------------

---

What You Learned

Today you built a fortress. You learned how to decouple your AI applications from the internet.