Local LLMs with Ollama

ollama run llama3.1

Note: The first time you run this, it will download about 4.7GB of data. This is the "brain" of the AI.

Once it finishes, you will see a prompt. Type "Why is the sky blue?" to verify it works. Then type /bye to exit.


Step 2: Python Integration

Now that the engine is running in the background, let's control it with Python. First, install the library:

pip install ollama

Now, create a file named local_bot.py. We will replicate the "Hello World" of local AI.


import ollama

print("Thinking locally... (this relies on your hardware, not the internet)")

response = ollama.chat(model='llama3.1', messages=[
  {
    'role': 'user',
    'content': 'Explain quantum computing in one sentence.',
  },
])

print("\nResponse:")
print(response['message']['content'])


Why this matters: This code looks almost identical to the OpenAI code, but it requires no API key and sends no data outside your machine.

Step 3: Streaming for Better UX

When you run the code above, you might notice a long pause before the text appears. This is because your computer is generating the entire answer in memory before showing it to you. Cloud APIs are fast; your laptop might be slower.

To fix the "frozen" feeling, we use streaming. This prints each word (token) as it is generated.

Update local_bot.py:


import ollama

print("Ask me anything (type 'quit' to exit):")

while True:
    user_input = input("\nYou: ")
    if user_input.lower() == 'quit':
        break

    print("AI: ", end="", flush=True)

    # We enable streaming=True
    stream = ollama.chat(
        model='llama3.1',
        messages=[{'role': 'user', 'content': user_input}],
        stream=True,
    )

    # We iterate over the stream as chunks arrive
    for chunk in stream:
        print(chunk['message']['content'], end='', flush=True)
    
    print() # New line at the end


Why this matters: This makes the application feel responsive, even if your hardware is older.

Step 4: Understanding Quantization

You might wonder: "GPT-4 is massive. How does Llama 3.1 fit on my laptop?"

The answer is Quantization.

Standard models store numbers (weights) as 16-bit or 32-bit floating-point numbers. That takes up a lot of space. Quantization reduces the precision of these numbers, say to 4-bits.

*   FP16 (16-bit): High precision, massive size.
*   Q4 (4-bit): Lower precision, much smaller size, surprisingly similar intelligence.

When you ran ollama run llama3.1, it defaulted to a 4-bit quantized version. This is why it is only 4.7GB instead of 15GB+.


You don't need to write code for this yet, but you need to know that q4 (4-bit) is the standard for running on laptops.


Step 5: Custom Modelfiles

This is the most powerful part of Ollama.

In the cloud, you have to send a "System Prompt" every single time you call the API (e.g., "You are a helpful assistant...").

With Ollama, you can save a System Prompt into a new model. This creates a dedicated tool.

1. Create a file named Modelfile (no extension) in your project folder.

Put this content inside it:

FROM llama3.1

PARAMETER temperature 0.1

SYSTEM """

You are Mario from the Super Mario Bros games.

You must start every response with "It's a-me, Mario!"

You use Italian-American slang and refer to the user as 'Player One'.

Keep answers short and enthusiastic.

"""

2. Build the model.

Run this command in your terminal (not Python, your actual terminal):

ollama create mario-bot -f Modelfile

import ollama

response = ollama.chat(model='mario-bot', messages=[
  {
    'role': 'user',
    'content': 'How do I bake a cake?',
  },
])

print(response['message']['content'])

"It's a-me, Mario! To bake a cake, Player One, you gotta mix the flour, eggs, and sugar like you're mixing power-ups! Put it in the oven until it rises like a platform! Wahoo!"

Now You Try

Here are three extensions to solidify your skills:

Challenge Project: The Local vs. Cloud Benchmark

You are going to build a tool to make business decisions. Your boss wants to know: "Is local AI good enough, or do we need to pay for GPT-4?"

* Define a complex prompt (e.g., "Write a 200-word email apologizing to a client for a delay caused by a supply chain issue").

* Record the start time and end time for the Local Llama generation.

* Print the outputs side-by-side.

* Print the time taken for each.

--- Local Llama 3.1 ---
Time taken: 4.2 seconds
Output: Dear Client, I am writing to sincerely apologize...

--- OpenAI GPT-4o-Mini ---
Time taken: 1.1 seconds
Output: Dear Valued Client, We regret to inform you...

--- Conclusion ---
Local model was 3.1 seconds slower.

(Note: You will likely find the Cloud is faster, but Local is free. This is the trade-off you need to understand.)

What You Learned

Today you broke the dependency on the cloud.

* Privacy: You processed data without it ever leaving your laptop.