Open Source Models via APIs

pip install --upgrade openai python-dotenv

import os
import time
from dotenv import load_dotenv
from openai import OpenAI

# Load environment variables
load_dotenv()

# Get the Groq API key
api_key = os.getenv("GROQ_API_KEY")

if not api_key:
    print("Error: GROQ_API_KEY not found in .env file")
    exit()

# THE MAGIC TRICK:
# We use the standard OpenAI client, but we change the base_url.
# This tells the library: "Use the OpenAI format, but send the data to Groq."
client = OpenAI(
    api_key=api_key,
    base_url="https://api.groq.com/openai/v1"
)

print("Client initialized successfully linked to Groq!")

Step 2: Your First Llama 3 Call

Add this to your code:

try:
    print("\nSending request to Llama 3...")
    
    completion = client.chat.completions.create(
        model="llama3-8b-8192",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Explain quantum computing in one sentence."}
        ]
    )

    print("\nResponse:")
    print(completion.choices[0].message.content)

except Exception as e:
    print(f"An error occurred: {e}")

Run this code. You should see a response almost instantly.

Step 3: Measuring the Speed

The main reason we are using Groq today is speed. But "fast" is subjective. As an engineer, you need to measure it.

We will wrap our call in a timer. We will measure the "wall clock time"—how long it takes from the moment we send the request to the moment we get the full answer.

Replace the previous request code with this detailed measurement block:

print("\n--- Speed Test ---")
prompt = "Write a short poem about a robot running very fast."

start_time = time.time()

completion = client.chat.completions.create(
    model="llama3-8b-8192",
    messages=[
        {"role": "user", "content": prompt}
    ]
)

end_time = time.time()
duration = end_time - start_time

content = completion.choices[0].message.content
token_count = completion.usage.completion_tokens

print(f"Prompt: {prompt}")
print(f"\nResponse:\n{content}\n")
print("-" * 30)
print(f"Time taken: {duration:.4f} seconds")
print(f"Tokens generated: {token_count}")
print(f"Speed: {token_count / duration:.2f} tokens per second")

Step 4: The Speed Chat Loop

Now, let's turn this into a proper chatbot. We want a continuous loop where we can chat with Llama 3. Because it is so fast, it feels much more conversational than older models.

We will also add a system prompt to give Llama a personality. Llama 3 is very obedient to system prompts.

def start_speed_chat():
    # 1. Define the system personality
    system_prompt = {
        "role": "system", 
        "content": "You are a concise, witty AI. You answer quickly and get straight to the point."
    }
    
    chat_history = [system_prompt]

    print("\n=== Llama 3 Speed Chat (Type 'quit' to exit) ===")
    
    while True:
        # 2. Get user input
        user_input = input("\nYou: ")
        
        if user_input.lower() in ['quit', 'exit']:
            print("Goodbye!")
            break
        
        # 3. Add user message to history
        chat_history.append({"role": "user", "content": user_input})
        
        try:
            start_ts = time.time()
            
            # 4. Make the API Call
            response = client.chat.completions.create(
                model="llama3-8b-8192",
                messages=chat_history,
                temperature=0.7,
                max_tokens=1024
            )
            
            end_ts = time.time()
            
            # 5. Extract and print response
            ai_message = response.choices[0].message.content
            print(f"Llama ({end_ts - start_ts:.3f}s): {ai_message}")
            
            # 6. Add AI response to history so it remembers context
            chat_history.append({"role": "assistant", "content": ai_message})
            
        except Exception as e:
            print(f"Error: {e}")

# Run the function
if __name__ == "__main__":
    start_speed_chat()

Common Mistakes

Now You Try

You have a working speed chat. Now let's push the boundaries.

Groq hosts other models besides Llama 3. They also host Mistral (another famous open-source model from France).

* Find the model ID for Mistral (Mixtral-8x7b) in the Groq documentation.

* Modify your code to ask the user at the start: "Choose your fighter: (1) Llama 3 or (2) Mistral".

* Pass the chosen model string to the API call.

Llama 3 is very sensitive to system prompts.

Even though Groq is free right now (or very cheap), let's practice tracking usage.

* When the user quits, print: "Total session tokens used: X".

Challenge Project: The "Drop-in" Test

The ultimate test of the "OpenAI Compatible" concept is to take code that was written specifically for OpenAI and make it run on Groq with minimal changes.

Take your chatbot script from Day 22 (The CLI Chatbot with History). Make it run on Groq's Llama 3 model.

client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))
# ... later ...
model="gpt-3.5-turbo"

client = OpenAI(
    api_key=os.getenv("GROQ_API_KEY"), 
    base_url="https://api.groq.com/openai/v1"
)
# ... later ...
model="llama3-8b-8192"

* If your Day 22 code used specific OpenAI features like "Functions" or "JSON Mode," they might behave slightly differently, but standard chat will work perfectly.

What You Learned

Today you broke the chains of vendor lock-in. You learned: