Fine-Tuning: When & Why
What You'll Build Today
Welcome to Day 61! You have come a long way. You know how to prompt models, and you know how to give them memory using RAG (Retrieval Augmented Generation).
Today, we are tackling the biggest strategic question in Generative AI: "Should I train my own model?"
Many beginners think they need to "train" a model to teach it new facts. Usually, they are wrong. Today, you are going to build a Fine-Tuning Decision Engine. This is a Python-based calculator and logic framework that analyzes a specific use case and tells you whether you should stick with Prompt Engineering, use RAG, or actually invest in Fine-Tuning.
Here is what you will learn:
* The difference between Context and Training: Why feeding a model a PDF is not the same as training it, and why that distinction saves you money.
* The "Smaller Model" Strategy: How to use fine-tuning to make a cheap, fast model outperform a smart, expensive one.
* Cost Analysis Logic: How to calculate the break-even point where training a model becomes cheaper than writing long prompts.
* Data Suitability: How to recognize when you simply don't have the right data to fine-tune effectively.
Get ready. Today we stop guessing and start calculating.
---
The Problem
Let's say you are building a customer support bot for a company called "TechWiz." TechWiz has a very specific way they want the bot to talk. They want it to be polite, use specific legal disclaimers, never use the word "guarantee," and always format the output as a specific JSON object so their frontend website can read it.
To achieve this with standard Prompt Engineering, your system prompt has to be massive.
Here is what your code currently looks like. Read this and feel the pain of the wasted tokens:
import tiktoken
# This is the "System Prompt" required to get the specific behavior we need
system_prompt = """
You are a customer support agent for TechWiz.
Tone: Professional, slightly formal, but empathetic.
Banned words: Never say "guarantee", "promise", or "fix".
Instead use: "resolve", "address", or "look into".
Formatting: You must ALWAYS reply in valid JSON.
The JSON structure must be: {"sentiment": "positive|neutral|negative", "reply": "text", "ticket_required": boolean}
Legal: If the user mentions a refund, add the refund disclaimer ID #442.
History: TechWiz was founded in 2010... (imagine 500 more words of company context)
Examples:
User: "My screen is broken."
You: {"sentiment": "negative", "reply": "I am sorry to hear your screen is damaged. We can address this.", "ticket_required": true}
User: "Thanks!"
You: {"sentiment": "positive", "reply": "You are welcome.", "ticket_required": false}
... (imagine 20 more examples to force the style)
"""
user_query = "Hi, my internet is slow."
# Let's count how many tokens we are wasting just to say "Hello"
encoder = tiktoken.encoding_for_model("gpt-4")
prompt_tokens = len(encoder.encode(system_prompt))
query_tokens = len(encoder.encode(user_query))
print(f"System Instructions Tokens: {prompt_tokens}")
print(f"Actual User Query Tokens: {query_tokens}")
print(f"Percentage of cost that is overhead: {prompt_tokens / (prompt_tokens + query_tokens) * 100:.1f}%")
There has to be a way to "bake" these instructions into the model so we don't have to repeat them every time. That is Fine-Tuning.
---
Let's Build It
We are going to build a Python tool that helps us decide when to switch from the method above (Prompt Engineering) to Fine-Tuning.
Step 1: Defining the Strategy Class
We need a place to store our assumptions about costs and performance. We will compare two strategies:
class StrategyAnalyzer:
def __init__(self):
# Costs are per 1,000 tokens (approximate market rates)
# Strategy A: Big Model + Big Prompt (e.g., GPT-4)
self.pe_input_cost = 0.03 # $0.03 per 1k tokens
self.pe_output_cost = 0.06 # $0.06 per 1k tokens
# Strategy B: Small Model + Fine-Tuned (e.g., GPT-3.5-Turbo FT)
# Note: Fine-tuned models are usually MORE expensive than base models per token,
# but we use fewer tokens and a cheaper base model class.
self.ft_training_cost = 0.008 # One-time cost per 1k training tokens
self.ft_input_cost = 0.003 # $0.003 per 1k tokens
self.ft_output_cost = 0.006 # $0.006 per 1k tokens
def describe(self):
print("Strategy Comparison initialized.")
print(f"Prompt Engineering Input Cost: ${self.pe_input_cost}/1k tokens")
print(f"Fine-Tuning Input Cost: ${self.ft_input_cost}/1k tokens")
analyzer = StrategyAnalyzer()
analyzer.describe()
Step 2: Calculating Prompt Engineering Costs
Let's calculate the cost of running the "Pain" scenario from earlier for one month.
* System Prompt: 800 tokens (The instructions)
* User Query: 50 tokens (The question)
* Model Response: 100 tokens (The answer)
* Volume: 10,000 queries per month
def calculate_pe_monthly_cost(analyzer, num_queries):
# Inputs
system_instructions = 800
user_query_avg = 50
response_avg = 100
# Total input per call
total_input_tokens = system_instructions + user_query_avg
# Calculate costs
# Divide by 1000 because prices are per 1k
cost_input = (total_input_tokens / 1000) * analyzer.pe_input_cost
cost_output = (response_avg / 1000) * analyzer.pe_output_cost
cost_per_call = cost_input + cost_output
total_monthly = cost_per_call * num_queries
return total_monthly
monthly_pe_bill = calculate_pe_monthly_cost(analyzer, 10000)
print(f"Monthly bill using Prompt Engineering: ${monthly_pe_bill:.2f}")
system_instructions (800) is much larger than the query (50). You are paying mostly for context, not content.
Step 3: Calculating Fine-Tuning Costs
If we fine-tune, we can remove almost all the system instructions because the model has "learned" the persona. We only need a tiny reminder.
* New System Prompt: 50 tokens (Just "You are the TechWiz bot.")
* Training Data: We need to train it first. Let's say we have 500 example conversations (approx 1000 tokens each).
def calculate_ft_costs(analyzer, num_queries):
# 1. One-time Training Cost
training_examples = 500
tokens_per_example = 1000
total_training_tokens = training_examples * tokens_per_example
# We usually train for 3 "epochs" (passes over the data)
epochs = 3
training_bill = (total_training_tokens epochs / 1000) analyzer.ft_training_cost
# 2. Recurring Inference Cost (Monthly)
# Notice: System instructions drop from 800 to 50!
system_instructions = 50
user_query_avg = 50
response_avg = 100
total_input_tokens = system_instructions + user_query_avg
cost_input = (total_input_tokens / 1000) * analyzer.ft_input_cost
cost_output = (response_avg / 1000) * analyzer.ft_output_cost
monthly_bill = (cost_input + cost_output) * num_queries
return training_bill, monthly_bill
train_cost, monthly_ft_bill = calculate_ft_costs(analyzer, 10000)
print(f"One-time Training Cost: ${train_cost:.2f}")
print(f"Monthly bill using Fine-Tuning: ${monthly_ft_bill:.2f}")
Step 4: The Break-Even Analysis
Now, let's write the logic that compares them. At what point does the investment in fine-tuning pay off?
def compare_strategies(pe_monthly, ft_monthly, ft_upfront):
savings_per_month = pe_monthly - ft_monthly
print(f"\n--- Analysis ---")
print(f"Prompt Engineering/mo: ${pe_monthly:.2f}")
print(f"Fine-Tuning/mo: ${ft_monthly:.2f}")
print(f"Fine-Tuning Upfront: ${ft_upfront:.2f}")
if savings_per_month <= 0:
print("DECISION: DO NOT FINE-TUNE. Prompt Engineering is cheaper.")
return
months_to_breakeven = ft_upfront / savings_per_month
print(f"Monthly Savings: ${savings_per_month:.2f}")
print(f"Break-even point: {months_to_breakeven:.1f} months")
if months_to_breakeven < 6:
print("DECISION: FINE-TUNE. ROI is positive in less than 6 months.")
else:
print("DECISION: HOLD. ROI takes too long.")
compare_strategies(monthly_pe_bill, monthly_ft_bill, train_cost)
You should see that despite the upfront cost, the fine-tuned model pays for itself very quickly (often in less than a month) because the per-call cost is drastically lower due to the reduced prompt size and cheaper model architecture.
Step 5: The Qualitative Filters (The "When" Logic)
Cost isn't the only factor. Sometimes you cannot fine-tune, even if it's cheaper. Let's add a logic gate for data suitability.
def suitability_check(use_case_type, data_available_count, knowledge_cutoff_sensitive):
"""
use_case_type: 'style', 'format', or 'knowledge'
data_available_count: number of high-quality examples
knowledge_cutoff_sensitive: Boolean (does the info change often?)
"""
print(f"\nChecking suitability for: {use_case_type}...")
# check 1: Data Volume
if data_available_count < 50:
return "STOP: Not enough data. You need at least 50-100 high quality examples."
# Check 2: Knowledge vs Style
if use_case_type == 'knowledge':
if knowledge_cutoff_sensitive:
return "STOP: Use RAG. Fine-tuning is bad for rapidly changing facts."
else:
return "CAUTION: Fine-tuning for facts is hallucination-prone. Use RAG unless facts are static."
# Check 3: Style/Format (The Sweet Spot)
if use_case_type in ['style', 'format']:
return "GO: Fine-tuning is excellent for style, tone, and strict formatting."
return "GO: Proceed to cost analysis."
# Test cases
print(suitability_check('format', 500, False))
print(suitability_check('knowledge', 1000, True))
---
Now You Try
You have the basic calculator. Now extend it to handle real-world complexities.
Currently, our calculator assumes the data is ready. In reality, creating 500 perfect examples takes human time.
* Add a variable hourly_wage ($50/hr) and minutes_per_example (5 mins).
* See how this impacts the break-even point.
Sometimes you need RAG and Fine-Tuning (e.g., a specific style, but accessing today's news).
* Create a function calculate_hybrid_cost.
* It should include the Fine-Tuning inference cost PLUS a Vector Database cost (approx $0.002 per query).
* Compare this against pure Prompt Engineering.
What if OpenAI drops the price of GPT-4 by 50%?
* Wrap your compare_strategies function in a loop.
* Run it three times with different pe_input_cost values ($0.03, $0.015, $0.005).
* Print at what price point Fine-Tuning stops being worth it.
---
Challenge Project: The Consultant's Report
You are a consultant for a hospital. They want to build an AI that summarizes doctor's notes into patient letters.
* Volume: 1,000,000 letters per year.
* Requirement: The output must be extremely empathetic and follow a strict legal template.
* Current State: They use GPT-4 with a 2,000-token system prompt.
* Proposal: Fine-tune GPT-3.5 using 1,000 existing perfect letters.
Your Task:Write a Python script that generates a text report comparing the two options over a 1-year period.
Requirements:* Define a class MedicalAIProject.
* Input variables: Volume (1M), Prompt Size (2k vs 100), Model Costs (lookup current OpenAI pricing for GPT-4 vs GPT-3.5-Turbo-Fine-Tuned).
* Include a "Human Review" cost:
* Assume GPT-4 is safer and only needs 5% of letters reviewed by a human ($10/review).
* Assume GPT-3.5-FT needs 10% reviewed initially, dropping to 5% after 3 months.
* Output: A printed report showing the total Year 1 cost for both options and a final recommendation.
Hint: The "Human Review" cost will likely dwarf the API costs. This teaches you that sometimes the "expensive" model (GPT-4) is actually cheaper because it makes fewer mistakes!---
What You Learned
Today you learned that "Training an AI" is a business decision, not just a technical one.
* Prompt Engineering is great for prototyping, low volume, and tasks requiring high reasoning (GPT-4).
* Fine-Tuning is for Form, Format, and Style. It reduces latency and token costs but requires upfront investment and maintenance.
* RAG (from previous lessons) is for Facts. Never fine-tune to teach a model the news; it will hallucinate.
Why This Matters:In a real job, suggesting "Let's fine-tune a model!" is a popular idea. You now have the framework to say, "Actually, let's run the numbers first," and save your company thousands of dollars and months of wasted time.
Tomorrow: We stop simulating. You will take a raw dataset, format it intoJSONL, upload it to OpenAI, and actually trigger a fine-tuning job to create your own custom model.