AI Lab 04: LLM Parameter Tuning & Hallucination Observation
- Estimated Time: 5 hours
- Tools Required: OpenAI Platform (Playground) or Cohere Dashboard (Both provide free testing credits and a no-code web interface).
- Hardware Required: A computer with an internet connection.
Lab Objectives
- Understand the core nature of Large Language Models (LLMs): They are not databases of facts, but statistical engines that predict the "next most likely piece of text (Token)."
- Experiment with and understand how core control parameters (Temperature and Top P) affect the AI's creativity and stability.
- Core Task: Induce "Hallucinations" (confident but false statements) by giving the AI tricky prompts and tweaking extreme parameters, then document how these parameters directly alter the output.
5-Hour Schedule & Step-by-Step Breakdown
Phase 1: Registration & Playground UI Setup (0.5 Hour)
Phase 1: Registration & Playground UI Setup (0.5 Hour)
- Open your browser, register, and log in to the OpenAI Platform (requires an OpenAI account) or the Cohere Dashboard.
- Navigate to the Playground interface (make sure you are in "Chat" mode).
- Familiarize yourself with the Parameters panel on the right side:
- Model: Select the AI engine you want to drive (e.g., gpt-3.5-turbo or command-light).
- Maximum Length / Max Tokens: Limits the maximum number of words the AI can generate.
- Temperature: Controls the "randomness" of the AI's predictions. Lower values make the AI conservative and predictable; higher values make it more creative (but also more prone to losing control).
- Top P: Controls the "scope" of vocabulary selection. A low value means the AI only chooses from the most probable words, while a high value allows it to use rarer, more obscure words.
Phase 2: Parameter Battles & Hallucination Induction (3 Hours)
Phase 2: Parameter Battles & Hallucination Induction (3 Hours)
In this phase, you will conduct three comparative experiments to test the AI's limits under different configurations.
Experiment 1: The Temperature Extremes (1 Hour)
- Set the Prompt: Prepare a task that requires divergent thinking. Example: "Write a 50-word promotional copy for a coffee mug designed to be used in zero-gravity space."
- Absolute Rational Mode (Temperature = 0):
- Slide the Temperature down to 0. Click Submit to generate the text.
- Delete the result, and click Submit again. Notice how the two outputs are nearly (or entirely) identical. This proves that at Temperature 0, the AI only selects the absolute highest-probability words.
- Madness Mode (Temperature = 2 or Max Value):
- Slide the Temperature up to its maximum limit (e.g., 2.0).
- Click Submit again. Observe how the AI becomes incoherent, grammatically incorrect, or even invents non-existent words. This is the extreme manifestation of an AI completely losing its statistical constraints. (Take a screenshot of this absurd result).
Experiment 2: The Top P Vocabulary Funnel (1 Hour)
- Reset Parameters: Change Temperature back to the default 1.0.
- Test Low Top P (Top P = 0.1):
- Enter the prompt: "Write a short poem about autumn leaves."
- Set Top P to 0.1 and generate. Observe how the vocabulary is highly generic, common, and lacks literary flair.
- Test High Top P (Top P = 1.0):
- Change Top P back to 1.0 and regenerate. Notice if the AI begins to utilize more sophisticated, rare, or symbolic vocabulary to describe the same scene.
Experiment 3: Inducing "Factual Hallucinations" (1 Hour)
- Reset Parameters: Set Temperature to 0.7 and Top P to 1.0.
- Design a Trap Question: Intentionally ask the AI about an event that sounds plausible but never actually happened in history.
- Example: "Please describe in detail the historical event of President Abraham Lincoln traveling to Japan aboard the Titanic in 1903."
- Or: "Summarize the research paper that won Jay Chou the Nobel Prize in Physics in 1980."
- Observe & Record: Does the AI honestly tell you "this never happened," or does it confidently fabricate a realistic-sounding fake history (Hallucination)? (Take a screenshot documenting its fabricated story).
Phase 3: Writing the AI Lab Journal (1.5 Hours)
Phase 3: Writing the "AI Lab Journal" (1.5 Hours)
Complete an "AI Lab Journal" in a Word or Markdown file. This will serve as tangible proof of your 10-hour weekly course workload. The journal must follow the structure below:
AI Lab Journal Report
AI Lab Journal: LLM Parameter Tuning & Hallucination Report
1. Lab Setup & Key Definitions
- Platform & Model Used: [e.g., OpenAI Playground / gpt-3.5-turbo]
- Key Definition: What is a "Token"? Explain what a Token is in 1–2 sentences using your own words.
2. Temperature Comparison Table
| Test Round | Temperature | Top P | Prompt Used | Output Description (Stable / Creative / Broken) |
|---|---|---|---|---|
| Rational Mode | 0.0 | 1.0 | [Your Prompt] | Generates the exact same text every time; accurate but conservative. |
| Madness Mode | 2.0 | 1.0 | [Your Prompt] | [Describe how the AI broke down or generated gibberish] |
3. Test Results & Screenshot Evidence
- Temperature Breakdown Test: Use about 100 words to describe what happened to the sentence structure when you maximized the Temperature.
[Insert a screenshot of the incoherent AI output here]
- Factual Hallucination Test: Describe the trap question you designed. How did the AI respond? Did it fall for the trick?
[Insert a screenshot of the AI confidently fabricating fake facts here]
4. Core Reflections
- Question 1: If you were building an internal "Legal Contract Review AI Assistant" for your company, would you set the Temperature closer to 0 or closer to 1? Why?
- Question 2: Through Experiment 3, you witnessed firsthand that AI can completely fabricate facts. What is the most important warning this gives you regarding your future use of ChatGPT for studies or work?
🎯 Pro-Tips for Students (How to Avoid Common Pitfalls)
- Keep an Eye on Your Tokens: The Playground operates on a "Pay-as-you-go" logic (though new accounts usually receive free test credits). Always monitor your token usage in the bottom right corner. If you receive a "Rate limit reached" error, simply wait a few minutes before trying again.
- Isolate Your Variables: In any scientific experiment, controlling variables is crucial. When testing the effects of Temperature, keep Top P locked at 1.0. When testing Top P, keep Temperature locked. Do not change both at the same time, or you won't know which parameter caused the output to change!