The Verification Habit

The Core Problem

AI has a dangerous trait: it's confidently wrong.

Unlike a search engine that admits "no results found," AI will always give you an answer. It will cite studies that don't exist, quote people who never said those words, and provide statistics that were invented on the spot. And it will do all of this with the same confident tone it uses for accurate information.

This is called hallucination, and it's not a bug that will be fully fixed—it's a fundamental property of how these systems work. Large language models predict what text should come next based on patterns. They don't "know" facts the way a database knows them.

The danger isn't that AI lies. It's that AI doesn't know the difference between what it knows and what it's guessing.

Your job: Be the fact-checker. The professional who uses AI effectively also uses AI skeptically.

Why Hallucinations Happen

Understanding the mechanism helps you catch the errors:

Pattern Matching, Not Knowledge Retrieval

AI doesn't look things up. It predicts what words typically follow other words based on training data. If "Einstein once said" is often followed by inspirational quotes, AI will generate an inspirational quote—whether Einstein actually said it or not.

Training Data Cutoffs

AI has a knowledge cutoff date. Anything after that date is unknown, but AI might still answer as if it knows. If you ask about a recent event, AI might generate a plausible-sounding response based on patterns rather than facts.

Confidence Without Calibration

AI isn't calibrated to express uncertainty. It doesn't say "I'm 60% sure about this." It answers confidently regardless of actual certainty. The less data AI has about a topic, the more it fills in with pattern-based guessing—without flagging that it's guessing.

Obscure Topics Are High Risk

The more niche or specialized your question, the higher the hallucination risk. AI has seen millions of examples of common topics but far fewer examples of specialized domains.

The Three-Check Rule

Every AI output should pass through three filters before you trust it:

Check 1: The Sniff Test

Ask yourself: Does this sound plausible given what I already know?

This isn't about being an expert on everything—it's about activating your existing knowledge. If AI says something that contradicts your professional experience, pause.

Red flags:

• Claims that seem too convenient for your argument
• Statistics that are suspiciously round numbers
• Quotes that sound too perfect
• Explanations that are too simple for complex topics
• Anything that triggers a "wait, really?" reaction

Check 2: The Source Check

For claims that matter: Ask AI for its sources, then verify independently.

The process:

1. AI makes a claim
2. Ask: "What's your source for that statistic?" or "Where did that quote originate?"
3. AI may cite a source (often plausible-sounding but not verifiable)
4. Search for that source independently
5. If you can't find it or it says something different, flag the claim

Important: AI may fabricate sources that sound real. A "2019 McKinsey study" might not exist. A "Harvard Business Review article" might be invented. Verify by finding the actual source yourself.

Check 3: The Stakes Check

Calibrate verification to consequences.

Not everything needs the same scrutiny. An internal draft can tolerate more risk than a published report. Match your effort to the consequences of being wrong.

High-Risk Categories for Hallucinations

Be especially vigilant in these areas:

1. Statistics and Numbers

AI frequently invents statistics. "Studies show that 73% of employees..." may be based on nothing.

Verification: Search for the specific study. If you can't find it, don't use it.

2. Quotes and Attributions

Famous quotes are often misattributed, and AI makes this worse. Einstein, Churchill, and Mark Twain are quote magnets for things they never said.

Verification: Use a quote verification site or search for "[exact quote] + [person's name] + source"

3. Historical Dates and Events

AI can mix up dates, conflate events, or describe things that didn't happen.

Verification: Cross-reference with reliable sources like encyclopedias or primary documents.

4. Legal and Regulatory Information

Laws change, vary by jurisdiction, and have exceptions AI may not know.

Verification: Consult official sources or legal professionals for anything binding.

5. Medical or Health Claims

AI may provide outdated or incorrect health information.

Verification: Always defer to medical professionals and current clinical guidelines.

6. Niche or Specialized Domains

The more obscure the topic, the more likely AI is extrapolating rather than recalling.

Verification: Check with domain experts or specialized sources.

7. Recent Events

Anything after AI's training cutoff is fabrication risk.

Verification: Use current news sources.

Verification Prompts

Use these follow-up prompts to probe AI claims:

For statistics:

"What's your source for that statistic? Provide the study name, 
author, and publication year."

For quotes:

"Where did [person] say that? Provide the source and context."

For factual claims:

"How confident are you in this information? What are you basing 
it on?"

For recent topics:

"This might be after your knowledge cutoff. What's your training 
data cutoff date, and should I verify this independently?"

Meta-verification:

"Flag any claims in your response that I should verify before 
using in a professional context."

Before and After: Verification in Action

Scenario: Preparing a Presentation Slide on Remote Work

AI Output (Unverified):

According to a Stanford study, remote workers are 13% more 
productive than their in-office counterparts, and 91% of 
employees say they're more productive at home.

Verification Process:

1. Sniff Test: Two statistics—one sounds specific (Stanford study), one sounds suspiciously high (91%).
2. Source Check:
   • Ask AI: "What's the Stanford study you're referencing?"
   • AI might say: "Nicholas Bloom's 2015 study"
   • Google it: The study exists! It found 13% productivity increase for call center workers at a Chinese company.
3. Stakes Check:
   • This is for a client presentation = High stakes
   • Verify the 91% figure: Searching shows this number varies widely by survey and definition of "productive"
4. Outcome:
   • The 13% Stanford claim is verifiable but narrow (call center workers, one company)
   • The 91% is from a specific survey with different methodology
   • Adjust slide to be accurate about sources and scope

Verified Output for Slide:

Stanford research (Bloom, 2015) found remote workers 13% more 
productive in one company's call center. Self-reported productivity 
surveys show varying results—context and role matter.

Exercise 1: The Hallucination Hunt

Generate AI content and actively look for hallucinations.

Step 1: Generate Content

Use this prompt:

Write a 200-word summary of the history and health benefits of 
[choose a topic: turmeric, cold water swimming, meditation, or 
intermittent fasting]. Include 3 specific statistics or studies.

Step 2: Verify Each Claim

Categories: Statistic, Quote, Historical Fact, Health Claim, Other

Verification Results: Confirmed, Partially True, Cannot Find Source, False

Actions: Keep, Modify, Remove, Replace with Verified Data

Step 3: Reflect

• How many claims were fully verifiable?
• Which category had the most issues?
• How long did verification take?

Exercise 2: Stakes Calibration

Practice matching verification effort to stakes level.

Rate These Scenarios

For each scenario, rate the stakes (Low/Medium/High/Critical) and describe your verification approach:

Scenario 1: You're writing a casual Slack message to your team about productivity tips AI suggested.

Stakes level:
Verification approach:

Scenario 2: You're preparing a client proposal that includes market size data AI provided.

Stakes level:
Verification approach:

Scenario 3: You're drafting a company blog post that includes a quote attributed to a well-known CEO.

Stakes level:
Verification approach:

Scenario 4: You're creating internal documentation on software architecture best practices, using AI explanations.

Stakes level:
Verification approach:

Scenario 5: You're generating a summary of legal requirements for a new compliance process.

Stakes level:
Verification approach:

Exercise 3: Build Your Verification Workflow

Create a personal verification checklist for your common use cases.

Step 1: Identify Your High-Stakes Outputs

Step 2: Create Verification Steps for Each

For your #1 high-stakes output:

VERIFICATION CHECKLIST FOR: [Work Product]

Before using AI-generated content:

□ Sniff test passed?
□ Key claims identified:
  □ Claim 1:
  □ Claim 2:
  □ Claim 3:

□ Sources requested from AI?
□ Sources verified independently?
  □ Claim 1 source:
  □ Claim 2 source:
  □ Claim 3 source:

□ Domain expert review needed? (Y/N)
□ Recent/current information verified externally? (Y/N)

Ready to use: ________

Step 3: Determine Your Shortcuts

What can you skip for low-stakes work?

LOW-STAKES VERIFICATION (internal, brainstorm, draft):
□ 
□ 

Quick Reference: The Verification Cheat Sheet

╔════════════════════════════════════════════════════════════════╗
║                    THE THREE-CHECK RULE                        ║
╠════════════════════════════════════════════════════════════════╣
║                                                                ║
║  1. SNIFF TEST                                                 ║
║     Does it match what I know? Any red flags?                  ║
║                                                                ║
║  2. SOURCE CHECK                                               ║
║     Ask AI for sources → Verify independently                  ║
║                                                                ║
║  3. STAKES CHECK                                               ║
║     Low stakes = light verification                            ║
║     High stakes = full verification                            ║
║                                                                ║
╠════════════════════════════════════════════════════════════════╣
║                    HIGH-RISK CATEGORIES                        ║
╠════════════════════════════════════════════════════════════════╣
║                                                                ║
║  • Statistics and numbers                                      ║
║  • Quotes and attributions                                     ║
║  • Historical dates and events                                 ║
║  • Legal and regulatory claims                                 ║
║  • Medical or health advice                                    ║
║  • Niche/specialized domains                                   ║
║  • Recent events (after training cutoff)                       ║
║                                                                ║
╠════════════════════════════════════════════════════════════════╣
║                    VERIFICATION PROMPTS                        ║
╠════════════════════════════════════════════════════════════════╣
║                                                                ║
║  "What's your source for that?"                                ║
║  "How confident are you in this?"                              ║
║  "Flag claims I should verify independently."                  ║
║  "Is this from before or after your knowledge cutoff?"         ║
║                                                                ║
╚════════════════════════════════════════════════════════════════╝

The Verification Mindset

Verification isn't about distrusting AI—it's about using it responsibly.

The amateur: Believes AI output is truth and uses it directly.

The skeptic: Dismisses AI because it sometimes gets things wrong.

The professional: Uses AI for efficiency and applies judgment to the output.

The goal isn't to catch every error. It's to catch the errors that matter for your context and stakes. A casual internal message doesn't need the same rigor as a published report.

AI makes you faster, not infallible. The professional who verifies keeps their credibility intact.

Key Takeaways

1. AI doesn't know what it doesn't know. It will provide confident answers regardless of actual certainty.
2. Hallucinations aren't bugs—they're features of the technology. Pattern matching generates plausible-sounding text, not verified facts.
3. The Three-Check Rule catches most issues. Sniff Test, Source Check, Stakes Check.
4. Match verification to consequences. Brainstorms need less scrutiny than client deliverables.
5. Your credibility is on the line. When you publish or share AI content, you own it. Verify accordingly.

Next Steps

• [ ] Practice the Hallucination Hunt exercise with one AI output this week
• [ ] Create a verification checklist for your highest-stakes work product
• [ ] Save the "Three-Check Rule" somewhere visible for quick reference
• [ ] Identify your personal high-risk categories (where you're most likely to over-trust AI)
• [ ] Build verification time into your workflow for important deliverables