Home › Blog › Website-Only Chatbot

Should Your Website Chatbot Only Know What’s on Your Website? Why “I Don’t Know” Beats a Confident Wrong Answer

By Andrew “Easy” Anderson, Founder, iQEverything 9 minute read

We hired a kid one summer to help on the farm. Nice kid. Worked hard. One problem: he answered every question like he’d been there twenty years. “Which gate to the back forty?” “The red one.” There is no red gate. Never was. But he said it with such confidence that my uncle drove around for ten minutes looking for it. :)

The other kid we hired that summer said “I don’t know, let me go get Easy” about forty times a day. Drove me a little nuts. But nobody ever drove around looking for a gate that didn’t exist.

Now here’s the question. Which one of those two do you want talking to your customers at 9 p.m. when you’re not around? Because that’s the whole chatbot decision, and most of the industry is selling you the first kid.

I call him the Know-It-All Bot. He’s trained on the whole internet, he’s got a personality, and he will never, ever tell a customer “I don’t know.” Let me show you what he’s cost some folks.

What happens when a chatbot makes something up?

Three stories. All real, all sourced, all in the last three years.

Air Canada, 2024: the bot invented a refund policy

A man named Jake Moffatt asked Air Canada’s website chatbot about bereavement fares after his grandmother died. The bot told him he could book now and apply for the discount afterward. The airline’s real policy said the opposite. When he applied, Air Canada said no, and pointed out the correct policy was right there on another page of the site.

He took them to the British Columbia Civil Resolution Tribunal. Air Canada’s defense was, and I am not making this up, that the chatbot was a separate legal entity responsible for its own actions. The tribunal member called that “a remarkable submission” and wrote that the chatbot “is still just a part of Air Canada’s website.” Air Canada paid CA$812.02. The dollar amount is small. The precedent isn’t. You own what your bot says.

Chevrolet of Watsonville, 2023: the $1 Tahoe

A dealership put a general-purpose AI chatbot on its site. A visitor told the bot to agree with everything he said, then asked it to sell him a 2024 Chevy Tahoe for one dollar. The bot replied that it was a deal, adding “a legally binding offer, no takesies backsies.” The screenshots went everywhere. The dealer pulled the bot down. Nobody got a Tahoe for a dollar, but that’s not the point. The point is a bot with no fence will say anything a clever visitor tells it to say, with your name on it.

New York City, 2024: the government bot told businesses to break the law

New York City launched a chatbot to help small business owners with city rules. The Markup tested it in March 2024. The bot said landlords could refuse tenants with housing vouchers, that employers could take a cut of workers’ tips, and that a business could go cash-only. All three are illegal in New York City. This wasn’t a garage startup. It was a city government running a major vendor’s AI, and it still handed out confident wrong answers on questions that could get a business owner fined.

Why do chatbots guess instead of saying “I don’t know”?

Because we trained them to. In September 2025, researchers at OpenAI published a paper called “Why Language Models Hallucinate,” and the explanation is almost too simple:

“Language models are optimized to be good test-takers, and guessing when uncertain improves test performance.”

Think about the kid with the red gate. If every test he ever took gave zero points for “I don’t know” and a chance at full points for a confident guess, what would you expect him to do? Guess. Every time. That’s the machine. The people who built it are telling you so in writing.

How often does it happen? It depends entirely on whether the bot is answering from a document or from memory, and this is the whole argument in two numbers:

So a fence cuts the problem way down. It does not cut it to zero. That third row is the honest one: even the best-grounded tools still slip on hard questions. Which is exactly why the fence needs a gate, and the gate is “I don’t know, let me get a person.”

What does “only knows your website” actually mean?

The industry word is grounded, or if you want the acronym, RAG. IBM defines it plainly: retrieval augmented generation is “an architecture for optimizing the performance of an AI model by connecting it with external knowledge bases.” In English: instead of answering from everything it ever read, the bot goes and looks it up in a specific pile of documents first, and answers from what it finds there.

For a business chatbot, that pile should be your website, and nothing else. Your services page. Your service area. Your pricing page if you have one. Your FAQ. Your hours. That’s the fence. And here’s the thing people get backwards about fences: a fence isn’t there to limit the cow. It’s there to keep her out of the road. The bot that only knows your site can’t quote a price you don’t charge, promise a service you don’t offer, or invent a refund policy on a Tuesday night. Not because it’s smart. Because it can’t see the road.

If this sounds familiar, it’s the same bias we wrote about in the 40-kilobyte app: do less, do it right, and skip everything that isn’t the job. A bot that does three things correctly beats a bot that does everything sort of.

How do you build the fence? Three steps

Step 1: Fence it

Train the bot on your website content only. Not the internet. Not a general model with your site “added in.” If a vendor can’t tell you in one sentence where the bot’s answers come from, that’s your answer. And keep the website current, because the fence is only as good as what’s inside it. If your site still lists a service you dropped last spring, so will the bot.

Step 2: Give it a gate

When the answer isn’t on the site, the bot says “I don’t know that one, let me get you to a person,” collects a name and number, and hands off the full transcript so the human isn’t starting from scratch. This is the step the Know-It-All Bot skips, and it’s the one customers actually care about. Salesforce surveyed more than 15,000 people in 2024: customers were 45% more likely to use an AI agent when there was a clear path to a human, and about three in four wanted to know up front whether they were talking to AI at all. So tell them. Honesty isn’t a feature. It’s the price of admission.

Step 3: Give it a job list, not a personality

Answer questions from the site. Book an appointment into your real calendar. Capture the lead, with a photo if the customer has one. That’s the list. No jokes, no discounts, no “legally binding” anything, no opinions about the competition. Every capability you add is a new stretch of fence you have to check. A dealership bot that could only answer questions about inventory and book a test drive would never have sold a Tahoe for a dollar, because selling wasn’t on its list.

Why does this matter more than being impressive?

Because trust is the whole product, and the average chatbot has already spent most of it. Gartner asked 5,728 customers in late 2023: 64% said they’d prefer companies not use AI in customer service at all, and 42% named wrong answers as the specific fear. Zendesk’s 2025 CX Trends found 63% of consumers will switch to a competitor after one bad experience. One.

Read those together and the Know-It-All Bot looks like what he is: a way to have a bad experience faster. The honest little bot that only knows your business and says “let me get a person” isn’t the exciting option. It’s the one that still has customers next year. Small and right beats big and confident. Every time. Ask Air Canada.

Frequently asked questions

Two ways to get the honest kid

Full disclosure: we built a chat widget on exactly these three rules, because our clients kept asking and everything we tested either made things up or charged by the conversation. Two doors. Pick the one that fits.