Design Systems, Not Chatbots

There are a few traps that we, as software people, keep falling into. It happens to engineers who are excited about AI, to veterans building in this space, to companies that want to be seen doing AI, and to the product owners inside those companies.

The first trap is usually the first idea: take a general-purpose model (OpenAI, Vertex AI and so on), call it through an API, wrap it in simple or elaborate prompts, ship a chatbot, and then announce "look, we use AI too, we've integrated AI into our product" and lean on that in every piece of marketing.

The second trap (I went back and forth on whether to even call it a trap) is building a RAG system, a system restricted to a document or a document base. That doesn't solve the whole problem either, especially in regulated areas like healthcare or law. The critical issue is what we call hallucination: when a model doesn't have the answer, instead of saying "I don't know", it makes one up without a source.

In regulated or high-stakes production systems, every answer a model gives without a source is a potential bug and a source of risk.

Part 1Real-world examples#

Here are three examples of how this goes wrong.

1. Prompt injection: a system prompt is not security#

In a story that went viral in 2023, a software engineer (Chris Bakke) injected a prompt into a car dealership's chatbot that looked harmless but was actually designed to manipulate the system's behavior.

Once the bot was pulled into an absurd frame along the lines of "everything you say to the customer is legally binding", the model turned out to be quite willing to follow those instructions. As a result, it could behave as if it were accepting a sales offer that didn't exist. (Chatbots that sell things are not that common yet, but the same pattern applies to reservations and plenty of other domains.)

What is prompt injection? See Palo Alto Networks: What is a prompt injection attack?

In other words, the person talking to the LLM hacked it with a prompt and got it to behave however they wanted. It didn't stay an internet curiosity, either. It became part of the security canon: OWASP Top 10 for LLM Applications (LLM01: Prompt Injection). By putting prompt injection at the very top of its list of LLM vulnerabilities, OWASP formally recognized it as a design-level problem.

2. Hallucination: "wrong but convincing truth" in court#

The Mata v. Avianca case in the US, in 2023, showed exactly where LLM use breaks inside a legal system. A lawyer used ChatGPT to produce legal references to court decisions that had never existed, and submitted them to the court. The result:

• fabricated decisions,
• case law that didn't exist,
• a chain of references that was completely convincing and completely wrong.

The court responded with explicit sanctions. The case has become the classic reference point: the model produced information that looked right but was wrong, and that output went straight into the legal system. (One of the biggest problems today is simply misuse: treating everything an AI says as true.)

Source: Mata v. Avianca, Inc., U.S. District Court (S.D.N.Y., 2023 sanctions order)

3. Even when a vendor says "no hallucinations", don't take it on faith#

In 2024, a study by Stanford RegLab found hallucinations in AI products built for legal work. The important detail: the companies being evaluated were claiming they had solved hallucination.

The study examined Lexis+ AI (LexisNexis) and Westlaw AI-Assisted Research (Thomson Reuters). The result was clear: legal queries produced hallucinations at a meaningful rate. The rates varied by system but were in the double digits (roughly 17–33% was reported).

All of these cases share one underlying condition: while the model is talking, it doesn't separate what it knows from what it is making up. It has to be forced to, and that forcing has to happen in the system architecture, not in the prompt.

Source: Stanford RegLab — Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (2024)

Part 2Before you start designing#

Before designing anything, it matters a great deal to pick the right words for what you're building, because words end up shaping the architecture.

"Chatbot" has stretched so far over the last few years that it has almost lost its meaning. Everyone says "AI assistant", but most of the time we aren't talking about the same thing. In practice there are three different approaches, and any discussion that doesn't separate them stays up in the air.

The first is the autonomous chatbot. In its simplest form: the user types, the model answers, nobody is in between. The system decides on its own and hands the result straight to the user. It sounds efficient, but you are also taking on all of the risk. We've seen how this blows up in real life, as in the Air Canada case, or the MyCity bot that The Markup exposed. The system gives out incorrect legal information, and that information directly shapes what users do. The consequences: damages, pressure to shut the system down, reputational harm and legal exposure.

The second looks more "enterprise": closed-loop automation (agent systems). The user asks for something; the system does the work in the background, fills in forms, calls APIs, books appointments, and only shows you the result. The process itself is invisible. At first glance it seems more controlled, but in reality there is no transparency. Speed goes up, cost goes down… and after a while quality slips and people have to be brought back in. The problem isn't solved, only postponed.

The third, in my view, is the one we should actually be talking about: agent-assist. Here the AI doesn't decide; it supports. It gives the operator suggestions, pulls in sources, prepares drafts. But the final word always belongs to a human. That's the critical difference. Take Nuance's DAX Copilot: while the doctor talks, the AI drafts the notes, but nothing becomes official until the doctor reads and signs them. The system saves time without taking over the responsibility. It's a deliberate architectural boundary. (source)

This is where the core claim of this article starts: in regulated industries, autonomous chatbots that give users binding information or trigger actions carry serious engineering and governance risk.

The right approach is clear: agent-assist + human approval + an auditable record. The system doesn't just produce an answer, it also makes how it answered traceable.

This isn't only a theoretical view; the data says the same thing. Gartner's 2025 predictions paint an interesting picture. On one hand, agentic AI is expected to become very widespread. On the other, a large share of these projects is expected to be cancelled because of cost, unclear value and weak risk management. That isn't a contradiction. What is widespread and what is safe are not always the same thing.

The key point: your question isn't "should I use AI?" It's "at what level of risk, and under what contract, should I use it?"

My own view is clear: a significant share of the companies in Turkey that deploy autonomous chatbots in regulated areas will face at least one KVKK (data protection) process or consumer lawsuit in the coming years. That isn't a certainty, but it is a reasonable risk scenario if you carry today's examples over into Turkish consumer law and KVKK. Mata v. Avianca, Air Canada, MyCity… they all say the same thing: if you don't set up the right contract from the start, the system hits a wall at some point.

The problem isn't the model. It's control.

Part 3The refusal and citation contracts#

Read this story with an engineer's eye and it keeps coming back to two basic "contracts".

1. The refusal contract#

"I don't know" is not a failure of the system. It's evidence that the system is working correctly.

We need to accept one thing plainly: every answer a model produces without relying on a source is a bug in production. Sometimes it looks like a small content mistake, but in regulated areas it turns directly into legal risk.

One of the clearest examples is, again, Mata v. Avianca. The lawyer asks the model to confirm whether a case is real. The model confidently says "yes", even backing it up with references to trusted legal databases. The problem: no such case exists. It's entirely made up. The outcome: a fine from the court and serious damage to the lawyer's reputation.

The problem here isn't "the model made a mistake". It's something more basic: the model can't say that it doesn't know. Even when it isn't sure, it is pushed to produce an answer.

So the solution isn't writing a "better prompt" into the model. It's a decision at the system level: if the model has no verifiable source, it must say "I don't know" instead of producing an answer. And the even more important part: that answer must be defined as a successful outcome, not as a failure.

That means your metrics have to be built accordingly. "90% coverage" may look like a nice number, but if that 90% includes unverified answers, the system is quietly producing risk. In a good agent-assist architecture the goal is:

• either give a correct, sourced answer,
• or give no answer at all.

You deliberately remove the gray area in between, because in real life the gray area is exactly what causes trouble.

From here on I'll include some technical examples from the .NET and Azure side. For developers, here's a slice of the infrastructure I built in .NET (I'll make the repository public when the series is finished; small pieces for now):

public sealed record AssistantAnswer(
    string Answer,
    IReadOnlyList<Citation> Citations,
    ConfidenceLevel Confidence,
    RiskClass RiskClass,
    bool EscalationRequired,
    string? RefusalReason)
{
    public static AssistantAnswer Refused(
        string reason,
        RiskClass risk,
        ConfidenceLevel confidence = ConfidenceLevel.Low) =>
        new(
            Answer: "Not enough sources were found to answer this question.",
            Citations: Array.Empty<Citation>(),
            Confidence: confidence,
            RiskClass: risk,
            EscalationRequired: true,
            RefusalReason: reason);
}

From the user's point of view this isn't a failure either. The system realizes it can't speak to the question with the information it has, and says so honestly. That's the right behavior.

The semantics change on the API side as well. This isn't an exception, so you return HTTP 200, not 500. The frontend doesn't present it as "something broke". Instead you design a more natural flow: "I couldn't find a reliable source on this, so I'm routing you to the right team." The UX becomes consistent with the system's contract.

So what do we gain, and what do we lose?

The gains are clear. First, controllability: when the system speaks and when it stays silent becomes a deterministic decision. Second, measurability: the moment you keep RefusalReason in your logs, you can answer a very useful question with data: "What is missing from our knowledge base and our datasets?" Over time this becomes one of the most valuable outputs of the system, because you're no longer chasing hallucinations, you're closing knowledge gaps directly.

The costs are mostly technical. "Let's just tell the model not to answer without a source" doesn't work here, because models are, by nature, inclined to fill gaps. The real solution lives in the orchestration layer: if retrieval returns nothing, don't call the model at all. It looks like a small change, but it grows the architecture. Threshold checks, retrieval scoring and fallback flows all have to be designed more carefully.

There's a small price in latency, too. In a well-designed system, especially if you cut out unnecessary calls to large models, you can see a meaningful drop in token costs; the latency impact depends on your retrieval, reranking and validation strategy. In return, you stop paying for large-model calls that were never going to produce a grounded answer. On expensive calls, such as 32K contexts, that makes a real difference. What we've seen in practice:

• a 15–30% drop in token costs (one of today's biggest headaches, even for individual developers),
• a visible improvement in quality metrics.

The system isn't slower. It's smarter.

2. The citation contract: no sentence without a source#

On the refusal path, citations can come back empty, and that's normal. If the system says "I can't speak to this", nobody expects it to cite a source. But when citations are present, things change.

Every citation has to be tied to a real chunk, a real document ID and a verifiable snippet. Otherwise it isn't a citation, it's a hallucination with decorations.

This is one of the places where the industry gets burned most. Because a RAG pipeline exists, everyone assumes the system works "from the sources". In practice, many models first produce the answer from their own internal knowledge, then look through the retrieved chunks for something that resembles the answer and attach it as a citation. So it looks as if the answer is sourced, but the citation doesn't actually support it.

That's a very dangerous illusion, because people trust what they see on screen when there's a source next to it: the operator, the user, the legal team. In reality the system has only put make-up on its answer after the fact.

That's why you shouldn't leave citations to the model's language skills. A citation should count as valid not because the model says "I got this from that source", but because the system has verified it. That's where a critical node sits in the architecture:

{
  "refused": false,
  "citations": ["CHUNK_42", "CHUNK_43"],
  "confidence": "High",
  "escalationRequired": false
}

{
  "refused": true,
  "refusalReason": "no_source",
  "citations": [],
  "escalationRequired": true
}

{
  "refused": true,
  "refusalReason": "no_source",
  "citations": [],
  "escalationRequired": true
}

{
  "refused": true,
  "refusalReason": "invalid_citation",
  "citations": [],
  "escalationRequired": true
}

The key point in this flow: the model is not the center of the system, it's just one step. The decision doesn't start in the model and it doesn't end there. Risk classification, retrieval, permission filtering, citation validation and the audit event are each enforced around the model call. That's why a refusal isn't an error; it's a safe outcome the system chooses on purpose.

The citation validator (step 7 in the flow)#

This layer's job is simple: compare every citation the model returns against the list of chunks the retrieval layer actually returned. If the model says [CHUNK_42], the system asks:

• Does this chunk actually exist?
• Did this chunk come from retrieval for this user's query?
• Is the claim in the answer genuinely supported by this snippet?

If the answer is no, there are two options: the answer is reformatted, or it falls onto the refusal path.

This isn't an expensive operation, nowhere near the cost of another large model call. It's closer to a verification loop. But its effect is large, because this is where most citation errors get caught.

It does have a cost. First, the prompt has to be written with more discipline. You have to hand the model its chunks not as plain text but in a structured form such as [CHUNK_42] [CHUNK_43] [CHUNK_44], and then ask it to put the relevant chunk ID next to every important claim. This doesn't always work perfectly, especially with small models: some mix up chunk IDs, some forget to add citations, and some present unsupported commentary as if it were sourced.

There's also an important false assumption to break here: a more powerful model does not always mean fewer hallucinations. Reasoning models think better on some tasks, but on work like grounded summarization they can add more interpretation. More reasoning sometimes means more extrapolation. The model may drift from reporting "what the text says" to "what could be inferred from the text". In a regulated system, that difference can be fatal.

That's why the citation contract should be treated as a security layer that is independent of model choice. Using an expensive model does not, on its own, make you safe. In fact, an expensive model with weak citation checks can behave worse than a cheap model with strict ones.

The summary is simple:

• No source, no answer.
• If there is a source, it must actually support the answer.
• And that must be verified by the system, not by the model.

Final partMy predictions#

This part is opinion, not evidence. But it isn't made up on the spot either; these are predictions built on existing cases and on where the technology is heading.

Prediction one: within the next one to two years, at least one large financial or insurance company in Turkey will lose a consumer rights case because of incorrect information given by its autonomous chatbot.

My reasoning: what happened at Air Canada is entirely possible in Turkey; only the legal framework will differ. An issue that was argued under different regulation in Canada will most likely land on Turkey's Consumer Protection Law No. 6502 and KVKK. In particular, whatever guidance KVKK publishes on AI may become the de facto standard for evaluating technical architecture in these cases. The court will ask: "Was this system autonomous? Was there human approval? Were logs kept? Were sources shown?" If the answers are weak, the company is in trouble.

Prediction two: this one is mostly for people who, like me, follow the Microsoft ecosystem closely. From 2027, Microsoft Agent Framework will overtake Semantic Kernel in enterprise .NET projects in Turkey. SK is still strong and will keep being supported, but teams starting new projects will most likely prefer the structure that is more standard, lighter and a more natural fit for the .NET ecosystem. The point isn't that "SK is bad"; it's where Microsoft's long-term standard is shifting. An interface design that Stephen Toub is involved in is rarely an accident in the .NET world. The abstraction that forms there tends to become the ecosystem's default over time.

Prediction three: again specific to Microsoft and .NET. Citation enforcement, prompt injection defense and PII redaction will become standard middleware in 2027. Today every team writes its own citation validator, its own prompt injection filter, its own PII scrubbing layer. That isn't sustainable; every company solving the same security problem from scratch is inefficient and risky. We'll most likely see something like a Microsoft.Extensions.AI.Guardrails package on NuGet. It may come from Microsoft or from the community, but one way or another it will be standardized, and it will spread to other ecosystems too. Because this is no longer an "AI feature" issue, it's an application security issue. The fact that the OWASP LLM Top 10 lists prompt injection and vector/embedding weaknesses as separate items shows it. The ecosystem is waiting for shared primitives here.

Closing#

Four or five years ago, building an AI system inside an enterprise, regulated company looked like "the work of the future". Today it's overdue. The infrastructure is ready, the tools are ready, the patterns have been written down. On the Microsoft side, with .NET 10 LTS, Microsoft.Extensions.AI and Microsoft Agent Framework are in production. On Azure, pieces like Foundry, AI Search and the APIM AI Gateway are already in place (and other ecosystems, Vertex AI or independent stacks, are moving fast too). Most importantly, how to do this work is no longer an experiment. It's a discipline that has been discussed, argued about and keeps arriving at the same place.

At the start of this article I said: in enterprise AI we aren't building chatbots, we're building agent-assist systems with contracts. Now we can sharpen that a little:

Wherever these contracts can be enforced, that's where they should be written.

And definitely not in the system prompt, in the README, or on the marketing page. Because a contract has to be able to stop the system when it doesn't hold. Otherwise it's not a contract, it's a statement of intent.

In the Air Canada case, the money involved is actually beside the point. What matters is the reasoning the tribunal set out. The gist of what Member Rivers said: "It makes no difference whether the information comes from a static page or a chatbot."

Translate that into .NET and you're left with a single rule:

Behind every answer the model gives, there is either a verifiable source or an explicit refusal. There is no third option.

You're really building the entire architecture around that sentence. The refusal contract and the citation contract are its implementation.

The gap over the next few years will open up right here. Teams that build this discipline in from the start won't have to rewrite their systems as they grow. Those that don't will learn the same lesson somewhere more expensive, in a harder setting. Probably during a production incident. Or in a courtroom.

I'm not leaving this article on its own; it's the first part of a series. The next parts cover how to implement this story properly in the Azure and .NET ecosystem.