Guardrails and Moderation for AI Apps

A language model will happily answer almost anything, which is exactly why a public-facing AI app needs boundaries. Guardrails are the checks and rules around the model that keep conversations safe, relevant and compliant. They protect users, your brand and your legal position. This article explains the main types and shows how to combine them.

What are guardrails?

Guardrails are controls applied before, during and after model calls to prevent undesirable behaviour. They are not a single tool but a layered system:

Input guardrails: inspect what users send.
Model-level guidance: system prompts, safe settings and model choice.
Output guardrails: inspect what the model returns.
Action guardrails: limit what tools and integrations can do.
Process guardrails: monitoring, human review and incident response.

What could go wrong?

Harmful content: hate, harassment, violence, self-harm or sexual material
Dangerous instructions (weapons, malware, fraud)
Medical, legal or financial advice beyond what the product should offer
Off-topic use that burns your budget (someone using your support bot as a free general chatbot)
Disclosure of personal data or confidential information
Prompt injection and jailbreak attempts
Hallucinated claims or promises that create liability (wrong prices, refunds)
Biased or disrespectful responses
Infringing content such as copied copyrighted text

Input guardrails

Length and rate limits: cap message size and requests per user to prevent abuse and cost spikes.
Moderation checks: use a moderation model or API to classify harmful categories before the main model sees the message.
Topic filtering: a lightweight classifier or rule determines whether the question is in scope for your app.
PII detection: detect personal identifiers, such as phone numbers, ID numbers and card numbers, and mask or reject them if you should not process them.
Prompt injection heuristics: flag messages that try to override instructions, though this is imperfect.
Authentication and authorisation: know who the user is and what they may access.

Model-level guidance

Write a clear system prompt defining role, scope, tone, forbidden topics, how to refuse politely and what to do when unsure.
Provide grounding documents and instruct the model to answer only from them.
Choose settings such as lower temperature for factual tasks.
Keep secrets out of prompts.
Use models with built-in safety training, but never rely on that alone.

Output guardrails

Moderation of responses: run the same safety classifiers on the output.
Format validation: if you expect JSON, validate against a schema and retry or fall back on failure.
Grounding checks: compare claims with retrieved sources, or require citations and verify them.
PII and secret scanning: block outputs containing patterns like API keys or personal data.
Link and image controls: allow only approved domains.
Policy rules: forbid commitments such as "I guarantee a refund" unless a rule engine confirms eligibility.
Tone and brand checks: detect rude or off-brand language.
Confidence and fallback: if checks fail, return a safe message and escalate.

A layered pipeline in Python

The following sketch shows the structure. In practice, moderate, in_scope and contains_pii would be backed by moderation APIs, classifiers or libraries.

import re

SAFE_FALLBACK = "Sorry, I can't help with that. I can connect you with a team member."

def contains_pii(text: str) -> bool:
    return bool(re.search(r"\b\d{10}\b|\b\d{13,16}\b", text))      # phone / card-like numbers

def moderate(text: str) -> bool:
    banned = ("how to make a bomb", "credit card dump")             # placeholder; use a real classifier
    return any(b in text.lower() for b in banned)

def in_scope(text: str) -> bool:
    topics = ("order", "delivery", "refund", "price", "product", "hours")
    return any(t in text.lower() for t in topics)

def handle_message(user_text: str, call_model) -> str:
    if len(user_text) > 1000:
        return "Please send a shorter message."
    if moderate(user_text):
        return SAFE_FALLBACK
    if not in_scope(user_text):
        return "I can help with orders, delivery, refunds and products. What would you like to know?"
    if contains_pii(user_text):
        user_text = re.sub(r"\b\d{10,16}\b", "[NUMBER]", user_text)  # never forward raw numbers

    answer = call_model(user_text)

    if moderate(answer) or contains_pii(answer):
        return SAFE_FALLBACK
    if re.search(r"\bguarantee\b", answer, re.I):
        return SAFE_FALLBACK
    return answer

This toy version shows the shape: check, transform, call, check again, fall back. Real systems log every blocked event for review.

Specialised guardrail tools

Several open-source and commercial frameworks help define rules, validators and conversation flows, and many model providers offer moderation endpoints. Use them to avoid reinventing basic filters, but still test them on your own use case and languages, since classifiers trained mostly on English may miss harmful content in Nepali or romanised Nepali.

Testing guardrails

Build a test set of safe, borderline and attack prompts in the languages your users speak.
Measure false negatives (harmful content allowed) and false positives (legitimate content blocked). Too strict guardrails make the product frustrating.
Include adversarial cases: rephrased requests, role-play framing, encoded text and prompt injection in documents.
Re-run tests whenever you change prompts, models or rules.
Review real conversations (with privacy safeguards) to find gaps.

Human escalation

Some situations need a person: users in distress, threats, complaints, legal issues or repeated blocked attempts. Define clear triggers, provide a quick path to a human and give agents the conversation context. For self-harm related messages, show supportive messages and local help resources rather than just refusing.

Balancing safety and usefulness

Over-blocking drives users away; under-blocking creates risk. Aim for graceful refusals that explain what the assistant can help with, and tune thresholds based on the risk of your domain. A children's education app needs much stricter settings than an internal developer tool.

Governance

Assign an owner for safety rules.
Document policies and review them regularly.
Keep logs for audits with appropriate retention and privacy controls.
Provide users with a way to report problems.
Update guardrails as new attacks and regulations emerge.

Frequently Asked Questions

Do I need guardrails if the model is already safe?

Yes. Models vary, can be manipulated and do not know your specific policies, so application-level controls are still needed.

Will guardrails slow my app?

Lightweight checks add little delay. Heavy classifiers add latency, so choose them based on risk and run some checks in parallel.

Are guardrails the same as a system prompt?

No. A system prompt guides the model but can be bypassed. Guardrails include independent checks outside the model.

How do I handle Nepali content?

Test moderation tools with Nepali, romanised Nepali and mixed text, and add your own keyword lists and examples where tools are weak.

Conclusion

Effective guardrails are layered: check inputs, guide the model, validate outputs, limit actions and escalate to humans. Test them with realistic and adversarial examples, tune the balance between safety and helpfulness, and review regularly.