Securing LLMs: Prompt Filtering, Rate Limits & Access Controls

Large language models are already part of many products. They write help text, summarize documents, and automate workflows. That power comes with risk. A single crafted prompt can change a model’s behavior. A misconfigured connector can leak data. A runaway script can wipe out your budget. This guide explains what to fix first, how to build layered defenses, and how to test that those defenses actually work. It is written for engineers and product leads who need copy-paste ready patterns and clear next steps. No links appear in the article body. Sources are listed at the end.

Why this matters — short and plain

LLMs parse plain text. They cannot tell which words are safe and which are an attacker’s trick. That simple fact creates three big problems:

1. Prompt injection. Attackers hide instructions inside user input or documents. The model follows them. The result can be data leakage or unsafe behavior.

2. Data exfiltration through connectors. When an LLM can read Drive or Slack, a poisoned document can make it reveal secrets. These attacks can be zero-click.

3. Resource abuse and privilege mistakes. High request volume wastes money. Over-privileged keys multiply damage when leaked.

The rest of this guide shows concrete ways to stop and detect these failures.

Prompt filtering — a practical, layered pipeline

Treat user input as hostile until proven otherwise. Use small, fast checks first. Add slower, smarter checks second. Only then let the model read the text.

Principles to keep front of mind

• Keep system instructions and user content separate. Never concatenate them into a single blob that the model must parse.

• Normalize input. Remove hidden characters. Convert markup to plain text. That prevents attackers from hiding payloads.

• Score for intent. Use a lightweight classifier that groups input into safe, risky, or unknown.

• Use similarity checks against known malicious patterns. This finds mutated attacks that pass regexes.

• Fail closed. When a check flags high risk, block or route to human review.

A simple pipeline you can implement today

1. Canonicalize input. Normalize whitespace. Strip unusual unicode. Decode base64 only after review.

2. Fast regex checks. Block obvious commands: “ignore previous”, “forget instructions”, “act as system”. These checks run in milliseconds.

3. Intent classifier. Run a small model that returns safe / suspicious / high_risk. For suspicious inputs, run step 4.

4. Embedding similarity. Compare the user text embedding to a curated set of malicious prompt embeddings. If cosine similarity passes the threshold, redact or escalate.

5. Connector vetting. If the request references a document or URL, fetch the document and scan it with the same pipeline before exposing it to the model.

6. Output scan. Scan the model’s response for secrets, PII, or URLs. Redact or block if found.

Examples and short patterns

• Instruction isolation: Send a short, immutable system prompt such as: “Follow this safety policy. Do not reveal secrets.” Send user input in a separate user field. Do not embed system prompts inside user text.

• Redaction strategy: If a user requests extraction from a file, pre-redact lines that match secret regexes (API keys, private keys, credentials) before sending content to the model.

• Human escalation: For any request that touches sensitive connectors or scores as high risk, require human approval. Do not rely on the model for the final decision.

Rate limits — protect budget and availability

Rate limiting is about fairness, cost control, and resilience. Design limits that stop abuse and let legitimate users keep working.

Multi-layer limits to implement

• Provider quotas. Know your cloud provider limits. Respect burst windows they allow.

• Tenant limits. Set per-organization quotas so one customer cannot exhaust your resources.

• Per-user / per-key limits. Stop a compromised key from burning tokens.

• Endpoint weights. Treat expensive endpoints as heavier. Long generation = higher cost. Embeddings = a different weight.

• Graceful degradation. Offer reduced-quality or cached responses rather than a hard fail when the system is overloaded.

Recommended mechanics

• Use a token bucket or leaky bucket implementation in a fast store such as Redis for distributed limits.

• For background or heavy jobs, put requests in a queue and process them with workers under a capacity limit.

• Return clear HTTP 429 responses. Include Retry-After and a human message that explains what happened.

• Implement exponential backoff with jitter for retries in clients to avoid synchronized retry storms.

Access controls and secrets hygiene

Minimize blast radius. When keys leak or humans make mistakes, good access control limits damage.

Concrete rules

• Least privilege. Give the minimum rights needed. Separate read from write. Separate model creation from billing.

• Short-lived credentials. Prefer time-bound tokens and rotate them automatically. Long-lived keys should be rare.

• Central secrets manager. Never hardcode keys. Use a vault. Enforce policy that disallows key leaks.

• Network controls. Restrict management consoles with IP allowlists or private network endpoints.

• Audit trails. Log every request, who made it, and what resources were accessed. Keep logs immutable for incident forensics.

Practical roles to create

• Reader. Can query models and read logs. No deploy rights.

• Operator. Can run workloads and manage queues. No billing or key creation.

• Admin. Can create keys and change RBAC. Limited to a small set of people and protected by MFA.

Detection, canaries, and red teaming

Prevention will eventually fail. Detect early and respond fast.

Canary tokens and honeystrings

Seed internal documents with unique, harmless strings. If a model ever outputs one of those strings in a public context, treat it as a probable leak. Canary tokens are simple and cheap. They give high fidelity when they trigger.

Monitoring to implement

• Track request rates per user, per key, and per tenant. Alert on sudden spikes.

• Monitor token usage trends and average response lengths. Unusual increases often indicate automated abuse.

• Scan outputs for sensitive patterns. If a model starts returning sequences that match internal keys, trigger a lockout.

Red teaming and tests

• Create an automated suite of prompt-injection tests. Run them as part of CI.

• Schedule periodic human red team exercises. Humans find creative paths that automated tests miss.

• Track gaps and log the fixes. Turn each red-team finding into a test that prevents regression.

A practical 7-day runbook

This is what to do next. Each day is one deliverable.

Day 1 — inventory and logging
List all model endpoints and API keys. Enable structured request logging. Send logs to a central store.

Day 2 — prompt hygiene
Isolate system prompts. Add canonicalization and basic regex filters for clear injection signatures.

Day 3 — rate limiting basics
Deploy a token bucket per key in Redis. Apply conservative default limits and test fail behavior.

Day 4 — access controls
Audit roles. Remove unused keys. Move secrets to a vault and require rotation for all production keys.

Day 5 — connectors and vetting
Block auto-ingest from external connectors. Add a review step. Scan any document content before the model sees it.

Day 6 — monitoring
Create alerts for canary triggers and token usage spikes. Add dashboards for request volume and error rates.

Day 7 — red team
Run an internal prompt-injection battery. Fix gaps, then automate the tests in CI.

Short example patterns — copy-paste friendly

Fast regex block (pseudo):

Rate limit header for clients:
Return 429 with:

Canary token example:
Place a unique phrase in a private doc like DO_NOT_SHARE_CANARY_2025_ABC123. Watch outputs for that exact phrase.

Common mistakes and how to avoid them

• Putting system rules inside long prompts. Fix: keep rules short and immutable.

• Trusting connectors by default. Fix: treat external files as untrusted until scanned.

• Only relying on model safety. Fix: add independent filters and human review gates.

• One-size-fits-all rate limits. Fix: weight endpoints and split quotas by tenant.

Closing: defense in depth and continuous testing

LLM security is not a single switch. It is a set of practices. Build multiple layers so that one failure does not lead to disaster. Start with isolation and simple filters. Add quotas and access control. Then instrument, test, and automate the response steps. Finally, make red teaming part of your release cycle.