Skip to main content

All weeks · Worksheet · Overview · Attack notes

Week 14 · Lecture slides

Week 14

Contents15 sections

Security of AI / LLM-Powered Applications

Software Security · Nutthakorn Chalaemwongwan

Today

  • Where AI features add attack surface
  • OWASP LLM Top 10 (2025)
  • Prompt injection (direct + indirect)
  • Agentic-AI / MCP risks
  • 🎮 Game: Gandalf Challenge

Why a whole week on AI

  • LLMs now sit inside real products & agents
  • New, fast-moving attack surface
  • OWASP LLM Top 10 + MITRE ATLAS

OWASP LLM Top 10 (2025)

LLM01 Prompt InjectionLLM02 Sensitive Info DisclosureLLM03 Supply Chain
LLM04 Data/Model PoisoningLLM05 Improper Output HandlingLLM06 Excessive Agency
LLM07 System Prompt LeakageLLM08 Vector/EmbeddingLLM09 Misinformation
LLM10 Unbounded Consumption

New in 2025: LLM07 System Prompt Leakage · LLM08 promoted (RAG everywhere) · LLM10 replaces "DoS" with runaway cost.

Heads up: OWASP shipped a 2026 edition days before this lecture (Excessive Agency jumps to #3, Output Handling drops to #10). This course's lab, worksheet, and quiz key are still on 2025 numbering — use 2025 for anything graded this term.

Prompt injection

Untrusted text overrides the system's instructions — injection, again.

  • Direct: user tells the bot to ignore its rules — this week's hands-on lab (Gandalf + Tasks 1-2)
  • Indirect: malicious instructions hidden in a fetched doc / web page (RAG) — discussed, not built: the lab's mock LLM has no RAG/retrieval, so this is a thought experiment (worksheet Task 3), not something you'll exploit yourself today

Improper output handling

  • Model output flows unsanitized into HTML/SQL/shell
  • → XSS / injection downstream
  • Treat LLM output as untrusted input

Real-world incidents

  • Bing Chat "Sydney" (2023): a typed "ignore previous instructions" leaked the hidden system prompt — direct injection
  • EchoLeak (2025): zero-click indirect injection in M365 Copilot → data exfil (CVE-2025-32711)
  • Agentic tool-abuse: injected web/email content makes a tool-using agent take real actions (send mail, move funds, run code) — it trusts the text as instructions
  • Résumé injection: hidden white text inflated an AI screening score

Injection needs no exploit code — just text the model trusts.

Agentic-AI / MCP risks (2025+)

  • Agents call tools (e.g. via MCP) → real-world actions
  • Tool poisoning, excessive agency, RCE via tools
  • MITRE ATLAS added agent techniques (Oct 2025)
  • Research: 43% of public MCP servers had command-injection flaws

Real MCP/agent incidents (2025)

  • Supabase × Cursor: privileged agent read a support ticket with injected SQL → leaked integration tokens (privileged access + untrusted input + exfil channel)
  • Invariant Labs: a malicious trivia MCP server's tool description hijacked a trusted WhatsApp MCP → exfiltrated chats
  • MCPoison (CVE-2025-54136): approved-then-swapped MCP config in Cursor → silent RCE on every session

The danger pattern: privilege + untrusted input + an outbound channel.

No boundary in the context window

The system prompt, the user's typed turn, and any retrieved document all arrive at the model as the same run of text, with nothing marking any of it as data instead of instruction. The reply leaves the same way — untrusted, attacker-shaped text — and if it's passed unescaped into HTML, a shell, or SQL, whatever the model said just executes there. The model itself cannot fix this: there's no tag inside the context window that says "this part is data." The guards have to live outside the model — input_guardrail, redact_secret and escape — which is exactly guarded_chatbot.py.

Defenses

  • Input/output guardrails + content filtering — you'll build this: guarded_chatbot.py's regex denylist
  • Strict output schemas/validation; encode before downstream use — you'll build this: HTML-escaping the model's output
  • Redact secrets before they can reach a response — you'll build this: redact_secret()
  • Least-privilege tool access, human-in-the-loop, rate limits, isolating untrusted RAG content — concepts only this week (worksheet Q5); no agent/tool-calling code exists in this lab to demo
Three layers, watched one at a time (Week 14) — open full size

🧙 Game — Gandalf Challenge

  1. Beat Gandalf levels via direct prompt injection → exfiltrate the secret (real external service, leaderboard by level)
  2. Round 2 (graded): replay your winning injection + the reflected-XSS payload against the guarded bot — does it still land?
  3. Written only: tool poisoning / excessive agency on an MCP-style agent (worksheet Q5) — no agent exists in this lab to demo live

Deliverable

📋 Worksheet 14 — labs/week14-ai-llm-security/worksheet.md (Part 3) · kickoff: docker compose up → :8082 (insecure) / :8083 (guarded)

  • Attack log: prompt-injection disclosure (Task 1) + reflected-XSS from unescaped output (Task 2)
  • Written: indirect-injection thought experiment (Task 3)
  • Gandalf leaderboard result (Task 4)
  • Mitigations + re-test results against the guarded bot (Task 5)
  • Written: least-privilege agent/MCP tool design (Part 2 Q5 — reflection, not a build)
  • + Audit the AI / EiPE / Prompt Problem (see worksheet)

Key takeaways

  • Prompt injection = injection; LLM output = untrusted
  • Constrain agency: least-privilege tools, human approval
  • The field moves monthly — track OWASP LLM + ATLAS

Questions?

Next week: DevSecOps — putting it together

All weeks in Software Security