Chapter 2: The LLM Application Ecosystem

2.1 Anatomy of an LLM-Powered Application

Before you can secure an LLM-powered application, you must understand its moving parts. While implementations vary, a typical architecture involves several key components that work in concert. A user's prompt is not sent directly to the LLM; it journeys through a series of systems, each presenting a unique attack surface.

LLM Application Architecture Diagram

Let's trace the lifecycle of a prompt through this ecosystem:

  1. User Interface (UI): The user enters a prompt into a web or mobile interface.
  2. Application Backend: The application's server receives the prompt. It may perform pre-processing, add contextual information (like conversation history), or apply business logic.
  3. LLM API: The backend securely calls the third-party LLM API, sending the processed prompt.
  4. Data Stores: The backend may read from or write to databases to retrieve user data or store conversation logs.
  5. External Tools/Plugins: In advanced cases, the LLM might be authorized to call external APIs (e.g., a search engine, a booking system) to fulfill the request.

2.2 Identifying the Attack Surfaces

Each component in our diagram is a potential entry point for an attacker. We must expand our view of security beyond just the backend code.

  • UI: Vulnerable to traditional web attacks like Cross-Site Scripting (XSS), but also the initial entry point for malicious prompts.
  • Application Backend: A primary target, susceptible to both classic vulnerabilities and new logic flaws related to how it handles prompts and LLM responses.
  • LLM API: The connection itself can be attacked (e.g., Man-in-the-Middle), and insecurely stored API keys can lead to a full compromise.
  • Data Stores: A prime target for data exfiltration if an attacker can manipulate the LLM or application logic.
  • External Tools: The blast radius of an attack increases dramatically if the LLM has agency to interact with other systems.

2.3 Mapping Threats to the OWASP Top 10 for LLM Applications

The Open Web Application Security Project (OWASP) has developed a Top 10 list specifically for the risks associated with Large Language Models. This provides an invaluable, industry-standard framework for understanding and prioritizing threats.

The following is a summary. We will cover many of these in greater detail in subsequent chapters.
  • LLM01: Prompt Injection: Tricking the LLM into executing unintended instructions via clever user input. (Covered in-depth in Chapter 4).
  • LLM02: Insecure Output Handling: Failing to properly sanitize or validate the LLM's output before passing it to other systems, potentially leading to XSS or backend code execution.
  • LLM03: Training Data Poisoning: Corrupting the model's training data to introduce vulnerabilities or biases. (A key supply chain risk covered in Chapter 5).
  • LLM04: Model Denial of Service: Overloading the LLM with resource-intensive queries, causing service degradation and high costs.
  • LLM05: Supply Chain Vulnerabilities: Compromises in the third-party packages or pre-trained models used by the application. (Covered in-depth in Chapter 5).
  • LLM06: Sensitive Information Disclosure: The LLM inadvertently revealing confidential data from its training set or context.
  • LLM07: Insecure Plugin Design: Granting plugins excessive permissions or failing to handle their output safely.
  • LLM08: Excessive Agency: Giving the LLM too much autonomy to perform actions without sufficient oversight, which can be hijacked by prompt injection.
  • LLM09: Overreliance: Trusting the LLM's output without proper human review, leading to the propagation of misinformation or harmful content.
  • LLM10: Model Theft: An attacker stealing the proprietary LLM model itself, a significant threat to the vendor and their customers.

2.4 Threat Modeling Exercise

Let's apply these concepts to a hypothetical customer support chatbot. How would we threat model it?

We would ask questions like:

  • "Where in our architecture could an attacker inject a malicious prompt? The main chat input? A user profile field that the LLM reads?"
  • "What sensitive information does the chatbot have access to? Could it be tricked into revealing a user's order history?" (LLM06)
  • "The chatbot can process returns by calling our internal RMA API. What happens if an attacker hijacks that functionality through prompt injection?" (LLM08)
  • "Are we logging every user prompt? If so, are we protecting that sensitive data?"

This process of proactively identifying threats is the first and most critical step in building a secure LLM-powered application. In the next chapter, we will focus on the most important asset we need to protect: the data.

Previous: Previous: Introduction Next: Data Governance