> Applying Security by Design and Cyber Immunity principles to protect AI systems | Kaspersky official blog

AI systems are much harder to protect than traditional IT — whether from operator mistakes, malicious interference, or just plain unstable behavior. There are two main reasons for this. First, AI outputs are probabilistic, so the same prompt can give different answers each time you run it. Second, today’s LLMs can’t reliably tell commands apart from data. Despite the demonstrated security issues of AI systems that we’ve covered many times, it’s impossible to ignore the potential economic payoff of adopting them. That’s why many companies are moving beyond simple AI assistants to AI agents. These agents carry out tasks on their own and have direct access to the same tools as humans — from reading databases to changing settings in production processes. Our experts, in their new study, examine how to clearly define and minimize the resulting risks to keep potential problems to a minimum. The approach borrows and adapts the principles of Cyber Immunity and Security by Design. The basic principles of Security by Design All the principles of this discipline are universally applicable to the design of any computer system — AI agents included. What differs is how deeply each principle needs to be adapted and reworked for AI scenarios. Security is a separate and important factor right from the design stage. Security goals and assumptions are set on day one and followed throughout every stage of development. For AI agents, these assumptions should include three key points: (i) the language model is treated as untrusted, (ii) external data of any kind is considered potentially malicious, and (iii) irreversible actions can’t be performed autonomously without mandatory human approval. The system is designed on the assumption that it will be attacked. The trusted computing base and attack surface are kept to a minimum. Agents get only the tools they actually need, limited permissions, and the smallest amount of context possible. Security policy enforcement is handled entirely by deterministic components: the policy monitor, the agent harness, and isolation mechanisms (sandboxes). The principle of least privilege applies. Each agent has its own set of credentials and access privileges (identity), limited to the bare minimum it needs to do its job. If an agent temporarily needs elevated privileges, it gets them only after external approval, and they last exactly as long as the task that requires them. Components are isolated. In agent-based systems, this is one of the most complex and multi-layered architecture problems. The agent harness should run in an isolated environment (sandbox or virtual machine), and agents should be kept as isolated from each other as possible. Data processing in trusted and untrusted contexts is also kept separate. For example, one agent reads a pull request and writes a draft of the code, while another reviews that work without ever seeing the potentially malicious text of the request. Default deny is the rule. Security policies follow the rule: anything not explicitly allowed is forbidden. This includes tool calls, interactions between agents, and network traffic. Defense in depth is in place. Different measures should work at the same time and reinforce each other. Probabilistic defenses — like AI guardrails (a “firewall” for language models) or alignment (the model’s own sense of what’s acceptable and what isn’t) — can’t offer any formal security guarantees. Nevertheless, they make the system more secure overall and serve as a valuable source of telemetry. The supply chain is protected. Some agents can install skills, libraries, and other software components on their own, so these abilities need strict rules. Either block them completely, or limit them to a closed, pre-approved list of specific versions of specific components. Systems built on cloud LLMs face an extra risk: model updates. One day, the same API request might be answered by a different language model with different security properties. Pitfalls of AI security The general principles of Security by Design are well known and look simple on paper. But applying them to AI systems is tricky because agents are incredibly flexible. For example, a security team might believe that an agent on a developer’s computer runs in a sandbox and can’t change anything outside it. But an attacker, or even the agent itself, can come up with dozens of ways around this seemingly solid protection: from running commands through a local MCP server that the sandbox doesn’t restrict, to reconfiguring another AI agent installed on the system. To make things even harder, the range of attack vectors is huge: from arbitrary fields in the agent’s configuration to innocent-looking text strings in a skill file. One serious problem that designers often underestimate is user fatigue. A busy agent can throw up dozens of requests every minute, asking the user to confirm potentially dangerous actions. Within an hour, the person is clicking Approve on autopilot without reading what they’re approving. That’s why it’s important to carefully tune both how often these requests appear and how they’re worded. Nobody will read a request description that’s too long. And one that’s too short can hide the key details the user needs in order to make the right call. Vulnerabilities in agents and related tools are a pressing problem — and it’s made worse by how quickly these categories of software evolve and receive updates. Industry-standard platforms and apps — such as LangGraph, MS Semantic Kernel, and Claude Code — have had critical vulnerabilities patched more than once. Architects should always consider swapping a large, do-it-all framework for a compact, stripped-down system that has only the features a specific agent actually needs. On top of that, carefully implementing standard, long-established security measures in the available tools greatly reduces the damage if an agent is attacked or behaves unexpectedly. One example is a strict allowlist of domains that the web fetch tool can open when asked by the agent. But designers rarely account for the hidden problems and overhead associated with maintaining the list. As a result, it slowly goes out of date and leaves loopholes for attackers. The most striking example is the EchoLeak attack. Microsoft 365 Copilot had a properly implemented and fairly effective allowlist for accessing external sites, but attackers circumvented it through a feature in a secondary tool and the MS Teams domain. Practical tips for architects and security teams AI system architects and heads of engineering and security teams should build in agent security from the design stage. They should rely primarily on verifiable, deterministic mechanisms: isolating components, controlling interactions, and keeping the trusted computing base to a minimum. Key architectural decisions include defining acceptable autonomy for the agents, limiting access to confidential data and dangerous combinations of capabilities, and using sandboxes, separate roles for different models, and access control gateways. Restrictions added as an afterthought reduce risks, but they’re no substitute for secure architecture. At the organizational and operational levels, companies need to keep an inventory of their agents, coordinate the efforts of security and deployment teams, protect container environments, manage the supply chain (AIBOM), and integrate multi-layered defense. Observability requires special attention. For AI systems, this should include logging the LLM’s requests, responses, and reasoning, as well as detailed telemetry from the hosts and cloud environments where the agent harness runs. The hardest part of designing agents is balancing security, performance, and usefulness. To learn how to strike that balance with threat modeling and clearly defined security goals, read the full version of the study.