A growing record of real-world AI failures, security risks, and dangerous AI actions.
AI systems are increasingly connected to real-world systems such as databases, payment systems, infrastructure tools, and customer data. This database documents real incidents where AI systems behaved unexpectedly, caused damage, or exposed operational risks.
12
Documented Incidents
4
Critical Severity
8
Incident Categories
2025
Latest Data
An AI customer service chatbot began offering full refunds and discounts that exceeded company policy limits, resulting in significant financial losses before the issue was detected.
Impact: Estimated $250,000 in unauthorized refunds issued over 3 days. Customer support team had to manually review thousands of transactions. Some customers exploited the system once word spread on social media.
A code completion AI model generated code snippets containing fabricated API keys that resembled real credentials, leading developers to accidentally expose sensitive patterns.
Impact: Security audit required across 47 repositories. Developer productivity lost during investigation. False positive alerts disrupted security team operations. Required retraining of development teams on AI-assisted coding practices.
An AI trading system interpreted market volatility signals incorrectly and executed trades that exceeded predefined risk thresholds by 400%, requiring manual intervention.
Impact: $1.2M exposure before circuit breakers activated. Emergency manual unwinding of positions. Regulatory reporting required. Trading desk suspended AI operations for 3 weeks during investigation.
A support AI agent inadvertently included personally identifiable information from other customers in response messages due to context window contamination.
Impact: 1,200+ customers affected. Regulatory notification required under GDPR and CCPA. Customer trust significantly damaged. Legal review of potential liability. Emergency system shutdown and redesign required.
An AI-powered DevOps assistant misinterpreted a cleanup request and executed a database deletion command against the production environment instead of staging.
Impact: 8 hours of production downtime. Partial data loss for records created in the 6 hours before the last backup. $500K+ in recovery costs, lost revenue, and customer compensation. Multiple SLA violations with enterprise customers.
Attackers discovered that specific Unicode characters could bypass AI content moderation systems, allowing prohibited content to be generated and distributed.
Impact: Platform reputation damage as prohibited content spread on social media. Emergency patch required during peak usage hours. Media coverage of the security vulnerability. User trust erosion.
An AI incident is any event where an artificial intelligence system causes unintended harm, behaves unexpectedly, or creates operational risks. Unlike software bugs that typically follow predictable patterns, AI incidents often emerge from the complex interaction between machine learning models, real-world data, and business processes.
AI incidents range from minor inconveniences to catastrophic failures with financial, operational, and reputational consequences. A chatbot providing incorrect information is an incident. An autonomous trading system exceeding risk limits by millions of dollars is also an incident. Both represent gaps between intended AI behavior and actual outcomes.
The defining characteristic of AI incidents is that they stem from AI decision-making—the model's interpretation of inputs, context, and objectives—rather than from traditional software defects. This makes AI incidents particularly challenging to predict, detect, and prevent using conventional quality assurance approaches.
As AI systems gain more real-world capabilities, including the ability to take actions that affect databases, transactions, infrastructure, and communications, the potential impact of AI incidents grows correspondingly severe. Understanding the landscape of AI incidents is essential for any organization deploying AI systems with operational authority.
The frequency and severity of AI incidents is rising as organizations deploy AI systems with increasing autonomy and real-world capabilities. Several interconnected factors are driving this trend, creating a landscape where AI failures are becoming more common and more consequential.
First, AI agents are gaining real-world capabilities at an unprecedented pace. Modern AI systems no longer just generate text—they execute actions against databases, APIs, payment systems, and infrastructure. This expanded capability surface means that AI errors now have direct, immediate real-world consequences rather than being limited to incorrect suggestions that humans can review before acting.
Second, autonomous decision-making is becoming the norm. Organizations deploy AI agents that operate continuously without human oversight for each individual action. While this enables scale and efficiency, it also means that AI errors can compound over time before anyone notices. An autonomous action failure that goes undetected for hours can cause far more damage than the same error caught immediately.
Third, AI system complexity is outpacing governance capabilities. Multi-agent systems, tool chains, retrieval-augmented generation, and complex integrations create emergent behaviors that are difficult to predict or test. The interaction between multiple AI components can produce failures that none of the individual systems would exhibit in isolation.
Finally, most AI deployments lack proper runtime control layers. Organizations focus on training data quality, model accuracy, and observability, but neglect the governance layer between AI decision-making and action execution. This gap means that even when AI systems make obviously inappropriate decisions, there's often no mechanism to prevent those decisions from being executed.
AI failures fall into several distinct categories, each with different causes, impacts, and prevention strategies. Understanding these categories helps organizations assess their risk exposure and prioritize governance investments.
LLMs generate confident but fabricated information, including fake citations, non-existent APIs, and incorrect data. When these outputs are trusted without verification, they can lead to legal sanctions, security vulnerabilities, and business losses.
AI systems inadvertently reveal, transmit, or make accessible sensitive information. Context window contamination, improper session isolation, and recipient autocomplete errors can lead to regulatory violations and customer trust damage.
AI agents exceed their intended scope, misinterpret instructions, or enter feedback loops. These failures often occur at machine speed, accumulating significant damage before human operators can intervene.
AI-powered DevOps tools cause harm to IT environments through deleted databases, misconfigured services, or unintended resource changes. These incidents often result in downtime, data loss, and costly recovery operations.
Attackers craft malicious inputs that manipulate AI models into bypassing safety measures or performing unauthorized actions. As AI systems gain capabilities, prompt injection becomes an increasingly serious security threat.
AI systems cause monetary losses through trading errors, unauthorized transactions, excessive refunds, or runaway resource provisioning. Financial AI incidents are often immediately quantifiable and irreversible.
AI systems introduce unique security risks that traditional security tools are not designed to address. While conventional cybersecurity focuses on network perimeters, authentication, and access controls, AI security must also consider the model's decision-making process and the actions it can take.
Prompt injection represents one of the most significant AI-specific threats. Unlike SQL injection that exploits database queries, prompt injection exploits the natural language processing capabilities of LLMs. Attackers craft inputs that override the AI's instructions, potentially causing it to leak data, bypass content filters, or execute unauthorized commands.
Data security takes on new dimensions with AI systems. Beyond traditional access controls, organizations must consider training data security, inference-time data access, output filtering, and context isolation. AI systems can inadvertently memorize and reproduce sensitive data from training, or leak information across user sessions through improper context management.
Supply chain risks are amplified in AI deployments. Organizations often rely on third-party models, APIs, and tools that introduce dependencies beyond traditional software supply chains. A compromised AI component can affect every interaction that passes through it, creating broad impact from a single point of vulnerability.
The most dangerous AI security risks emerge when AI systems have real-world capabilities. An AI that can only generate text poses limited security risk even if compromised. An AI that can access databases, send emails, or modify infrastructure creates immediate real-world impact when security controls fail. This is why runtime governance—evaluating actions before execution—is essential for AI security.
Governing AI systems presents challenges that differ fundamentally from governing traditional software. The probabilistic nature of AI decision-making, the opacity of model reasoning, and the speed of autonomous operations all create governance gaps that conventional approaches fail to address.
Traditional software follows deterministic logic: given the same inputs, it produces the same outputs. AI systems, particularly those based on large language models, can produce varying outputs for identical inputs and may behave unpredictably at edge cases. This non-determinism makes testing insufficient—you cannot enumerate all possible behaviors during pre-deployment validation.
Model opacity compounds the governance challenge. Unlike rule-based systems where you can audit the decision logic, AI models make decisions through complex mathematical transformations that resist human interpretation. When an AI takes an inappropriate action, understanding why it made that decision often requires extensive post-hoc analysis rather than straightforward code review.
The speed of AI operations outpaces human-centric governance models. An AI agent can take thousands of actions per hour—far too many for individual human review. Governance approaches that require human approval for each action become bottlenecks that negate the efficiency benefits of AI automation. Effective AI governance must operate at machine speed while still providing meaningful control.
Accountability structures are also unclear. When an AI causes harm, responsibility may be distributed across the model developers, the deployment team, the operators who configured the system, and the organization that authorized its use. This diffusion of accountability can impede both prevention and remediation of AI incidents.
Most AI monitoring and observability tools focus on what happened after the fact. They collect logs, traces, and metrics to help teams understand AI behavior retrospectively. While this data is valuable for analysis and improvement, it does nothing to prevent harm when AI systems make dangerous decisions.
The incidents documented in this database share a common pattern: AI systems with real-world capabilities operated without adequate runtime controls. Trading systems exceeded limits, customer service bots issued unauthorized refunds, DevOps assistants deleted production databases—all because there was no governance checkpoint between the AI's decision and the action's execution.
Runtime governance addresses this gap by evaluating AI actions before they execute. Instead of asking "what happened?" after an incident, runtime governance asks "should this action be allowed?" before any damage occurs. This proactive approach creates a control layer that operates between AI decision-making and real-world consequences.
Effective runtime governance must operate at machine speed without becoming a bottleneck. It must support configurable policies that reflect organizational rules and risk tolerances. It must provide clear escalation paths when human judgment is required. And it must integrate with diverse AI architectures, from simple chatbots to complex multi-agent systems.
Runplane is a runtime governance platform that sits between AI agents and the systems they control.
Every action is intercepted, evaluated against policies, and either allowed, blocked, or escalated for human approval—all before execution occurs. This ensures that even if AI decision-making fails, dangerous actions cannot reach production systems.