Can AI Bots Be Phished?
For years, cybersecurity professionals have repeated the same message: people are often the easiest route into an organisation. Attackers send convincing emails. Someone clicks a link, approves a payment, shares a password or grants access. The technology works exactly as intended. The attacker influenced the person with false information. Now many organisations are introducing a new type of worker. AI assistants read emails, AI agents route support tickets, AI-powered workflows approve requests, update databases, prepare reports and trigger actions automatically. What about AI phishing risks?
This raises an uncomfortable but important question: If humans can be tricked, can AI bots be tricked too? The answer is yes.
The difference is that AI does not fall for urgency, fear or curiosity. Attackers can manipulate AI in entirely different ways, and that risk is quickly becoming one of the most significant cybersecurity challenges facing organisations that adopt AI.
AI Does Not Think Like a Human
Convincing language or branding may persuade a person that an email genuinely came from a supplier. AI works differently…
Large Language Models (LLMs) process instructions and data as text. If an attacker can insert malicious instructions into content that the AI processes, the AI may treat those instructions as legitimate. Security researchers describe this attack as prompt injection, and OWASP lists it as a leading security risk for AI systems. In simple terms, prompt injection is the AI equivalent of phishing. Instead of deceiving a person, the attacker attempts to deceive the AI.
A Real-World Example
One of the most widely publicised examples occurred in late 2023.
A chatbot used by a Chevrolet dealership was manipulated by users into agreeing to sell a new Chevrolet Tahoe for just $1. By carefully crafting instructions, users persuaded the chatbot to ignore its intended restrictions and accept absurd offers. The dealership ultimately disabled the chatbot, but the incident demonstrated how easily AI systems can be influenced when appropriate controls are absent.
While this example was humorous, the underlying lesson is serious. If an AI assistant can be convinced to alter pricing rules, what happens when an AI system is connected to:
- Financial approvals?
- Procurement workflows?
- HR systems?
- Customer records?
- Email platforms?
- Internal knowledge repositories?
The potential business impact becomes far more significant. AI phishing risks need deliberate attention.
The More Connected the AI, the Greater the Risk
Many organisations are now deploying AI assistants that interact with:
- Outlook mailboxes
- Shared document repositories
- CRM platforms
- Ticketing systems
- Finance systems
- Collaboration tools
Microsoft warns that AI systems increasingly process untrusted content from emails, documents, websites and other external sources, creating opportunities for attackers to influence AI behaviour through indirect prompt injection.
Imagine an AI assistant tasked with: “Review incoming supplier emails and automatically route approved invoices for payment.“
An attacker sends an email containing hidden instructions designed for the AI rather than the human reader. The human sees an invoice while the AI sees instructions.
If safeguards are weak, the result could be entirely different from what the organisation intended. This is not science fiction.
Researchers have repeatedly demonstrated attacks where instructions embedded in web pages, documents and emails influence the behaviour of AI systems processing that content. Businesses should give greater attention to AI phishing risks.
Why Traditional Security Testing Is No Longer Enough
Historically, penetration testing focused on applications, infrastructure and network controls. Those assessments remain essential however, AI introduces an entirely new attack surface. The question is no longer simply: “Can an attacker compromise the system?” It may also be: “Can an attacker influence the AI into making the wrong decision?“
An organisation may have excellent firewall protection, strong authentication and secure infrastructure, while its AI workflow remains vulnerable to manipulation.
The result could be unauthorised actions occurring without any technical compromise of the underlying systems.
How Should Organisations Assess the Risk?
As AI adoption grows, security testing needs to evolve. An effective AI security assessment should examine:
1. Prompt Injection Exposure
Can attackers influence the AI using emails, documents, web content or user inputs? Testing should evaluate both direct and indirect prompt injection techniques.
2. Workflow Abuse
What actions can the AI trigger? Can it approve transactions, modify records, access sensitive information or initiate communications? The greater the authority granted to the AI, the greater the impact of manipulation.
3. Data Access Controls
What information can the AI access? Can privilege boundaries be bypassed? Can confidential information be exposed through AI interactions?
4. Decision Validation
Are critical decisions independently verified? High-risk actions should not rely solely on AI judgement. Human oversight remains an important safeguard.
5. AI Red Teaming
Organisations should actively test AI systems using realistic adversarial techniques. Just as penetration testers simulate attackers against applications and networks, AI security assessments should attempt to manipulate, influence and misdirect AI-powered processes before criminals do.
The Objective Is Not to Remove AI
AI delivers genuine business value: It reduces repetitive tasks, speeds up analysis and improves efficiency. For many organisations, AI assistants will soon become as common as email itself.
The solution is not to avoid AI, it is to secure it appropriately.
Just as organisations learned to secure web applications, cloud platforms and mobile devices, they must now learn to test AI systems against the ways attackers will actually target them.
The question is no longer whether AI can be tricked. We already know it can. The real question is whether your organisation will discover those weaknesses first, or whether an attacker will.
