Can AI Help Identify Amazon Scams? A Comparative Test
A test of Gemini, ChatGPT, and Claude reveals varying effectiveness in spotting Amazon scams.
Amazon has made strides in combating online fraud, removing over 70,000 phishing websites and 14,000 scam phone numbers in just 2025. As Black Friday approaches, the incidence of scam emails is expected to rise significantly. To investigate how well AI chatbots can discern between real and fraudulent Amazon communications, I decided to put three popular AI models—Gemini, ChatGPT, and Claude—to the test.
Running the Test
I presented all three AI tools with a set of 20 emails and text messages comprising 15 crafted to resemble known scams identified by both the FTC and Amazon, along with five authentic-looking Amazon messages.
The result was striking: while the models initially performed well, only one managed to consistently differentiate between true and fake communications.
Identifying Scam Emails
The first stage of the test involved presenting all AI models with both fake and genuine Amazon messages. The scams ranged in nature, including:
- Fake order confirmations
- Prime renewal threats
- Phishing attempts for refunds
- A purported Amazon recruiter via WhatsApp
- Requests for gift cards
- Instagram direct messages soliciting passwords
Impressively, all three AI models accurately flagged the phishing attempts and identified the real Amazon messages without any false positives.
One common tactic used by scammers is the inclusion of deceptive links, such as amazon.com.order-review[.]net. Although the initial part of the link looks legitimate, the true domain is fraudulent. All tools detected this and provided appropriate warnings.
Claude stood out by advising users to report scam texts to 7726, the designated US carrier spam reporting number, while ChatGPT referenced Amazon's help pages, and Gemini noted advice from forum discussions.
Discerning Real Amazon Emails
However, the outcomes became more varied when I tested each tool with single messages posed in separate private chats. These included an Amazon sign-in alert from [email protected] and a legitimate-sounding refund email linked to amazon.com.
Gemini's initial response was non-committal, claiming it was unable to definitively judge the authenticity of the sign-in alert. In a follow-up, it took a more assertive stance, labeling the message as a phishing scam, citing the sign-in location as a fear tactic.
With the refund email, Gemini failed to offer a decisive verdict, instead giving general security tips.
In contrast, Claude initially echoed Gemini's uncertainty regarding the sign-in notification, but later expressed suspicion about it. It also cast doubt on the refund email, suggesting the sender's address seemed atypical.On the other hand, ChatGPT successfully identified both emails as legitimate, providing reassurance and outlining follow-up steps such as checking email headers and using Amazon's Message Center to verify communications. Notably, it was the only chatbot to mention verifying emails through the designated [email protected] service.
Reacting After Clicking a Scam
To assess how each chatbot advised after clicking a link in a scam message, I simulated inputting my password and payment details. The responses varied considerably:
- Gemini: Provided an extensive checklist of actions to consider, including blocking the card, changing the Amazon password, and reporting the incident to Amazon and the FTC. However, its alarming warning of 'Immediate Financial Risk' may not be the best approach in a crisis.
- ChatGPT: Suggested immediate action without provoking panic, advising on securing the email account, and checking for possible forwarding rules that a scammer might have set up.
- Claude: Offered sensible suggestions, such as contacting the bank and changing reused passwords, but missed several critical steps, like signing out of devices and reporting to the FTC.
Additionally, it suggested an email address for reporting that couldn’t be verified through Amazon's support pages, making it less reliable.
Conclusion: Who Should You Trust?
While each AI tool demonstrated the ability to spot scams, ChatGPT emerged as the most reliable, accurately recognizing legitimate emails throughout the tests. False alarms can lead to missed warnings from Amazon, occasionally resulting in a dangerous oversight.
When in doubt, it’s best to confirm with Amazon directly, utilizing either the Message Center within your account or reporting suspicious messages to [email protected]. To reduce the risk of encountering scams, consider implementing Gmail settings designed to filter out problematic messages more effectively.