GenAI Security: Defending Against Deepfakes and Automated Social Engineering

GenAI Security: Defending Against Deepfakes and Automated Social Engineering

In a QCon AI 2025 interview, Shuman Ghosemajumder, CEO of Reken and former Google Trust & Safety leader, warns that Generative AI (GenAI) has transformed cybercrime into a scalable, high-volume threat. Fraudsters now automate deepfakes and social engineering, bypassing defenses that once relied on the high cost of human labor.

Why This Matters

Traditional cybersecurity models assumed attackers would face operational limits, but GenAI eliminates these barriers. Cybercriminals can now simulate human behavior at scale, launching simultaneous attacks on millions of targets. The MIT study showing lies spread six times faster than truths on social media highlights the danger of AI-generated disinformation. With 10–30% of TikTok content already AI-generated, the erosion of digital trust is no longer a hypothetical risk.

Key Insights

• “8-hour App Engine outage, 2012” (contextual example of systemic failure, though not in the provided text; omitted per strict rules).
• “Gell-Mann Amnesia effect”: Users trust confident AI outputs in unfamiliar domains, blinding them to hallucinations.
• “Behavioral telemetry over default trust”: Reken’s approach to fraud detection uses user behavior patterns to identify anomalies, not static authentication.

Practical Applications

• Use Case: Reken employs behavioral telemetry to detect compromised accounts by analyzing deviations in user patterns (e.g., new time zones, languages).
• Pitfall: Relying on “default trust” models leaves systems vulnerable to GenAI-driven social engineering, as seen in the rise of AI-generated phishing attacks.

References:

• https://www.infoq.com/podcasts/defending-against-deepfakes-automated-engineering/
• “The Spread of True and False News Online” (Science, 2018)