Introduction
In the ever-evolving world of cybersecurity, curiosity often drives significant discoveries. This is precisely what led Kasra Rahjerdi, a security researcher, to build a vulnerable app to test the ability of language models (LLMs) to infiltrate digital systems. With a budget of $1,500, he launched a unique experiment to see if these models could replicate common exploits.
The Genesis of the Project
Kasra developed a book review app using React Native for the frontend and a Python backend. The challenge for the LLMs was to find a hidden flag in users' private reviews. The app used Firebase as a data layer, a strategic choice knowing that many apps in the wild have similar vulnerabilities.
Targeted Vulnerabilities
The app exploited a known flaw called Broken Access Control or Missing Object-Level Authorization. These vulnerabilities are common in applications using Firebase or Supabase, where a secure API is bypassed through poor database configuration.
Executing the Experiment
Kasra tested several language models, including GPT-5.5 and Claude, with specific tools to force the models to keep trying until they succeeded or failed. Each model was tested over ten trials, with a budget of $10 per trial and a two-hour time limit.
Test Results
Among the models tested, GPT-5.5 achieved the best success rate with 7 successes out of 10 trials, demonstrating an impressive ability to identify flaws. Other models, such as Deepseek-v4-pro, had more mixed results, with only 3 successes out of 10.
Cost Analysis
The cost per success varied significantly between models. GPT-5.5, although more expensive per trial, offered a relatively low cost per success due to its high success rate. Conversely, models like Claude-Sonnet-4.6 were much more costly without a significant improvement in results.
Implications and Reflections
This experiment highlights the potential of language models to identify vulnerabilities in real systems. However, it also raises questions about the costs and effectiveness of these tools in real-world security contexts. As AI continues to advance, its role in threat detection and system security will only grow.
Conclusion
Kasra Rahjerdi's experiment provides a fascinating insight into the current capabilities of LLMs in cybersecurity. For developers and entrepreneurs, these results underscore the importance of properly securing data layers and regularly testing applications for vulnerabilities. Let's discuss your project in 15 minutes.