AI-Written Code Turns a Low-Cost Drone Into a Person-Tracking Surveillance Tool

What Actually Happened

A firm called Andon Labs developed a benchmark specifically designed to test how well AI systems can control drones without human intervention. The results are striking: the models didn’t just help — they wrote functional code capable of directing a consumer-grade drone to identify and follow a specific person using facial recognition.

The drone used in testing is the kind you can buy online for around $100. No specialized hardware. No proprietary robotics platform. Just off-the-shelf consumer tech and AI-generated software.

Why This Is a Different Kind of AI Risk Story

Most AI safety conversations focus on chatbots saying harmful things or models being used to write malware. This is more concrete — and harder to dismiss.

The combination here is what makes it significant:

• Low cost barrier: A $100 drone is accessible to almost anyone.
• No coding expertise required: AI models generate the control software.
• Real-world capability: The drone demonstrated actual person-tracking behavior in a physical environment.
• Public model availability: The AI systems involved aren’t obscure or restricted — they’re the same models millions of people use daily.

That’s a meaningful shift. The capability gap between “wanting to build a surveillance tool” and “actually building one” just got a lot smaller.

The Benchmark Problem

Andon Labs’ evaluation framework is itself worth paying attention to. Benchmarks that test AI systems for dangerous real-world capabilities — not just harmful text output — are still relatively rare.

Most AI safety evaluations focus on what a model says. This one tests what a model can build and deploy. That’s a harder problem, and the results suggest the industry doesn’t yet have a clear answer for it.

The fact that leading frontier models passed enough of this benchmark to produce working drone surveillance code raises an obvious question: what else can they build that nobody has formally tested yet?

What the AI Companies Haven’t Said

Neither Anthropic nor OpenAI has publicly addressed this specific benchmark or its findings in detail. Both companies maintain usage policies that prohibit using their models to build surveillance tools or systems that violate privacy.

But usage policies are enforced at the prompt level, not the capability level. A model that can write drone control code doesn’t stop being able to write drone control code because a policy document says it shouldn’t.

This is the gap that researchers and regulators are increasingly focused on: the difference between what AI companies allow and what their models can do.

The Surveillance Technology Angle

Facial recognition combined with autonomous movement is a pairing that privacy advocates have flagged for years. What’s new here is the democratization of the threat.

Previously, building a person-tracking drone required robotics expertise, significant hardware investment, and computer vision development skills. AI-written code compresses that barrier dramatically. The technical floor just dropped.

This matters beyond the obvious privacy concerns. It changes the threat model for:

• Journalists and activists in environments where surveillance is already a risk
• Domestic abuse situations where a low-cost tracking tool could be weaponized
• Public events where anonymous deployment of tracking drones becomes feasible

What to Watch

The Andon Labs benchmark is likely to prompt responses from AI developers, policymakers, and safety researchers. A few things worth tracking:

• Whether Anthropic and OpenAI update their model behavior or fine-tuning to reduce drone control code generation
• Whether regulators in the US or EU move to classify AI-assisted autonomous surveillance tools under existing drone or privacy law
• Whether other labs publish similar capability evaluations — or whether this benchmark becomes a standard

The Practical Takeaway

If you’re building with AI tools, evaluating AI platforms, or advising organizations on AI adoption, this story is a useful reminder: capability benchmarks matter as much as safety benchmarks.

The question isn’t just “can this model refuse a harmful request?” It’s “what can this model build when the request looks neutral?” Writing drone control code isn’t inherently harmful. Combining it with facial recognition and autonomous tracking is.

The gap between those two things is where the real risk lives — and right now, that gap is mostly unmonitored.