We’re living in a time when AI tools like GitHub Copilot and Cursor have become a regular part of the software development workflow. Developers can now crank out code at lightning speed—sometimes in half the time it used to take. But as the pace picks up, something worrying is starting to emerge. Underneath all the “LGTM” approvals and clean pull requests, cracks are forming in the foundations of software quality, system reliability, and security.
1. More Code, Less Clarity
AI tools are great at producing loads of boilerplate and quick solutions. But here’s the thing: they often generate code without truly understanding the context. That means:
- The code might technically work, but no one really understands why it works;
- It misses edge cases or fails under less common conditions;
- It doesn’t adapt well to complex or evolving environments.
This becomes particularly dangerous in parts of a system that really matter—infrastructure, security, configuration, or anything financial. What seems fine on the surface could be a ticking time bomb.
2. Code Reviews Are Becoming a Checkbox
A lot of seasoned engineers are now finding themselves reviewing AI-generated code that’s hard to follow. Not because it’s complex—but because it was written by a model, not a person who deeply understands the system. It’s even worse when reviewers start to blindly trust AI output too.
What happens then?
- Reviewers give the thumbs-up without asking hard questions;
- Bugs sneak into production because no one really owned the logic;
- SDKs and APIs silently break because assumptions were never clarified.
Over time, these small cracks add up to a fragile codebase.
3. The Real Trouble Comes When Things Go Wrong
In AI-heavy codebases, everything often looks smooth—until it doesn’t. All it takes is one weird input, one stale system state, or one poorly-timed request, and the system collapses.
That’s because the last layer of defense—the layer that’s supposed to validate, recover, or fail gracefully—was never really built. Or worse, it was suggested by an AI model and no one thought to double-check it.
This is where things get dangerous. Systems are being stitched together by auto-suggestions, not design. And that’s not sustainable.
So What Should We Do About It?
AI-assisted coding isn’t going away—and it shouldn’t. But we need to be a lot more intentional about how we use it. Here are some practices worth considering:
- Log AI-Generated Code: Track which code was generated by AI, who prompted it, and when.
- Require Real Tests: If AI writes it, someone needs to write a test for it. Especially for anything with side effects.
- Protect the Critical Stuff: For anything involving security, networking, or authentication, AI-generated code should be off-limits.
- Rethink Code Review: Make sure at least one reviewer actually understands the domain, not just the syntax.
In the rush to ship faster, we can’t lose sight of the things that keep systems stable and secure. It’s not just about producing more code—it’s about building software that lasts, even when things go sideways.
Potential Technical Black Swan
I have heard of people in some tech companies like Google, Microsoft and Meta generating code using AI as described above. I predict this will lead to a disaster in these tech companies’s infrastructures and products and then a huge dump on their stock prices (aligned with some potential financial crisis). Fortunately, if such black swan really happened, I think there are new opportunities to build a better AI-generated world.