How LLMs are secretly reshaping the code of tomorrow, and what to do about it
The next revolutionary programming framework might be dead before you even discover it. AI coding assistants are reshaping software development practices in some unexpected ways.
While many software engineers love using AI assistants to automate basic coding tasks and make our lives easier, we are inadvertently creating a feedback loop that discourages innovation in programming languages and frameworks. By understanding this feedback loop and deliberately designing for it, we might be able to fix the future of software development and avoid the pitfall of convenient stagnation.
A Golden Age for Incumbents, A Looming Crisis for Everyone Else
AI has fundamentally transformed the developer experience. Tools like GitHub Copilot, Amazon Q Developer, and other AI coding assistants have become as essential as IDEs themselves. After a short learning curve with LLM prompting, you can generate entire components, debug complex issues, and even architect system designs. What once took hours of referencing documentation and searching StackOverflow can now happen in seconds through a quick prompt.
Consider a typical frontend development workflow: a developer needs to create a data table component with sorting, filtering, and pagination. Today, an AI assistant could generate the entire React component in seconds, complete with TypeScript types, accessibility features, and modern best practices baked in.
But this incredible productivity boost comes with a catch that many haven’t recognized. While these tools feel like magic with mainstream frameworks like React, Express, or Django, try using them with a brand-new framework or lesser-known technology. The difference is stark: instead of confident, complete solutions, you get hesitant, error-prone suggestions. This performance gap is reshaping the entire development ecosystem.
The Dark Side of Biased Training Data
Here’s the brutal truth beneath the magic of AI coding assistants: they’re only as good as their training data, and that stifles new frameworks. Popular frameworks like React, Express, or Django come with decades of example code and documentation. LLMs excel at reading and writing code for these classic frameworks because of the countless variations of code they’ve been trained on. However, AI coding performance drops dramatically when working with newer or less mainstream technologies.
This creates a powerful feedback loop:
- Developers choose popular incumbent frameworks because AI recommends them
- This leads to more code being written in these frameworks
- Which provides more training data for AI models
- Making the AI even better at these frameworks, and even more biased toward recommending them
- Attracting even more developers to these incumbent technologies
The result? A winner-takes-all effect that’s already becoming visible in the ecosystem.
For example, I am a long-term JavaScript developer. For nearly 15 years, I’ve coded in JavaScript for Node.js. The JavaScript ecosystem is known for continual innovation and framework churn—a blessing and a curse that I’ve seen play out over the years. Recently, I’ve been writing more JavaScript for the Bun runtime. I’ve seen firsthand how LLM-based assistants try to push me away from using the Bun native API, back to vanilla JavaScript implementations that look like something I could have written 10 years ago. I have to explicitly push to get LLMs into the latent space of working with the latest cutting-edge JS technology.
The Unintended Consequences of Convenience
This feedback loop is concerning because it creates an uphill battle for innovation. Why would a team invest years of developer effort toward launching a new, potentially better programming language runtime or framework when they know it will face headwinds from AI assistants that are biased toward incumbent technologies? And why would a team invest in adoption of something new when they know there is an “AI productivity tax” in terms of efficiency lost from choosing a technology that AI assistants don’t understand well?
Here’s a provocative example: Imagine a world where ChatGPT was invented before Kubernetes reached mainstream adoption. In this world, I don’t think there would have ever been a Kubernetes. Some people might say that would have been a better world! But imagine if developers and operations folks of the time had been using ChatGPT, and they were funneled toward Mesos, bare Docker containers launched as services via systemd, or other pre-existing tooling. Without broader Kubernetes adoption, there would have been fewer examples of Kubernetes manifests, LLMs would have hallucinated endlessly while attempting to write Kubernetes manifests, and as a result, Kubernetes adoption would have been stifled before it even got off the ground.
We’re witnessing the emergence of two distinct development velocities: lightning-fast development in established technologies where AI assistants shine, and traditional, slower development in newer frameworks where developers must rely more on human capabilities.
Meanwhile, in the toxic swamps of old-school IT practice, some developers are choosing to write complex, abstracted code as a form of job security. After all, if your core systems are built on top of a bespoke framework that only you understand, then not only is your job secure from other humans, it’s also pretty secure against those meddling AIs! When major vendors overmarket with fear-based messaging about replacing SWEs with AI, it encourages this type of toxic “job security through obscurity” approach, leading to even more unmaintainable codebases and increased technical debt.
The New World of Framework Design
To thrive in this transformed landscape, framework designers who want their startup framework to reach a significant audience need to adapt their approach to designing and launching frameworks.
Framework design must start with carefully designed semantics. For a new framework to be successful, its API must be designed with method and parameter names that are easy for both humans and AI to understand. Instead of clever abstractions, consider clear, descriptive naming that follows well-established patterns. No more slapping together an API with the first parameter name that comes to mind! The framework design loop should involve providing samples of proposed code or API requests for your new programming language or framework to a baseline AI chat prompt. Ensure that the AI can understand what the API does without access to any documentation RAG or other unique knowledge.
Documentation also needs to evolve to serve both human and AI readers. There should be clear, consistent terminology, abundant examples covering all common use cases, and explicit relationship mapping between concepts. A new programming language or framework will require dramatically more investment in documentation compared to how we launched runtimes or frameworks in the past. Human documentation writers must treat LLM mastery of the framework as part of their mission. Documentation owners should regularly test to ensure that LLMs are finding, training on, and accurately recalling and applying details from the documentation.
But there’s another interesting problem: knowledge cutoff dates. LLMs take months to train, so if your framework makes a major breaking change, it could be six months to a year before the model even knows about this change. Even once the model knows about this breaking change, how does it know which version of the framework to promote? Most likely it will give additional weight to the version of the framework that is older, with more examples trained into the model.
Each major breaking “split” in a framework will dramatically decrease the quality of AI-generated results, just like it confuses and hinders your human software developer adopters. This means frameworks need to innovate with much more care and move slower. Maybe major breaking changes can no longer be launched via semver update but rather require a complete framework name change.
Proposal: Ensuring Accurate Framework Knowledge in LLMs
I do think there is a lot of room for improvement in documentation delivery mechanisms and integration of documentation with LLM-based tools. We have to solve the “knowledge cutoff date” problem if new frameworks are to have any hope of competing against incumbents. Fortunately, we have the tools to fix this, but it’s an integration problem that will require standardized coordination between several different components.
Remember in the old days of Linux, before widespread internet access, when the best way to access documentation was using the man command? Whenever you installed a Linux package, the package would also install its documentation locally. We need to go back, Marty! Back to the future!
In my ideal world, there is a package manager for documentation. It knows the versions of runtimes, libraries, or frameworks that are installed in your project. It “installs” the right documentation for the specific version of each framework or tool that you have installed, either by explicitly downloading and storing the documentation files locally for RAG purposes or by keeping an index of references to a remote Model Context Protocol (MCP) server that provides the versioned documentation for the packages you installed.
I’m particularly bullish on MCP, as it is an open standard designed by Anthropic. If we can get enough incumbent and startup frameworks to make versioned documentation available via a standardized MCP distribution mechanism, and we can get major AI coding assistants and code editors to integrate with this protocol, then whenever you chat with a model about a software framework, the model can refer to the specific documentation about the specific version of the frameworks you have chosen. This will be huge for AI accuracy (you will be dramatically less likely to get bad AI answers for an older framework version). It was also assist new framework owners in getting past the hurdle of early adoption. A new framework can bring its docs to the user’s LLM context, rather than needing to wait for the LLM to train the docs in.
In the meantime, this is something you can kludge together yourself. I want to write some proper content on how to glue together a solution for yourself, but in the meantime consider maintaining a set of system prompts for your LLM to explictly tell it what frameworks you are using and what versions of those frameworks. You may also need to supplement your LLM context with your own carefully curated RAG that contains the documentation for the specific framework versions you have adopted.
For Amazon Q Developer, consider adopting “Amazon Q customizations” as a way to supplement the LLM with additional information it may require.
Advice for Developers Working with AI
When evaluating new technologies, the software landscape has fundamentally shifted. Beyond traditional metrics like performance, community support, and technical capabilities, development teams must now consider AI compatibility as a critical factor. This means assessing how well AI coding assistants understand and generate code for the technology, how comprehensive the documentation is, and whether the technology follows conventional patterns that AI models can readily grasp.
During technology selection, carefully weigh the tradeoffs between choosing AI-friendly architectures versus newer, innovative frameworks. While established frameworks like React or Django offer immediate productivity gains through AI assistance, newer frameworks might provide breakthrough capabilities that justify the temporary loss in AI-assisted development speed. Consider creating a formal evaluation matrix that includes “AI compatibility” as a weighted factor alongside traditional evaluation criteria.
Contributing back to newer frameworks has become more critical than ever. The success of an innovative framework now depends not just on its technical merits but on how motivated its community is to help it reach the critical mass of examples and documentation that AI models need to train on. If you like a new framework, consider:
- Contributing documentation and examples, not just code
- Creating tutorials that explicitly compare patterns in the new framework to established ones
- Participating in community discussions to help establish clear, consistent terminology
- Building and sharing template projects that demonstrate best practices
- Creating training materials that help both humans and AI models understand the framework’s concepts
Call to Action
The future of software development doesn’t have to be controlled by today’s training data.
By understanding and actively addressing the impact that AI convenience has on software development practices, we can create a future where AI accelerates rather than stifles innovation in software development. The key lies in updating our tools and practices to work harmoniously with both human and artificial intelligence.
Last but not least: the code we write today becomes the training data of tomorrow. Let’s make every line count.