The Abstraction Pattern

---

Over a year ago, I transitioned from product data scientist to AI engineer. The role is more technical than I expected, but it revealed something interesting: we’re not experiencing a new disruption—we’re following a pattern that’s been repeating for decades.

---

The transition

Moving from product data science to AI engineering was a significant shift. Product DS is broad and less technical—you’re wrangling data, building quick models, interpreting metrics, talking to stakeholders. You can get away with knowing how to use the tools without understanding the machinery beneath them.

AI engineering is different. It’s deeper, more technical. You need to understand model architectures, fine-tuning strategies, inference optimization, and system design. The problems are harder. The details matter more.

That said, my background as a data engineer helped tremendously. Working with big data systems—distributed computing, data pipelines, infrastructure—taught me how to think about scale, system design, and the tradeoffs between different approaches. Those skills translated directly. AI engineering is, in many ways, data engineering at a different layer.

---

The real story: abstraction levels

But here’s what’s actually happening: AI isn’t disrupting software engineering. It’s just the latest step in a century-old pattern of abstraction.

Consider the history:

Assembly and low-level languages — You wrote CPU instructions directly. You managed memory, registers, every clock cycle. The programmer’s burden was enormous, but so was the mastery required and the control afforded. The people who did this well were exceptional.

Higher-level languages like C and C++ — Suddenly, you didn’t have to manage memory manually (well, mostly). The language abstracted away register allocation and low-level details. You could focus on functionality instead of mechanism. But those assembly experts? Many became C experts and systems programmers—their low-level knowledge made them invaluable when performance mattered. We still needed them.

Python and scripting languages — The computing power became cheap enough that raw CPU efficiency didn’t matter as much. Python abstracted away even more: memory management, complex type systems, compilation. You could write more code with fewer people, focusing almost entirely on what you’re building rather than how the machine executes it. But the infrastructure underneath—the systems optimized by people who understood C and assembly—still existed. Those engineers didn’t disappear. They built the platforms the rest of us use.

And now: AI and prompting — The pattern continues. Why write 100 lines of Python when you can write a prompt and let a model generate it? Why debug intricate business logic when you can describe the desired behavior and let the AI figure it out? But someone still needs to build and optimize the models, manage the infrastructure, understand the constraints and failure modes.

---

The pattern is consistent

Each transition follows the same logic:

1. Previous era: Programmers spend 80% of their effort on mechanics, 20% on functionality
2. New abstraction: Someone says “what if the machine handled the mechanics?”
3. Transition period: Skepticism, hybrid approaches, figuring out the tradeoffs
4. New era: Programmers spend 20% of effort on mechanics, 80% on functionality

When C arrived, assembly programmers said “it’s too slow, you can’t do real things in it.” When Python arrived, C programmers said the same. Now, engineers hesitate about AI, saying “it’s unreliable, you can’t use it in production.”

They’re right that the tradeoffs exist. But the trajectory is clear.

---

What changed

The job of a software engineer hasn’t disappeared with each abstraction—it’s transformed.

• Assembly programmers didn’t vanish when C arrived; they became systems programmers and compiler engineers. We still need people who understand the metal.
• C programmers didn’t disappear when Python arrived; they built the runtimes, databases, and infrastructure that Python depends on. That work is harder and more specialized, not less valuable.
• Now, engineers who understand ML and system design can become AI engineers—building applications that leverage these models. But we still need people deep in the stack: training engineers, infrastructure teams, people optimizing the models themselves.

AI engineering isn’t about building autonomous agents or systems that act independently. It’s about building applications with AI as a component. The shift is in how you construct software: instead of implementing business logic directly, you might use a language model to generate code, classify data, or produce text. But the lower-level work—the harder, more specialized work—that’s where the real constraints live. The people doing that work are essential.

---

A closing thought

This is the continuation of a 70-year trend: make it easier for humans to express intent, let machines handle more of the mechanics. The role shifted from “how do I make the machine do this?” to “what do I want the machine to do?”

Each time we’ve made this jump, we’ve gotten better at building applications—because people at every level kept doing their work. The assembly experts became systems engineers. The C programmers built the infrastructure Python runs on. The skill for each generation has been recognizing which problems benefit from which abstractions, and having the expertise to implement at that level.

AI engineering is that: understanding where and how to integrate AI into applications you’re building. And behind it all, the people doing the lower-level work—training models, building inference systems, optimizing infrastructure—that work is harder, more specialized, and absolutely critical. Those are the engineers I have tremendous respect for. The abstraction only exists because they built it.