All posts
Jeremy Stover

Measuring what AI is doing to your code

engineeringai

AI coding assistants are astonishingly productive and almost completely unmeasured. Teams can tell you how much AI-authored code merged last week. Almost none can tell you what that code did to the shape of the codebase. That gap is the problem Semfora was built to close.

From volume to structure

Lines merged is a vanity metric. The questions that matter are structural:

  1. Did this change land in critical code, the load-bearing paths where a mistake ripples furthest?
  2. Did it raise or lower the structure score of the files it touched?
  3. Did it follow the codebase's existing patterns, or invent new ones?

Because we rebuild the map on every merge, each of these is a time series rather than a snapshot. You can watch a module's complexity climb commit by commit and step in before it becomes the file nobody wants to touch.

The shape of the problem

The numbers in this table are invented; the trajectory is the shape the metrics are designed to catch:

MergeStructure scoreCritical?
Week 10.71no
Week 40.63no
Week 70.52yes

By week seven the file is both harder to reason about and load-bearing. Complexity alone is survivable, and criticality alone is survivable. The combination is what shows up at the top of the percentile chart before an incident, and it arrives gradually enough that no single review catches it.

Governing it

The point is anything but slowing people down. AI assistants are going to keep shipping code faster than any team can review it, and the volume argument is already lost. What a team can still control is whether the structural cost of each change is visible while the change is cheap to reverse: on the pull request, next to the diff, computed the same way every time. That is where we put it.

Comments

to join the conversation.