How to Measure AI’s Business Impact

Measuring AI’s business impact means tracking whether an AI-assisted workflow changes an outcome the organization already values, such as inventory or delivery performance. That matters because usage counts and token totals show activity, not whether the work produced a better business result.

AWS executive Colleen Aubrey has argued for assessing AI with familiar business measures rather than creating AI-specific scorecards. Start with an outcome the team already tracks and record its baseline. A before-and-after change shows what moved during a period, but on its own it cannot separate AI’s contribution from other changes to the work.

Compare equivalent work, not just before and after

IBM describes a more informative comparison: similar groups perform the same project, with one group using AI and the other using traditional methods. IBM says to compare speed, quality, cost, and accuracy. Where practical, keep the task and conditions as similar as possible, then examine results by task-specific skill and account for how people adapt to the tool and integrate it into existing team workflows.

Choose the business outcome first, then compare equivalent work before calling faster output a win.

This approach helps distinguish a change observed during an AI rollout from a result associated with AI-assisted work. It does not make every difference attributable to the tool; the comparison still depends on how closely the groups and their work match.

What reported AWS examples measured

SiliconANGLE reported AWS-attributed results for Wells Vehicle Electronics: inventory fell 7% over three months while the number of actively monitored SKUs grew from 20 to 2,000. The inventory measure describes a business outcome; the SKU count describes the expanded monitoring scope.

The same interview account said Aubrey’s group doubled its production-push velocity after improving test coverage and pipeline health. That account connects the change to broader engineering workflow improvements, rather than isolating a coding assistant’s effect. SiliconANGLE also reported that TE Connectivity had scaled Connect Decisions to 2,000 planners, a measure of adoption rather than a business outcome on its own.

Software teams need measures across the delivery cycle

For software work, code volume and the speed of a first draft can miss time spent reviewing, testing, maintaining, or addressing security issues. GitLab recommends evaluating AI-assisted development across the software development life cycle (SDLC), from concept through production, and balancing delivery measures with quality, security, and downstream effort.

GitLab’s DORA measures are specific to software delivery, not a universal scorecard for every AI project. For a software pilot, pairing them with review, testing, maintenance, and security considerations helps account for work that may happen after code is generated.