Learning Analytics: From Vanity Metrics to Decisions

Most L&D dashboards are an exercise in creative writing. They are filled with healthy-looking charts showing completion rates, satisfaction scores, and hours logged that ultimately tell us nothing about business value. These are vanity metrics: they make the department look busy, but they rarely trigger a meaningful change in strategy or investment.

As a project manager, I see the fallout of this data gap every day. Companies spend six figures on high-fidelity content only to measure success by whether an employee reached the final slide. If we want learning to be taken seriously at the board level, we have to stop reporting on activity and start reporting on outcomes.

The Great Analytics Disconnect

We have become addicted to completion rates because they are easy to track. Most Learning Management Systems (LMS) are built to tick boxes, not to track growth. However, a 100% completion rate may simply mean your team is good at clicking 'Next' while multitasking. It does not mean they have mastered a new skill or corrected a dangerous work habit.

0.21 — Average correlation between course satisfaction scores and actual on-the-job performance change.

The data is sobering. When you look at the correlation between 'smile sheets' - those post-course satisfaction surveys - and actual performance lift, the connection is nearly non-existent. A learner might love a course because it was entertaining or easy, yet still fail to apply a single concept once they return to their desk. We must shift our focus from how the learner felt to what the learner can now do.

The Four Metrics That Actually Matter

If we are going to retire vanity metrics, we need to replace them with 'decision-grade' data. These are the indicators that help a project manager or a CEO decide whether to double down on a program or scrap it entirely. In my experience, four specific metrics provide the clearest picture of learning health.

• Time-to-competency: The duration from the start of training until a learner can perform the target task unaided and without error.
• Behaviour change at 30 days: Evidence of new habits or refined processes observed in the workflow, not just reported in a survey.
• Performance lift: The measurable improvement in the specific business KPI the course was designed to solve, such as lower error rates or higher sales conversions.
• Cost per behaviour change: A calculation of the total spend divided by the number of employees successfully demonstrating the new skill.

Focusing on cost per behaviour change rather than cost per head shifts the conversation. It forces us to look at the efficiency of the instructional design. A cheap course that results in zero behaviour change is infinitely more expensive than a premium intervention that solves a technical bottleneck for 50% of the workforce.

How to Capture Decision-Grade Data

Moving to these metrics requires a shift in how we build courses. You cannot measure behaviour change if the course was never linked to a specific behaviour in the first place. Every project kick-off should start by identifying one observable action and one existing business metric that we intend to move.

Practical measurement workflows

We use xAPI (Experience API) to go beyond the 'Pass/Fail' binary. By capturing decisions made inside branching scenarios, we can see exactly where learners are struggling. If 70% of your managers choose the 'avoidance' option in a conflict simulation, you have discovered a specific training gap that a simple completion report would have missed.

Stop surveying on day one. A survey sent immediately after a course only measures short-term recall and mood. Instead, push your evaluation to the 30-day mark. At this stage, the novelty has worn off, and the learner can give you an honest assessment of whether the training survived the reality of their daily workload.

Triangulate your data points. Do not rely solely on what the employee says. Compare their self-assessment against manager observations and hard system data, such as CRM logs or support ticket resolutions. If all three lines point upward, you have proof of impact. If they diverge, you have a diagnostic tool to find out why the training isn't sticking.

Performance data is the only language the rest of the business speaks. If L&D wants a seat at the table, we have to stop talking about 'engagement' and start talking about 'output'. — Paul Bohanan, Senior PM

The Discipline of Metric Pruning

Adding new metrics is easy; the hard part is removing the old ones that clutter your reports. To maintain clarity, you must be disciplined about what you present to stakeholders. A dashboard with twenty charts is a dashboard where nothing is important.

I recommend a 'one-in, one-out' rule for each quarter. Look at your current reporting suite and identify the metric that has the lowest impact on decision-making. Usually, this is something like 'Average time spent in module'. Retire it deliberately. Replace it with a metric that actually tells you what to do next, like 'Drop-off rate at the key decision point'.

This process of subtraction forces your team to focus on quality over quantity. It turns your quarterly reviews from a routine 'Update' into a strategic 'Intervention'. When you present fewer, higher-quality data points, you build trust with leadership. They begin to see L&D not as a cost centre, but as a precision tool for performance engineering.