Lean Certification Framework

In the world of Customer Education, we often view professional certification as the “final boss” for validating our products and our customers themselves. It can involve six-figure partnerships with testing agencies, the grueling psychometric validations, and the expensive proctoring platforms. Having led education teams at companies like Amazon and Unity, I have seen that high-budget world firsthand. At Unity, we partnered with third partners for developing certification exams at a cost well exceeding $150,000 a year.

But what happens when you are in a lean SaaS environment and you need that same level of rigor without the budget?

In one of my roles, I saw an opportunity for a medium-stakes certification exam to aid retention and advocacy with customers, as well as increase brand credibility in the market place. I started from scratch, using existing tools, and only my time alongside leading a learning team.

This project, and the toolkit available at the end of it, gives an overview of the project.

The Battle for the MQC: Product Knowledge vs. Job Performance

Building a certification is not simply a trivial task of writing questions across a set of product features. It is also not a way to train users on how to use a product or help them overcome common challenges. It is a strategic exercise in defining what competence actually looks like, and testing whether someone has the knowledge and skills to apply those competencies at a specific level in their job.

When developing customer certifications, one of the biggest challenges isn’t the technology, but the Expert Blindspot. Internal teams often believe customers need to know every toggle and sub-menu in the product. However, customer education is about people, not product. It requires understanding their roles and what they need to be able to do to get their job done to an acceptable level. This is where Job Task Analysis (JTA) comes into play.

Creating a JTA is a collaborative exercise, and I’ve partnered with customer-facing teams, business leaders, and actual customers to build a JTA and define the Minimum Qualifying Candidate (MQC) for an exam.

By involving experts in the JTA and question writing, I ensured the exam was a measure of professional utility, not just product trivia.

A Defensible Methodology

To set the passing score, I adapted well-trodden theories into a simplified framework. I used a combination of the Ebel method for question selection and a simplified Angoff process for setting the threshold.

By asking SMEs to predict the performance of an MQC, we moved away from arbitrary pass marks and toward a defensible, data-backed standard. Coupled with the JTA, I call this my **Lean Certification Framework **(download link further down this page 👇🏼).

The 60-Second Proctoring Hack

One of the biggest costs in certification is proctoring (supervising exams to ensure candidates don’t cheat). To bypass this, we delivered the exam live during virtual sessions via our existing LMS. The instructor would set the scene and explain the process at the start of the hour.

We told candidates we would randomly join their sessions to watch them answer a question and ask for their thinking. In reality, we were only on the call for about one minute of their hour. This cheap approach to proctoring maintained the integrity of the exam and discouraged cheating without the need for intrusive, expensive software.

For a small incentive, we awarded a branded mug to those who passed. It was a tangible, low-cost trophy that drove higher engagement than a digital badge and added a nice personal touch.

The Outcome: Almost One Year On

At the time of writing this, we are now nearly a year into the program, running sessions every three months. The uptake has been consistently strong, proving that there is a significant appetite for formal validation when the curriculum is aligned with real-world utility.

The Data: A 78% Pass Rate

Our pass rate currently sits at approximately 78%. In the world of psychometrics, for us this is our Goldilocks number (it’s just about right!). It is high enough to show that our training effectively tests candidates at our MQC (Minimum Qualifying Candidate) benchmark (as much as we can be sure the majority of exam takers meet the threshold), whilst rigorous enough to ensure that the Hornbill Certified title retains its value. It isn’t a participation trophy it is a hard-earned badge of competency.

Maintenance Without the Churn

One of the most powerful outcomes of this methodology is how easy it is to maintain. Because every question is mapped to a specific competency, we don’t have to guess why an item is performing a certain way. If a question we rated as “Hard” has a 100% success rate, we don’t change the question; we re-rate it. We trust the questions because they speak to the job, not just the interface. This allows us to maintain a robust, defensible standard with minimal effort.

The Human Element

The learner feedback has been the ultimate validation of this scenario-over-interface approach. Candidates have shared:

• “I really liked how the questions related to scenarios, not just about the interface.”
• “Great support and an excellent experience.”
• “Everything ran really smoothly.”

By keeping the process human, having an instructor set the scene and engage with them for that vital 60-second spot check. We removed the sterile, often anxiety-inducing atmosphere of third-party proctoring.

Closing Thoughts

This project started with a zero-dollar budget and a spreadsheet. It grew through deep collaboration with customer-facing teams and business leaders to define what a customer actually needs to do to be successful.

Today, it stands as a core part of our customer education strategy and we are now building out specialised credentials based on this foundational-level exam.

If you are waiting for a massive budget to start your certification journey, my advice is simple: Stop waiting. Map your competencies, engage your SMEs, and start building!

---

Download the Framework

Ready to build your own defensible certification program without the huge price tag? I documented my Lean Certification I used, from Job Task Analysis to the “60-second proctoring” hack into a repeatable toolkit.