Planning an exam prep app: why trust is the product

An exam prep app has a sharper job than most learning apps. Its users have a date, a fee and a pass mark. They aren’t browsing; they want to know what to study and whether they’re ready. That makes the planning unusually concrete, and it puts one thing above everything else: whether people can trust what the app tells them.

We built CertBrain, a study app for the AWS Certified AI Practitioner exam, on the web and Android. These are the decisions we’d plan for first.

Trust is the product

Question banks of unclear origin are common, and a wrong answer key costs people marks on the real exam. An app with a polished interface and one confidently wrong question has failed at the thing it’s for.

So decide your content rules before anyone writes a question. For CertBrain we set three:

• Original content only. Everything is written from the exam guide and the official documentation. Nothing is copied from another course, and nothing comes from the real exam.
• Every item has a source. Each lesson, question and flashcard records its exam domain, task statement, difficulty and the documentation page that backs it up.
• Outdated material is never the right answer. Services that have been retired can still come up in study material, so lessons mention them with a note, and no question uses one as the correct answer.

Every item was reviewed against its cited source. Most of the problems that review caught weren’t wrong answers. They were sources that didn’t actually say what the question claimed. You only find those if the source is recorded next to the question. We wrote more about this in introducing CertBrain.

Sourcing and reviewing questions

Plan where questions come from and who checks them. Writing them is one job; reviewing them against a source is another, and it’s worth having a second pass done by someone sceptical. Decide what happens when an exam changes, too. If every item records which part of the exam guide it covers, finding the affected questions is a filter, not a reread.

Weight by the exam guide

Most exam guides divide the exam into domains, each with a share of the scored content. Your practice material should follow the same proportions, or learners will over-prepare for small topics and under-prepare for large ones. CertBrain’s course has 70 lessons and 300 practice questions weighted to the exam guide, with one course per exam domain.

Keep those weightings as data, not code. In CertBrain, the content importer reads each domain’s weighting from the course pages and stores it on the course. When the guide changes, the content changes, and the app follows.

A diagnostic to start

A diagnostic quiz gives a new visitor something useful in a few minutes: which domains they’re strong in and which need work. It’s also the obvious first step before asking anyone to sign up.

Decide early whether it needs an account and whether it stores anything. CertBrain’s diagnostic needs no account and stores nothing: the browser scores it and shows per-domain results straight away. We considered saving results to email them later, and left it out, because it was real infrastructure and a real privacy question for a feature nobody had asked for. The Android app has the same diagnostic, reachable from the sign-in screen, scored on the phone. CertBrain on Google Play covers how.

Readiness, honestly scored

Every exam learner eventually asks: am I ready to book it? A readiness score is the feature that answers that, and it’s where trust matters most.

CertBrain’s Readiness page scores each domain from the spaced-repetition data the app already had, weighted the way the exam weights its domains. “Ready to book” needs a strong overall score and a minimum in every domain, because an average can hide one weak domain that costs a lot of marks on the day. Below that, the page names the domain holding you back and its weakest lessons. The details are in scoring exam readiness from spaced repetition.

Two decisions are worth copying:

• Don’t predict a pass without outcomes to calibrate against. CertBrain’s page says plainly that it’s a study guide, not an exam score or a guarantee.
• Make sure the underlying data means what you think. Building Readiness exposed a bug that let cramming push cards to long review intervals. Fixing it first was the difference between a score that measured study and one that measured clicking.

Free and paid

If some content will be paid, decide how much stays free and whether locked content stays visible. In CertBrain, every course and lesson title shows, each exam domain has a free sample, and a single check gates the rest. That gate was in place before billing existed, so the paywall behaved correctly from the first day.

Say what you aren’t

Most exam prep apps are independent of the body that sets the exam. Say so, in the same words, on the site, in the app and in the store listing. In CertBrain that sentence lives in one place in the site’s settings, so it only ever needs changing once.

If you’re planning exam prep for a certification you teach, our education apps page describes how we approach it, or get in touch and we’ll talk it through.