Technical manual

Edition 0.2, 17 October 2026. We revise the text at least once a year. The live figures refresh from recorded results each hour.

A figure we have not measured is marked pending, with the reason. Pending never means zero.

Applied AI Foundations

14 exams

Only exams taken for a result count. Practice and review records are excluded.

Edition 2026-09 of the Standard rates six areas on two axes. Value covers reaching further, briefing the assistant and the finished piece. Control covers checking its work, safety and confidentiality, and knowing what to delegate.

Each area states what it includes, what it leaves out and what counts as evidence. A question writer and a reader of the Standard work from the same definition.

Each exam version sets the share of its points each area carries. The definitions and each exam's shares are on the Standard.

One AI examiner marks Applied AI Foundations, the Applied AI Badge and Applied AI Certified. Three AI examiners mark Applied AI Specialist and Applied AI Strategist independently; a majority sets each score, and a named person countersigns the result.

Each score of 1 or 2 must quote the candidate's own words. We check every quote against the candidate's messages, draft and answers. A quote that is not there does not count, and the score goes to a person.

A person reads every cheating flag before a result goes out. On the paid exams, a person also reads every safety event and every result on the pass mark.

The Examination Board chair approves each marking configuration: the models, the instructions, the settings and the AI tool candidates use. A result marked by a configuration nobody approved waits for a person. The marking engine is version 4.4.0; the panel of three AI examiners is version 4.4.2.

How we use your data for marking is in the privacy notice.

A prompt is never scored for style. Two events read the candidate's turns before they rely on the output. Context covers the reader, the purpose and the source facts. Specification covers the form, length and required contents. Each scores one behaviour 0, 1 or 2.

What the candidate keeps out of the tool is scored under Safety, and whether they checked the output under Checking its work. On the free exam the assistant is set up to make one stated mistake in each task. The checking event counts only when that mistake reached the candidate; otherwise it is marked not applicable.

Reliability is the share of the spread in scores that comes from real differences between candidates rather than error. On a fixed exam we use Cronbach's alpha over its scored events. On an exam drawn from the question bank we split each exam by task and apply the Spearman-Brown correction. Each event counts with its weight in the overall score, so the figure describes the score a reader sees.

The standard error of measurement is SD × √(1 − reliability). Two scores less than two SEMs apart do not show a difference. Figures use the first marking of 30 or more completed, scorable results per exam version. Previews and test seats are left out. A coefficient outside 0 to 1 is not printed.

Pending. No exam version has 30 graded results yet. The closest has 8.

Internal structure: we correlate the area scores across results. Areas that correlate above 0.9 are not telling a reader different things. These print with the reliability figures, from 30 results.

Marking error is measured separately, by marking the same work twice (see Marking the same work twice). The SEM above does not include it.

For every question we compute its difficulty, the share of the points candidates earn. We also compute its corrected item-total correlation: how it moves with the rest of the exam, leaving the question itself out. A question that strong candidates do no better on usually has an ambiguous prompt or a wrong key, and it is reviewed. For multiple-choice questions we record how often each option is chosen and the average score of those who chose it.

We do not publish figures for single questions, because the questions are secure. Item response theory needs about 200 candidates per question, and no question has that many yet.

Validity belongs to a use of a score, not to the exam. For each use we follow the chain from the work to the decision (Kane, 2013) and name the weakest link.

Scoring: was the work scored correctly?

The AI examiner is checked against an anchor set and a blind sample re-marked by people. The anchor set was drafted with AI and reviewed by the Institute; independent experts have not yet scored it.

Generalisation: would other tasks of the same kind give the same score?

Reliability and the SEM above. The free exam has two tasks, a narrow sample; there is no parallel form or retest evidence yet.

Extrapolation: does the score predict real work?

Not yet shown for any use. This is the weakest link. The planned study compares scores with structured-interview or manager ratings of AI-assisted work three months later.

Decision: does the score justify the action taken?

Pass marks are provisional until a panel sets them. Employer reports say to use a score as one input, never as the only screen. They treat scores within two SEMs of the top as level.

Known threats: slow typing and less fluent written English can lower a score. Follow-up questions are chosen by keyword. Correcting the assistant on a specific point is not yet scored on its own. All known limits.

One result in 10 of every exam is chosen by its exam id. A person marks it again without seeing the scores the AI examiner gave. The choice is fixed by the id, so nobody can pick which results a person checks.

Once 100 marked events are compared, the AI examiner must reach exact agreement of 70% and a quadratic weighted kappa (QWK) of 0.70. Below that line, every new result of that exam waits for a person.

Pending. No result has been chosen for a second marking yet.

Some chosen exams are marked by two people. How often two people agree is the line an AI examiner is judged against.

Pending. 0 of 20 events marked by two people.

Before a model marks Applied AI Foundations, it marks 12 practice exams that our staff wrote and scored (27 September 2026). To be chosen, it must meet four lines:

Exact agreement

At least 70%

QWK

At least 0.70

Checked quotes

Behind 95% of its positive scores

Speed

An answer within 45 seconds

• openai/gpt-5.6-luna-pro (exemplars): exact 87%, QWK 0.89, quotes checked 100%, 37 seconds. Met the line.
• openai/gpt-5.6-luna-pro (anchors): exact 90%, QWK 0.92, quotes checked 99%, 45 seconds. Did not meet the line.

Practice exams written by staff are easier to mark than real ones. These figures show a model can do the work; the agreement with people above shows whether it does.

Pending. We are building a set of attacks on the AI examiner. They include hidden instructions, padded answers and answers that copy the marking guide's wording. The first run is not published yet.

Pending. Before the chair approves a new marking configuration, it will mark a fixed set of exams twice. It must give the same scores both times (an intraclass correlation of at least 0.90). The first run is not published yet.

Candidates may share attributes such as age group, with consent. We compare each group's average score on a question with everyone else's, among candidates who did equally well on the rest of the exam, once a group has 5 candidates. A gap of 0.25 points or more sends the question to the Examination Board.

This is a screen, not a full study. Matching on a few candidates is rough, so a gap can have an innocent cause, and a group that too few others match prints no figure. 0 of 0 comparisons are flagged today.

Pending. A full study of differential item functioning needs 200 candidates in a group and 500 in all. 0 candidates have consented so far.

Pending. Agreement with people for each group prints once a group has 50 compared events from 5 candidates.

• Applied AI Badge, exam version 1: a provisional mark the Examination Board set, next review 4 September 2028.
• Applied AI Foundations, exam version 5: a provisional mark the Examination Board set, next review 18 October 2028.
• Applied AI Certified, exam version 2: a provisional mark the Examination Board set, next review 18 October 2028.

A panel replaces a provisional mark once enough people have taken the exam. We never publish the mark itself.

We are not accredited by any body. We work to these standards and publish the evidence here instead:

ISO/IEC 17024

Bodies that certify people. Our policies were drafted against the 2012 edition.

AERA, APA and NCME Standards

Validity, reliability and fairness of tests used to make decisions about people.

ITC guidelines

Tests delivered online and technology-based testing.

NCCA guidance on AI (2025)

Using AI in certification programmes, including telling candidates when AI marks.

What we have and have not proven

You can ask for a review within 14 days of your result. A person who did not mark your exam reads your work again. A formal appeal follows within 21 days of that decision. People, not AI examiners, decide every review and appeal.

Every deadline, before you take an exam

• The human-to-human baseline: 0 of 20 events re-marked by two people.
• The first attack run against the AI examiner.
• The first repeat-marking run on the anchor set.
• A differential item functioning study: it needs 200 candidates in a group and 500 in all.
• Agreement with people for each consented group of candidates.
• Pass marks set by a panel of subject experts for every exam.
• Reliability and SEM for Applied AI Foundations, version 4, form 10: 8 of 30 results.
• Reliability and SEM for Applied AI Foundations, version 4, form 9: 2 of 30 results.
• Independent experts scoring the anchor set the AI examiner is checked against.
• A study of whether a pass predicts how someone works with AI on the job.

• Edition 0.2, 17 October 2026. Edition 2026-09 of the Standard. New sections: what is measured, how a brief is scored, reliability and the SEM, question statistics, and the validity argument for each use.
• Edition 0.1, 15 September 2026. First edition: marking, the blind re-marking sample for every exam, the judge bench, the fairness screen, pass marks, standards and what is still pending.