The Review Step

Software produces flags. A person decides what they mean. That person's time, training and instructions are the programme. A review queue can be managed with the same discipline as online timesheets: show who handled each case, how long it waited and what changed, while keeping the decision itself evidence-led.

What the reviewer is for

Deciding whether a flag warrants any further step at all, which for most flags it does not.

Distinguishing between explanations with the recording in front of them.

Recording a reason, briefly, for whichever way they decide.

And escalating the small number that need an academic judgement.

What they need

Time. Two to five minutes per flag with surrounding context, not thirty seconds with a clip.

The whole session available, not only the flagged moment.

Context: access arrangements, known technical problems, declared circumstances.

Written criteria: what counts, what does not, what is inconclusive.

And permission to close a flag as nothing without justifying it to anybody, which sounds trivial and determines whether the step is real.

Who should do it

Trained staff with no stake in the cohort.

Not the module leader in most cases: they know the students, hold views about the assessment, and are usually the person who will bring the case.

Separating the reviewer from the case-bringer is basic and is frequently not done, because the same person is the cheapest option.

The criteria

Write them down before deployment, in terms a reviewer can apply consistently.

Gaze alone: not a case.

Audio with content bearing on the exam: escalate.

Second person visible and interacting with the material: escalate.

Connection failure: never a case.

Inconclusive: closed, and recorded as inconclusive rather than as suspicion.

Calibration

Have several reviewers assess the same twenty sessions and compare.

Disagreement will be substantial at first, which is the point of doing it.

Repeat quarterly. Reviewer drift is real and invisible without this.

An institution that has never calibrated does not know what its criteria mean in practice.

Recording the decision

One line per flag: what was seen, what was concluded, by whom, when.

This is the evidence base for any appeal and the data for any fairness analysis.

A system where flags are closed with no record cannot demonstrate that review happened, which is a problem at the first challenge.

The workload reality

If review is under-resourced, reviewers confirm rather than assess, because confirming is faster.

This is a predictable response to workload and not a failure of individuals.

The fix is capacity or fewer flags, and the honest choice between them belongs at the point of purchase.

What to check

How many minutes per flag are budgeted, and how many are actually spent?

Are the criteria written down?

Is the reviewer separate from the person who would bring the case?

And has calibration ever been run?

The point

If review is under-resourced, reviewers confirm rather than assess, because confirming is faster.

That is a predictable response to workload rather than a failure of individuals.

Worth stating

Give reviewers explicit permission to close a flag as nothing without justifying it to anybody.

That sounds trivial and determines whether the review step is real or a formality with a name.

Also worth knowing

Not the module leader in most cases: they know the students, hold views about the assessment, and are usually the person who would bring the case.

Separating reviewer from case-bringer is basic and frequently not done because the same person is cheapest.

And finally

Write the criteria down before deployment: gaze alone is not a case, connection failure is never a case, inconclusive is recorded as inconclusive rather than as suspicion.

Reviewers cannot apply criteria that exist only as intentions.

Summary

Give reviewers time, context, written criteria, and explicit permission to close a flag as nothing. The last one determines whether the step is real.

In summary

The review step is what is actually being bought.

Time, training, criteria and independence are the whole of it, and each is measurable before deployment. For wider institutional context, consult IEEE.