Higher Education /Document processing interactive demo field guide · 9 min

Course-Eval Theme Analyser

Turns thousands of open-ended course-evaluation comments into themes with counts and verbatim quotes, separating strengths from friction — and marks any thin theme as low-confidence rather than overstating it.

Open the live demo

theater/demos/higher-ed_course-eval-theme-analyser.html · sandbox · read-only

Open

FIG. 1

The live demo, running on fabricated data. Open it to step through the full flow — every output is shown for a person to approve before anything happens.

How it would work

Reads a unit's free-text evaluation comments, clusters them into themes with sentiment, counts and verbatim quotes, holds back any wellbeing disclosure, and lays it all out for the subject coordinator to approve before it reaches the unit's improvement record.

Input 01

The free-text comment pile

The full set of open-ended student responses for one unit and teaching period — exported from the evaluation survey as a CSV or pasted in — with the response and enrolment counts that tell you how representative they are.

Agent 02

Clusters, scores, holds back

Groups comments into recurring themes, tags each as positive, mixed or negative, counts the mentions and pulls a verbatim quote, marks thin themes low-confidence, and pulls any comment that reads as personal distress out of the summary entirely.

Output 03

A draft summary, working shown

A theme report — each theme with its count, sentiment, quote and a confidence rating — plus any held-back wellbeing comment routed for escalation, for the subject coordinator to review, edit and approve before it goes to the unit's improvement record.

Where it works well

It reads every comment, every time — the step that gets skipped when the pile is big.

Best for a coordinator or department chair reviewing units with more than 30 responses, where manual grouping is the step that gets dropped at end of teaching period.
It separates what students valued from where they hit friction, so a unit review starts from evidence instead of the three comments someone happened to remember.
Across a faculty's worth of units, the recaptured hours go back into acting on the feedback — redesigning the assessment, fixing the lab — not into reading it.

The invisible cost of qualitative evaluation is that nobody reads it. A coordinator with 200 free-text responses across three units skims the first page, catches the loud complaints, and the quiet patterns — the ones a handful of students each raised — never surface.

Where it works badly

It is confidently wrong when a handful of comments get dressed up as a finding.

Weak on sarcasm, mixed sentiment and in-jokes — "the 8am lecture was a real treat" can land as positive. It should flag the ambiguous, not resolve it.
A small or skewed response set is not the student body. The tool reports who answered, not who enrolled; reading a 30% response rate as the cohort's verdict is a human error it cannot prevent.
It clusters by language, so two comments about the same problem in different words can split into two thin themes — or two unrelated gripes merge under one label.

The honest test

If you would not act on a theme without reading its count and its quotes first, then this tool is a faster way to the comments — not a verdict you can publish unread.

Give it forty responses and it will still produce clean, authoritative theme cards. A theme backed by three comments looks identical to one backed by sixty unless you read the count — and a low response rate can make a vocal minority look like the cohort. That is the trap.

What it doesn't do — and shouldn't

It drafts the summary. A coordinator approves it. Distress goes to a human, always.

WHAT IT DOES

Shows the mention count, sentiment and a verbatim quote behind every theme

Marks themes supported by only a few comments as low-confidence

Pulls any comment disclosing distress out of the summary and routes it for escalation to student wellbeing support

WHAT IT WON’T

Publish the summary or push it to the unit record on its own

Score, rank or appraise the teaching staff named in comments

Respond to, or make a welfare judgement on, a student in distress

Course evaluations feed the monitoring, review and improvement that the Higher Education Standards Framework (Threshold Standards) 2021 requires of providers, and they can feed staff performance conversations — so a mislabelled theme is not harmless. And a student disclosing distress in a survey box is a duty-of-care moment: the framework also requires providers to foster student wellbeing, which means a person, not a classifier, decides what happens next. The accountable academic stays on the decision.

What your data has to look like

Open-text responses tied to one unit and period, with the counts that show how representative they are.

Typical readiness

across orgs we see, before the first job

Open-ended comments as exportable text

Usual weak point

Response and enrolment counts per unit

Needs shaping

Comments separated by unit and teaching period

Usual weak point

A consistent question set across the comment fields

Usual weak point

Identifiers stripped before analysis

Needs shaping

The real first job

The weak point is almost always the export: evaluation comments locked inside the survey platform's on-screen report, pooled across units, or carrying student and staff names in the free text. Getting the comments out as clean, per-unit, de-identified text — and capturing the response and enrolment counts alongside them — is usually the real first job, larger and more valuable than the clustering on top. Once that is in place, every teaching period after is a few minutes, not a lost afternoon.

Right fit if…

You review units with more than 30 free-text responses and the manual read gets skipped

You can export the open-ended comments as text, per unit, per period

You want strengths and friction separated as a starting point for a unit review

You will read the count and quotes before acting on any theme

Walk away if…

Your evaluation comments only exist inside an on-screen report you cannot export

Your response rates are low enough that the answers aren't the cohort

You want a number that ranks or appraises teaching staff

You need it to handle a student in distress, rather than route it to a person

Open questions

The worried-buyer questions, answered straight

It can over-weight a small, vocal set — which is exactly why nothing is published on its say-so. Every theme shows its mention count, a verbatim quote and a confidence rating, and thin themes are marked low-confidence rather than presented as findings. A subject coordinator reads those before approving, and the tool never scores or ranks the staff named in comments. It surfaces patterns; a person decides what they mean.

It works from open-text responses exported as rows, one per student. It tolerates blanks and short answers, but it clusters by language, so wording that changes every period or comments pooled across units produce muddier themes. The first piece of work is usually getting the comments out of the survey platform as clean, per-unit, per-period text — and that pays off across every teaching period after.

No. It removes the hours of reading and grouping 200 comments by hand, so the coordinator spends their time on the judgement: which friction theme to act on, whether the assessment redesign is the fix, what to tell the teaching team. The unit review is still theirs. The capacity it frees goes back into acting on the feedback, not into wading through it.

Each run is anchored to one unit and one teaching period — that is the point. The risk isn’t staleness so much as mixing periods: feed it last year’s comments alongside this delivery’s and the themes describe a unit that no longer exists. The honest test is whether you can say which unit and which teaching period a given comment set belongs to before you run it.

Student evaluation comments are personal information handled under the privacy law that applies to your provider — the Privacy Act 1988 and the Australian Privacy Principles for private providers and ANU, or the relevant state or territory privacy law for most public universities — so any deployment runs against your own systems and data handling, not a shared pool — we scope where the data sits and who can see it as part of the build, and identifiers should be stripped before analysis. A comment that reads as personal distress is pulled out of the theme summary and routed for escalation to your student wellbeing support, because that is a duty-of-care moment for a person, not a classifier. The demo runs entirely on fabricated units and comments.

No. It organises the qualitative feedback that feeds course monitoring, review and improvement under the Higher Education Standards Framework (Threshold Standards) 2021 — it does not certify that a unit meets the framework or close the review for you. It gets the evidence base together; the accountable academic draws the conclusions and owns the improvement record.

What it takes to build

2–3 weeks · 4 phases

Reused from template~70%

Bespoke to this skin~30%

stack · Claude · text clustering · review export

What it would cost

Fixed scope, fixed price, fixed dates.

Bite-sized first piece

One contained change, low risk

Pilot build

Most builds land here

Embedded support

Scale on proof

Considering this for your faculty?

The honest place to start is a bite-sized first piece — one unit's comments, one teaching period, low risk. Tell us where the feedback piles up unread; we'll play it back, scope it, and show you what's possible.

Book a call How we work