Structure Beats Prompting: Lessons from an AI Assessment Platform

"Generate a question paper" is a demo. It works on stage, it gets applause, and it collapses the moment a real faculty member needs the paper to satisfy an accreditation audit.

Questify is an AI question paper, quiz, and adaptive assessment platform built for Anna University affiliated institutions in Tamil Nadu — part of a government-sanctioned project, built by a ten-person team where I work as lead developer. Most of what I have learned about applied LLM systems came from the distance between the demo that impressed people in month one and the system that survived committee review in month fifteen.

The short version of that distance: the model is the least important component.

What a university actually needs from a question paper

An Indian engineering exam paper is not a list of questions. It is a document that has to satisfy a set of constraints simultaneously:

• Every Course Outcome (CO) in the syllabus carries a defined weightage, and the paper must hit those weights
• Questions distribute across Bloom's taxonomy levels L1 to L6 in a specified proportion — you cannot ship a paper that is entirely recall
• The structure is fixed: Part A is 2-mark questions, Part B is 8-mark, Part C is 15-mark
• Content must be traceable to the prescribed textbook
• The whole thing has to produce NBA/NAAC accreditation matrices on demand, because the college is audited on it

A free-form prompt cannot hold that. You can describe it in a prompt, and the model will produce something that looks compliant — and then the CO weightage is off by 12%, or Part B has four L2 questions where the blueprint called for two, and nobody notices until an auditor does.

The fix is not a better prompt. The fix is to stop asking the model to satisfy the constraints.

Blueprints first, generation second

Faculty do not prompt Questify. They build a blueprint: a matrix declaring, per unit, how many questions at which Bloom level, mapped to which CO, at which mark weighting.

The blueprint is a structured object. It is validated before a single token is generated — CO weights must sum correctly, Bloom proportions must be achievable given the question counts, mark totals must match the paper structure. An invalid blueprint fails immediately, with a specific error, at a moment when the faculty member is still in the editor and can fix it.

Generation then runs against the blueprint. The model is asked for one question at a time with a fully specified slot: this CO, this Bloom level, this mark value, this unit, grounded in this retrieved context. It is a much smaller and much more reliable request than "write me a paper."

The practical consequence is that failure becomes local. If one generated question is weak, that is one slot to regenerate — not a paper to discard and re-prompt.

Grounding, with citations that point at pages

Questions have to come from the prescribed textbook, not from whatever the model absorbed during pretraining. That is a RAG problem, and the specifics matter more than the acronym.

Two of those are worth arguing about.

Hybrid retrieval, weighted toward keyword. Alpha 0.6 leans BM25. Pure vector search is excellent at "find me passages about this concept" and mediocre at "find me the passage that defines Kirchhoff's Current Law," because textbook content is dense with exact terminology that semantic similarity smears. Engineering syllabi are full of named laws, named theorems, and standard notation. Keyword matching is not the old way here — it is the right tool for a corpus where the words are the content.

1400-character chunks with 220 of overlap. Small chunks retrieve precisely and lose the context that makes a passage answerable. Large chunks carry context and dilute the embedding. The overlap exists so a definition that straddles a boundary survives in at least one chunk intact. These numbers came from trying values and reading the retrieved passages by hand, which remains the only reliable way to tune a chunker.

Every generated question carries its source citation through to the reviewing faculty member — chapter and page. That is not a nice-to-have. A faculty member signing off on an exam paper is putting their name on it, and "the AI said so" is not a defensible position in front of a committee. The citation is what makes review possible in seconds rather than minutes.

Difficulty, computed rather than guessed

Ask a model how hard a question is and it will tell you. Ask it twice and it may tell you something else. Ask it about two questions in different sessions and the answers are not on a common scale.

That is fine for a demo and useless for adaptive practice, which needs difficulty to be a stable number comparable across the whole question bank.

So the model does not output a difficulty. It scores six rubric features on a 1-5 scale:

• Conceptual load
• Reasoning steps required
• Prerequisite depth
• Linguistic complexity
• Answer precision required
• Source density

Those six numbers go into a deterministic backend calculation. Marks contribute a base, the Bloom level contributes an adjustment, the rubric features contribute a weighted term, and the result is clamped into a bounded range to produce an Item Response Theory b parameter.

initial_b = MARKS_BASE + BLOOM_ADJUST + rubric_adjustment   (clamped)

easy    : below -1.0
medium  : -1.0 to 0.75
hard    : above 0.75

The split is the point. The model does the part it is genuinely good at — reading a question and judging qualitative properties. The arithmetic that has to be consistent is arithmetic. Same question, same features, same b, every time, forever. Change the calibration constants and every item in the bank moves together instead of drifting apart.

That b value then drives IRT-based adaptive practice: a student's ability estimate updates as they answer, and the next item is selected against it. None of that works on a difficulty score that wobbles.

Generation runs on chat infrastructure

An unexpectedly good decision: question generation does not have its own request pipeline. It runs through the same chat session infrastructure that powers the assistant.

Which means generation gets, for free:

• Persistence. A paper in progress is a session you can close and come back to.
• Undo and redo. A snapshot stack over the session, so a bad regeneration is one step back, not a restart.
• Streaming. Server-sent events with a rich event vocabulary — assistant segments, tool activity, warnings, failures — so the UI shows what is happening rather than a spinner.
• Context compaction. Long sessions get summarized in a background job before they hit the context limit, so the session does not simply die at the boundary.
• Credit accounting. Token spend is metered per user against a policy, because a government project with real budget cannot have unbounded inference cost.

The alternative — a fire-and-forget generation endpoint — would have needed all five of those rebuilt separately within a few months.

What a five-layer prompt is for

The prompt sent to Bedrock is assembled from five ordered layers:

1. Global system prompt — the assistant's identity and hard rules
2. Per-user behaviour prompt — faculty preferences that persist across sessions
3. Retrieval context — the passages pulled for this specific request
4. Conversation summary — present only if the session has been compacted
5. Message history

The layering matters because each has a different lifetime and a different owner. The global layer is versioned and changed by the team. The behaviour layer belongs to the user. Retrieval is per-request. Summary is machine-generated. History is append-only.

Flatten those into one string and every prompt change becomes a merge conflict between five concerns that update at completely different rates. Keeping them separate is the difference between a prompt you can edit and a prompt you are afraid to touch.

Where the honest limits are

The system does not remove the faculty member. Every generated question goes to review before it enters a paper, and a meaningful fraction gets edited or rejected. What the platform changes is the unit of work: from writing forty questions to reviewing forty questions with sources attached — which is faster, and is also a task the faculty member is genuinely qualified to do.

Retrieval quality caps output quality. A badly scanned textbook produces bad chunks, and no amount of prompt work recovers from that. A surprising share of the engineering effort went into ingestion — extraction, syllabus parsing, fingerprinting uploads so the same textbook isn't indexed four times.

And the constraint validation catches structural violations, not conceptual ones. A paper can hit every CO weight and Bloom proportion perfectly and still be a mediocre exam. That judgement stays human.

The transferable part

If you are building anything LLM-backed beyond a demo:

• Put constraints in a schema and a validator, never in a prompt. Validate before you generate.
• Ask the model for judgements, compute the numbers yourself. Anything that needs to be comparable across time must be deterministic.
• Cite everything. A generated claim without a source cannot be reviewed, and unreviewable output cannot be used where it matters.
• Make generation a session, not a request. You will want persistence, undo, and streaming eventually — retrofitting them is worse.
• Structure the prompt by lifetime. Layers that change at different rates should not live in the same string.

The team is at 272 commits and roughly nineteen months on this. The demo took a fortnight. Everything since has been the difference between output that looks right and output that stands up.