Spaced Revision

The problem

An educator writing practice material by hand produces maybe a dozen good MCQs in an evening. A course needs hundreds, they need to match the notes the students actually have, and they go stale the moment the syllabus moves.

So the platform generates them. Students ask questions in natural language and get flashcards, MCQs or structured notes back, grounded in that course's own notes rather than in whatever the model remembers.

Where the agent actually lives

The system is six services. The one that matters here is a Python and FastAPI service running a LangGraph ReAct agent with five tools:

• search_vector_db for retrieval over course notes
• get_course_structure for the course, subject and topic hierarchy
• create_flashcards, create_mcqs, create_notes for generation, one to ten items, mutually exclusive within a turn

Responses stream to the client as NDJSON. The service subscribes to LangGraph's event stream and translates it: on_chat_model_stream becomes a token frame, on_tool_start and on_tool_end become structured tool frames, and a final done frame carries the token count and thread id. The client renders tool output through dedicated components rather than as text.

The interesting constraint is ordering, not the graph

Forcing the lookup first turns "the model invented a course" from a silent empty-result bug into an impossible state. It is a prompt-level constraint rather than a graph edge, which is not elegant, but it is the thing that actually stopped the bug.

The second ordering rule is that the agent must not restate generated content. Cards and MCQs come back as structured tool output and get deduplicated server side. If the model also wrote them into its prose the student would see everything twice.

The doubt solver is a real state machine

Separately from the chat agent, there is a hand-built LangGraph graph for answering student doubts about a specific MCQ. This one has explicit nodes:

detect_mcq_issues
  ├─ MCQ is flawed ──────────────→ file a flag, stop
  └─ MCQ is fine ────────────────→ generate_doubt_answer
                                        ↓
                                   verify_doubt_solution
                                        ├─ not verified → generate_doubt_answer (max depth 3)
                                        └─ verified → done

detect_mcq_issues is the node I care about. Before answering a question about an MCQ, it checks whether the MCQ itself is broken: no correct option, more than one correct option, an ambiguous stem. If it is, the graph does not answer. It files an automated flag against the main backend and stops.

The alternative is a model confidently constructing a defence of a question that has no right answer, which is worse than silence.

Retrieval

Notes are chunked and embedded locally with all-MiniLM-L6-v2 at 384 dimensions, into a FAISS index of 25,000+ chunks drawn from 20,000+ course notes. Cosine similarity via inner product on L2-normalised vectors.

Model routing is a correctness decision

The chat default is grok-4-fast, chosen because it is cheap and quick, and users can switch it.

Content generation and answer evaluation are pinned to gpt-4o with structured output, and users cannot switch those. The split is deliberate: conversation tolerates a weaker model, but an MCQ with a wrong answer key is a defect that reaches every student who sees that question.

The same instinct shows up in the answer-evaluation schema, which is generated per request so the score bounds are baked into the structure the model has to satisfy. The model cannot return 12 out of 10 because the schema will not accept it.

Billing an agent is harder than billing a call

Tools make their own model calls. Generation runs gpt-4o inside the tool, so the agent's own reported usage under-bills by a wide margin. Usage is aggregated across the agent turn and every tool call before the balance is touched.

Reading that usage at all needs a four-level fallback, because LangChain surfaces it inconsistently across providers: structured usage metadata, then response metadata, then the raw SDK usage object, then a character-count estimate as a last resort.

The charge itself discounts input four to one:

usedTokens = ceil(prompt_tokens / 4) + completion_tokens

The platform sells conversation credits, and charging a student full rate for a long MCQ stem they did not write would feel like a penalty for using the product.

Three failures worth designing around

An empty answer must never be cached. A reasoning model can burn its entire completion budget and return nothing with a length finish reason. Storing that alongside a valid embedding poisons the cache for that question permanently, and every future student asking it gets the blank. There is an explicit emptiness check before any credit is deducted and before anything is written.

A cache is an optimisation, never a dependency. The similarity lookup is wrapped so that any error at all falls through to a live model call. An embedding dimension change should degrade to slower, not to a 500.

A write must never discard an answer the user paid for. Persisting the response and updating the balance are both allowed to fail and log. The user already spent the credits; losing their answer to a database blip is the worse outcome.

The same instinct runs through startup: optional dependencies are wrapped so neither can prevent the service from booting. Chat works without its history store, losing history. Practice grouping simply disappears without its vector store. Nothing takes the server down on the way up.

The correction loop

There is no human approval gate before generated content reaches students. What exists instead is a correction that propagates backwards.

It is a small piece of precedence logic and it is the highest-leverage thing in the content pipeline, because the correction cost is paid once and the benefit compounds.

The chat layer

Real-time chat runs on Socket.IO in a separate Node service, supporting one-to-one and group channels. One feature, three stores, two services:

• MongoDB in the chat service owns messages, read markers and bug reports
• MySQL in the main backend owns identity, peer relationships and entitlement, which the chat service never duplicates and always asks for
• Redis in the main backend backs the queues that fan out push notifications

Coordination is by derived naming rather than negotiation. A direct message is not delivered into a shared room; it is emitted to both participants' personal rooms in a single call. Read state and unread counts key off a sorted pair identifier:

const conversationId = [a, b].sort().join("-");

Sorting makes it order-independent, so both clients compute the same key for the same conversation without agreeing on one first. That key is never a room, only an identifier.

One schema stores both room messages and direct messages, discriminated by a boolean, with a validation hook enforcing the invariant: a direct message needs a sender and a recipient and must not carry a room id, a room message needs a room id, and every message needs either text or a file.

The honest limitation: the socket layer runs as a single process today. Room membership and connection state are held in memory, so scaling out horizontally means sharing that state first, otherwise a second instance would not route direct messages correctly. It is a known constraint rather than an oversight.

What I would change

The Socket.IO layer trusts client-asserted identity rather than verifying the token at the socket boundary. It should verify. That is the change I would make before any scaling work, because scaling a trust problem just distributes it.

Beyond that: the vector index wants metadata filtering so retrieval stops over-fetching, and the long-running jobs that currently run inline want a queue.