How a case is scored

A case does not give you a mark out of ten. It gives you a score on each of its rubric criteria, and each criterion records a proficiency against one knowledge component on your profile.

This page explains what gets judged and how. It does not cover what the result then becomes — that is Bloom levels and proficiency.

There is no overall case score

Nothing in Fito adds your criteria up. A case’s rubric declares a weight on each criterion, and the validator insists those weights sum to 100, but no part of scoring, submission or your profile reads them. The weights are a signal to the case’s author about relative importance, not an input to a total.

What you get back is a row per criterion, and what is kept is a proficiency per knowledge component. That is deliberate: a component you can apply and a component you have only heard of are two different facts, and one averaged number hides both.

What a rubric criterion is

A case carries its rubric in its own repository, at .case/rubric.yaml. Each criterion declares:

One knowledge component — what this criterion is about.
One Bloom level — the depth it certifies for that component.
Four level descriptions, one for each score from 0 to 3. Every criterion must describe all four; a rubric missing one does not validate. They are a quality ladder at that one Bloom level — not_achieved, partial, achieved, exemplary — not a climb from one level to the next.
An evaluation method — how the score is arrived at.

You can read a case’s criteria before you start. The case cover at /cases/<id> lists each criterion’s knowledge component and Bloom level. The commands and the level descriptions live in the case repository, so you see them once you open it.

A score measures how well, not how deep

A criterion’s score and its Bloom level measure two different things. The score, 0 to 3, says how well your work met the criterion. The Bloom level says how deep a met criterion certifies.

So a score of 2 or 3 records the criterion’s own Bloom level as your proficiency, and a score of 0 or 1 records NOT_ACHIEVED. A 2 on an APPLICATION criterion records APPLICATION. A 3 on a KNOWLEDGE criterion records KNOWLEDGE, never more: the 3 changes the feedback, not the depth.

The full table, and what NOT_ACHIEVED means beside a component you were never assessed on, is on Bloom levels and proficiency.

The four evaluation methods

A criterion is judged in one of four ways, and the method is the case author’s choice, declared per criterion.

Method	How the score is reached
`automated`	A command runs in your checkout. Its exit code and output are the evidence.
`ai`	Your artifacts and your `git diff` are read and judged against a hint the author wrote.
`hybrid`	The automated gate runs first. Pass takes a fixed level; fail falls through to `ai` or `human`.
`human`	Nobody local can score it. It comes back marked Needs human review.

An automated command that exits non-zero is evidence, not a crash. A failing test suite is exactly the input the scorer needs, so the run carries on and the exit code and output go into the report.

Each command gets a 15-minute budget. One still running after that is terminated and reported with exit code 124. That too is evidence — a build that hangs scores like a build that fails, rather than stopping the whole run.

Who does the scoring

The scoring runs on your machine, on your own LLM subscription. Fito’s server holds no LLM key and never sees your code or your prompts. That is why scoring needs an LLM provider configured in pi, and why it costs you rather than Fito.

The judging is done by a worker persona called fito-performance-reviewer, which reads the evidence report and your checkout. It may read the exemplar solution’s notes.md for the author’s rationale while judging, and it never puts solution code in front of you.

Preview, then submit

The same scoring runs twice, under two commands.

/fito-evaluate <case-id>

A dry run against your local checkout. No pull request, no submission, nothing written to your profile, and no login needed. Each criterion comes back with its score out of 3, the label that score matches, a one-sentence justification, and the proficiency a submission would record. Where an automated criterion scored below 3, the captured output is shown so you can see why. Run it as often as you like.

/fito-submit <case-id>

The same scoring, for record. It runs after the pull request is opened, posts the results to the backend, moves the submission to EVALUATED, and writes the proficiency to your profile.

For what /fito-submit does before it scores, see Submit and read your result.

The quiz, when a case has one

A case may carry an optional knowledge-check quiz at .case/quiz.yaml. Many cases have none, so a case with no quiz is not a case missing something.

Questions come in three types:

Type	How it is scored
`multiple_choice`	Deterministically, in code: met or not met. No model is involved.
`short_answer`	0–3 by the same reviewer persona that scores the rubric, at the question’s own Bloom level.
`code_snippet`	0–3 by that same reviewer.

Every question names at least one knowledge component and one Bloom level, so a quiz records proficiency the same way a rubric criterion does: a score of 2 or 3 records the question’s Bloom level, and a score of 0 or 1 records NOT_ACHIEVED.

A multiple-choice answer is met or not met

A correct multiple-choice answer is met. It scores 2 and records the question’s own Bloom level. An incorrect or blank answer is not met. It scores 0 and records NOT_ACHIEVED. There is no partial credit, and no answer scores 3.

So the result reads as correct or incorrect, never as a fraction: 2/3 would show a right answer as a lost point. An open-ended question can reach 3, so its result keeps the score out of 3.

Where the answer key lives

The answer key never reaches the part of the session you talk to. The public GET /cases/{id}/quiz endpoint serves the prompts and the choices and omits the answers entirely, and during a run the key is held only by the scoring step.

In /fito-evaluate the quiz is practice — you answer it and see the scores, and nothing is recorded. In /fito-submit the answers are scored for record.

When a rubric and a quiz cover the same component

They are merged by taking the higher level of the two, whatever the scores behind them. If the rubric records KNOWLEDGE for a component and the quiz records COMPREHENSION for the same one, COMPREHENSION is what lands on your profile.

A case that carries a quiz and no scorable rubric is marked EVALUATED by the quiz alone.

What’s next

Bloom levels and proficiency — what the 0–3 score becomes once it is recorded.
Submit and read your result — the command that runs this for record.
Knowledge components — what every criterion points at.