Quality Scoring: Making Editorial Judgement Countable

Ask most content teams whether their output has got worse since they started using AI and you’ll get a confident answer in both directions. The head of content says it’s fine. The sceptical editor says it’s all starting to sound the same. Neither can prove anything, because “quality” lives in people’s heads as taste.

That is the problem. If quality is taste, you can’t detect AI-driven decline until traffic falls, and by then you’re six months of published work behind. Rankings lag. Engagement lags. Sales feedback lags even more. A content quality score framework gives you something that moves earlier: a number, produced the same way every month, that you can put in front of a finance director without flinching.

Here is the version I’d run with a team of one to five people. It takes about three hours a month.

The five criteria

Keep it to five. Every extra criterion adds scoring time and blurs what the number means. Score each from 1 to 5.

Total is out of 25. Write one-line anchors for each score, not just for 1 and 5. A rubric with only the extremes produces a pile of 3s.

Notice what’s missing: SEO, word count, readability scores. Those are measurable already and Semrush or Surfer will happily give you them. This rubric covers what tools can’t.

Two editors, a blind sample, a fixed cadence

The mechanics matter more than the criteria.

Sample size. Pull 10 published pieces a month. If you publish fewer than 10, score everything. If you publish 60, take 10 by random draw (in Google Sheets, =RANDBETWEEN against a numbered list does the job). Don’t let anyone pick favourites.

Two scorers. Each piece is scored independently by two people. In a three-person team that might be the content lead plus whoever didn’t write or edit the piece. Scores go into a shared sheet, hidden from each other until both are in. Anyone who has worked on the piece sits out.

Strip the labels. Remove author names and whether AI was involved. You’re scoring the output, not the process. If the editors know a piece was AI-drafted they will mark it down by reflex, and that bias will poison the trend.

Same day each month. First Thursday, say. Put it in the calendar like a finance close.

A worked example

A four-person in-house team at a UK B2B payroll software firm adopted Claude for first drafts in January. They scored a baseline of 10 pieces from the previous quarter, all human-written, then continued monthly.

(Disagreement is the average gap between the two editors’ totals, in points.)

Organic traffic over the same months was flat to slightly up. Nobody would have noticed anything from Search Console in Google Analytics 4. The score said otherwise: a 2.4-point fall by March, driven almost entirely by specificity. Drafts were coming out tidy and generic, and editing had become a proofreading exercise.

They found the cause in half an hour of reading the lowest scorers: the prompt had no source material in it. Writers were asking for “a guide to statutory sick pay changes” and accepting what came back. The fix was a brief template that required three inputs before drafting: an internal expert quote, one customer scenario, and the HMRC or GOV.UK page to cite. By April specificity was back to 3.7. Without the trend line, the first sign would have been a ranking drop around September.

Making the number defensible

A sceptical stakeholder will ask three things. Have answers.

Is it reliable? Track editor disagreement, as above. If the average gap between two scorers stays under 1.5 points out of 25, the rubric is doing its job. Above 2, your anchors are vague; rewrite them. Run a calibration session in the first month: both editors score the same three pieces, then talk through every difference.

Is it valid? Check it against something external once a quarter. Take your top and bottom third by score and compare engaged sessions, conversion to demo or newsletter signup, and time on page. You’re not looking for a perfect correlation. You’re looking for the top third to beat the bottom third more often than not. If it never does, the rubric is measuring the wrong thing. How to set up that comparison properly is covered in Measuring AI Content Performance and ROI.

Is the sample enough? Ten pieces is small. A single month’s movement of half a point is noise. Read the three-month direction, not the single month. A drop of more than 1.5 points sustained across two consecutive months is the trigger for action.

Set thresholds before you need them

Decide in advance what happens at each level, while nobody is defensive about it.

• Mean 20 or above: carry on.
• Mean 17 to 20: review the lowest criterion and adjust briefs or editing checklists.
• Mean below 17, or any criterion mean below 3: pause AI-first drafting on the affected content type until it recovers.
• Any single piece scoring 1 on accuracy: pull or correct it that week, whatever the average says.

Those numbers are starting points. After six months, reset them to your own distribution.

Keeping it cheap

You can build the whole thing in a Google Sheet: one tab for raw scores, one for monthly means, one line chart. Airtable works if you prefer forms for scorers. Don’t pay for software to do this.

You can use an LLM as a third, noisy scorer to flag candidates for human review, and it’s fast for a first pass over fifty pieces. Treat it as triage only. In my experience models rate fluent, generic prose generously, which is exactly the failure you’re trying to catch. The humans’ numbers are the record.

Start with last quarter

Don’t wait for a clean month. This week, pick ten pieces from last quarter, have two people score them blind against the table above, and argue about the disagreements. You’ll have a baseline, a first draft of your anchors, and, quite possibly, an uncomfortable discovery about which pieces your team actually rates.