Large Language Models (LLMs) are being – or are certainly on their way – to being trusted with decisions that once belonged solely to human experts: who gets hired, who receives care, who wins a case. But behind the scenes, a quiet instability threatens the entire foundation. 

A new report from software engineer and researcher James Padolsey, published via The Collective Intelligence Project, titled LLM Judges Are Unreliable, delivers a damning analysis: today’s frontier models – from OpenAI, Google, Anthropic, and others – are intrinsically ill-suited to making sensitive or high-stakes judgments.

Despite the hype, LLMs aren’t machines of perfect reason. They are closer to statistical guessers – and when asked to judge, rank, or classify – important tools in parsing information that informs decision-making – they show troubling patterns of inconsistency, positional bias, and prompt sensitivity. 

These issues, the study warns, are not only real – they’re measurable.

The Fragility Within

Padolsey conducted a suite of evaluations across leading LLMs, including GPT-4.1, Gemini Flash, Claude Sonnet, and Mistral Large. The aim: to stress-test their ability to judge.

Here’s what the study found:

Pairwise Choice – Positional & Labeling Biases
LLMs tend to favor the second option when comparing two responses – picking it about 61% of the time. Even small changes to the wording or labels can shift that preference by 5–10%, showing that how choices are presented can quietly influence the outcome, much like in human decision-making.

Rubric-Based Scoring – Order & Context Effects
The order and context in which traits are scored can sway results — traits rated last tend to get lower scores, and judging multiple traits together often leads to softer, less critical evaluations than judging them individually.

Essentially, how and when a trait is scored through an LLM – whether it’s rated first or last, alone or with others – can significantly influence the result, often leading to softer or lower scores due to context and order effects.

If you’re using LLMs for something like hiring, moderation, peer review, etc. these effects could skew the results just based on how you structure the prompt. That’s a serious problem if you need fair, consistent judgments.

Scale Interpretation & Negative Trait Biases
LLMs tend to assume that higher numbers mean better – like a 5-star review – which creates problems when scoring negative traits like toxicity or sexism. When “5” means “very toxic,” models often understate the severity, giving lower scores than they should. In one test, the same sexist content scored much higher when rated on a letter scale (A–E) than on a 1–5 number scale, showing that the format of the scale can strongly affect results.

System Prompt Unpredictability
Telling an LLM to “avoid bias” doesn’t always work – and can sometimes make things worse. In one test, a prompt meant to reduce position bias actually made the model favour the second option even more. In another, the same instruction caused a big swing toward the first option being picked far less often. The effects of de-biasing prompts are unpredictable and vary depending on how the task is set up.

Classification Instability & Ambiguity
LLMs often change their classification of ambiguous content just because the prompt wording or category order changes. In some tests, the same item got a different label every time the setup was tweaked — showing that the result may reflect the prompt, not the content itself.

ELO/Dynamic Ranking
ELO rankings – where items are compared in pairs over and over – while often seen as reliable, can be unstable if the pairwise comparisons feeding into them are biased. In Padolsey’s tests, simply changing the wording of the comparison prompt caused major shifts in rankings – showing that what looks like a strong preference may actually be fragile and prompt-dependent.

These findings underscore a simple truth: how you ask a model something deeply affects what it tells you, 

But why is this happening?

A System Not Built for Consistency

The problem isn’t just bad prompting or poor model tuning. It’s architectural.

LLMs do not operate with the mechanistic precision of traditional programmes. Every output from an LLM reflects a complex dance of statistical inference drawn from billions of parameters and terabytes of training data – data that pushes and pulls on every decision in subtle, and terrifyingly unknowable ways.

In fact, a piece by the BBC on AI consciousness saw Prof Murray Shanahan, principal scientist at Google DeepMind and emeritus professor in AI at Imperial College, London, tell the media outlet that “we don’t actually understand very well the way in which LLMs work internally, and that is some cause for concern.”

“We are in a strange position of building these extremely complex things, where we don’t have a good theory of exactly how they achieve the remarkable things they are achieving.”

Recommended reading

• UK Policing AI and Automation Called Out Over Racial Bias
• Civil Rights Alarm Bells Raised over PM’s Surveillance Plans
• UK Police Use of Facial Recognition Found to be ‘Unethical’

LLMs don’t treat system instructions – like “be a fair judge”- as special or guaranteed to guide behaviour. They read them the same way they read everything else in the prompt. So, the content you’re asking the model to evaluate can easily confuse or override those instructions. That’s why tricks like telling a model “you’re a pirate now” (called jailbreaks) can work.

In critical areas like law or finance, this design flaw means models can quietly misfire, even when they seem to follow the rules. Padolsey’s report shows these aren’t rare glitches – they’re baked into how LLMs operate.

What You Can Do About It

The report doesn’t just diagnose; it offers a playbook. Here are some of the most actionable countermeasures:

• Use Neutral Labels: Avoid “Response 1” or “A/B” — use abstract labels like “X” or “ID_123”. Randomise the order of responses to reduce position bias.
• Test Every Part of Your Prompt: Even small wording changes can affect results. Always test system instructions, rubrics, and “de-biasing” language.
• Choose the Right Rating Scales: Try different scoring formats (numbers, letters, descriptive labels). Order and format can change how the model scores, especially sensitive traits like toxicity or bias.
• Avoid Over-Reliance on Pairwise Rankings: Rankings like ELO can hide bias from earlier comparisons. When possible, rate items individually instead.
• Build Clear and Flexible Categories: For classification tasks, use simple labels with clear options like “none” or “uncertain” to handle edge cases better.
• Use More Than One Model: Don’t rely on just one LLM. Different models have different biases. A diverse set of models is more reliable.
• Repeat Runs Carefully: Repeating model runs can reduce randomness, but won’t fix deeper systemic biases. Use thoughtfully.
• Don’t Assume Humans Are Fair: Human judgment has bias too. Always test for fairness, even if results “match” what people say.
• Watch Out for Groupthink in Consensus Models: Combining multiple outputs helps with noise, but not if all models share the same bias. True diversity matters.
• Think Beyond the Output: If AI is used in real-world decisions (hiring, healthcare, law), you’re accountable. Be ready to explain and defend the results – regulators will ask.

The Padolsey report tells a familiar story: technology is being deployed faster than it is being understood. 

LLMs are not impartial judges. They are sensitive, unpredictable systems, heavily influenced by prompt structure, label semantics, and even the ordering of inputs – that doesn’t create solid footing for important decision-making.

If we want LLMs to make fair, consistent judgments, we must stop treating prompt engineering as an art – and start treating it as a science.