When answers aren't enough - G-Eval

Introduction

When evaluating AI or building a chatbot or writing assistant, you see the model is responding perfectly. You can even feel it when reading the outputs, but when someone asks “how do we know it works?” there’s no clear answer to that. How do we know that the model is generating text fine based on what we’ve asked.

G-Eval fills this gap. It is a framework for LLM based evaluations. Using the magic of probability and chain of thought we get the answer to such subjective questions.

What is G-Eval

G-Eval is a framework for LLM based evaluation of LLM generated text. It is useful for evaluating outputs where there isn’t one correct answer, or more precisely, where we don’t have a golden reference or a fixed set of rules to compare the output against.

It is based on 2023 paper by Yang Liu and coauthors at Microsoft, which is also open source in this github repository.

This framework has 3 main components.

• A prompt that states the task and defines the evaluation criterion properly. For example “Rate this summary tone on a scale of 1 to 5”.

• The judge then prepares the steps to proceed, evaluate and score the example. It makes use of chain-of-thought (CoT) to prepare the steps for the evaluation.

• Then the scoring function is called and instead of simply taking the score generated by the model as a definite value, G-Eval uses the probabilities assigned to the possible score tokens to calculate a weighted score, that takes the prompt, generated CoT, input task, output and returns the probable score instead of a definite score.

G-Eval in practice

Now let’s take a look at the example and try to make use of this framework.

Example

Task:
Summarize this paragraph.

Criterion:
Rate the tone of the summary from 1 to 5.

Output:
"The company grew rapidly. It hired 500 people.
This caused its revenue to increase."

1. Define the criterion

We first define the criterion before sending anything to the LLM.

Criteria: Tone

1 = soft
5 = harsh

2. Generate the CoT

The model then generates the evaluation steps

1. Check whether the ideas are logically connected.
2. Check whether there are contradictory statements.
3. Check whether the summary is easy to follow.
4. Assign a score from 1–5.

3. Evaluate the output

Now the evaluation prompt contains the original task, the generated answer, the criteria, and the evaluation steps.

The judge model processes all of that and produces a score. You might see something

score: 4

But how did the model actually scored? Why only 4 and not 3 or 1 or 5?

Instead of treating the generated score as an absolute value, G-Eval looks at the probability of each possible score token.

But, how do we get the probability distribution of tokens? This requires an underlying API to expose token-level log probabilities (such as OpenAI’s logprobs parameter), and a lot of chat-completion endpoints either do not expose them at all or only expose them awkwardly.

Example

1 → 0.02
2 → 0.04
3 → 0.14
4 → 0.55
5 → 0.25

G-Eval uses the formula below to get the weighted score.

$$Score = \sum_i p_i \times i$$

Score = (1 × 0.02) + (2 × 0.04) + (3 × 0.14) + (4 × 0.55) + (5 × 0.25) = 4.0

Why this calculation?

Why not just take the highest-probability score?

For example:

1 → 0.02
2 → 0.04
3 → 0.14
4 → 0.55
5 → 0.25

The most probable score is clearly 4.

But imagine another evaluation gives us:

1 → 0.00
2 → 0.00
3 → 0.01
4 → 0.97
5 → 0.02

Both would give us 4 if we simply picked the most probable score.

But they don’t represent the same judgment.

In the first case, the model is somewhat split between 4 and 5. In the second case, it is almost certain that the score should be 4.

The weighted calculation:

$$Score = \sum_i p_i \times i$$

lets us use the whole probability distribution instead of throwing away everything except the most probable score.

So the final score is not simply “the model said 4”. It takes into account how likely the model thinks every possible score is.

Can we trust the score?

Now we have a score. But how do we know that the score actually means anything?

An LLM is judging an LLM-generated output. We are essentially asking a model to make a subjective judgment and then using that judgment as an evaluation signal.

So the question isn’t really:

Is the score objectively correct?

There is no easy way to answer that.

A better question is:

How closely does the score agree with human judgment?

The G-Eval paper compares its results with human evaluations and reports a stronger correlation than several existing evaluation metrics in its experiments. For example, the GPT-4 based G-Eval setup achieved a Spearman correlation of 0.514 on the summarization task.

This does not mean that G-Eval is 51.4% accurate. Spearman correlation measures how closely the rankings produced by G-Eval agree with the rankings made by humans.

And this is also where the limitations of using an LLM as an evaluator start to appear. The model can have its own biases, the evaluation can change depending on how the prompt or criteria are written, and different models can produce different judgments.

G-Eval does not magically turn subjective evaluation into an objective truth. It gives us a structured way to turn subjective criteria into a measurable signal.

Conclusion

Evaluating AI-generated text is not always as simple as checking whether the answer matches a predefined answer. Sometimes there is no single correct answer. We care about things like relevance, coherence, tone, and overall quality instead.

G-Eval gives us a way to handle these cases by using an LLM as an evaluator, generating evaluation steps, and using the model’s probabilities to produce a score.

It is not a perfect or objective measure of quality. The evaluator itself can have biases and its scores still need to be validated against human judgment. But it gives us something useful: a structured way to turn a subjective question like “Is this answer good?” into a measurable signal.