Software Quality · AI Assurance · Software Failures and Quality Decisions
Gemini’s Image Failures: AI Assurance Needs Better Questions Than “Does It Generate?”
A practical AI Assurance analysis of Gemini’s 2024 image-generation failures: task-specific acceptance criteria, evaluation sets, refusals, and regression evidence.
A successful response can be a failed task
An AI feature can return a polished result, use the right output format, and still fail the user. The transport succeeded. The product promise did not.
In February 2024, Google paused Gemini’s generation of images of people and acknowledged problems including inaccurate historical depictions and inappropriate refusals. Its explanation described overcompensation in some situations and excessive caution in others. Google’s explanation of the image-generation issues
That public account supports a specific quality discussion. It does not reveal the complete internal evaluation process, and I would not use it to claim that the team performed no testing.
The practical question is how to judge an AI-enabled feature when several reasonable goals can conflict.
Separate the product goals before scoring the output
A product might aim to represent people fairly, follow a user’s request, avoid harmful output, and preserve accuracy where the task calls for it. Passing one goal does not establish the others.
Historical reconstruction, contemporary illustration, and fictional world-building are different tasks. Applying identical acceptance criteria to all three can produce contradictory decisions.
I would start an evaluation by making those task categories explicit. What is the user asking the product to do? Which constraints are essential? Which variations are acceptable? When is uncertainty unavoidable? Who is qualified to assess the answer?
This is requirements work. A model score cannot repair an undefined product promise.
Define an oracle appropriate to the task
A test oracle is the basis used to decide whether an observed result is acceptable. Conventional software often gives us exact values or formal rules. Generative outputs may need a combination of deterministic checks and structured judgement.
For a hypothetical image feature, a rubric could distinguish:
| Dimension | Evaluation question |
|---|---|
| Instruction adherence | Does the output address the task actually requested? |
| Contextual accuracy | Does the task require factual constraints, and are they respected? |
| Representation | Are unjustified assumptions or exclusions introduced? |
| Refusal behaviour | Is refusal appropriate for this request? |
| User control | Can the person understand limitations and revise the request? |
The rubric should define what an unacceptable result looks like for each category. “Looks good” is not a reliable release criterion.
These are proposed evaluation dimensions, not a claim to reproduce Google’s internal rubric.
Build a small, purposeful evaluation set
A large prompt list is not automatically a strong test set. I would begin with cases chosen to distinguish the behaviours the team needs to understand.
Include ordinary requests, requests requiring contextual constraints, fictional prompts, ambiguous prompts, and cases where the expected behaviour is clarification or refusal. Keep the intended task and rationale with each example.
Counterfactual pairs can help investigate consistency: change one relevant property while preserving the rest of the request, then compare whether the response changes for a justified reason. A difference is evidence to inspect, not automatically proof of bias or a defect.
Domain-sensitive cases need appropriate reviewers. General-purpose evaluators should not make confident accuracy claims in areas where they lack the necessary knowledge.
Evaluate variability rather than one screenshot
A single output demonstrates one observation. It does not establish the distribution of future outputs.
For non-deterministic features, repeat selected cases under recorded conditions. Keep the model or service version where available, relevant settings, input, output, evaluation result, and reviewer notes. Distinguish a repeated failure from a one-off observation without dismissing a serious single example.
A useful report describes the sampling method and uncertainty. It should not call a small unrepresentative set a universal accuracy rate.
Automated checks can verify output structure, required fields, error states, and stable workflow constraints. Human review can assess dimensions that need context. The evaluation should explain which judgement came from which method.
Refusal is product behaviour too
Teams often evaluate harmful compliance and forget harmful obstruction. A system that declines legitimate tasks can fail people even while appearing cautious.
Expected refusal behaviour needs its own cases and rationale. The test should inspect what the user sees: a clear explanation, an appropriate next step, and an opportunity to clarify where that is useful.
A generic refusal message may conceal several different states: a policy decision, an unavailable dependency, an unsupported capability, or uncertainty. Those states need different investigation and sometimes different interface behaviour.
This is why AI Assurance includes the surrounding product experience, not just the generated content.
Evaluate changes as regression risks
A change intended to improve one dimension can worsen another. A model upgrade, prompt revision, retrieval change, or policy adjustment should be evaluated against the relevant baseline with the same case definitions.
I would compare the results by task category rather than compress everything into a single average. An improvement on common creative prompts can hide a regression on constrained factual tasks. The release decision needs to see that trade-off.
Preserve examples of newly failing behaviour and record the decision taken. Some limitations may be acceptable within a clearly defined scope; others should block release or require a narrower feature boundary.
Make the assurance claim match the evidence
A defensible conclusion might be that a feature met specified criteria on a documented evaluation set, with certain unresolved cases. That is useful evidence for a product decision.
It is different from declaring the model safe, unbiased, or accurate in every context. Those words can imply a breadth of assessment that the exercise did not perform.
The need for QA does not disappear when output becomes probabilistic. It becomes more important to define the task, make judgement explicit, document variability, and connect evaluation results to the decisions users will experience. Read about my software quality consulting.