How to review an AI-generated bug investigation

An AI-generated bug investigation is useful only when a human can check how it reached its conclusion. Review the evidence first, confirm that every source claim refers to the deployed revision, compare the conclusion label with the strength of the evidence, and keep important unknown facts visible. Do not approve a change merely because the explanation sounds confident.

A good review should answer five questions: Did the investigation examine the right source? Do the citations support the claims? Is the conclusion stronger than the evidence allows? Is the recommended action safe and proportionate? Is the draft response accurate without exposing private technical detail?

Confirm that it investigated the right version

A source citation is meaningful only when it identifies the version of the code that the visitor used. A file path and line range from the current main branch may be irrelevant to a report created before several deployments.

Check the full repository revision attached to the investigation. It should match the revision recorded for the report when the investigation entered the queue. A later deployment or repository check should not silently replace it.

Then ask:

• Was this revision actually deployed when the visitor reported the problem?
• Does each source citation use that same full revision?
• If the report is being investigated again after a change, is the newer revision clearly part of a new investigation rather than a rewrite of the old one?

Read why the deployed commit must stay attached to a report for the complete version-mapping problem.

Read the evidence before the conclusion

Start with the observable facts, not the summary. Relevant evidence may include the visitor statement, a visitor-approved screenshot, a bounded console error, failed request metadata, release information, or a cited source excerpt.

For each item, check three things:

1. Identity. The conclusion should cite a specific evidence item, not refer generally to "the logs" or "the code".
2. Relevance. The cited item should concern the reported action and visible result.
3. Support. The excerpt should establish the claim being made, not merely mention the same feature.

A real file and line number do not automatically make a conclusion correct. Code can be relevant without being causal. For example, a cited error-handling branch may explain the message a visitor saw while leaving the original failure unexplained.

Treat browser evidence the same way. A request returning an error establishes that the request failed. It does not, by itself, establish why it failed. The guide to browser evidence for website problems explains what each category can and cannot show.

Match the conclusion label to the evidence

Automated investigations should use calibrated labels instead of presenting every explanation as a fact.

• Confirmed needs direct evidence that excludes reasonable alternatives. It should have repository evidence and direct visitor-browser evidence.
• Likely needs a strong source-backed chain, but may leave a reasonable alternative open.
• Possible means the evidence supports the explanation without establishing it as the leading cause.
• Unknown means the available evidence does not support a cause.

Challenge a confirmed conclusion first. Ask what competing explanation the evidence rules out. If the answer is missing, the conclusion should usually be likely or possible.

Do not treat unknown as a failed review. An honest unknown can still record what was checked, identify the evidence that is missing, and recommend the next safe observation. That is more useful than a specific but unsupported cause.

Inspect the unknown facts

Unknown facts should be a separate section, not qualifications hidden in a paragraph. They define the boundary of the result.

Look for unknowns that could change the recommended action. Examples include whether the problem affects one account state or every visitor, whether the failure still occurs on the current deployment, and whether the source path is active for the reported page.

Each important unknown should explain why it remains unknown and what evidence could resolve it. Reject vague entries such as "more testing is needed". A useful unknown names the missing fact and the next bounded check.

Review the recommended action as a proposal

A recommendation is not an instruction to release. It should state one proportionate next step, explain its risk, and require human approval.

Check that the action:

• Cites the evidence it depends on.
• Addresses the reported behaviour rather than an unrelated code smell.
• Does not assume an unknown fact is true.
• Includes a focused verification step when a change is proposed.
• Uses the normal review and release process.

If the conclusion is unknown, the recommendation can be to collect one missing observation, review a product decision, or give safe visitor guidance. It does not need to invent a code change.

Check the draft response separately

The draft response is customer-facing language, not a technical summary. Review it independently even when the investigation is sound.

It should describe the behaviour in plain language, avoid blaming the visitor, avoid promising a release date, and never say a problem is fixed before verification. It should not name private source paths, filenames, credentials, internal tools, or operational details.

A human should be able to edit the draft without changing the underlying investigation result. The evidence and conclusion remain the record of what was found; the draft is communication prepared for review.

Use a short approval checklist

Before accepting or acting on an automated investigation, confirm:

• The report and investigation refer to the correct deployed revision.
• Every important claim cites registered evidence.
• The source excerpts actually support the claims.
• The conclusion label matches the evidence strength.
• Important alternatives and unknown facts remain visible.
• The recommended action is bounded and requires human approval.
• The draft response is accurate, plain, and free of private technical detail.

If any answer is no, send the result back for more investigation or record that the conclusion is not supported. Fluency is not a substitute for a checkable evidence chain.

How IssueProbe presents a result for review

IssueProbe separates the raw report, safe browser evidence, conclusion, evidence citations, unknown facts, recommended action, and draft response. A likely or confirmed conclusion needs repository evidence, and a confirmed conclusion also needs direct visitor-browser evidence. The result can state that the cause is unknown when the available evidence does not support one.

Each eligible investigation is bounded and read-only. It can read and search the approved repository and register source citations, but it cannot push a change, run arbitrary repository scripts, release code, or send the draft response to the visitor. A human reviews the result, can revise the draft separately, and records the final disposition.

See the illustrative example report for the complete review structure. For the context an automated investigation needs before it starts, read how to give a coding agent enough context to find a fix.