I used to decide that a prompt revision was better because the new output looked better. The language was smoother. The structure was tighter. The answer felt more complete. I saved the new version, discarded the old one, and moved on.
Then I started keeping both versions and testing them on fresh inputs. Some of the “improvements” held up. Many did not. A few made the prompt more fragile or more prone to the exact problems I thought I had fixed. The feeling that the revision was better had been unreliable. I needed a clearer way to decide.
The method below is the one I now use before I replace any prompt I rely on. It is deliberately simple. It does not require special tools. It only requires the willingness to run the old version and the new version under the same conditions and to judge them by the same practical standard.
Why “Looks Better” Is Not Enough
A single improved output can be produced by luck, by a softer interpretation of the instructions, or by the model leaning into a more fluent style that happens to match your preference on that day. None of those factors guarantee that the revised prompt will perform better on the next input, or the one after that.
Worse, a revision that increases polish can simultaneously increase quiet failures—softened requests, invented details, dropped exceptions, or over-confident inferences. Those failures are easy to miss when you are only reading for surface quality. They become expensive when you later treat the output as reliable.
A revision is actually better only when it improves the results you care about across more than one input, without introducing new problems that cost you more time or more risk than the original.
The Four-Question Test
Before I keep a revised prompt, I ask four questions. All four must be answered with evidence from side-by-side runs, not from the impression left by a single good output.
1. Does it improve the specific problem I was trying to solve?
I write the problem down before I make the change. “The summary keeps mixing facts and inferences.” “The email rewrite softens direct requests.” “The action list invents owners.” The revision is judged first by whether that named problem occurs less often or less severely. General improvement in style does not count unless the original problem is also reduced.
2. Does the improvement appear on more than one input?
I run the original and the revised prompt on at least three real inputs of the same type. If the advantage appears on only one of them, I treat the revision as unproven. Consistent improvement across varied material is the minimum bar.
3. Does it keep the strengths of the original?
A revision that solves one problem by creating three new ones is not better. I check whether the new version still avoids the failures the original already handled well—especially invention of facts, commitments, or owners, and omission of critical exceptions. If those strengths erode, the revision is usually not worth keeping.
4. Is the output easier or harder to review?
I notice whether I spend less time or more time checking and correcting the new outputs. A prompt that produces slightly cleaner first drafts but requires longer, more anxious review is often a net loss. The total time from prompt to trusted result is what matters.
If the answer to all four questions is favorable, I keep the revision and make it the new baseline. If any answer is negative or unclear, I discard the change or redesign it as a smaller, more targeted edit.
How to Run the Comparison Cleanly
The test only works when the conditions stay the same.
Use the identical model and settings for both versions
Use the same real inputs, not idealized examples
Change only the prompt text between versions
Generate at least two outputs per version per input so a single lucky run does not dominate
Judge the outputs against the four questions above, not against which one feels more impressive
I keep the comparison notes in the same file as the prompt. A few lines are enough: “V3 reduced invented owners on all three note sets. Review time unchanged. Kept.” The record prevents me from re-testing the same dead-end change later.
A Concrete Example: Softened Requests in Email Rewrites
I had a rewrite prompt that reliably produced clear, short emails but occasionally softened direct requests into polite suggestions. I added one sentence: “Keep direct requests direct. Do not replace them with softer phrasing.”
I ran the original and the revised prompt on three real drafts that contained clear asks. On all three, the revised version preserved the directness of the request. It did not introduce new commitments or lengthen the emails. Review time stayed the same. The change solved the named problem, held up across inputs, preserved the original strengths, and did not increase review cost. I kept it.
A different revision I tested in the same period—adding a longer tone guide—produced prettier language on the first draft and re-introduced soft phrasing on the second and third. It failed questions 1 and 2. I discarded it.
The difference was visible only because I ran the side-by-side. The prettier version would have won if I had judged by first impression alone.
What Counts as Evidence of Improvement
I treat the following as positive evidence:
The named failure mode occurs less often across multiple inputs
The severity of the remaining failures is lower and easier to catch
Review time decreases or stays the same while quality on the target problem rises
The shorter or simpler version of the revision performs as well as the longer one (in which case I keep the shorter one)
I treat the following as warning signs:
Improvement appears on only one flattering input
New types of invention or omission appear
Review becomes more effortful or more uncertain
The revision adds length without a clear, repeated gain
Feeling that the prompt is now “more professional” or “more complete” does not count as evidence.
When to Stop Revising
A prompt does not need to be perfect. It needs to be reliable enough for the work you actually do, with failure modes you understand and can catch quickly. I stop revising when additional single-variable changes stop producing clear gains on the four questions. Further edits at that point usually add complexity without reducing total cost.
The same stopping rule applies to prompts I find elsewhere. A well-written example is a useful starting point. It becomes a working prompt only after it survives the four-question test on my own material.
Why This Standard Matters
Without a clear test, prompt libraries fill with versions that once looked good and then quietly degraded. With the test, the library stays smaller and more trustworthy. You stop collecting clever wording and start collecting evidence about what actually improves the results you care about.
The process is slower than simply saving the latest output you liked. It is faster than debugging an over-revised prompt three weeks later when you no longer remember what changed.
Starting With the Smallest Useful Version
The easiest revisions to evaluate are the ones that start from a short, clear baseline. When the original prompt is already bloated, every change interacts with multiple untested instructions and the comparison becomes noisy. That is one more reason to begin with the smallest useful version of any new prompt: it gives you a stable foundation against which later revisions can be measured cleanly.
Make one change. Run the old and the new under the same conditions. Answer the four questions with evidence. Keep the revision only when the answers support it. That sequence turns the feeling that a prompt is better into a decision you can defend and, if necessary, reverse.
A revision is actually better when it solves the problem you named, does so across more than one input, preserves what already worked, and does not increase the cost of review. Everything else is preference or first-impression bias. Test accordingly. Start with the smallest useful version, then let repeated evidence—not a single cleaner output—decide what stays.
No notes on this sheet yet.