Reviewing Prompts in a Team: Treat Prompts as Code
When working with Large Language Models (LLMs) as an individual developer or researcher, the temptation to adjust prompts ad-hoc is strong. A word here, an instruction there, until the output looks good in the playground. However, once AI applications form the core of production applications, this "cowboy approach" is no longer sustainable. Prompts must be handled, reviewed, and approved in a structured way within a team.
In this article, we discuss why you should review prompts with the same discipline you use to review software code (such as Python or JavaScript). We cover setting up test sets, separating subjective taste from measurable quality, and we provide a concrete template that you can integrate into your pull requests today.
The Mindshift: From Solo Project to Team Product
A prompt is not just a piece of text; it is declared logic. Where traditional code defines the how, a prompt defines the what. A seemingly innocent adjustment in a prompt can have far-reaching consequences for the reliability and security of the application. That is why it is essential to work with proper prompt version control.
Within a team, the following problems often arise if prompts are not structurally reviewed:
- Regression: A prompt change solves a specific edge-case problem, but breaks the correct functioning for three other common scenarios.
- Loss of context: The original author leaves or is on leave. No one on the team understands why a specific instruction (for example: "Answer as if you were a tree") was ever added.
- Security risks: New prompt instructions accidentally open the door to prompt injection attacks because input validation or boundaries are not sharply defined.
Why Every Change Needs a Test Set
A fundamental rule when reviewing prompts is: you cannot judge a change without evidence. Because the output of an LLM is non-deterministic (even at a temperature of 0, absolute consistency is not guaranteed over time), a reviewer cannot simply look at the prompt and say with 100% certainty: "This will work."
For every significant prompt change, the developer must run the prompt against a gold standard: the test set. A good test set for prompts contains:
- The 'Happy Path' cases: Typical input where the model must perform the standard task perfectly.
- Edge cases: Unusual, complex, or unclear input where the prompt must guide towards failing gracefully or asking for clarification.
- Adversarial cases: Input intentionally designed to bypass the guidelines of the system prompts.
During a review, the author must not only provide the new prompt, but also a diff (difference) of the generated output compared to the previous prompt version, based on the agreed test set. You can read more about structured testing of LLM outputs on our sister platform in the guide on LLM quality assurance and validation.
Subjective Taste vs. Measurable Quality
One of the biggest pitfalls during a prompt review is the discussion about taste. Phrases like "I don't think this answer sounds friendly enough" or "Maybe it should be a bit shorter" are disastrous for progress. They lead to endless iterations without a clear end goal.
To review effectively in a team, subjective taste must be converted into measurable rubric criteria. When assessing common prompt errors, we apply objective standards.
| Subjective Feedback (Incorrect) | Measurable Feedback (Correct) |
|---|---|
| "The tone is too boring." | "The output lacks the prescribed brand identity: use a maximum of 2 sentences per paragraph and address the reader as 'you'." |
| "The answer is too long." | "The output exceeds the strict limit of 150 words specified in the system instruction." |
| "It feels like the model is hallucinating." | "The model claims facts in paragraph 3 that are not in the provided RAG (Retrieval-Augmented Generation) context." |
The Use of 'LLM-as-a-Judge'
To further curb discussions about subjectivity, mature AI teams often deploy another LLM (often a heavier, slower model like GPT-4 or Claude 3 Opus) to score the output of the modified prompt. This principle, where an independent prompt evaluates the quality of another prompt on a scale of 1 to 5 based on established criteria, provides a quantitative basis for the human reviewer.
Dealing with Model Updates that Change Behavior
A prompt never lives in a vacuum; it is inextricably linked to the underlying model. When OpenAI, Google, or Anthropic updates a model (for example, from version 0613 to 1106), the semantics of your prompt can suddenly be interpreted differently. What previously yielded a perfect JSON output might now result in JSON with unwanted markdown wrappers.
A good team practice is to always pin prompt versions to a specific model version. When a model upgrade is required, this is treated as a major breaking change. The existing prompts must go through the entire test set, and the review process focuses specifically on "model drift": the subtle ways in which the new model version deviates from the old one. You can find more in-depth information on this phenomenon in our overview article on the impact of model updates on production environments.
Prompt Ownership and Responsibility
If everyone is responsible, no one is responsible. Within a development team, there must be clarity about the ownership of prompts. In practice, we often see the following division of roles:
- The Domain Expert (Subject Matter Expert): Determines the *content* accuracy. If the prompt summarizes medical advice, the doctor or researcher is the one who determines whether the output is correct.
- The Prompt Engineer / AI Developer: Is responsible for the syntax, efficiency (token usage), and robustness against edge cases. They structure the prompt.
- The Software Engineer: Ensures that the input and output of the prompt are correctly handled and integrated into the rest of the application code.
During a pull request (PR) for a core prompt, there must be at least an approval from the person managing the application logic, and—in the case of content changes—the domain expert.
Crucial Documentation for a Prompt
A prompt without documentation is legacy code from the moment it is committed. To properly review and maintain a prompt, contextual information is essential. This documentation ideally belongs in a central prompt library or at least in a docstring above the code where the prompt is called.
A complete prompt documentation always contains:
- Goal: What is the exact, defined task of this prompt? (e.g., "Extracts dates and amounts from raw invoice text").
- Assumptions: What conditions must be true before this prompt works? (e.g., "Assumes that text has already been stripped of HTML tags" or "Only works well with Dutch or English material").
- Known Weaknesses: Where does the prompt structurally fail? (e.g., "Sometimes the model confuses VAT amounts with the total amount for receipts from Belgium"). This helps reviewers understand the scope of a problem and prevents them from blindly searching for a "perfect" prompt that does not exist.
The Light Review Checklist and Template
To ensure the review process runs smoothly in practice, we recommend setting up a standard Pull Request template in platforms like GitHub, GitLab, or Azure DevOps. Below you will find a lightweight template specifically designed for prompt changes.
pull_request_template.md or make it a specific issue template within your repository.
Conclusion
Building robust AI applications requires us to move away from the idea that prompts are magical pieces of text that we iteratively adjust in a web interface. By treating prompts as a fundamental part of your codebase—including version control, objective test sets, clear ownership, and formal peer reviews—you increase the stability of your application and reduce stress within the development team. Implementing a review template as described above is an excellent first step toward a mature AI development process.



