Research · Published · Updated
Testing Whether a Weekly Review Finds More Defects Than a Monthly One
A small before-and-after design for review cadence, with a fixed sample, a defect dictionary, and an honest account of seasonality.
On this page
Headline signal
A local comparison design, not evidence that one cadence fits every team. (https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final). Treat this as context for planning, not a guarantee about a particular assistant or operating result.
Analysis and operating context
Every quality program eventually argues about cadence. One side says weekly reviews catch problems early; the other says they consume the time that could be spent fixing them. The argument rarely produces evidence because the team changes several things at once. This design isolates the cadence question enough to support a local decision, without pretending that a small sample settles it for everyone.
Define the defect dictionary first. List the categories the review will look for, such as missing evidence, wrong source, incomplete instruction, unsafe access, customer-facing error, and formatting failure. Give each a one-sentence test and an example. Freeze the dictionary before the comparison. If the categories change halfway through, the before-and-after numbers describe different things.
Choose the work lanes and the observation period. Use lanes with stable volume and comparable task types. Record the volume, the staffing, the client mix, and any tool changes during the period. If a new client arrives in week three, the comparison is confounded and should be reported as such rather than adjusted away.
Randomize or alternate the review cadence if the operation allows it. Alternating weeks is often more practical than random assignment: one week reviewed weekly, the next reviewed at the end of the month. This spreads seasonal effects across both conditions. If alternation is impossible, use a before-and-after design and state that time is a competing explanation.
Keep the review sample size constant. A weekly review of ten items and a monthly review of forty items are not the same intervention. Decide the number of items reviewed per period in advance and hold it steady. Otherwise the cadence effect is mixed with a volume effect and the result cannot be interpreted.
Measure detection, not only correction. For each reviewed item, record whether a defect was found, its category, its severity, the time between the work and the review, and whether the defect had already reached the client. A cadence that finds more defects before delivery is valuable even if the total defect count is unchanged.
Use a second reviewer on a subset to check consistency. The same item should receive the same category when reviewed twice without discussion. Report agreement counts and disagreements. If agreement is low, the dictionary is not ready for a cadence comparison. Improving the dictionary is a legitimate outcome of the pilot.
Analyze with counts and rates per hundred reviewed items, and show the raw numbers. With small samples, avoid precise claims; a difference of two defects is not a trend. Report the confidence interval if the design supports one, and describe what would change the conclusion. Include the cost side: reviewer hours, rework hours, and delay caused by the review itself.
Account for the learning effect. Reviewers get better at finding defects as they practice, regardless of cadence. If the weekly period comes first, part of the improvement may be experience. Alternating helps, and a second cycle after the team is familiar with the dictionary gives a cleaner comparison.
The conclusion should be a decision rule rather than a slogan. For example: if weekly review finds at least a stated share of severe defects before delivery and does not delay accepted work beyond the agreed window, keep it for the high-risk lane. Apply the rule to the evidence and record the date for the next review. The study is a way to make the cadence choice reviewable, not a permanent verdict.
Apply this to a research support lane
Pick one lane and freeze the defect dictionary before the first cycle.
Design a cadence testSources
Frequently asked questions
How long should the pilot run?
Long enough to cover at least two full cycles of each cadence and one ordinary disruption. For most teams that means six to eight weeks, with the limitation stated.
What if the team cannot alternate?
Use a before-and-after design, keep the sample size constant, and treat time as a competing explanation rather than claiming a clean causal result.