Research · Published:
Do Editors Agree on Why an Assistant Draft Was Returned?
This reliability study separates repeatable return categories from judgments that depend on unstated editorial context.
Headline signal: Eight return reasons coded independently before calibration (Outsourced Assistants research protocol).
Research question and scope: Do two editors independently assign the same primary return reason to assistant-supported article drafts? The unit of analysis is one dated article assignment and its review record. The study does not measure an assistant's general ability, predict traffic, or establish a result for a client. Publication judgment remains with the editor; the assistant may collect approved evidence, maintain the record, and flag a mismatch.
Methodology: Select returned drafts across Blog and Research, remove the original labels, and have two editors code each item using eight written reasons. Record exact agreement, split decisions, confidence, and whether the original brief contained the rule. Calibrate on disagreements, then test revised definitions on a fresh sample. Before observation, define the article family, reader question, material claims, source classes, owner, and pass rule. Sample ordinary work alongside at least one correction and one exception so an apparently smooth queue does not hide its weak cases. Preserve the original brief and timestamps rather than reconstructing them after the outcome is known.
Evidence handling: distinguish a source fact from a local observation, calculation, interpretation, example, and recommendation. Map every material factual statement to the passage that supports it, including publisher, date, population, definition, and access date. Repeated pages that rely on the same underlying document count as one evidence lineage, not independent confirmation.
Relevant standards: Google Search Central's people-first content questions inform usefulness and sourcing; NIST's AI RMF informs documented risk and oversight; WCAG 2.2 informs accessible presentation; and FTC guidance informs claim substantiation. These sources provide principles, not a benchmark for OutsourcedAssistants.com and not proof that the proposed workflow improves commercial performance.
Finding and interpretation: Disagreement is most informative when it clusters around changed briefs, source mismatch, and style. Those cases may need clearer definitions or a requirement to split one return into separate corrections. Agreement does not establish that either reviewer is correct; it shows whether the shared rule travels. Treat the result as evidence about the named sample and conditions. If a source, reviewer, deadline, or definition changes, record the change before comparing periods. An escalation can show that a control worked; a fast completion can still fail when its evidence cannot be reconstructed.
Role boundary: an assistant can assemble the sample, normalize agreed fields, calculate a defined measure, and surface disagreement. The owner decides materiality, acceptable uncertainty, public wording, access, policy, and any change to the operating rule. Consensus or a passing technical check does not expand that authority.
Limitations: The sample may overrepresent difficult drafts, reviewers may share assumptions, and post-calibration agreement is not independent. Sentence-level and assignment-level reasons can also differ. Search ranking, inaccessible sources, small samples, changing web pages, reviewer judgment, and unrecorded private context can also affect the result. The method cannot prove completeness, causation, or future performance. Report exclusions and unresolved questions beside the conclusion rather than burying them in working notes.
Evidence-led conclusion: use the finding to choose one bounded next action—continue the rule, clarify the brief, narrow a claim, add review coverage, or pause. Repeat the observation on a fresh sample before widening the workflow. This supports a transparent management decision while keeping daily Research distinct from practical Blog guidance.
Sources
- Google Search Central: Creating helpful, reliable, people-first content
- NIST AI Risk Management Framework
- W3C Web Content Accessibility Guidelines 2.2
- FTC advertising and marketing guidance
Frequently asked questions
Is a high agreement rate a quality score?
No. It measures consistency of the coding rule, not the truth or quality of the draft.