Research · Published:
Research: How Consistently Do Reviewers Judge Article Alt Text?
A blinded review records agreement on whether image descriptions match the page context and avoid unsupported guesses.
Headline signal: Two independent ratings before discussion for each sampled image (OutsourcedAssistants.com study protocol).
Research question and scope: How consistently do reviewers classify article alt text as useful, redundant, unsupported, or mismatched to the rendered image? The unit of analysis is one rendered article image with its surrounding heading, caption, and alt text. This protocol covers one declared article operation. It does not measure a worker's general ability, compare nationalities, or predict outcomes for other publishers.
Methodology: Draw a sample of live images and remove author identity from the review packet. Give two reviewers the same written categories based on WCAG guidance. Record independent ratings, reasons, disagreements, adjudication, broken assets, and whether surrounding text already conveys the image purpose. Set the observation dates, eligibility rule, outcome fields, reviewer, exclusions, and pass threshold before reviewing results. Keep original inputs, timestamps, decisions, and corrections.
Evidence treatment: classify each item as a primary-source fact, local observation, calculation, interpretation, example, or recommendation. Record the publisher, page or dataset location, displayed date, and access date. The cited guidance informs the protocol; it does not supply the local result.
Inference boundaries: Agreement reflects the supplied rubric, sample, and reviewers. It does not prove that readers using assistive technology experience the descriptions in the same way. Report counts with their denominators and keep ordinary, returned, escalated, overridden, and unresolved items separate. Do not use causal language unless the design addresses competing explanations.
Limitations: Small samples may omit uncommon image purposes, reviewer training affects agreement, and the protocol does not replace user testing. Public guidance can change, records may be incomplete, and reviewers may apply categories differently. Disclose missing records, deviations, small samples, and unresolved cases.
Decision use: use the finding for one limited workflow choice, then observe the next cycle. Editors and accountable owners retain publication, access, privacy, policy, legal, payment, and exception decisions.
References: the primary guidance pages listed below define the content, accessibility, security, and personal-information considerations used to frame this protocol.
Sources
- Google Search Central: Creating helpful, reliable, people-first content
- W3C Web Content Accessibility Guidelines 2.2
- NIST Cybersecurity Framework 2.0
- U.S. Federal Trade Commission: Protecting Personal Information
Frequently asked questions
Does this protocol establish cause and effect?
No. It describes a bounded observation. Any causal conclusion would require a design that addresses selection, timing, topic difficulty, reviewer availability, and other plausible explanations.
What records should the team retain?
Keep the question, sample rule, original inputs, timestamps, source notes, exclusions, reviewer decisions, outcomes, corrections, and stated limitations.