Research · Published
Testing Whether Assistant Priority Decisions Are Consistent
A priority rule is only useful if two people apply it the same way. Test the levels, the evidence, and the conflicts.
On this page
Headline signal
One bounded protocol, two independent review paths, and zero provider performance claims (Declared OutsourcedAssistants.com research protocol). Treat this as context for planning, not a guarantee about a particular assistant or operating result.
Analysis and operating context
Research question: do different reviewers assign the same priority to the same request when they use the written rule? The unit is one request and one priority decision. The method tests the rule and the evidence, not the reviewer's seniority.
Start with the written levels and examples. A rule that says urgent when the customer is affected is too broad. Define the impact, the deadline, the workaround, and the consequence. Include examples of requests that look urgent but are routine.
Build a case set with routine, deadline-driven, customer-impacting, and ambiguous requests. Include at least one request where the requester used the word urgent without evidence. Include a request with a real external commitment. The set should cover the boundaries, not only the easy cases.
Ask two reviewers to assign a level and write the evidence. Compare the levels and the reasons. If they agree on the level but for different reasons, the rule may be producing the right answer by accident. Record both the decision and the reasoning.
Measure agreement by level and by evidence. A high agreement on routine cases and low agreement on ambiguous cases is expected. A low agreement on the definition of customer impact is a rule problem. Do not average the levels or treat the majority as correct.
Test the conflict rule. Give the reviewers two requests at the same level and ask which goes first. Compare the answers. If the rule does not resolve the conflict, the owner will be asked to choose every time. That is a design gap, not a reviewer failure.
Review the priority decisions against the actual outcomes. Did the urgent requests actually need same-day work? Did a routine request become a problem because it waited? The outcome data can show whether the levels are calibrated, but it does not prove that a different decision would have changed the result.
Limitations: the case set is small and may not represent the full queue. Reviewers may learn the expected answer during the exercise. The rule may be clear to the owner but not to the assistant. The method measures consistency, not the correctness of the business priority.
Decision use: if agreement is low, rewrite the examples and retest. If the conflict rule is missing, add one. If the levels are too broad, split them. Keep the rule and the test record with the queue documentation so a new assistant can see how the decisions are made.Record the requester's stated urgency separately from the reviewer's assigned level. A request can be marked urgent by the requester and routine by the rule. The gap is useful evidence about how urgency language is used in the organization.
Test the rule with a new reviewer who has not seen the examples. A rule that only works for the person who wrote it is not ready. The new reviewer should be able to apply the levels with the written guide and a short orientation.
Review the priority decisions against the actual work. A request marked urgent but completed three days later without consequence may indicate that the level was inflated. A routine request that caused a missed deadline may indicate that the rule is too narrow.
Record the time between the priority decision and the first action. A correct level that is applied after the deadline does not help. The measure should include the response expectation for each level and whether it was met.
Test the rule with a request that has a real external commitment and a request that only claims urgency. The contrast shows whether the reviewers are using evidence or language. If both receive the same level, the rule is not discriminating.
Decision use: rewrite the examples, add a conflict rule, or narrow the levels. If the rule is clear but the owner overrides it routinely, the written rule may not reflect the business. Review the overrides with the owner and update the rule rather than blaming the assistant.
Compare the priority level with the queue position. A request marked urgent but placed behind three routine items suggests that the queue is not following the rule. The mismatch is a useful finding even if no deadline was missed.
Record the override rate by owner. A high override rate may mean the rule is wrong, the owner is not using it, or the queue is under pressure. Discuss the pattern with the owner rather than assuming one cause.
Apply this to a research support lane
Use the method to define the evidence, sample, and owner decision before expanding the assistant lane.
Review assistant operationsSources
Frequently asked questions
Does this study report provider performance?
No. It is a method and decision protocol. It does not report a provider's performance, a universal benchmark, or a legal conclusion.
Who owns the final decision?
The accountable business owner or adviser. The assistant may prepare evidence, but consequential decisions remain with the named owner.
When should the protocol be repeated?
After a material change in tools, scope, staffing, policy, or evidence quality, and at the review date recorded in the study packet.