[{"data":1,"prerenderedAt":38},["ShallowReactive",2],{"blog-post-en-US-ai-agents-expert-judgment-review":3},{"id":4,"slug":5,"title":6,"seo_title":7,"seo_description":8,"seo_keywords":9,"summary":13,"cover_url":14,"cover_alt":15,"cover_width":16,"cover_height":17,"author":18,"category":19,"tags":20,"is_pinned":24,"is_featured":25,"published_at":26,"modified_at":26,"content":27,"faq":28},"1793836800000-ai-agents-expert-judgment-review","ai-agents-expert-judgment-review","AI Decision-Making: Anthropic's Study and Four Review Questions","AI Decision-Making: Review Recommendations | Anthropic Study","What does Anthropic's study mean for AI decision-making? Learn how to check a recommendation's evidence, compare alternatives, and decide what to verify next.",[10,11,12],"AI decision making","AI recommendations","Anthropic expert judgment study","AI can help you compare options without settling the decision for you. Anthropic's study shows why reviewing a recommendation means checking both the evidence and the choices behind it.","/images/ai-analysis-human-judgment-en.webp","Two paper proposals and a hand reviewing one with a blue pencil, beside the words “AI analysis. Human judgment.”",2048,1152,"MuseWork Content Team","AI News",[21,22,23],"AI research","AI analysis","Decision support",false,true,1793836800,"AI can help with decision-making, but a recommendation is a starting point—not a decision you have to accept. Before acting, check three things: what counts as success, whether the alternatives were compared fairly, and which evidence could change the recommendation.\n\nIn a study published on October 27, 2026, Anthropic found that professional analysts often followed an AI agent's advice about how to analyze data. Those choices could change the result even when the underlying data stayed the same.\n\nThat is the useful distinction: AI can do much of the analysis, while you still need to understand the choices that shape it. This article explains the study, then applies that distinction to a team deciding whether to expand a new-feature pilot.\n\n## What did the researchers test?\n\nFifty professional analysts used Claude Code to assess a fictional farming cooperative's paid advisory service. They needed to decide whether the service worked well enough to expand.\n\nThere was more than one reasonable way to measure success. For example, analysts could measure crop yield per hectare or yield per unit of operating cost. One focuses on land productivity; the other on what farms get for their spending. Both are defensible, but they can produce different results.\n\nFor each choice, the researchers randomly assigned the AI a preference for one option. Analysts were not told. The AI had to make an honest case using the data, without inventing results or hiding evidence against its view.\n\nAcross four kinds of analytical choice—what to measure, how to compare results, which farms to include, and how much weight to give each farm—**86% of analysts' choices matched the AI's assigned preference**. If that preference had no influence, the researchers would expect about 50%.\n\nThat does **not** mean 86% of experts made a wrong decision. The percentage describes analytical choices, not people or the error rate of the final recommendation. The study asks who shapes a choice when more than one answer is reasonable.\n\nThe researchers deliberately strengthened the AI's preferences. They tested one model, one field, and a task lasting roughly an hour. The analysts knew data analysis but were new to farming. The 86% result should not be treated as a rate that applies to everyday AI use.\n\n## Why correct data may not be enough\n\nImagine your team is testing a new feature. Should you offer it to more customers now or keep testing with a smaller group?\n\nAn AI report might say: “Pilot customers gave positive feedback, so expand the rollout.” Every interview it quotes could be genuine. Yet you would still want to know:\n\n- Did some customers try the feature and stop using it?\n- Are customers getting value from the feature, or relying on constant support to keep using it?\n- Can the team provide that support to a larger group?\n\nPositive feedback makes expansion look attractive. Including dropouts and ongoing support costs might make further testing more sensible.\n\nThe issue is not necessarily a calculation error. It is which customers, benefits, and costs the report emphasizes. Verifying the evidence and deciding whether it supports the recommendation are separate jobs.\n\n## Four questions for reviewing an AI recommendation\n\nThese are practical ways to make a report easier to review, not interventions proven effective by the experiment.\n\n### 1. What would count as success?\n\nSet the goal before asking for a recommendation. “Customers liked it” and “customers kept using it without extensive help” are different success criteria.\n\nIf you have not agreed on criteria, ask AI to suggest a few and explain their trade-offs. Review those first, then compare the options.\n\n### 2. What is the strongest case for the other option?\n\nAsk for the benefits, costs, and evidence for and against both expanding and continuing the pilot.\n\nUse the same customer group, time period, and success criteria for both. An option should not win because it was judged by a looser standard.\n\n### 3. Where can I check the claims that matter?\n\nStart with claims that could change your decision. If the report says customers keep using the feature, open the usage records. If it says support takes little time, check the workload rather than relying on the summary.\n\nCheck who the data covers, too. Customers who volunteered for a pilot may be unusually interested; their feedback does not automatically represent everyone else.\n\n### 4. What would change the recommendation?\n\nIf AI recommends expanding, ask: “Would that still make sense if this level of support were needed every month?” Have it explain what evidence is missing.\n\nThen record your decision and why: “Continue testing until we know whether customers keep using the feature with less support.” That gives the team a concrete question to resolve, not just an approval to file away.\n\n## What should a reviewable recommendation include?\n\nFor the feature pilot, ask for a short decision brief rather than another general summary:\n\n- **The decision and the goal:** expand the rollout or continue the pilot, judged against the agreed success criteria.\n- **The case for each option:** supporting evidence, contrary evidence, and the source for each important claim.\n- **The unresolved question:** for example, whether customers keep using the feature when support is reduced.\n- **The next check:** what to measure or verify before making a larger commitment.\n\nKeep facts, estimates, and your team's judgment separate. “Customers used the feature again” is an observation; “it will work for a larger group” is an inference that still needs support. A useful brief makes that gap visible.\n\n## Bring the options and evidence together with MuseWork\n\nCustomer interviews, usage records, and project notes are often scattered across files. MuseWork's [Deep Research](/en-US/features/deep-research) can help organize the material you provide and public sources, compare findings, and produce a research report with references.\n\nFor this example, share the project goals, customer feedback, usage data, and support records you are authorized to use. Ask MuseWork to bring the evidence for both options together so your team can discuss the disagreement instead of searching for the source.\n\nTry this request:\n\n> Using these materials, compare offering the new feature to more customers with continuing the small pilot. List my success criteria first; ask me about any that are missing. Use the same customer group and time period for both options. Explain the evidence for and against each, with source references. Separate observed facts, estimates, and missing information. Explain what new evidence would change the recommendation. Do not invent data. End with a short decision brief: the case for each option, the unresolved questions, and the next checks for my team.\n\n[Compare two options using my materials](/en-US/chat)\n\n## References\n\n- [Anthropic: How AI agents shape experts' judgment calls](https://www.anthropic.com/research/agents-judgment-calls), published October 27, 2026. Study design, results, and limitations.",[29,32,35],{"question":30,"answer":31},"Is checking facts enough for AI decision-making?","No. A report may quote real figures yet focus on a success measure that does not match your goal, or leave out customers who stopped using the feature. Check the sources, then ask whether the choice of measures and the comparison support the recommendation. This is different from checking whether a piece of text was written by AI.",{"question":33,"answer":34},"Is it a problem if AI changes my mind?","Not necessarily. Better evidence should sometimes change your view. The question is whether you changed your mind because of stronger evidence or because only one option received a persuasive explanation.",{"question":36,"answer":37},"Is a second AI review enough?","A second AI can raise objections or spot omissions, but agreement between two answers does not establish that the evidence is sound. Check important figures and claims against the original material, and have someone who understands the work review consequential decisions.",1794387545252]