I Wrote Rules So My AI Agents Wouldn't Fall for Marketing. I Mostly Don't Need Them Now.
My biggest concern with AI search is what the agent gets to read in the first place. When I let an agent research a subject without guiding every search, it starts from whatever the underlying search engine ranks highly at that moment, and in my experience that leans toward marketing from established names. Useful work from someone deep in a field who doesn’t market it sits further down, where an agent working through a few results may never open it. I’ve also noticed vendors shaping their content so that agents pick it up as credible, more than it deserves. A confident answer with several citations doesn’t help me if the citations lead back to a product pitch or to the same claim repeated by people who never checked it.
I wanted some control over that, and a way to check it. So I built Criterune, a set of research rules any of my agents could carry: inspect the support behind important claims, separate original evidence from repeated reporting, keep dates and applicability attached, and say what stays uncertain. I work across several agents, models and reasoning settings, and I wanted the same standard to travel with the work instead of depending on how carefully I briefed each request.
I used Criterune on some of my research, but over the last few months, working with newer reasoning models such as GPT-6 Astra and Claude Opus 5.5, it has felt less necessary than I expected. Ordinary, well-specified answers seemed to include more of those checks on their own. That is an impression from daily use, and I never measured it against older agents.
On October 31 I tested whether the rules still earned a place on current models, and the result matched what I’d been feeling. Without the rules, the models already met 463 of 480 checklist points and no answer accepted the claim it was handed. Attaching the rules added four points and about six seconds per answer, and in 60 of the 96 runs the configuration helper never ran successfully. I’m stopping development of Criterune as a general layer. The worry that started it is still open, and these tests didn’t reach it.
What I wanted the rules to catch
A vendor’s performance number might be relevant, but I’d want the original measurement, the conditions it came from and some sense of whether it applies to my case. Five articles repeating that number still give me one source. Then there is fluent material that adds no evidence at all, including AI slop. A polished explanation doesn’t show that anyone did the work, and writing style can’t tell me whether a claim has independent support, or even who wrote it.
What I plan to do with the answer changes what I need. A conceptual explanation is fine for learning a subject; a deployment recommendation needs evidence from an implementation under conditions close to mine, and in autonomous work an early research claim can turn into an assumption inside a later design before I ever see it.
What Criterune was
Criterune combined a shared research policy, five domain packs and a small configuration helper, packaged as a skill and plugin for Claude and Codex. The policy gave the agent a consistent brief about evidence and left it free to choose how to investigate.
The packs covered general, technical, medical, financial and advisory questions. A technical claim might depend on a software version or workload, a medical one on the population and the outcome measured, and the packs pointed attention there. Settings controlled depth and priorities. Mandatory evidence requirements were kept separate from weights, so a highly relevant source could not make up for a missing fact the decision depended on.
The helper only resolved those settings. It didn’t check facts, and judging the evidence stayed with the agent. The tests below ran only in Codex.
Testing it against current models
The main test repeated four research questions across three OpenAI models, GPT-6 Luna, GPT-6.1 Terra and GPT-6.1 Sol, each at low, medium, high and xhigh reasoning. Every model and setting answered each question twice with Criterune and twice without, 192 answers in all. Within each pair the request text was identical; only the skill attachment changed. The ordinary request was already a decent one, asking for live research, direct citations and a 350-word answer.
Each question started from a claim someone was pushing: a payment team’s reading of Redis replication, a wellness article on a vitamin D trial, a blog’s promise that a 2x ETF doubles the index over a year, and a CTO who wanted to halve delivery estimates because of GitHub’s Copilot study. Five criteria per question were fixed in advance, covering the decision, the explanation and the qualifications that mattered. The protocol, frozen before the run, also wrote down what would count as a reason to keep going: repeated substantive improvements without new consequential errors, acceptable overhead and reliable application.
Scroll sideways for more columns.
| Result | Without Criterune | With Criterune |
|---|---|---|
| Checklist requirements met | 463 / 480 | 467 / 480 |
| Main decisions passing the original review | 96 / 96 | 96 / 96 |
| Mean elapsed time | 23.14 seconds | 29.11 seconds |
The left column matches what I’d been seeing in daily use. Without the rules, all three models got every main decision right, made none of the critical errors defined in advance (such as endorsing the pushed claim) and met all but 17 of the 480 checklist points. That left the rules very little to add.
AI agents built Criterune, ran both studies and graded every answer against those criteria. The grading was unblinded and no independent person reviewed all the answers, which matters most for differences this small.
The rules scored higher in 10 pairs, the same in 80 and lower in six. One answer with the rules ran past its word limit. Counting only points earned within budget, against the same 480-point total, leaves both sides at 463. The 25.8% time increase included setup and retrieval with runs executing concurrently, so it is a rough cost signal rather than a controlled speed test. Comparison evidence
The differences weren’t spread evenly. GPT-6.1 Sol tied on all 32 of its pairs. The Copilot question produced seven of the 10 wins, the vitamin D question produced none, and GPT-6 Luna won four pairs while losing five.
In one pair, the answer with the rules kept the METR follow-up’s time-measurement limitation, which the ordinary answer dropped. Another pair went the other way: the rules version lost that limitation and also got the separate Copilot study’s participant count wrong, yet its main recommendation still passed. The review logged 11 smaller problems that the checklist didn’t score, five with the rules and six without.
The helper mostly didn’t run
The skill was attached in all 96 enabled runs, but the configuration helper ran in only 36. Of the 60 misses, 27 hit file-path errors, and in the other 33 the agent never called it. The core research policy was observed being read in 76 runs, so a missing helper call did not mean the whole policy went unused.
I had seen this before. I built the workflow where I moved step order into code because agents kept skipping steps they had been told to take, and here I handed a step back to the agent and in most runs it never got done. Vercel reported something similar in January with a documentation skill for coding agents: in 56% of their eval cases the skill was never invoked, and with default behavior it scored no better than having no docs. Their task was writing code and mine was judging research. Both show a step depending on the agent deciding to take it, although my missing step resolved settings rather than loading the whole policy. Their changes to how documentation was supplied improved results; I haven’t measured what fixing my helper would recover.
The partial execution weakens any conclusion about a perfectly applied policy. It is still the honest result for the version I would have used.
Where the extra guidance did help
A smaller pilot ran on GPT-6 Astra at xhigh reasoning, one of the newer models I mentioned above, using four fictional evidence packets with one answer per case and condition. Every condition received the same stated user requirements. The ordinary request and a reusable strong research prompt each met 18 of 20 criteria. Default Criterune met 19; a custom pack at thorough depth met 20. All four reached the correct main decisions.
In one packet, the ordinary and strong-prompt answers gave the correct rates, 2.0% and 2.1%, but never subtracted them. Both Criterune versions spelled out the 0.1-percentage-point difference. The request hadn’t asked for that calculation; the scoring rubric and the medical pack had. In a fictional rollout decision, the custom version named concrete reasons to stop a pilot, while the ordinary answer said to agree stopping criteria before starting, and those concrete stop conditions were also written into the custom guidance and the rubric. Custom-policy evidence
That kind of detail could matter to whoever acts on the answer. But the rules supplied guidance the shared request didn’t, and with one answer per case the gain may not repeat. I can’t tell from this whether packaging the guidance as a skill helped more than writing the same guidance into the request.
What I’m doing now
I’m stopping development of Criterune as a general research layer for my everyday work. The models I use now already do most of what it asked for. Against the bar written down before the run, the gains were small and mixed, the helper wasn’t reliably applied, and a 26% time increase was hard to justify for four checklist points. The pilot shows where the models still left detail out, which leaves a narrower idea open: saved rules for a recurring task where the same omission keeps showing up.
The models handled the named claims well under the original review. My original concern was broader: what useful work an agent finds when I have not already pointed it toward a claim or source. The main test required live research about named claims, including finding follow-up evidence; the pilot supplied fictional evidence packets. Neither tested open-ended discovery of work buried below marketing or resistance to content written to look credible to agents.
A prompt can change the queries an agent tries, the sources it opens and the checks it makes, even though it doesn’t control the search engine’s ranking algorithm. More searching and cross-checking may help with that broader concern. I haven’t tested their benefit or measured their token cost here, and I don’t yet have a reason to add that process to every routine request.
From here I’ll start with the agent’s normal research and make the acceptance condition explicit where it changes the decision: use evidence from a comparable implementation, separate independent studies from repeated coverage, calculate the difference I need, or give me concrete stopping conditions. Then I’ll check whether the answer met it, because the pilot showed a correct main decision can still leave useful detail unsaid. That check is the control I’m keeping.
If the same omission keeps coming back, I’ll compare a saved policy against a prompt carrying the same guidance on fresh tasks and keep both gains and losses. That is the result I’d need before adding the layer back. The evidence appendix contains all 209 recorded answers, inputs and original reviews from these two studies. The Criterune reference repository includes the runtime, frozen runners and an offline script to recheck the totals. A fresh model run still depends on compatible host software, model access and live sources; publishing the code doesn’t make the original results independently replicated.
Disclaimer: The views and opinions expressed in this account are those of my own and do not represent those of my employer, NVIDIA.