Nine domains added named AI-crawler blocks. All eleven groups were identical to the default
A lot of the AI-crawler advice circulating right now tells you to add a named User-agent: block for each AI crawler in your robots.txt, one for GPTBot, one for ClaudeBot, one for each of the rest. We tested it, and on the sites we checked, it does nothing for the crawlers it names, and the naive version of it actively opens access it was meant to close.
Nine domains in the set we swept had added named blocks. Between them they carried eleven named User-agent: groups. Every one of the eleven was byte-identical to that site's * group, the default rule that already applies to everything. Under how robots.txt matching actually works, a request from a named agent is evaluated only against its own named group, so an identical copy changes nothing for that crawler. It's not a stronger rule. It's the same rule, typed out twice.
That would be a harmless waste of a few lines if it stopped there. It doesn't. Robots.txt matching is specific-group-wins, not additive. The moment you add a named group for an agent, that agent stops reading * entirely. If the named group is a partial copy, if it's missing the Disallow: /api/, /admin/ or /_next/ lines the * block carries, that agent now has access to paths the site owner thought were closed to everyone. The naive fix, applied without checking the named group is a complete copy, silently lifts restrictions for exactly the best-behaved crawlers, the ones that actually honour robots.txt in the first place.
So the advice isn't wrong in principle. Named groups can do useful things, different rules for different agents is a real capability. It's wrong as a copy-paste instruction, because the instruction that's circulating doesn't say "and make sure the named group is a complete superset of your default rule, not a partial one." If you've already added named crawler blocks, the check is quick: diff each named group against your * group and confirm nothing's missing, not just that something's there.