Every so often a directory like mine stops adding tools and starts shipping broken links instead. I run one: about 4,323 AI tool listings. I had been assuming "the crawler refreshes the descriptions" was close enough to "the links still work". So I spent a morning checking whether the stored URLs actually load. The method was simple, and it collapsed on the first control group.
What I ran
Sample: 300 of 4,323 listings (6.9%) — the top 150 by the traffic number I display, plus 150 random from the rest. One machine, one pass, Node fetch, concurrency 16, follow redirects, 15-second timeout.
outcome
count
2xx / 3xx
193
4xx
21
5xx
2
no TCP or TLS answer at all
84
First read of that table: "28% of my catalog is dead." Wrong, and I can prove it's wrong.
The control group that killed the method
Two of the 84 were notion.so and writesonic.com. Both are undeniably alive companies. They timed out from my machine exactly like the other 82 did.
So that b
Discussion
Start the conversation
Your voice can be the first to spark an engaging conversation.