“Bots are costing me $50 a day”. What scrapers taught me about being visible.

Bots crossed half of all web traffic in 2025. By June 2026, Cloudflare Radar put the figure at 57.5%, a threshold its own CEO said arrived years ahead of what anyone had forecast. On TollBit’s network, AI bot traffic went from 1 in 200 web visits at the start of 2025 to 1 in 31 by the end of it. Roughly 80% of that AI crawling, per Cloudflare’s own data, is model training. Not a user. Not a click that ever sends anyone back to the page it scraped.

Numbers like that read as an abstraction until they show up in your own invoice.

What the trend looks like from inside a one-person product

JobMentis had 0 traffic in January. Realistically, by August it was doing something like 500 visits a day, built almost entirely on owned content: close to 100,000 interview questions, plus directories of career coaches and recruiting agencies, some of it published in 6 languages. That is the AI-first, solo-founder version of growth. No ad budget, no sales team, a lot of content shipped fast because the tools finally let a single founder do it.

The strategy worked, and it also made the site exactly the kind of target a crawler is built to find: a large, structured, constantly expanding set of pages, most of them cheap enough to serve that nobody had reason to think twice about how they were built.

“Bots are costing me $50 a day.”

I wrote that in public, mid-incident, because I did not have a clean answer yet and was not going to pretend I did. The traffic looked like the obvious villain: 5 IPs concentrated in Singapore, hammering my sitemap and a long tail of malformed URLs, straight to the address, no browsing behaviour at all.

That was all true. It was also not the whole story. When I finally sat down and read the incident properly instead of reacting to it, the worst day on record turned out to be about 100 times a normal one, and roughly 2/3 of that number was mine to own.

A cheap site stopped being cheap

A page assembled fresh for a visitor costs roughly 14 times what the same page costs served from cache. At normal traffic, that gap is invisible.

In the 2 months before the incident, on top of the climb from 0, JobMentis grew again, roughly 3 times its size, as I shipped new content libraries fast. Some of those new pages went out with no caching instruction at all, which is the one setting where doing nothing produces the most expensive possible behaviour, silently, by default. Nothing broke. Nothing warned me. The cost just sat there, small enough to hide in a bill I wasn’t reading closely.

The day everything came due

2 things landed on the same day. A large release changed tens of thousands of page addresses at once, which discarded the entire cache in a single stroke. And the crawler traffic I’d already flagged as suspicious arrived at scale while every page on the site was cold.

The 19 million edge requests I mentioned publicly were the multi-day total. The worst single hour inside that window ran at 189,132 edge requests, rebuilding pages from scratch against a database that was never sized for that. That single day cost about what 100 normal days cost.

I do not think the crawlers meant to find that seam. I think they hit the sitemap the way crawlers always do, and the sitemap happened to be pointing at a site that had just lost its cache.

3 ways to fail without an error

None of the 3 causes threw an error, which is exactly why none of them got caught earlier. New pages had no required caching decision, so skipping it was free until it wasn’t. The cache-invalidating release shipped as a single step instead of a staged rollout, so the whole catalogue went cold at once instead of in batches. And the least comfortable one: some pages that did request caching were being silently downgraded by a single unrelated line elsewhere in how they were assembled. The code read correctly. The build did not do what the code claimed.

What actually brought it down

I removed the automated load at the edge, carefully enough to leave real search and AI crawlers untouched so search visibility didn’t take the hit too. I moved the most expensive reads off the live database and baked that content into the release instead of fetching it per visitor. I narrowed what each query returned so pages stopped pulling data they never displayed. Then I made the build enforce the caching rules instead of trusting myself to remember them, so the failure mode now breaks a build instead of quietly costing money.

Requests missing cache went from 63% to 26%. Database queries per hour fell 56%. Rows returned per hour fell 78%. It took roughly 3 days of actual engineering attention, spread thin across 2 weeks of a mostly one-person, AI-assisted schedule.

A caveat, because the honest version matters more than the tidy one: the old baseline isn’t really the right target anymore. Zero to 500 a day is not a stable line, it’s a curve, and some of the new steady state is legitimate growth, not leftover leak. I’d rather admit that than claim a cleaner win than I actually got.

Where this leaves a solo founder

None of this excuses the crawlers, and the wider pattern behind them is real, the same 57.5% and the same 1-in-31 I opened with. Kinsta’s CTO Daniel Pataki summed up the dilemma in a single line: “block everything or leave it alone. Neither works at scale.” Block too hard and you lose the search visibility that got you the traffic in the first place. Leave it alone and you’re funding an invoice for a customer you will never have.

The bots did not create the seam I fell into. They just found it faster than I would have, because finding seams is what a crawler is for. Publish anything worth scraping while you’re growing that fast on your own, and the gap in how you built it is more likely to exist than not, on a timeline you don’t control.

Silence is the failure mode worth engineering against. 3 separate defects here produced no error, no warning, and no failed build, and 2 of them were fixes I’d already written that were quietly doing nothing. More care would not have caught any of them. Only a check that turns silence into a hard failure would have.