← Back to Blog

Most of My Traffic Was Bots

Decide what a human is before you celebrate a number.

I built flydirectfrom.com. You pick an airport and it shows you where you can fly nonstop from there. It is public, it is free, and for a while I was quietly pleased with how many people seemed to be using it.

Then I measured it properly and found out most of those people were not people.

This post is about that gap: what the first numbers said, what was actually behind them, and the decisions I made once I stopped trusting the first chart I saw. None of it is exotic. I would have caught all of it on someone else's dashboard and missed it on my own.

The number I wanted to believe

The first traffic number I had came from the edge, the layer that sits in front of the site and sees every request. After filtering out the obvious junk (known bot signatures, my own health checks, one scraper hammering the site from a single region), the edge still reported what looked like a healthy stream of human visitors. Mostly the homepage, nothing from search yet, but enough to feel like a signal for something with no launch and no marketing behind it.

The problem with that number is in the word "human". At the edge, a human is mostly a request that carries a browser user agent and does not match a known bot signature. That is a low bar. Anything that bothers to pretend to be Chrome clears it, and pretending to be Chrome costs one line of configuration in any scraping library.

I have seen teams make real decisions on that bar. A common version: a team celebrates a traffic jump after a redesign, and only later discovers that a partner's monitoring service had started polling the new pages around the clock. The redesign had done nothing. The chart had done a lot.

Counting people who run JavaScript

So I added a second measurement: a lightweight, privacy-friendly analytics script on the page. No login, no cookies, no reason to follow anyone around. I only wanted to know whether a person loaded a page and did something on it.

Shortly after it went live, it had seen roughly one human running JavaScript.

That was the moment the edge number stopped meaning anything to me. If the browser-agent "humans" were real, a good share of them would have executed the page and fired an event. They did not. Most of what the edge counted as browsers were bots that fetched the HTML and left.

Once the picture settled, it was steadier and less flattering: a handful of humans for every hundred hits the edge reported. Not the audience I thought I had. A small one.

The bursts

Even after switching sources, the charts were not clean. Every so often a single day would spike far above the baseline. That looks like a launch, or a link going around somewhere. It was neither.

When I pulled those sessions apart they had a clear shape. Old Chrome versions, long after normal browsers had auto-updated past them. Traffic routed through residential proxies, so the addresses looked like home connections. A cluster of countries and browser timezones that had nothing to do with the audience the site is built for. My reading is a scraper wearing a reasonable disguise, rotating through real-looking addresses so that per-IP rate limits never trip.

I did not try to block it. Blocking scrapers is an arms race a small public site does not win, and the content is public anyway. Instead I flag those sessions as suspected bots and exclude the flag from every chart. Every chart, not most of them. The one dashboard tile you forget to filter is the one you will screenshot and get excited about.

There is a tradeoff here worth naming. A bot filter is a heuristic, and a heuristic will eventually catch a real person on an old corporate laptop behind a proxy. I accept that. Undercounting a few real visitors is a smaller error than building plans on thousands of fake ones, and the direction of the error matters more than its size when you are deciding where to spend time.

Analytics tools also lie in the other direction

Here is the part that surprised me more than the bots.

Before trusting the setup, I wanted a smoke test: a headless browser that loads a page and confirms the event shows up. The events never arrived. I spent longer than I would like to admit assuming my code was broken.

It was not. Many analytics libraries quietly drop traffic they think is a bot, and a headless browser announcing itself through the webdriver flag is exactly what they look for. The test only registered once I ran it in a real browser. That is reasonable behaviour from the library. It is also invisible, which means automated checks can tell you tracking is broken when it is fine, and you will never see the filter that decided so.

The second surprise went the opposite way. A real visitor on a phone showed up in the data as a single page-leave event. No pageview, no clicks, just the exit. Libraries often batch events to send them together, and on a mobile browser that closes a tab abruptly, the batch holding the pageview can simply never make it out. Only the final event survived. The fix is boring: send the events you care about immediately. On a low-traffic site the extra requests cost nothing, and losing one of the few real sessions you get costs a lot.

Between those two, the lesson is that your analytics tool has its own definition of a human, and it is not the same as your edge provider's. Neither of them is the same as yours until you write yours down.

Search Console was the honest one

The most useful signal turned out to be the one I looked at least: Google Search Console. It does not count visits. It counts how often the site appeared in search results, where it ranked, and what people typed. Scrapers do not generate impressions, because impressions happen on Google's side of the wall.

The absolute numbers were small, but the trend was real and it pointed the right way: impressions climbing steadily, clicks following slowly behind, average position improving.

The pattern inside those numbers was more interesting than the totals. Pages for small regional airports ranked well, near the top of the first page or just below it. Pages for big hubs sat several pages deep. That makes sense once you see it. Nobody is going to outrank the major travel sites for nonstop flights from a giant hub. For a regional airport, a page that just answers the question cleanly has a chance.

Even Search Console needed a filter. Some queries were exact, quoted strings of a distance in kilometres. Nobody types that. Those were scrapers too, probing search results for pages that contain a specific value. So the honest source still had noise in it. It was just noise that was easier to spot.

Cut thin pages instead of adding more

The site generates a page per airport and per country, thousands of them, and adding more is cheap. That is exactly the temptation. When a site underperforms, the reflex is to make more of it.

But Search Console had already told me how Google saw those pages. The coverage report showed most of them discovered and not indexed. Google was looking at the bulk of the site and deciding not to bother. Adding more of the same would only have made that ratio worse.

So I went the other way. A page now stays indexable only if it has enough real content to justify existing. That rule cut the sitemap by about 40%. Thin pages stay reachable for anyone who clicks through, but they carry noindex, follow, so search engines can pass through them without counting them against the site. Later I tightened the rule again, so that the indexed set is pages with fresh data plus pages that have already earned impressions on their own.

The same move works on a bloated product. Volume is the cheapest thing to produce and the least convincing thing to show.

Data honesty is the same question

Some of the route data on the site is older archive data, and only part of it comes from fresh, observed schedules. The archive is useful for exploration, and the site labels it as archive data. But I decided never to write an article about an airport from archive data alone. An article implies someone checked. If nobody did, it should not exist.

The same question came back from a different direction, which I found almost funny. Fresh flight data includes private jet and cargo legs, which inflates the destination count on some pages. "How many nonstop destinations does this airport have" has the same trap as "how many humans visited": you have to decide what counts before the number means anything.

What I'd tell you to do

Decide what counts as a human before you look at the first number. Write it down. Mine is now: a session that runs JavaScript, fires at least one real event, and is not flagged as a suspected bot. Your definition will be different. The point is to have one before the chart has a chance to flatter you.

Then check it from both sides. Find out what your edge provider counts as a browser, and find out what your analytics library silently drops. Run one manual test from a real phone and one from a real desktop browser, and confirm each shows up as a complete session. If a real visit can arrive as a single orphan event, your funnel numbers are fiction no matter how clean the bot filter is.

Pick one source that bots cannot easily inflate and treat it as ground truth. For a content site that is search impressions. Rankings by page type will tell you more about where your site can win than any traffic total.

When the signals say your content is thin, believe them before you believe your page count. Cutting what you ask search engines to index is a faster fix than writing more of what they have already declined.

And when your data is old, say so on the page. Readers forgive old data that is labelled. They do not forgive an article that pretends someone checked.

The uncomfortable part is that the honest number is much smaller than the one I started with. A handful of people for every hundred hits is the real audience. I would rather build on that than on a crowd of scrapers pretending to be Chrome.