Accidental SEO and how I got 400,000 page views without even knowing it

Accidental SEO and how I got 400,000 page views without even knowing it

Until quite recently I knew about as little as one could about SEO. If anything, the term gave me the creeps — I associated it with link farms and domain squatters and those old pop-up ads that asked you to shoot at monkeys.
As I started to read more about it, both on the web (e.g., Ray’s excellent SEO Strategy That Works guide) and in books (like O’Reilly’s SEO Warrior), the key takeaway at every turn seemed to be that “content is king,” the upshot of every chapter to make your site as readable, usable, and just plain interesting as you could.
That imperative resonated with me, first because it’s undeniably pure and noble, but also because it helped explain two SEO coups of mine, both total accidents, one a bit more of an accident—and a much bigger accident—than the other.

It turns out I sort of know what I’m doing

About a year ago I wrote a post about the phrase “it turns out,” in which I tried to explain how lazy writers could exploit it, consciously or not, to give their speculations the sheen of discovered truth. Within a few weeks I was the number one Google search result for the phrase (with or without quotes), and I’ve held that position ever since. The question is, how’d a nitwit like me make that happen? (Keeping in mind that ranking highly for “it turns out” is worth exactly zero dollars.)
The first thing I did right was to use Paul Graham’s writing as an example. It’s probably the best SEO tip you won’t find in the literature: if you want to rank highly for some keyword, try to connect it in some way to a critique of Paul Graham. You’ll shoot right to the top of Hacker News.
What I did most right, though, was to make my post indisputably about that single phrase. The title of the post was “It turns out.” The page title was “It turns out.” The WordPress slug (which becomes the post’s URL) was “it-turns-out”. The Hacker News submission title was “It turns out.” And, of course, between the text and the comments the phrase “it turns out” appeared 55 times on the page.
Which meant that every time a Twitter link went out linking to the post, or a pingback came in, it invariably included the anchor text “it turns out.” And since Google probably uses some sort of vectorial semantics to improve its rankings, and with the post itself so full of the title phrase, it was only natural for Google to treat it as the authority on the subject.
It helped a great deal that the post got picked up by the excellent Language Log and later by the economics blog Marginal Revolution, two sites with a lot of Google juice. But that only happened because I wrote Geoff Pullum and Tyler Cowen e-mails asking them to read it.
Point being, I had accidentally done what most SEO guides ultimately tell you to do, which is to write something people like, target it directly at one or two keywords, and engage a suitable audience.

Surely You’re Joking, AwStats!

But that’s nothing compared to what I’ll call “The Feynman Incident.”
On my personal website I used to somewhat sheepishly store a collection of books that are not exactly in the public domain. That way no matter where I was I could always access, say, Douglas Hofstadter’s incomparable masterpiece, Godel, Escher, Bach: An Eternal Golden Braid. Now for all intents and purposes they were for my personal use only—I used a password-protected directory and wrote a “robots.txt” directive that hid it from search engines.
It so happened that in late April of last year I stumbled upon the complete three-volume set of the Feynman Lectures on Physics in PDF form. Thinking that I might eventually want to read these on the road, too, I uploaded them to my private /books directory. Or that’s what I thought I did.
What really happened was that I put them in my not-quite-as-private /papers directory, which at that point was completely free to be indexed by search engines.
I would have had no idea of the impact of their crawls if I hadn’t had the narcissistic reflex to check the stats on my site, a site which around that time was getting on the order of ~200 pageviews per day. Here’s what I saw when I logged into AwStats:

Holy crap! Had I become an overnight Internet demicelebrity without even knowing it?
Nope: a moment or two later I scrolled down the page to discover that 99.9% of that traffic was going into the “/papers” directory. I had a hunch about what was happening, but just to confirm, I Googled “feynman lectures pdf” and saw that my site was the #1 hit. Oops!
(Luckily I was using Site5’s “$5 per month for 5TB of transfer” plan, because otherwise I probably would have had to dole out a whole lot of cash.)
I also did a bunch of searches for obscure physics keywords, and the scope of Feynman’s lectures being what it was, these too linked to jsomers.net. I had accidentally become the global authority on everything that Feynman taught.
The lesson there being that Feynman’s content is so good that its mere un-linked-to presence (as PDFs, no less) on some kid’s obscure homepage is enough for it to pretty much instantly make its way to the top of Google’s search results.