What the bleep is a Web Crawler?

Before we begin, let’s agree on one thing. SEO stands for Search Engine Optimization. Not Seals, Eels, and Octopi. Say it with me. SEARCH…ENGINE…OPTIMIZATION…Good. Let’s move on.

First things first…as they should be…

In order to better understand what a web crawler is, we need to at least have a basic grasp of how Search Engines like Google, Bing, and Yahoo work. The entire point of these sites is to get you EXACTLY what you’re searching for. To do this, they scour the internet for content, and create an index of everything that currently exists.

Google is, in a somewhat arbitrary way, the ultimate Encyclopedia Britannica.

We say arbitrary because one of the unfortunate differences between Google and an actual Encyclopedia is that, in the case of the internet, “knowledge” isn’t so much vetted as allowed.

Flat earth theories are given as much weight as Einstein’s theory of relativity. Mermaid videos and their associated back channels woo the gullible. And ghost “documentation” abounds.

The world wide web is a place where mediocrity lives hand-in-hand with sophistication. Some fruitcake in their basement can rather suddenly write blog posts about a race of lab rats that have begun to self organize, and Google will map it out just as dutifully as it does the Library of Congress.

Speaking of…where are our editors? We’re rambling again, aren’t we?

IN ANY CASE…ahem…this process of dutifully mapping out online content is often referred to as Web Crawling. Some people will refer to web crawling as the use of spiders.

By this, of course, we mean there are actual spiders crawling through fiber-optic cables the world over, spying on you to see what you’re doing. Spiders are intelligent, AND they have many eyes, so they see everything. They quite literally put the Optic into Fiber-Optic. They also have many legs, so they’re fast.

After watching you fumble between pornography and cute cat videos (while you simultaneously try to buy a fidget spinner on your iPhone), they report back to a motherspider at Google headquarters. It is her task to assimilate all of the information into a megalithic spindle of thread we rather naively refer to as the World Wide Web.

Seriously…where are our copy editors? Bueller? Bueller? … Bueller?

The Truth: Web Crawling is completed by spider bots, which we might as well call internet bots. They aren’t actual spiders. They’re just bits of software that browse the web on your behalf to give you whatever your heart desires (regardless of whether or not it’s good for you). So yeah, the reality of web crawlers isn’t as exciting as actual spiders.

But reality is what we ultimately aim to serve. Hopefully in a lighter, fluffier, and far more tastier form than you’re used to. Like pancakes…

That’s it. We aim to serve you delicious Truth Pancakes…

But back to those mindless bots. They are the very entities that retrieve information across the internet, from recipes to news sites. And all for the purposes of Web Indexing (sometimes referred to as Web Spidering).

Which brings us to INDEXATION.

Want a definition? We thought you’d never ask.

INDEXATION: The process whereby search engines such as Google, Bing, and Yahoo gather and rate content on the internet according to the keywords users are searching for.

This information is then compiled on a Search Engine Results Page (techies will refer to this as the SERP because we rather enjoy that fleeting notion we might just be superior to everyone else…which is…well…you know…what’s the word for it…sad).

So what have we learned?

Spider bots (aka internet bots, web crawlers, SkyNet) represent the AI that searches through sites to index links and content. The result of this indexation is the results page (SERP) you interface with when you search for anything on Google, Bing, or Yahoo. If you’ve ever wondered why search result pages sometimes surf up broken links from websites, it’s because the last time webcrawlers indexed the websites in question, those links likely weren’t broken. The SERP is more or less static information (versus some sort of live feed).

But…Why Do Web Crawlers Matter To You?

Excellent question!

These elements matter to YOU, because they are specifically what affects the user experience on YOUR site. If the ad copy on a page is completely random* (*see any of our blog posts), or there are technical issues which prevent web crawlers from properly scanning your site, it will either be considered not relevant to merit top ranking on anyone’s search or completely invisible.

And we aim to be relevant…and visible…right? Right! Alrighty then.

Let’s just agree that making our sites as searchable as possible is of the highest priority when it comes to the online success of any business.

So… Wondering how to make your site more searchable?

Good! We want to keep you hungry! Stay tuned for next week’s post on easy wins for making your site more indexable!

That’s All For Now Folks…Let Us Know If We Can Help!

First Name (required)

Last Name (required)

Email (required)

Message