Let's talk about spiders for a second.
You know, the same technology that the search engines use to index your
pages. When's the last time you ran one on your own site? Did you know
that your custom "404 not found" error page could be driving them away?
You'll want to sic a spider on your own site, before letting MSN,
Google or Yahoo get at it. Why? To make sure its spiderable, that's
why! Huh??? Didn't think about validating your
linking structure, did you?
There are many page whackers available that download a whole site to
your hard drive, but who knows what kind of "forgiving" spidering
technology they're using. The best one, the one that replicates the
crawling spiders like Google and Yahoo the closest, is a tool like
OptiSpider.
Here's a tip, if OptiSpider can't spider your site... neither can
Google. That's right, it's probably just about the best reason to get
and use a tool like OptiSpider, just so you can see the same stuff the
spiders see.
If you manage multiple sites for clients, it's a great weapon in your
toolbox! I can't count how many times I've done coaching sessions for
clients who claim search engines are not finding all their pages. A few
seconds later - using OptiSpider - I can tell them... dude!!! You
didn't link to the pages properly, or your custom 404 not found page is
killing the search engine spiders.
No Google, Yahoo or MSN spider is going to find all your pages, when
you put the crawler in an endless feedback loop. You're lucky if the
spiders ever come back.
Stop doing whatever it is (even if it's reading this article ;-) and go
look at your custom 404 page. Everyone loves to use them.
Unfortunately, many people use them badly. The biggest sin of all -
even I was guilty of this - is using a relative link on the 404 page.
It's so common that relative linking should be banned altogether.
Imagine the spider is several directories down in your site and
encounters a missing page or broken link, up comes your custom 404
error page. Too bad sucker! You used a relative link instead of an
absolute one. The spider is totally miffed at your site and resets,
leaving all the rest of your pages behind... unspidered, unindexed and
unfound.
Stop whatever it is you're doing and go do it now. Put an absolute link
on your 404 page, one that includes the http://www. and your full .com
web address to your home page. That way, when your custom 404 error
page comes up, the spider won't choke. Instead, it finds your home page
from the custom 404 page and continues spidering like normal.
If you want to make spiders happy and be sure they can crawl your
entire site, then use a spidering tool like
OptiSpider to test your internal
linking structure. Just don't be surprised if you end up getting a
whole lot more pages indexed by the search engines. What's on your 404
page?
by Michael Campbell
Author
of....
Revenge of the Mininet...
Advanced search engine linking
strategies and
diagrams for
increased revenue.
Clickin' it Rich... The
complete work from home business
training system for new affiliates.
Nothing but 'Net... Simple internet marketing
strategy that made
$750,000 in less than a year.
Internet Marketing Secrets Newsletter...
Learn
how to harness
the money making power of multiple internet revenue streams like search
engines, affiliate programs, paid advertising, opt-in email,
newsletters, ebooks and lot more in internet
marketing secrets. Subscribe now and get a free lifetime
membership.