Home > Tips > Search Engine Tips > Search Engine Friendly Web Design
Search Engine Friendly Design
by Tim Mayer of Yahoo!
with commentary by Jerry
Updated: July 30, 2011
of all, you should understand the basic
differences between a "spider" and an
Finds web pages, follows links, and the
page content does not matter, it will be indexed later. The spider won't
crawl password-protected sites, it usually
avoids dynamic content due to "spider
traps", and respects robots.txt and
Meta Tag exclusions.
Primary job is to evaluate and score
content (text, meta tag info, anchor
text of incoming links, etc.). Removes
duplicates, aliases and filters spam.
When the indexer detects spam, it adds
the domain to its "do not crawl list".
Flash, etc. make sure you use text links
in the footer of your page for the navigation to ensure crawlability.
Spiders love hierarchies and users prefer
a search box. Have both. Always, always,
always link back to your home page.
The reason? So the spider can find the
rest of your site and lost visitors
can re-orient themselves easily.
Don't require cookies or session IDs to browse your site.
Doing so can harm your site's ability
to be indexed. Why? Since the spider
does not accept either, your content
management system may start to feed
the spider the same page over and over
again with different session IDs. This
causes a "spider trap" and an endless
loop, which eventually
times out and the spider backs out. A strong warning
is if one engine has a page saturation
that is out of proportion to the others.
That is usually a sign that a "spider trap"
has occurred. You risk having your site
dropped by that search engine within
the next 90 days due to "indexing bloat" if it isn't corrected.
use a 404-error page, use only one generic
page designed for that purpose, but
do not, for any reason, redirect the
404 to your home page. Ensure that the
title of this page is named "Error 404"
this will alert the spider and it will
move on without indexing the error page.
have moved your site or moved content,
use a permanent 301 redirect to point to the
new content/domain. This will inform
the spider and it will make note and
crawl the site properly on its next
visit. Doing so also has benefits, as
it will transfer all of the link credit
from the old site to the new one. User
bookmarks still function, as well as
all old links. 301 redirects can be left for
long periods of time that is why they are known as "permanent redirects".
Excessive URL Depth
Having a deep site structure decreases the chance
that the spider will find all of your
pages. Very deep URLs tend not to rank
would be classified as Depth One. It
is NOT suggested here to have all pages
on the root of the domain. Having depth
of two or three levels would be acceptable.
is six levels and probably would NOT
Static URLs get crawled, and dynamic
pages that have incoming links from
static pages will get crawled. However,
links between dynamic pages are often
problematic and sometimes do not get
crawled. Limit the "URL Depth" when
using a dynamic-to-static internal linking
It is vital that your site has unique
content. The titles that you use should
be page specific, meaning that the title
of each page should be unique to the
content on that page. This is also true
with your Meta Tags, specifically with
the description and keywords. If you
update the page's content in the future,
review your title, description and keyword
tags to ensure they are still relevant
to the content. Only separate pages
when there is separate content. Yahoo!
would rather see one long page than
five small pages.
should avoid "spam" at all costs. This
includes using "doorway pages" and "doorway
domains". Keyword stuffing is another
area that should be avoided. Hidden
text (text that is the same color as
the background), hidden links, and even
deceptive CSS can be detected by the
indexer and viewed as spam. Link Farms,
massive domain interlinking, which includes
off-topic links (which tend to dilute
valuable links), and cloaking are also
areas to avoid.
"Spam" to Yahoo!
To report Spam send an email with as
much information as possible (the keywords
used in the search, offending URL, and
why it is considered spam) to: firstname.lastname@example.org.
Yahoo! Content Guidelines
It would be highly recommended to keep
up-to-date on the content
guidelines from Yahoo! at least
once per quarter.
Yahoo! does not currently crack open
Flash files to either follow links or
extract textual elements. Even with
the SDK from Macromedia the extracting
text from SWF files provided little,
if any, value. It was found that content
providers weren't optimizing the content
for the search engines.
You can use programs like Anawave's
WebSnake to mimic a "crawl" through
your website to determine if a spider
could deep crawl your site.
Search Phrases: Yahoo! Search Engine Friendly Web Design
© 2005-2011, Web
Jerry West is the Director of Internet
Marketing. He has been consulting on
the web since 1996 and has assisted
hundreds of companies gain an upper-hand
over their competition. Visit Web
Marketing Now for the latest in
marketing tips that are tested and proven.
The above article
can be reproduced on your site or e-zine
as long as the signature file remains.