Indexing Issues that Keep your Company from Millions of $$
Saturday, January 19th, 2008I recently consulted with an executive from a lead generation company here in San Diego about a problem they were having with their site not being properly indexed into the search engines. The site only had about 35 pages of their several hundred article and tip pages indexed for organic search. They even had pretty good internal site linking, from what he was telling me. This is what I found after reviewing and FTPing into their site:
- Because they recently switched domain names (within the last 6 months), they had successfully 301 redrected their old domain to their new one but were not 301 redirecting their domain.com to their www.domain.com, creating two indexed domains and potentially tripping duplicate content filters.
- The site was using Google’s Sitemaps application. Unfortunatley, the sitemaps.xml file being served up on their domain had an XML parsing error caused by some rogue “&” being placed into some of the URL strings listed in the XML file. The error was preventing the search engines from reading the file and indexing the corresponding pages! But here is the kicker: becuase the sitemaps.xml file was returning an error, Google decided not to index the rest of the site - even though running through a simple Xenu link check revealed good internal site linking to several hundred pages on the site. (as pictured below)

In fact, there were about 35 properly formed XML page URL’s listed in the XML sitemap (about the same as listed on Google’s site: command) before the first “&” character was found in the file, stopping the indexing process dead in it’s tracks! This just goes to show that Google’s sitemap application supercedes a site’s internal linking structure and that you need to be very careful when dealing with sitemaps for indexing purposes.






