Archived Articles

	Sorting Out SiteMatch!
	Barry Lloyd | 24 April 04
	1. Sorting Out SiteMatch. A Review of Our Experiences with the New Yahoo Search and Overture PFI Program! i) Introduction: At the beginning of March, Yahoo rolled out a new and somewhat controversial form of paid inclusion called SiteMatch. They also introduced a new Trusted Feed program called SiteMatch XChange™ to replace the existing Inktomi, AltaVista and AllTheWeb feed programs, combining them into one. In this article, we will be looking at how the standard SiteMatch program works, the impact (if any) it has on results and other issues that may have caused concerns with the new Yahoo! Search Engine. ii) History: Yahoo has been acquiring search technology aggressively over the past 2 years. Starting with their acquisition of Inktomi at the end of 2002, the search engine industry was shocked when Yahoo announced their purchase of Overture, the leading PPC provider, in the middle of 2003. Overture had also been busy buying search engines, having purchased both AltaVista and AllTheWeb earlier in the same year. By purchasing Overture, Yahoo gained control of three major crawling search engines, along with their various technologies, as well as the leader in pay-per-click search. It was obvious that Yahoo were gearing themselves for a major re-launch into the lucrative field of search. In February 2004, Yahoo dropped showing Google results on their searches for web pages (the default search) and replaced them with results from their own database. This appeared to be very similar to the old Inktomi data but with additional results which seemed to come from fresher crawls undertaken by both AllTheWeb and AltaVista. However, ranking criteria appeared to be a little different from the old Inktomi formula - but all-in-all, people who had done well with Inktomi tended to do well in the new Yahoo results. Were we seeing a revamped Inktomi? According to Yahoo, no - this was an entirely new search technology created by combining the best elements of their recently acquired crawlers. In fact, Inktomi - as a brand, came to an end. Yahoo Search, along with their crawler Slurp (hmmm - seem to remember that from Inktomi) replaced the Inktomi crawler. iii) The Launch of SiteMatch: At the beginning of March, Yahoo announced that they would be replacing the 3 paid inclusion programs for the 3 crawling engines they had purchased by a single paid inclusion program, SiteMatch. Administered by Overture, SiteMatch could be purchased via Overture directly or through a selection of partners including the majority of those who previously provided paid inclusion programs to the replaced programs. In the past, each of the engines had different prices per URL and submitting a page for a year to all 3 engines could cost around $115 per page for guaranteed inclusion. SiteMatch offered an inclusion price of $49.00 for the first page, $29 for the pages 2-10 and $10 for all subsequent pages of the same domain. On the face of it, this appeared initially to be a substantial discount for sites who wanted the fast inclusion and 48 hour refresh provided by paid spidering. Instead of an initial outlay of $1,115 for a 10 page site, you paid $310! This would get you inclusion in all the search portals previously covered by AltaVista, AllTheWeb and Inktomi - with the addition of Yahoo! To good to be true? Well, yes - there was a catch! Depending on the category your website/pages fell into, you would also have to pay an additional 15 cents or 30 cents per click! Webmasters world-wide were shocked. Pages receiving hundreds of referrals per day could cost hundreds of dollars per week instead of the fixed annual fee. To make things worse, Yahoo announced that existing Inktomi paid inclusion listings which had started to show on Yahoo since they dropped Google would no longer be included in Yahoo results after 15th April although they would continue to appear on other Yahoo partner site (such as MSN and HotBot) for the duration of the unused part of the Inktomi subscription (as long as these partners used Yahoo results). Similarly AltaVista PFI would continue to show on AltaVista and AllTheWeb PFI would continue to show on AllTheWeb until these PFI subscriptions expired. If webmasters wished for their PFI pages to continue to be included in Yahoo, it was suggested they upgrade to SiteMatch! People were not happy, but Yahoo started rolling people out to answer some pretty hostile questioning primarily on WebmasterWorld and JimWorld's Search Engine Forums. iv) The Main Questions and Answers from our own Observations: Q: Is SiteMatch the only way to get indexed by Yahoo? A: Yahoo state that 99% of all Yahoo's listings are derived from freely crawling the web. This is done by following links, although, to assist webmasters, Yahoo have now introduced a free submission location. We have tested crawl speeds by putting a new site into the Yahoo directory in March to see how long it would take to get a site crawled from this single link. The root directory was crawled within 2 weeks and we were getting referrals from pages on this site from MSN and Yahoo by week 5. We also put another site into free submission with no incoming links at the same time. Although the home page has been crawled (and no further pages as I write), I have yet to see it rank anywhere. Q: Will your existing PFI Inktomi listings be removed from Yahoo after April 15th. A: People who did not sign up for SiteMatch will no longer have their Inktomi PFI listings in Yahoo results after April 15th. But, provided your pages have been crawled by Yahoo's free crawler, naturally crawled pages will not be dropped. Certainly we have had no pages which were in Inktomi PFI removed from Yahoo listings after April 15th - all pages are there, although they have no PFI tracking code, which indicates that they are there through "natural" inclusion! This includes pages put into Inktomi PFI as recently as 20th February this year. Q: Will Penalties from Inktomi carry forward to Yahoo Search? A: Inktomi started imposing some heavy penalties last year via both manual reviews and automated methods. If you were in paid inclusion, this meant that you would appear last on any targeted search term. Another indication is that Slurp will only read your robots.txt file and proceed no further. These penalties have been carried forward to Yahoo. Yahoo have stated that they intend to start a re-review process and have offered the e-mail address: reportsearchspam@yahoo-inc.com for webmasters who wish to appeal a penalty. SiteMatch users can also contact their SiteMatch vendor to find out if they have a penalty and most try to find what has caused it and offer assistance. We have used the standard appeal procedure for a site that we felt had received an unfair penalty. We received a reply confirming Yahoo would review the site within 3 weeks of us making the request and (fortunately) saw the penalty lifted within another 7 days. So, it is possible to appeal their penalties, but ensure the site is squeaky clean! Q: Will use of SiteMatch drop my other site pages or prevent pages from being crawled? A: Simple answer to that is not in our experience. Q: Will SiteMatch pages receive a boost in the results? A: This used to be asked about all PFI programs in the past. SiteMatch pages get crawled and re-indexed every 48 hours. A good SEO or webmaster will use this frequent re-spidering to "adjust" the page to see how that affects the page rankings. That is the sole advantage SiteMatch (or any PFI program) gives you. A poorly optimised page will get poor results and vice-versa! So SiteMatch gives no inherent ranking advantage but it does give the webmaster the opportunity to adjust their pages frequently to find the ideal mixture of content to ensure decent listings. It's in your hands! Q: If Yahoo is crawling the web for free, why do you need SiteMatch? A: It all depends on how quickly you want to see results! If you have a commercial site with content changing frequently and you want to be showing this content in search engine results within a couple of days of publication, then SiteMatch gives you a great advantage. If you have static content and you are happy to wait a few weeks for the natural crawl and updates then SiteMatch is of no real benefit. Certainly, we have sites where we use SiteMatch on some pages, yet others remain updated only by the free crawl as they change infrequently. Basically, it all boils down to Return on Investment. For some sites/pages SiteMatch makes economic sense, for others it would be far too costly an exercise. Q: How long does it take for Yahoo to crawl a site in full? A: Currently, this is a difficult question to answer as Yahoo Search has been out crawling for only a few weeks! Yahoo seems to crawl a site in "layers". First the root directory, with pages linked to by the home page seems to be crawled, then index pages from sub-directories and then some internal pages. The initial crawl seems to take around 2-3 weeks, the second "level" crawl another 2 weeks and we have just started to see Slurp nibbling at some more internal pages. Updates seem to occur on a rolling basis but spidering is not as aggressive as Google can be. As soon as we have seen a sizeable site indexed in its entirety we will post an update. Q: How does the SiteMatch submission process work? A: SiteMatch paid inclusion is different to previous PFI programs as pages have to go through a review process to identify or check you have selected the correct category and also to check for spam. This is done manually by human reviewers. When putting a page into SiteMatch, you are asked to select the most appropriate category for your site or pages. Categories (and their associated CPC rates) are: Categories Adult ~ $0.15 Apparel ~ $0.30 Automotive ~ $0.15 Books ~ $0.15 Computers & Software ~ $0.15 Dating ~ $0.15 Education & Career ~ $0.15 Electronics ~ $0.30 Entertainment & Attractions ~ $0.15 Financial Services ~ $0.30 Flowers, Gifts & Registry ~ $0.30 Health, Beauty & Personal Care ~ $0.30 Home & Garden ~ $0.30 Jewellery & Watches ~ $0.15 Music & Video ~ $0.15 Office ~ $0.15 Professional Services ~ $0.30 Real Estate ~ $0.30 Reference ~ $0.15 Sports & Outdoors ~ $0.15 Telecom & Web Services ~ $0.30 Toys & Baby Equipment ~ $0.15 Travel ~ $0.30 Other ~ $0.15 Having selected the correct category, you have the option of selecting the areas or countries you wish to have your results displayed in. As such, you can exclude your listings from markets where you do not want visitors or enquiries. As a default, all listings are displayed worldwide - so make sure you select just the UK or Europe if this is what you want! After giving SiteMatch the pages you want included, you are asked to pay the appropriate fee along with a $50.00 deposit to be used for click-through costs. Your pages are then put in a queue for editorial review. In our experience, this takes around 5 working days. The review process appears to consists of not only checking pages that were submitted, but also other aspects of the site. This can include contact information, links to other websites and (probably) a search for related sites or possible cross-linking. So this is not purely a check for spam on the pages submitted. Your whole site is checked out and (if you have a network of sites) possibly your whole network. If you have other sites with what the reviewer may consider substantially the same content, you could have them all penalised! Sites that have affiliate links may also receive penalties. Any attempt to submit old fashioned doorway pages is doomed to failure. The full list of rules laid down are listed in the SiteMatch Guidelines. In summary, unless you know that your site is squeaky clean, don't use SiteMatch. If you have any doubts as to what constitutes "squeaky clean" get a second opinion from a professional SEM company who can show that they understand SiteMatch fully! If you attract a penalty, remember it can affect your whole site and (possibly) other web sites you may own! Be careful. However, there is some good news! If you are accepted, pages appear within a matter of hours on all Yahoo partner sites and Yahoo itself. For sites we have submitted, the average has been around 12 hours from acceptance. So expect your pages to appear within 7 working days from initial submission. As far as ranking criteria are concerned, we have noticed the following: On previous Inktomi partners (such as MSN and HotBot) the old Inktomi algorithm appears to still be the main criteria in ranking well. Pages laid out to the formula we wrote about in Optimising for Inktomi, still work just as well as they did previously. On Yahoo, things do work differently. Although we still achieved top 20 rankings for our pages, it appears that rankings only increased when external links were picked up by Yahoo's crawl. A Yahoo directory listing appears to have consistently boosted the rankings by several places over the initial placement in the Yahoo results, but other directories (such as the ODP) could well have the same effect. Results on AltaVista and AllTheWeb seem to follow the Yahoo results fairly closely but we expect these portals to be used for algo testing, so expect some inconsistencies! Regardless of if you have selected various countries or regions to show up in, geo-targeting algo changes are used. For example a .co.uk site will get a boost in MSN UK and Yahoo UK results and a slight demotion in the main .com MSN and Yahoo sites. The same will occur for other TLDs and the search portal that is used. All-in-all though, SiteMatch does what it says. After approval, inclusion is rapid and spidering is frequent. Q: What happens when your deposit runs out? A: Well, this hasn't happened to us yet! The official line is that your PFI pages will be removed from inclusion but, as happened with the Inktomi PFI drop, if your pages are included in the natural crawl, these will replace your PFI listings. If this is correct, and if you play your cards right, this could be a major way to reduce your spend on SiteMatch as you could use it for a 3 month period until a full crawl has been established on a new site and then allow pages that change infrequently to drop from the program. v) Conclusion Yahoo's new search engine has brought some healthy competition to the search engine field. Their fundamentally different way of ranking sites since the changes on Google some months ago have been a lifeline to many on-line businesses. Many observers have applauded their relevance, though others have complained at the ease at which they can be spammed. It is obvious that Yahoo is taking a very pro-active stance against spam (some would say that they are becoming way too harsh), so it is going to be interesting to see how this develops over the next few months. Certainly, few can continue to state that SiteMatch is going to flood the results with "paid spam". The criteria for inclusion are tough. Gloomy predictions that existing Inktomi PFI customers were all going to be forced into having to pay for SiteMatch have been incorrect. Yahoo have included the majority of those pages through their free crawl provided they are linked to adequately and are on sites that have reasonable incoming links from authoritative sites. The much vaunted Yahoo Search crawl has yet to prove itself, however. Inktomi had a reputation for taking anything up to a year to fully crawl a site. Yahoo needs to show that their iteration of Slurp is not going to be so lazy! Yahoo employees are making a concerted effort to engage the webmaster community. Although this obviously has great PR benefits (let's face it SiteMatch is pitched towards webmasters), the fact that they seem to taking webmaster comments seriously is to be applauded. Provided Yahoo can improve the freshness of their main index, searchers could well gravitate to their search engine. They have a good start. SiteMatch will remain a controversial program for many and can be confused with PPC by webmasters who are inexperienced with SEM. You can't target specific keywords without optimisation of the pages - and for many webmasters, this could cause dissatisfaction with the program. Similarly, many will expect a ranking boost for paying the fee. This just doesn't happen - you are paying for a service which can be used as a tool in your optimisation and SEM portfolio. This, too, may lead many webmasters to question the value of the program. For most it appears that SiteMatch will be unnecessary, for some - it can be of immense benefit. Only time will tell if this model becomes a success. But if it does, expect other search engines to follow suit (particularly MSN).
	Article by Barry Lloyd, CEO of MakeMeTop. Barry Lloyd founded the MakeMeTop brand of SEO services in 1999 although he began search engine optimisation in 1997. Barry posts on several forums as a Senior or Veteran Member under the name MakeMeTop, is a Moderator on the IHelpYouServices Forums, has spoken at leading conferences on various aspects of search engine marketing and has written articles on specific aspects of the industry for webmasters world-wide. MakeMeTop now has offices in Northern Ireland, mainland UK and Singapore and does search engine marketing for over 200 clients throughout the World www.makemetop.co.uk
	Reproduction rights You may reproduce this article in any format as long as the content is not edited and the "About the author" portion above remains intact.