Of the major engines in the search engine universe, FAST -- aka, www.alltheweb.com -- is arguably the most unique. In many ways, it operates differently than typical engines and, in light of recent developments, it's timely that we probe a little deeper into the inner workings of this strategically important engine.

To assist us with this article we invited Stephen Baker, Fast's Director of Business Development and Marketing, to lend his perspective. As you will see, Mr. Baker was quite upfront with his answers.


FAST powers other engines

Like Inktomi, FAST powers other search engines. For example, you're probably already aware that FAST provides primary results for the Lycos search engine. But what other engines does FAST power? Mr. Baker explained,

"In North America, we power all of the Lycos and Lycos Latin properties as well as the Telus portals and AllTheWeb. Our focus over 2002 will be to establish broader reach in the US (currently 35%). 2001 was focused on European expansion, where we power over 75% of searches. This includes Lycos Europe and all of the Terra sites, T-Online, Tiscali, including Excite.it, Scandinavia Online and Web.de. Overall, we appear on over 70 portals in the European market."

...obviously an important consideration for companies doing business in Latin or European markets.

FAST also has its own engine.

Unlike Inktomi, FAST has its own public search engine. As you may know, you can't submit directly to the Inktomi engine itself. Instead you must submit through an Inktomi partner as well as pay a fee for inclusion.
And, only recently did Position Technologies make available the pure Inktomi results for searching (http://169.207.238.189/search/) although certainly not in mainstream usage since the public is generally unaware this specific search option even exists.
FAST, however, provides the means to submit your site directly to them via alltheweb.com -- http://www.alltheweb.com/add_url.php -- which they use basically as a "testing" engine. AlltheWeb is where they first consider new features such as spam detection or the use of ODP categories as a training set or even linguistic analysis of non-ODP categories.

According to Mr. Baker, AlltheWeb has also begun indexing dynamic content and supports 48 languages with over 700 million full-text HTML pages in the index. Using a technique called query analysis, the new version of AlltheWeb will remember your choices and give you a more personal search experience. An interesting possibility is that query analysis could potentially supplant the importance of link popularity sometime in the future.


I asked Mr. Baker if the results at one FAST powered engine will match results at another?
He replied that, like Inktomi, the partner engines powered by FAST use their own algorithms. He said,

"Our engine is entirely tunable, allowing customers to pre-establish offensive content, language, etc., settings. In addition, most of our customers blend FAST results with directories, PFP, and licensed results."

Fast's new pay inclusion program.

On February 25, FAST officially launched their paid inclusion program. However, typical of FAST, their paid inclusion program is unique.
Administered by partners such as Terra Lycos, Fast's PartnerSite program has four versions designed to benefit a variety of online businesses based on size. Partner Lycos calls the PartnerSite program Lycos InSite Select.

The first level of the program provides a means by which you can pay per page and offers the following benefits:

*Guaranteed inclusion in the Lycos index within 48 hours of submission.
*Full refresh of your site content every 48 hours or less.
*Automatic notification that your site has been submitted and indexed.
*Guaranteed inclusion for a full year.
*Personalized submission reports to see exactly what has been added to the index and when.
*Access to Search Services Central, which is an online account where you can make changes and request support.

Cost?

For a "limited time" (they don't say when the offer expires), the annual membership is $18 plus $12 per URL. This is assuming you use Lycos InSite Select to initiate your service. Prices may vary according to reseller.

When you actually sign up for the service, you'll be asked to choose between Standard Submission where the spider begins at a starting point that you determine, and then indexes as many pages deep as you specify. Advanced Submission allows you to list all pages you specifically want indexed after which the FAST spider will continue to "discover" URLs within your site up to the total number you've specified.

This option assures that your most important pages will make it into the FAST index rather than leave the indexing to chance.

Here's how Lycos InSite's spider works:

*The spider goes to a Web site and follows hyperlinks throughout the site.
*The spider parses and downloads certain pages as it moves through the site.
*The spider has reached the limit of URLs determined by the user's subscription service, it stops spidering the Web site.
*The file of spidered URLs are then built into a search catalog.
*If your account is missing pages and you use JavaScript heavily, you can submit individual pages using "Advanced Submission."
If your pages use META refresh tags, enter the target destination pages instead.
Further details can be found at the web site -- http://searchservices.lycos.com/searchservices/

Besides the basic version of PartnerSite I (explained in the list above), FAST offers three additional versions of the program, all geared toward larger sites with different needs. Lycos calls these versions Lycos InSite Pro and within this category they have their PartnerSite II, III, and IV classifications.

Mr. Baker explained, "Recent studies indicate that 80% of shopping carts are abandoned because of poor site search." So, FAST is providing sites with a search-ability solution through its PartnerSite packages.

PartnerSite II: Designed for sites of up to 250 pages and includes a FAST-hosted onsite search engine. Cost: $189 month.

Continued in Part 2. Contact RobinN@acws.com for complete article.