In most cases, it is good for an enterprise engine to behave like the WWW engines because users can intuitively transfer their searching skills. But, this is a case where doing the right thing is more expensive for the WWW engines, and more reasonable for enterprises.

This is a scalability vs. usability tradeoff. Requiring all terms to match allows the engine to run faster, but risks giving no results at all if the user makes even a minor mistake. A spelling error on one word means no matches, and that means no hits (assuming that your pages have good spelling!). This can be a serious issue. About 10% of all search queries contain some sort of spelling error.

Why is “all terms” matching faster? It’s faster because the search engine needs to process significantly less data. For a multiple-term query, requiring all terms to match means fewer matching documents. The engine handles shorter lists of matches, which uses less memory and CPU. In a WWW search engine, defaulting to “all terms” requires less hardware for search, saving a lot of money in hardware costs.

Why is “any term” matching better? It’s better because people frequently misspell and mistype queries. With an “all terms” search, one mistake can mean no results at all. Most users are completely stymied when they don’t get results. They may try the same query again, or they may just leave.

With “any term” matching, a single error still gives results. It may even give the right results. A search for Watler Underwood will still match Underwood even though Watler is a misspelling. If there is only one Underwood on the site, the error doesn’t even need to be corrected.

What if your manager insists on “all terms”? First, look at your search logs and reports. Use your percentage of “no hits” queries as a baseline (it should be around 10% or below). You can run an experiment, changing the default to “all hits” then check the new percentage of no hits. Or, you can run an experiment with your top 100 or 200 queries. Choose the ones which are obvious misspellings, and try them in “all hits” mode to see if the results are worse.

If you do change to “all hits” mode, you will be very dependent on the quality of the spelling suggestions. You will need to actively manage those, adding Quick Link suggestions for common misspellings which show up in your query logs.

To change the default to “all hits” mode, go to the Interface>Query page in the admin UI, choose the Style you want to modify and change the “Default required search terms” setting.

Walter Underwood Ultraseek Principal Software Architect

When search is part of a site, we need to understand how it helps the users of that site. Can they find things quickly? Are they comfortable with the search?

Focusing on user satisfaction helps avoid manager centered design, but you also need to know how the search engine helps your users. There are two main aspects of this: effectiveness and trust. You change different things to improve each of these.

In order to improve relevance, you must be very clear about what it is, and what it means to make it better. You might end up tweaking the engine, changing what content is indexed, adding editorial results (“Best Bets” or “Quick Links”), or changing the presentation.

I look at relevance two ways.

UI Effectiveness: Relevant results reduce the number of clicks before visitors reach their goal. With every click, you lose visitors, maybe as many as 10%.

Relevant results at the top mean fewer clicks. Ultraseek can measure the number of clicks per result page and report that. Fewer clicks is better, though zero clicks is not good, because it means the visitor left without visiting any results.

To put specific results at the top, use Quick Links. But make sure this is based on user behavior, not on the org chart or datasheets. Quick Links must be more relevant than the first result.

Transparency and Trust: When users have some clue about why the results are presented, they trust the engine more. This is a transparency issue, and I think it is the biggest advantage of passage-based summaries. The passages are the engine explaining, ‘this is why I’m showing you this document.’ It makes a huge difference in how comfortable visitors are.

Relevance also increases trust. Irrelevant Quick Links will decrease trust, so be careful.

By Walter Underwood Principal Software Architect

For each valuable page of information on your network, there are at least 10 pages of useless information, such as log files, zip archives, and dynamic pages.

For example, do you need to index every page of an on-line calendar that goes to the year 2023? Probably not.

In fact, with the inclusion of this low-value content, the user can be overwhelmed by the number of results that match their query term. Especially if it is a common one.

One of our customers had 4.5 million pages in their index. They adjusted their spider to skip a lot of low-quality content, and ended up with 1.5 million pages. The quality of the search results improved dramatically. A smaller index, but better results.

To choose what should be in or out of your index, try some of your most popular queries. For each irrelevant result, ask yourself "should this be in the index?"

It is easy to control what gets indexed: adjust the filters on the Collections > Filters tab in the admin console. You can disallow content by matching its URL against a wildcard or regular expression pattern. The filters are matched from the top down, so make sure your specific disallow filters come before your more general allow filters.

by Ryan Weisenberger
Manager, Software Development

Here, you'll find tips, tricks and proven best practices from those who know search engines best—the team of developers who build Ultraseek, the sales engineers and technical support managers who have worked with you to optimize your implementation, fellow customers and independent enterprise search engine experts.

Content on www.ultraseek.com will be updated regularly with stories on how you can:

• Implement better search for your enterprise
• Design sites to be searchable
• Improve the quality of your Ultraseek search
• Better understand your search users

If you have a question or suggestion about the site, or would like to submit a story on how you are using Ultraseek, please email webmaster@ultraseek.com. We welcome your feedback and contributions to our site.

The Verity Team