Search Done Right

As organizations roll out search applications to end-users in order to provide a single point of information access, a funny thing happens - users become reliant on it and expect that search will be always “on” - just like it is on the web. However, many search applications on the market were not designed to address the demands of high availability, leaving customers to develop their own solutions and workarounds to address the problem. Given that search is becoming much more ubiquitous within the enterprise, designing search software with system availability should be a requirement for vendors. Let’s take a look at the challenges of how to make search reliable and the problems associated with an adhoc approach.

In parallel with our release of Velocity version 5.5, we at Vivisimo have been speaking with a number of analysts and journalists (e.g., InformationWeek, ComputerWorld). My own comments have focused on two thoughts about the state of enterprise search.

This post focuses on the first thought: enterprise search has grown up. That is, enterprise search has reached a state of development that makes possible the speedy deployment and painless ongoing administration of a search engine that handles the full complexity of enterprises and delivers a great end-user experience. A little elaboration:

1. speedy deployment means days or weeks, not many months or years

Last week, Fast Search and Transfer announced a big push into the Business Intelligence market, with the introduction of their Adaptive Information Warehouse (AIW). This comes a few months after Google and Endeca both announced partnerships with BI vendors. Is Business Intelligence a natural extension for enterprise search platforms?

There are clear overlaps between search and business intelligence. First, each application has the need to access disparate information sources. All of the top enterprise search vendors have developed valuable connectors to many repositories that can be reused for other tasks. Second, the need to normalize this data is common to both applications. And finally, recent breakthroughs in terms of creative techniques to display meta-data visually via the user interface within the context of search (see Vivisimo announcement from May 2006) can be put to good use in a BI application.

I just stumbled onto a post from December on the Google enterprise search blog arguing that of the two primary methods for implementing document-level search security, ACL indexing (early binding) and search-time result by result checking (late binding), only the latter is truly secure. Sounds to me like a prime example of a vendor trying hard to make a virtue out of a serious product limitation (Google search appliances currently do not support the early-binding method):

“While we agree with Mark [reference to this article] on some of the benefits with using early-binding security filtering, there are certain limitations that make it impractical (if not impossible) to use for most deployments today”.

John Battelle’s Searchblog mentions Zibb.com, launched by Reed Business Information to use “proprietary categorization, entity extraction and taxonomy management software … [which] automatically organizes content from hundreds of Internet sites into a vertical search engine.”

Vertical search by publishers is a worthy attempt to deal with this problem: the success of general web search engines Ask, Google, Live, and Yahoo (AGLY) - or even Clusty - as go-to places for all kinds of information threatens to diminish the brands and mindshare of specialty publishers, turning them into commodity suppliers of AGLY search results.

A previous post introduced a distinction between high-value enterprise information and all the rest, arguing that users and the enterprise can benefit by:

1. identifying the high-value information
2. indexing it in an appropriate, custom way
3. spotlighting it within the search results page

For example, let’s take LECG, a NASDAQ-traded company with a market capitalization of around $438M. They have “more than 1,000 experts and professionals across 30 disciplines in 10 countries” and offer this profile:

“LECG is a leading expert services firm. Our highly credentialed experts and professional staff conduct economic and financial analyses to provide objective opinions and advice that help resolve complex disputes and inform legislative, judicial, regulatory and business decision makers.”

Every enterprise has to decide how to handle its high-value, searchable information. What is high-value information, you ask? Some examples:

• descriptions of staff expertise within large service firms that essentially sell knowledge, like Accenture, law firms, and other consultancies
• product descriptions for the products for sale by a retail firm
• majors, course descriptions, research projects, centers etc. at a university
• clinician backgrounds, specialties, contact information, and photographs at a medical center

Nice interview of search industry commentator Steven Arnold in Government Computer News. I know and like Mr. Arnold and find his conference talks fun and informative. This remark of his is close to the mark:

This has been a time when people are realizing that enterprise search doesn’t work. Folks with enterprise search systems are really on the lookout for technologies that make search more useful for the users.

Right. If enterprise search doesn’t make users more productive, then what’s the point? But his brush paints too broadly. Enterprise search can and does work, and there are actually independent experts who test products very closely and conclude that enterprise search does work.

Lynda Moulton of the Gilbane Group writes today on how confusing the enterprise search market is for buyers. She is right. My own conversations with prospects and attendees at tradeshows and conferences reveal that people just don’t know what questions even to ask about a product or vendor. But there are two different sources of confusion:

1. products that aren’t quite up to the job (Lynda’s example)
2. the fog of buying from a myriad of options, terminologies, technologies, etc.

Let’s take Lynda’s example of a really difficult search problem:

• Looking up an address in a directory

Welcome to the Search Done Right blog, devoted to the challenges of enterprise search. There are many related blogs by John Battelle, Matt Cutts, Gary Price, Barry Schwartz, Danny Sullivan, Read/WriteWeb, and others. However, these excellent blogs mostly focus on web search: Google, Microsoft, Yahoo, etc.

Some questions we’ll address are: Why is enterprise search often so poor in comparison to web search? What are best practices in evaluating and buying enterprise search? What are new, good, or worthless trends and technologies? Where does enterprise search fall short? Which public web sites have good and bad site search, and why?