Search Done Right

Many enterprise search vendors have announced that clustering of search results is now part of their product and user experience. The most recent case is Google (press center, blog post, blogosphere reaction). Microsoft researchers have also experimented with clustering, without these experiments finding their way yet into Microsoft’s products.

By definition, a clustering engine analyzes the top (say 200-500) search results from a query and displays the main themes, typically as folders that may consist of subfolders.

The spread of clustering engines is gratifying, since Vivisimo was founded on a breakthrough clustering algorithm, has been refining the approach and educating and selling into the search market since 2001, and has evolved into a complete enterprise search provider.

The definition of Knowledge Worker seriously needs updating, so that corporations can best figure out how to enhance knowledge-work productivity. In particular, it needs a good tie-in to search, which is becoming the dominant means to leverage prior knowledge. Here’s the Knowledge Worker entry from Wikipedia as of today:

“Knowledge worker, a term coined by Peter Drucker in 1959, is one who works primarily with information or one who develops and uses knowledge in the workplace.”

“Develops and uses”? Most knowledge workers don’t develop knowledge as a main activity, or even at all. Instead, they seek out – or search for – knowledge developed by others.

In parallel with our release of version 5.5 of Velocity, we at Vivisimo have been speaking with a number of analysts and reporters. My own comments have focused on two thoughts about the state of enterprise search. A previous post developed the claim that search has grown up.

This post focuses on a second thought: search is spreading its wings. There are various applications in which search has been seen as 10% or even 0% of the challenge, but which are being transformed into search-centric tasks where search is recognized as 50% or 80% of the challenge. Two examples:

In parallel with our release of Velocity version 5.5, we at Vivisimo have been speaking with a number of analysts and journalists (e.g., InformationWeek, ComputerWorld). My own comments have focused on two thoughts about the state of enterprise search.

This post focuses on the first thought: enterprise search has grown up. That is, enterprise search has reached a state of development that makes possible the speedy deployment and painless ongoing administration of a search engine that handles the full complexity of enterprises and delivers a great end-user experience. A little elaboration:

1. speedy deployment means days or weeks, not many months or years

John Battelle’s Searchblog mentions Zibb.com, launched by Reed Business Information to use “proprietary categorization, entity extraction and taxonomy management software … [which] automatically organizes content from hundreds of Internet sites into a vertical search engine.”

Vertical search by publishers is a worthy attempt to deal with this problem: the success of general web search engines Ask, Google, Live, and Yahoo (AGLY) – or even Clusty – as go-to places for all kinds of information threatens to diminish the brands and mindshare of specialty publishers, turning them into commodity suppliers of AGLY search results.

A previous post introduced a distinction between high-value enterprise information and all the rest, arguing that users and the enterprise can benefit by:

1. identifying the high-value information
2. indexing it in an appropriate, custom way
3. spotlighting it within the search results page

For example, let’s take LECG, a NASDAQ-traded company with a market capitalization of around $438M. They have “more than 1,000 experts and professionals across 30 disciplines in 10 countries” and offer this profile:

“LECG is a leading expert services firm. Our highly credentialed experts and professional staff conduct economic and financial analyses to provide objective opinions and advice that help resolve complex disputes and inform legislative, judicial, regulatory and business decision makers.”

Every enterprise has to decide how to handle its high-value, searchable information. What is high-value information, you ask? Some examples:

• descriptions of staff expertise within large service firms that essentially sell knowledge, like Accenture, law firms, and other consultancies
• product descriptions for the products for sale by a retail firm
• majors, course descriptions, research projects, centers etc. at a university
• clinician backgrounds, specialties, contact information, and photographs at a medical center

Nice interview of search industry commentator Steven Arnold in Government Computer News. I know and like Mr. Arnold and find his conference talks fun and informative. This remark of his is close to the mark:

This has been a time when people are realizing that enterprise search doesn’t work. Folks with enterprise search systems are really on the lookout for technologies that make search more useful for the users.

Right. If enterprise search doesn’t make users more productive, then what’s the point? But his brush paints too broadly. Enterprise search can and does work, and there are actually independent experts who test products very closely and conclude that enterprise search does work.

Lynda Moulton of the Gilbane Group writes today on how confusing the enterprise search market is for buyers. She is right. My own conversations with prospects and attendees at tradeshows and conferences reveal that people just don’t know what questions even to ask about a product or vendor. But there are two different sources of confusion:

1. products that aren’t quite up to the job (Lynda’s example)
2. the fog of buying from a myriad of options, terminologies, technologies, etc.

Let’s take Lynda’s example of a really difficult search problem:

• Looking up an address in a directory

Welcome to the Search Done Right blog, devoted to the challenges of enterprise search. There are many related blogs by John Battelle, Matt Cutts, Gary Price, Barry Schwartz, Danny Sullivan, Read/WriteWeb, and others. However, these excellent blogs mostly focus on web search: Google, Microsoft, Yahoo, etc.

Some questions we’ll address are: Why is enterprise search often so poor in comparison to web search? What are best practices in evaluating and buying enterprise search? What are new, good, or worthless trends and technologies? Where does enterprise search fall short? Which public web sites have good and bad site search, and why?