Search Done Right

In a Forrester Research survey published last March, almost 50% of users surveyed said that the first thing they did when navigating an unfamiliar website was to use the site search function to find information. Yet, according to Forrester, more than half of the sites they recently reviewed failed to provide a comprehensive site search or search interface. It’s not uncommon for companies to spend upwards of $1M on a website redesign and then neglect or skimp on providing high quality site search.

In a Gilbane blog post, Lynda Moulton offers great advice to purchasers of enterprise search:

“I keep wondering how many enterprise search vendors use the technologies they build and sell to support their rapidly growing enterprises. That’s a great question to ask your potential search vendor as you decide what tools to procure for your enterprise. Get them to tell you how they use their tools and the benefits they see in their own enterprise. If they aren’t at least using their own search technology in their customer relationship management and technical support knowledge-base operations, think carefully about what that might mean concerning ease of deployment and utilization.”

Earlier this month, Tom Foremski of Silicon Valley Watcher wrote a piece on search titled “Is Search Broken?” lamenting the efforts required of humans to make search better. It seems to me that Foremski does not show that search is broken, but merely that search can always be improved. As I argued in my previous post, the sky is the limit. Search is infinitely perfectible and “solving search” would amount to creating an omniscient being. So anything that can be used, should be used to make search better.

Since the introduction of the Google Mini and the IBM/Yahoo partnership in enterprise search, some have wondered whether enterprise search is becoming commodified, i.e., that crawling information and presenting search results in a reasonable ranked order is easy to do and is “good enough” for the enterprise.  Others dissent from this view, including enterprise search vendors themselves.

I’d like to analyze the issue by using the framework of Clayton Christensen’s wonderful books on technology strategy, a key insight from which is that technology becomes commodified only when its capabilities overshoot what customers need, or believe they need.

Two corollaries are that:

Many enterprise search vendors have announced that clustering of search results is now part of their product and user experience. The most recent case is Google (press center, blog post, blogosphere reaction). Microsoft researchers have also experimented with clustering, without these experiments finding their way yet into Microsoft’s products.

By definition, a clustering engine analyzes the top (say 200-500) search results from a query and displays the main themes, typically as folders that may consist of subfolders.

The spread of clustering engines is gratifying, since Vivisimo was founded on a breakthrough clustering algorithm, has been refining the approach and educating and selling into the search market since 2001, and has evolved into a complete enterprise search provider.

The definition of Knowledge Worker seriously needs updating, so that corporations can best figure out how to enhance knowledge-work productivity. In particular, it needs a good tie-in to search, which is becoming the dominant means to leverage prior knowledge. Here’s the Knowledge Worker entry from Wikipedia as of today:

“Knowledge worker, a term coined by Peter Drucker in 1959, is one who works primarily with information or one who develops and uses knowledge in the workplace.”

“Develops and uses”? Most knowledge workers don’t develop knowledge as a main activity, or even at all. Instead, they seek out - or search for - knowledge developed by others.

What makes enterprise search challenging (but also very rewarding) is its open-ended nature: the behavior of a search engine cannot be easily specified. Predicting and specifying the behavior of a search application for every possible query is not humanly possible. Unlike a document management system or a database, the performance of a search engine is hard to measure and quantify. Speed and coverage are important factors, but the primary function of the application highly depends on qualitative and subjective factors.

This open-ended nature comes along three dimensions: what, when and how.

1. What Data is Searched

In parallel with our release of version 5.5 of Velocity, we at Vivisimo have been speaking with a number of analysts and reporters. My own comments have focused on two thoughts about the state of enterprise search. A previous post developed the claim that search has grown up.

This post focuses on a second thought: search is spreading its wings. There are various applications in which search has been seen as 10% or even 0% of the challenge, but which are being transformed into search-centric tasks where search is recognized as 50% or 80% of the challenge. Two examples:

As organizations roll out search applications to end-users in order to provide a single point of information access, a funny thing happens - users become reliant on it and expect that search will be always “on” - just like it is on the web. However, many search applications on the market were not designed to address the demands of high availability, leaving customers to develop their own solutions and workarounds to address the problem. Given that search is becoming much more ubiquitous within the enterprise, designing search software with system availability should be a requirement for vendors. Let’s take a look at the challenges of how to make search reliable and the problems associated with an adhoc approach.

In parallel with our release of Velocity version 5.5, we at Vivisimo have been speaking with a number of analysts and journalists (e.g., InformationWeek, ComputerWorld). My own comments have focused on two thoughts about the state of enterprise search.

This post focuses on the first thought: enterprise search has grown up. That is, enterprise search has reached a state of development that makes possible the speedy deployment and painless ongoing administration of a search engine that handles the full complexity of enterprises and delivers a great end-user experience. A little elaboration:

1. speedy deployment means days or weeks, not many months or years