Search Done Right

Enterprise search - or search within businesses - is a decade behind web search in terms of usage. Interesting questions are why? and whither? - what’s the trend?

There is roughly one web search per person per day in the U.S., counting the web searches at Google, Yahoo, Live, etc., but excluding searches at eBay, YouTube, CMU.edu, WashPost.com, USA.gov and the like. The analogue in business is searches done on the general intranet search, not at point solutions like Outlook search, desktop search, single-repository search, and so on. Our experience is that daily, general intranet searches lag web searches by orders of magnitude.

Vivisimo introduced high-quality text clustering into the search engine market in the year 2000, after a couple of years of computer science research on new algorithms by the founders at Carnegie Mellon. The research breakthrough was labelling the clusters, i.e, grouping search results into folder topics. Before that breakthrough, search result clusters had poor labels and so the technology was unusable. The technology was first demonstrated on a university website and later at vivisimo.com, with excellent reviews.

Microsoft has stated that it bought Fast to become a one-stop shop for enterprise search at the high end as well as at the low/medium end (”infrastructure search”). This is a compelling $1.2B argument that high-end search is what enterprises need, contrary to previous claims about infrastructure search being good enough.

[An alternative hypothesis is that Microsoft thinks low-end search is good enough for the enterprise and will try eventually to switch over the customer base from Fast, but wanted to immediately jump-start its play into enterprise search. If so, both Microsoft and Fast customers may well be in for a surprise.]

I believe that some purchasers of enterprise search are making a mistake, understandable as it may be, which they don’t make in other purchasing decisions: they go for point solutions which will prove inadequate soon after deployment.

A point solution solves an immediate, specific need. Now, what could be wrong with that??

Let’s consider point solutions in daily life that few people would find satisfactory:

• Starting work at a company that will go out of business shortly.
• Buying a thumb drive that only stores 5Mb because it will store your current PPT presentation. But it won’t store three of them.

As an entrepreneur and CEO, I am often asked by interviewees, interviewers, or acquaintances what keeps me up at night. Instead, I wish people would more often ask: What keeps me fired up during the day?

What fires me up is the same as what fires up everyone else in Vivisimo and in search: the world of online information is transforming utterly—for the better—how people learn and discover and profit from information.

In a Gilbane blog post, Lynda Moulton offers great advice to purchasers of enterprise search:

“I keep wondering how many enterprise search vendors use the technologies they build and sell to support their rapidly growing enterprises. That’s a great question to ask your potential search vendor as you decide what tools to procure for your enterprise. Get them to tell you how they use their tools and the benefits they see in their own enterprise. If they aren’t at least using their own search technology in their customer relationship management and technical support knowledge-base operations, think carefully about what that might mean concerning ease of deployment and utilization.”

Since the introduction of the Google Mini and the IBM/Yahoo partnership in enterprise search, some have wondered whether enterprise search is becoming commodified, i.e., that crawling information and presenting search results in a reasonable ranked order is easy to do and is “good enough” for the enterprise.  Others dissent from this view, including enterprise search vendors themselves.

I’d like to analyze the issue by using the framework of Clayton Christensen’s wonderful books on technology strategy, a key insight from which is that technology becomes commodified only when its capabilities overshoot what customers need, or believe they need.

Two corollaries are that:

Many enterprise search vendors have announced that clustering of search results is now part of their product and user experience. The most recent case is Google (press center, blog post, blogosphere reaction). Microsoft researchers have also experimented with clustering, without these experiments finding their way yet into Microsoft’s products.

By definition, a clustering engine analyzes the top (say 200-500) search results from a query and displays the main themes, typically as folders that may consist of subfolders.

The spread of clustering engines is gratifying, since Vivisimo was founded on a breakthrough clustering algorithm, has been refining the approach and educating and selling into the search market since 2001, and has evolved into a complete enterprise search provider.

The definition of Knowledge Worker seriously needs updating, so that corporations can best figure out how to enhance knowledge-work productivity. In particular, it needs a good tie-in to search, which is becoming the dominant means to leverage prior knowledge. Here’s the Knowledge Worker entry from Wikipedia as of today:

“Knowledge worker, a term coined by Peter Drucker in 1959, is one who works primarily with information or one who develops and uses knowledge in the workplace.”

“Develops and uses”? Most knowledge workers don’t develop knowledge as a main activity, or even at all. Instead, they seek out - or search for - knowledge developed by others.

In parallel with our release of version 5.5 of Velocity, we at Vivisimo have been speaking with a number of analysts and reporters. My own comments have focused on two thoughts about the state of enterprise search. A previous post developed the claim that search has grown up.

This post focuses on a second thought: search is spreading its wings. There are various applications in which search has been seen as 10% or even 0% of the challenge, but which are being transformed into search-centric tasks where search is recognized as 50% or 80% of the challenge. Two examples: