Search Done Right

In a recent editorial, Thomas Friedman suggested that the current crisis goes beyond the financial system: it represents a paradigm shift in our conception of growth. For decades, and especially in the 1980s and 90s, growth has been all about “more:” more money, more resources, more consumption. Today, as we reach the limits of this model “more” needs to be replaced by “better” and “smarter:” better, smarter, higher efficiency use of our limited resources.

In many ways this also applies to the information world and may very well explain why “enterprise search” and providers like Vivisimo are doing so well in this difficult economy.

Back in the early to mid 90s, during the emergence of the world-wide web, the number of websites was exploding. This profusion led to the January ‘94 founding of Yahoo, which brought some order to the web by offering a directory whose categories people could browse top-down, much like taxonomies in libraries. Then, Altavista was founded in late 1995, offering users single-search-box access to the world-wide web, bypassing the need for directories like Yahoo’s. Later Google was founded in late 1998 based on a better ranking algorithm than Altavista’s, soon sweeping it and others away

Smart and experienced IT executives know they can’t ignore the different styles of categorization their end users demand when searching. To increase discovery, users wish to navigate through a search result using faceted or structured navigation. Some users want dynamic categorization – clustering – while others refuse to trust anything that is not pre-defined within a taxonomy structure. Many organizations fail to meet the challenge of accommodating these competing search strategies, alienating one group or another.

For years now, people with a vested interest in information access—search vendors, librarians and other knowledge management professionals—have debated whether taxonomies will survive or if technologies like search eliminate the need for them.

I’ve been working in the enterprise search space for a number of years—not as long as some, but long enough to have heard many iterations of this debate. One version is, “Will auto-classification and semantic search technologies become sophisticated enough to replace taxonomies?” Another version transforms the question to, “Can folksonomies created by users via tagging replace the need for structured taxonomies?”

Brian Babineau of ESG (Enterprise Strategy Group) recently published a research brief on enterprise search, including many statistics, charts and graphs that will help organizations produce a strong business case to executive management on the need to gain control of information.

He also makes one of the clearest and most compelling arguments for search that I’ve seen lately:

An interesting conversation has emerged across the blogs this week regarding how well—or not—enterprise search solutions perform at finding information to the satisfaction of the individual user within the confines of an organization.

The ball got rolling when Udi Manber, a vice president of engineering at Google said that his company used its own solution for internal search, adding: “It’s not that good—I’m complaining about it.” (Ouch. Well at least he has those great cafeterias!)

We at Vivisimo use a CRM system from a vendor I won’t name.  For a long time, we have been crawling its contents in order to enable searching our CRM account as a tab on our own internal enterprise search; CRM search results are also returned on our default ‘All’ tab.  Clicking on a search result takes me into the native CRM system, when I can’t get what I want just from the search results page itself.  Of course, our internal search is powered by Velocity, since we eat our own dog food.

I once joked with a boss of mine that everything bought and sold by businesses can be boiled down to having just one of two benefits—it either saves you money or makes you money. Last Monday, I was reminded of this conversation while sitting on a panel, “Enterprise Search: Running Your Own Search Engine”, with a few other folks at the Search Engine Strategies Conference in San Jose. One of the panelists, Bill French, CTO from Myst Technology Partners, brought up statistics from IDC in his presentation, observing that employees may spend up to two hours a day searching for information, to illustrate the point of why enterprise search is so critical. Now these are not new numbers, they are the same figures that everyone in the industry has bandied about for quite some time as the reason d’etre for search—if enterprise search solutions can save each employee x amount of time, then multiply x by y (employee hourly salary) to get the theoretical dollar savings per employee search can provide. This is the “saves you money” argument.

In my last post, The Search Gap – Part 1, I discussed the enormous gap between web search and “behind the firewall” search—i.e enterprise search. Now, as promised, I’ll lay out how you can bridge the search gap in three critical areas—coverage, findability and usability.

Do a simple experiment: Ask yourself or a colleague who works in a large organization this question: “How often do you use a web search engine versus a search engine that is ‘behind the firewall’ (i.e., to find information only available inside your organization)?” The two answers are typically very different. In fact, most internet-savvy people use a web search engine every day. The same people rarely or never use a search engine behind the firewall. But maybe they just don’t need it?