Search Done Right

Six months ago, I was one of these guys who never carried around their cell phone. Today, my office is calling me a smartphone addict because I email them while vacationing off in the Caribbean in tropical paradise.

A few times in the past I tried to use PDAs but they always ended back in their box after a few days of fiddling with them. What made a difference this time? Three things: a mobile device (my Motorola Q is even lighter and slimmer than my previous phone), a fast internet connection and a usable web browser. Very quickly I realized that I could use my smartphone for 90% of the tasks I did from my laptop while carrying less than 5% of the weight.

Earlier this month, Tom Foremski of Silicon Valley Watcher wrote a piece on search titled “Is Search Broken?” lamenting the efforts required of humans to make search better. It seems to me that Foremski does not show that search is broken, but merely that search can always be improved. As I argued in my previous post, the sky is the limit. Search is infinitely perfectible and “solving search” would amount to creating an omniscient being. So anything that can be used, should be used to make search better.

What makes enterprise search challenging (but also very rewarding) is its open-ended nature: the behavior of a search engine cannot be easily specified. Predicting and specifying the behavior of a search application for every possible query is not humanly possible. Unlike a document management system or a database, the performance of a search engine is hard to measure and quantify. Speed and coverage are important factors, but the primary function of the application highly depends on qualitative and subjective factors.

This open-ended nature comes along three dimensions: what, when and how.

1. What Data is Searched

Last week, Fast Search and Transfer announced a big push into the Business Intelligence market, with the introduction of their Adaptive Information Warehouse (AIW). This comes a few months after Google and Endeca both announced partnerships with BI vendors. Is Business Intelligence a natural extension for enterprise search platforms?

There are clear overlaps between search and business intelligence. First, each application has the need to access disparate information sources. All of the top enterprise search vendors have developed valuable connectors to many repositories that can be reused for other tasks. Second, the need to normalize this data is common to both applications. And finally, recent breakthroughs in terms of creative techniques to display meta-data visually via the user interface within the context of search (see Vivisimo announcement from May 2006) can be put to good use in a BI application.

I just stumbled onto a post from December on the Google enterprise search blog arguing that of the two primary methods for implementing document-level search security, ACL indexing (early binding) and search-time result by result checking (late binding), only the latter is truly secure. Sounds to me like a prime example of a vendor trying hard to make a virtue out of a serious product limitation (Google search appliances currently do not support the early-binding method):

“While we agree with Mark [reference to this article] on some of the benefits with using early-binding security filtering, there are certain limitations that make it impractical (if not impossible) to use for most deployments today”.