Search Done Right

I was honored to attend last week’s Jobs and Economic Growth Forum (or Jobs Summit) at the White House, convened by President Obama to explore near-term interventions to decrease the 10% unemployment rate in the U.S., with comparable rates seen in the European Union.

Secretary of Labor Hilda Solis, Vice President Biden, and President Obama gave opening remarks to the 130 invited attendees and Administration staff, followed by breakout sessions in six groups, and then reconvened everyone into a final wrap up session.  Notably, President Obama personally attended two of the breakout sessions:  Green Jobs, and Infrastructure.

Back in the early to mid 90s, during the emergence of the world-wide web, the number of websites was exploding. This profusion led to the January ‘94 founding of Yahoo, which brought some order to the web by offering a directory whose categories people could browse top-down, much like taxonomies in libraries. Then, Altavista was founded in late 1995, offering users single-search-box access to the world-wide web, bypassing the need for directories like Yahoo’s. Later Google was founded in late 1998 based on a better ranking algorithm than Altavista’s, soon sweeping it and others away

We at Vivisimo use a CRM system from a vendor I won’t name.  For a long time, we have been crawling its contents in order to enable searching our CRM account as a tab on our own internal enterprise search; CRM search results are also returned on our default ‘All’ tab.  Clicking on a search result takes me into the native CRM system, when I can’t get what I want just from the search results page itself.  Of course, our internal search is powered by Velocity, since we eat our own dog food.

Enterprise search – or search within businesses – is a decade behind web search in terms of usage. Interesting questions are why? and whither? – what’s the trend?

There is roughly one web search per person per day in the U.S., counting the web searches at Google, Yahoo, Live, etc., but excluding searches at eBay, YouTube, CMU.edu, WashPost.com, USA.gov and the like. The analogue in business is searches done on the general intranet search, not at point solutions like Outlook search, desktop search, single-repository search, and so on. Our experience is that daily, general intranet searches lag web searches by orders of magnitude.

Vivisimo introduced high-quality text clustering into the search engine market in the year 2000, after a couple of years of computer science research on new algorithms by the founders at Carnegie Mellon. The research breakthrough was labelling the clusters, i.e, grouping search results into folder topics. Before that breakthrough, search result clusters had poor labels and so the technology was unusable. The technology was first demonstrated on a university website and later at vivisimo.com, with excellent reviews.

Microsoft has stated that it bought Fast to become a one-stop shop for enterprise search at the high end as well as at the low/medium end (”infrastructure search”). This is a compelling $1.2B argument that high-end search is what enterprises need, contrary to previous claims about infrastructure search being good enough.

[An alternative hypothesis is that Microsoft thinks low-end search is good enough for the enterprise and will try eventually to switch over the customer base from Fast, but wanted to immediately jump-start its play into enterprise search. If so, both Microsoft and Fast customers may well be in for a surprise.]

I believe that some purchasers of enterprise search are making a mistake, understandable as it may be, which they don’t make in other purchasing decisions: they go for point solutions which will prove inadequate soon after deployment.

A point solution solves an immediate, specific need. Now, what could be wrong with that??

Let’s consider point solutions in daily life that few people would find satisfactory:

• Starting work at a company that will go out of business shortly.
• Buying a thumb drive that only stores 5Mb because it will store your current PPT presentation. But it won’t store three of them.

As an entrepreneur and CEO, I am often asked by interviewees, interviewers, or acquaintances what keeps me up at night. Instead, I wish people would more often ask: What keeps me fired up during the day?

What fires me up is the same as what fires up everyone else in Vivisimo and in search: the world of online information is transforming utterly—for the better—how people learn and discover and profit from information.

In a Gilbane blog post, Lynda Moulton offers great advice to purchasers of enterprise search:

“I keep wondering how many enterprise search vendors use the technologies they build and sell to support their rapidly growing enterprises. That’s a great question to ask your potential search vendor as you decide what tools to procure for your enterprise. Get them to tell you how they use their tools and the benefits they see in their own enterprise. If they aren’t at least using their own search technology in their customer relationship management and technical support knowledge-base operations, think carefully about what that might mean concerning ease of deployment and utilization.”

Since the introduction of the Google Mini and the IBM/Yahoo partnership in enterprise search, some have wondered whether enterprise search is becoming commodified, i.e., that crawling information and presenting search results in a reasonable ranked order is easy to do and is “good enough” for the enterprise.  Others dissent from this view, including enterprise search vendors themselves.

I’d like to analyze the issue by using the framework of Clayton Christensen’s wonderful books on technology strategy, a key insight from which is that technology becomes commodified only when its capabilities overshoot what customers need, or believe they need.

Two corollaries are that: