search engine meeting

I attended the 12th annual Search Engine Meeting in Boston this week. Finally, nice weather in Boston! It was a beautful spring 70 degrees yesterday, while rainy and colder in my hometown of Boulder Colorado. Eventually travel karma comes around to reward the faithful.

The conference was very informative. It might be cast as more of what was absent than what was present. This is the premier conference for search engine technology, and was presented at by Larry Page and Sergei Brin back in the early days of Google. Now they send a product manager to talk about OneBox, Google's conquest of Enterprise data discovery, and as Stephen Arnold put it, an amazingly content free presentation. The absence was in the realm of the new, the ground breaking. Google leaves such a large wake, that instead of tackling the next generation of search, the competition has nearly seeded this to the domain of Google, while determining how to work within that hierarchy.

Two areas on my mind recently have been relevancy and intent. Relevancy is much too generalized in current search engines and too easily manipulated by search engine optimization efforts. Two primary examples come to mind. The first is blog search rankings. If you go to Technorati and look for the top blogs number 21 on the list is perezhilton.com, a celebrity gossip site. This should actually be number 1 for some people, but for most wouldn't even appear on the list. Generalized relevance sucks. It doesn't address the real issue of what is relevant to me. In order to get this the search engine needs to understand a lot more about me. This brings me to intent. Intent is also a personal thing that cannot be captured with three key words entered into a search field. A number of vendors at the conference brought this up in the form of figuring out what single page or object to send a person to based on their search. For instance, one vendor brought up the scenario of someone in the enterprise entiring "quarterly west coast sales" which, instead of bringing up documents that include the words in the search string, would bring up a graph showing the last few quarters of west coast sales for the company. In this case you can infer some additional context because they are searching internal to the enterprise. You should also be able to pick up their role in the company, past results from similar queries that were selected, and selected results from similar queries by other people similar to the searcher (by role, demographic, recency). Google even talked about a few of these things.

Noticably absent from the presentation stage were Microsoft, Yahoo, and Ask.com, or even that new "google killer" search company currently in stealth mode, Powerset.com. I did see a business development manager from Microsoft approach one of the analyst luminaries in the field, Sue Feldman from IDC Research, and ask her what they could do to beat Google. I was standing right next to her in a discussion with Sue about Collective Intellect's technology. I kid you not, this is what she asked.

Bottom line is that vertical and social search are heating significantly. Natural language processing is neat, but nobody has figured out how to scale it to Google proportions, or even if they should, and the majority of startups are trying to figure out how to work within the Google world. I overheard a Goldman Sachs person once use the metaphor, "Dancing between the raindrops". I think it applies here as well.