Open Data Anyone?

I presented at the OpenData conference last week in New York. The event was sponsored by Reuters and organized by Seth Goldstein. This was one of the better events I have ever attended in terms of two way communication and idea generation. In attendance where a veritable who's who in the attention economy space including the seminal author, Michael Goldhaber.

The immense amount of exhaust information generated by the creation and consumption of social media offers big opportunities for companies capable of mining that information for commercial purposes.

The mining of clickstream data provided for some fairly contentious conversations. The data is mostly gathered clandestinely through browser widgets or purchasing clickstream data from ISP's. I say clandestinely because even though disclosure is probably provided in the EULAs from the service or widget vendors, these are rarely read through (For instance, I sample, reading through probably 1/100). This data is then mined for a variety of purposes, some innocuous, others could be considered invasive. There is actually an organization set up that describes sort of a bill of rights for the end user called Attention Trust. I particularly liked the quote in Wandering Stan's blog from the conference:

Chris Law (Aggregate Knowledge): I wish AttentionTrust compliance was widespread...we don't want to surprise people.
Steve Gilmore: This is bullshit. Have you signed/endorsed the AttentionTrust principles?
Chris Law: No, we're looking into it.

There were a few companies mining media (social or otherwise) as well as closed data (proprietary data sets) and delivering this information to the financial service sector. This was labeled "Information Arbitrage". Other companies in this space (besides Collective Intellect) at the conference included Majestic Research, Gerson Lehrman Group, and Monitor110

A lot of innovation is going on in this area, bringing together the fields of text analytics, data mining, social psychology, and finance. The sleeping giants of media are beginning to awake. It should be an interesting few years.