Measurement Matt

At long last, there’s a follow-up to the 2010 Q2 Forrester Listening Platform Wave. I’m sure I can only begin to imagine the hours and hours of interviews, research, travel, and testing that goes into creating a new Wave. Based on my brief interactions with Zach, he clearly knows and cares quite a bit about the social intelligence space and its potential. I hold him in high regard, even if he’s a Gooner. I’m sure this report was a labor of love.

But when I see that Radian6’s Data score has dramatically increased to the point where it leads the competitive set, you need to take this evaluation with a Planetary Resources asteroid-sized grain of salt.

This is probably an important time to make clear: this is not sour grapes. My new employer did not meet the selection criteria, so this is not some whining about getting a score of 3.46 instead of a 3.75. Look in the sidebar at my bio and my previous posts on assessing listening platforms to understand where I’m coming from.

The imprimatur of Forrester is akin to “No one ever got fired for buying IBM.” Vendors tout it on their websites and sales materials because it gives them credibility and impresses clients. As much as their business model may be threatened by independant “open research” firms like Altimeter Group, they’re not going anywhere.

Somewhere in the methodology, I really hope that beneath data sources or functionality is consideration of trust and transparency. As I’ve harped on in the past, some tools either inadvertently or deliberately make it difficult to expose the limitations of their content reservoir. I’ve singled out Radian6 by name, but they’re by no means the only perpetrator of this. The social intelligence space is so new that there is not yet generally-accepted research methodology to combat bias and error. As someone in the field, I own some measure (zing!) of responsibility for that shortcoming. We shouldn’t get a free pass because of that.

But even grading on a curve, and having put NM Incite/Buzzmetrics and Radian6 through their paces (and then some), I simply can’t believe that a) R6 beat out NMI on data sources at all or b) that R6 beat NMI on data sources by so much. Unless there has been major degradation in the content reservoir in the last year*, there’s no way that there was a sudden massive influx of spam and duplicates, or that NMI no longer covers source that R6 has never had (LinkedIn forums, Amazon Reviews, etc.). *And I can’t rule this possibility out entirely. There has been significant staff turnover at NM Incite since the last Wave (myself included), and there’s bound to be side-effects to the events that led to them taking such a tremendous hit in the Strategy score category relative to 2010.

On that note, a really interesting exercise is to compare the scores of the holdovers from the last report: Radian6, Visible, NM Incite, and Converseon. I’m not sure what the restrictions are on sharing data from the reports, but if you can get your hands on both Waves, I strongly suggest you do. Here’s a public link to the 2010 one. If the past is any indication, recent and future M&As will not result in the hoped-for synergies.

Seth Grimes is a smart dude, and he makes some thoughtful points. Of course, I would prefer if he had mentioned my employer…

But on a more serious note, for a long time there was an apocryphal stat floating around that college-educated people will agree on sentiment only 85% of the time. I have no idea how valid it was, but it’s a point that needs to be investigated and considered.

The great pain of every analyst is when they send their work off into a black hole. That isn’t what happened here. This is a story of the second greatest pain an analyst experiences: when their work is no longer necessary, but nobody bothered to tell them.

On the other hand, this is also a story of one of an analyst’s favorite geek-out moments: sexy new sources of data to inform even better analysis.

The article reads kind of press release-y, but it’s interesting to get a peek into the heads of the people proving the value of crowdsourced math.

I’m still not particularly thrilled with the black box nature of Klout scores, but they’re not going away. PR firms are going to go nuts for this new feature; it will drastically reduce the amount of time they need to spend identifying influencers for outreach. They can spend their time vetting existing lists instead of trying to find them in the first place.

I don’t talk much about data visualization here, but I really ought to. This video reminds me of the replays at the end of a game of Civilization.

This is a big deal to anyone interested in ex-US social media data. While I didn’t call it out by name, it was one of those big sites like Quora and Orkut that have been challenging for social media data collectors to obtain.