Measurement Matt

THE Social KPI: Time to Insight

Thu, 11 Oct 2012 09:12:34 -0400

Here is the one criteria you should use to select a product that collects social media data: time to insight.

Time to Insight defined: How long does it take me to get the information I need to exercise judgment? After that, it’s up to governance models and my degree of empowerment to do something about it (or decide not to do something about it, as the case may be). I can control how long it takes to get to that point. Whether I am dealing with a crisis, working on a campaign, troubleshooting, looking for sales leads…you name it. Give me the info I need, in a form I can use and trust, when it’s useful to me. And that can happen IF I have the right tools at my disposal.

Annnnnnnnd we can pretty much end this post here.

OK,OK

For some people Google alerts, Google Reader, and/or Tweetdeck are good enough. They may even be able to get away with it at an enterprise. Though it’s probably not sustainable.

There are tons of guides out there purporting to recommend “the best social listening tool.” The problem is that they consistently either focus on the wrong measures of “best,” or their methodology for determining “best” is flawed. And sometimes both!

Some frequent sins:

Not identifying the target audience of the evaluation
Watering things down so much that the evaluation no longer represents a real life situation. Think: using one very dumbed-down search on a relatively unambiguous topic. One topic = one industry.
Trying to compare things that are not actually comparable. Think: comparing spam rates on a Twitter-only tool vs. a more comprehensive tool that analyzes blogs.
Failing to realize that things the guide thinks are comparable actually aren’t. Think: one output measuring the number of relevant posts and another measuring the number of relevant mentions. Subtle but very important difference: a post can contain multiple mentions, but a mention is always part of only one post.
(Also, have you noticed how few evaluations out there talk about languages other than English?)

I was a big fan of the men’s clothing store Syms. Their motto was “an educated consumer is our best customer.” Sadly they went bankrupt, but hopefully the idea is not.

So what measure will help you make an informed decision?

I just about fit it all into a tweet today: precision, recall, latency, accuracy, usability. They all roll up into Time to Insight.

Consider some examples:

I work in PR. In a crisis, I need fresh data NOW NOW NOW to decide if I need to respond, with what message, and to whom. If I spend 2 hours building a Boolean query or weeding out spam and duplicates, I’m dead in the water. I don’t mess with Excel; I might mess with Cision or Factiva. If some automatic measures aren’t 100% correct, I can live with it, if that means getting the data faster and trusting that it’s directionally correct. I want to recall every last blog post, and then some. My clients are trained to focus on impressions, and some extra content will only boost those numbers. So I have some wiggle room in what level of precision is acceptable.

I’m willing to compromise on:

Precision: YES, Recall: NO, Latency: NO, Accuracy: YES, Usability: NO

I work in Customer Service. I need to identify cries for help that require my intervention. I may have 24 hours to respond to a valid support message under my SLA, so I need to get it sooner or later. Having a filtering mechanism to waste less time wading through irrelevant messages would be a big help. I should have a way to verify my hunch that there is an emerging issue I might need to escalate. I know how to pronounce GUI.

I’m willing to compromise on:

Precision: YES, Recall: NO, Latency: YES, Accuracy: NO, Usability: YES

I work in Market Research. I want to test the hypothesis that there is a correlation between social media sentiment and sales so I can recommend marketing mix adjustments. I am looking at historical data from the first quarter of 2012. I’m going to take a random sample of messages and code them for positive and negative expressions of sentiment. Because I’m a masochist (but I digress…). I need fine-grained control of the data to segment it into forms I can export and manipulate.

I’m willing to compromise on:

Precision: NO, Recall: NO, Latency: YES, Accuracy: YES, Usability: NO

I work in Market Research. I want to test the hypothesis that there is a correlation between social media sentiment and sales so I can recommend marketing mix adjustments. I am looking at historical data from the first quarter of 2012. I’m going to use my vendor’s sentiment analysis. I crunch numbers and visualize data using software with names that read like the early stages of a game of Wheel of Fortune.

I’m willing to compromise on:

Precision: NO, Recall: NO, Latency: YES, Accuracy: NO, Usability: YES

Getting the picture? Understanding the strengths and weaknesses of different solutions only matters if they help or hinder you in ways that matter. Over time, your needs will likely change, and tool features will (hopefully) also grow and change as well. So this is not a one and done process. But if a feature, either by its presence or absence, doesn’t impact how long it takes you to figure out what the course of action is, you can ignore. Focus on the ones that impact your Time to Insight.

Making Waves

Sun, 06 May 2012 18:03:08 -0400

Making Waves:

At long last, there’s a follow-up to the 2010 Q2 Forrester Listening Platform Wave. I’m sure I can only begin to imagine the hours and hours of interviews, research, travel, and testing that goes into creating a new Wave. Based on my brief interactions with Zach, he clearly knows and cares quite a bit about the social intelligence space and its potential. I hold him in high regard, even if he’s a Gooner. I’m sure this report was a labor of love.

But when I see that Radian6’s Data score has dramatically increased to the point where it leads the competitive set, you need to take this evaluation with a Planetary Resources asteroid-sized grain of salt.

This is probably an important time to make clear: this is not sour grapes. My new employer did not meet the selection criteria, so this is not some whining about getting a score of 3.46 instead of a 3.75. Look in the sidebar at my bio and my previous posts on assessing listening platforms to understand where I’m coming from.

The imprimatur of Forrester is akin to “No one ever got fired for buying IBM.” Vendors tout it on their websites and sales materials because it gives them credibility and impresses clients. As much as their business model may be threatened by independant “open research” firms like Altimeter Group, they’re not going anywhere.

Somewhere in the methodology, I really hope that beneath data sources or functionality is consideration of trust and transparency. As I’ve harped on in the past, some tools either inadvertently or deliberately make it difficult to expose the limitations of their content reservoir. I’ve singled out Radian6 by name, but they’re by no means the only perpetrator of this. The social intelligence space is so new that there is not yet generally-accepted research methodology to combat bias and error. As someone in the field, I own some measure (zing!) of responsibility for that shortcoming. We shouldn’t get a free pass because of that.

But even grading on a curve, and having put NM Incite/Buzzmetrics and Radian6 through their paces (and then some), I simply can’t believe that a) R6 beat out NMI on data sources at all or b) that R6 beat NMI on data sources by so much. Unless there has been major degradation in the content reservoir in the last year*, there’s no way that there was a sudden massive influx of spam and duplicates, or that NMI no longer covers source that R6 has never had (LinkedIn forums, Amazon Reviews, etc.). *And I can’t rule this possibility out entirely. There has been significant staff turnover at NM Incite since the last Wave (myself included), and there’s bound to be side-effects to the events that led to them taking such a tremendous hit in the Strategy score category relative to 2010.

On that note, a really interesting exercise is to compare the scores of the holdovers from the last report: Radian6, Visible, NM Incite, and Converseon. I’m not sure what the restrictions are on sharing data from the reports, but if you can get your hands on both Waves, I strongly suggest you do. Here’s a public link to the 2010 one. If the past is any indication, recent and future M&As will not result in the hoped-for synergies.

Where’s Your Forum Strategy?

Thu, 03 May 2012 06:30:52 -0400

Where’s Your Forum Strategy?:

A great article from Jason Falls (and sure, I’m biased*). There’s a real danger to thinking about an amorphous mass of the “Social Media Landscape.” I’m glad when I hear someone at a brand say they want to benchmark themselves against someone else, but all too often they want to benchmark against an “aspirational brand” like Starbucks, Nike, or Apple. This ignores two critical facts:

Consumers don’t necessarily exhibit identical social media behaviors across industry verticals
Consumers don’t necessarily exhibit identical social media behaviors across platforms

Jason calls out a great example from the financial services vertical. Why would you assume Starbucks or Nike conversation has a similar distribution across platforms to this brand? How many social media campaigns for any brand run on forums? And how many run on Twitter or Facebook? Receptivity is going vary based on the channel. Trying to cram the same content in the same format is not going to end well.

*He’s citing NetBase data.

Bitly to Launch New Consumer Version

Wed, 02 May 2012 05:50:01 -0400

Bitly to Launch New Consumer Version:

I’m a fan of bitly. They’re looking at an area of social media activity that most other tools ignore. As much insight as one can derive from public action and sharing, there’s a whole realm of intent and consumption that also has value. This is where bitly shines.

The article also ends with a list of cool tidbits that I suggest you check out.

Social Media Sentiment: Competing On Accuracy | Social Media Explorer

Sun, 22 Apr 2012 06:01:07 -0400

Social Media Sentiment: Competing On Accuracy | Social Media Explorer:

Seth Grimes is a smart dude, and he makes some thoughtful points. Of course, I would prefer if he had mentioned my employer…

But on a more serious note, for a long time there was an apocryphal stat floating around that college-educated people will agree on sentiment only 85% of the time. I have no idea how valid it was, but it’s a point that needs to be investigated and considered.

Why Netflix Never Implemented The Algorithm That Won The Netflix $1 Million Challenge

Sat, 21 Apr 2012 07:01:07 -0400

Why Netflix Never Implemented The Algorithm That Won The Netflix $1 Million Challenge:

The great pain of every analyst is when they send their work off into a black hole. That isn’t what happened here. This is a story of the second greatest pain an analyst experiences: when their work is no longer necessary, but nobody bothered to tell them.

On the other hand, this is also a story of one of an analyst’s favorite geek-out moments: sexy new sources of data to inform even better analysis.

"Now enter Google Consumer Surveys, a potentially disrupting force in market research that appears to..."

Fri, 20 Apr 2012 05:01:09 -0400

“Now enter Google Consumer Surveys, a potentially disrupting force in market research that appears to do to survey sampling what Adwords did to search monetization. GCS appears on the surface to provide near-real-time sampling capabilities, demographic targeting, custom screening for low-incidence segments, all at a ridiculously low CPI. Their basic “general population” sample for the US costs $.10 per completed survey. Game over, right?”

- MKTGR » Blog Archive » Google Consumer Surveys

Great analysis by my former Nielsen colleague Dave Martin on the implications of Google’s move into market research.

What You Can Learn From Kaggle's Top 10 Data Scientists

Wed, 18 Apr 2012 06:19:34 -0400

What You Can Learn From Kaggle's Top 10 Data Scientists:

The article reads kind of press release-y, but it’s interesting to get a peek into the heads of the people proving the value of crowdsourced math.

Klout Launches Brand Pages To Help Companies Engage Influencers

Wed, 18 Apr 2012 04:02:50 -0400

Klout Launches Brand Pages To Help Companies Engage Influencers:

I’m still not particularly thrilled with the black box nature of Klout scores, but they’re not going away. PR firms are going to go nuts for this new feature; it will drastically reduce the amount of time they need to spend identifying influencers for outreach. They can spend their time vetting existing lists instead of trying to find them in the first place.

Incredible video shows how 100 years of shipping changed the world from 1750-1850

Tue, 17 Apr 2012 23:55:27 -0400

Incredible video shows how 100 years of shipping changed the world from 1750-1850:

I don’t talk much about data visualization here, but I really ought to. This video reminds me of the replays at the end of a game of Civilization.

Gnip Adds Sina Weibo, 300-million member Chinese Microblogging Service « Gnip Blog

Mon, 16 Apr 2012 06:04:37 -0400

Gnip Adds Sina Weibo, 300-million member Chinese Microblogging Service « Gnip Blog:

This is a big deal to anyone interested in ex-US social media data. While I didn’t call it out by name, it was one of those big sites like Quora and Orkut that have been challenging for social media data collectors to obtain.

On Correlation vs. Causation

Sun, 15 Apr 2012 17:19:48 -0400

Interesting article in this week’s The Atlantic on the birth and evolution of microtargeting:

All of these online movements contribute to what Gage calls “data exhaust.” Email, Amazon orders, resume uploads, tweets — especially tweets — cough out fumes that microtargeters or data brokers suck up to mold hyper-specific messaging. We’ve been hurled into an era of “Big Data,” Gage said. In the last eight years the amount of information slopped up by firms like his, which sell information to politicians, has tripled, from 300 distinct bits on each voter in 2004 to more than 900 today. We have the rise of social media and mobile technology to thank for this.

The Creepiness Factor: How Obama and Romney Are Getting to Know You - The Atlantic

But I think the response from GigaOm is equally thought-provoking:

Finding the why answer matters, though, especially if you’re in the business of trying to convince people to buy products or vote for you. You need to understand what really drives them. And as we move away from telephone and in-person surveys toward web-based sentiment analysis, it might get more difficult to find out. A marketer can’t just email someone and say, “I’ve been tracking your online activity (that’s how I got your email address), and I noticed you’re a Diet Dr. Pepper-drinking Republican. Why Diet Dr. Pepper?” We have to figure out ways to find answers to obvious questions that we can’t just ask.

Republicans love Diet Dr. Pepper. Now tell me why. — GigaOm

All these neat patterns in data remain just that until you turn a connective thread into an arrow. It’s the difference between a fun fact to whip out at the bar and a proof point for a business meeting.

Big (Data) News

Sun, 08 Apr 2012 20:40:24 -0400

I’m excited to tell the collected internets that I have decided to join the team at NetBase. I’ll miss my team at Porter Novelli, but the top-secret stuff I’m here to do was mind-bendingly awesome enough to tear me away. (The PN Strategic Digital Analytics team will be coding this post as Sentiment: Mixed)

The only changes here will be disclosures when competitive issues come up and access to new sources of data. And potentially lighter posting if I am too busy circling back offline providing actionable social business intelligence to shift paradigms into Web 3.0 (by putting birds on things).

Sunday Times Analytics Wrap

Sun, 08 Apr 2012 19:14:04 -0400

There was a nice assortment of articles in today’s Times worth sharing.

Mining Our Own Personal Data, for Self-Discovery

I’m not sure how I managed not to share his original blog post, but the Times has just caught up on Mathematica inventor Stephen Wolfram’s quest to quantify his life. I feel like he and Nick Felton (aka Feltron) need to join forces for the good of humanity. Felton’s data is presented in a more easy to grok visual style, and it comes from the approach of human relationships and not just vast longitudinal data series.

Trying to Find a Measure for How Well Colleges Do

It’s not clear that anyone has thought hard enough about what measurements are actually worth taking, and how quantifiable they are. What if the goal of a school is to teach its students social justice and not a trade? The inscription on the gates of my alma mater reads: ”They only are loyal to this college who, departing, bear their added riches in trust for mankind.” What units do you use to measure that? Number of protests organized? Probably not.

Finally, there was an interesting pair of articles on the values and limitations of algorithms that I happened to read back to back:

Consolidation in the social listening tool space

Sat, 07 Apr 2012 17:51:00 -0400

I was in an airport desperately trying to catch up on my Twitter stream when I saw the news that Visible Tech is buying Cymfony. I remember a period in 2009 where Cymfony seemed like it was outflanking my then-employer Nielsen Buzzmetrics on tons of opportunities, and Visible kept us from gaining much traction within Microsoft. And then suddenly in 2010 all these market researchers and marketers fed up with Cymfony started switching over and Microsoft’s Looking Glass was edging out of the realm of vaporware. But other than their appearance in a 2010 Forrester report (with an update from Zach Hofer-Shall here), I haven’t heard much from either of them, bar getting a Visible demo in my role at Porter Novelli. If Visible had any traction within the agencies of Omnicom, IPG, or Havas before, WPP’s bigger ownership stake will probably dampen it. The holding companies will not want to risk having the rug pulled out from under them. I know some really smart people at Visible, so hopefully this either removes looming uncertainty and doubt, or gives them a kick in the pants to do something even more awesome.

How much does that penny for your thoughts weigh?

Thu, 05 Apr 2012 06:29:00 -0400

I’m surprised that neither Stephen Wolfram nor Nick Felton haven’t yet tackled the “change in my pocket” analysis.

What’s a Pound of Change Worth?

Spam-churian candidates

Fri, 30 Mar 2012 10:00:00 -0400

I’ve talked before about issues with using social media data to predict election outcomes. Yesterday Mashable’s 78th infographic of the day looks at a new wrench in the gears: spam:

The same techniques used by social spammers advertising free iPads and Viagra are now being used to spread bogus political messages across social media, blogs and news sites.

Yet another instance that should bring home that point that social media mention volume isn’t everything. A low-noise data source (likely coming from a tool with robust filtering capabilities) is the only way to reduce the impact of this nonsense on data quality.

Data on Rails

Wed, 28 Mar 2012 19:51:00 -0400

This is the blog post-equivalent of arriving on the platform just as the 3 train arrives. Two items on NYC Subway data came on my radar this morning:

The infographic above uses ridership data to reveal the busiest and calmest stations.
The MTA is going to open up real-time data on trains to developers. I have an app on my phone that does some whiz-bang things with augmented reality and planning trips, but it has a kludgy system for notifying of delays and can’t tell me when the next train will actually arrive.

"To that end, if the ongoing competition in computer security between those uncovering..."

Sat, 17 Mar 2012 10:29:00 -0400

“To that end, if the ongoing competition in computer security between those uncovering vulnerabilities and those patching vulnerabilities is any indication, these bots might be the initial glimmerings of a larger emerging competition between “truth black hats” — discovering and leveraging social exploits in groups online and “truth white hats” — developing the active infrastructure to “patch” these cognitive weaknesses in the same communities. Whether the strategic advantage in this space of “social security” (sorry) accrues to the astroturfer or to those attempting to block those efforts remains to be seen.”

- I’m Not a Real Activist, But I Play One on the Internet | Truthiness in Digital Media

Here’s a Big Problem for practitioners of social listening to solve: what happens when the “people” responsible for Consumer-Generated Media aren’t actually people? Whether you’re taking a sample or analyzing in aggregate, the pool is contaminated.

Let’s End The Magical Thinking About Social Media ROI

Fri, 16 Mar 2012 19:28:00 -0400

Let’s End The Magical Thinking About Social Media ROI:

Yes, please