The problem with sentiment analysis as a KPI

Here’s a brief summary:

Automated sentiment analysis uses a combination of two approaches to determine sentiment: lexical analysis (which looks for emotionally-weighted words) and machine-learning (which relies on a “training corpus” of manually scored documents to predict the emotional content of new documents that it processes.) Continued human intervention (“training”) in the machine learning process may improve results, but makes it more or less meaningless to compare the results over time.

No two sentiment analysis tools on the market agree closely; with most disagreements occurring and around “sentiment-neutral” comments. Since these form the bulk of the content, there is much room for disagreement.

Sentiment analysis on short content (e.g. Tweets) lacks sufficient context for accurate judgments, whereas analysis of longer content often lacks sufficient relevance (e.g. the search term may only be mentioned in passing, and the sentiment score refer to a different object)

Human-based/manual sentiment analysis also faces reliability challenges. There is often as much disagreement between two human analysts as there is between two automated systems. Worse still, research demonstrates that the same person – when presented with the same text on different occasions – may score it differently each time.

Conclusion: Sentiment Analysis may be used to guide customer service engagement, but should not be used as a KPI

iPhone 5: a vicious feedback loop

Why was everyone expecting the iPhone 5? Was it because journalists and bloggers were picking up on Google search trends and writing the stories that people wanted to hear?

Have a glance at this chart showing three years of search traffic (click for a live version

In previous years, search traffic has more or less followed the announcement – but for the (still mythical) iPhone 5, search traffic began to grow well ahead of time (a situation — as Scott Thompson has pointed out to me — probably only exacerbated by the delayed announcement.)

Bloggers and journalists are increasingly using social signals (search & twitter volume) to determine their editorial policy. After all, search and social are a good indicator of what interests your audience – and major traffic drivers.

In this case, however, the process created a vicious feedback loop. Even the more sensible commentators and analysts found that — when the conversation was about the iPhone 5, there was no virtue or value in writing about it in any other way.

I used Google to count the posts that mentioned “iPhone 5” in the title on some of the top tech blogs, MSM and online news sources — the numbers tell a story.

(What is) The Value of a Fan

One of the most worrying questions in Social Media today seems to be, “what’s the value of all these fans? How can I mark them down as an asset, and not a cost?”

Recently I was invited to talk about these at Warc’s “Social Media: Beyond the Hype” conference and at Webit Congress 2011 (Webit is the digital industry’s get-together for CEE. It’s huge: very energetic, very exciting and great fun.)

Below are the slides, and below that, my script. I know that it’s hard to read the two together. Actually, someone told me that I mumbled a bit during the presentation (I wasn’t used to my Madonna-style headset mike to be fair) so the script is for them (and for anyone else who was too polite to complain.) You can download the script here.

NB: I write my scripts (like all my stuff) in a kind of lazy version of Markdown. This works great, but can make for a fairly ugly layout.

The Value of a Fan: Webit 2011
View more presentations from Mat Morrison

Continue reading

Why are we so ready to criticise? (or “No Social Media Guru is an Island”)

As has often been observed, there’s something irresistable about schadenfreude. That’s one reason for the obsessive finger-pointing by the digerati every time a new brand experiences a social media crisis.

Another may be our unholy desire for traffic. After all, I’m writing this blog post in response to today’s Nestlé-Greenpeace-Facebook storm (if you’re coming to this story late, Sam Ismail’s post offers background). More to the point, at least four bloggers cleverly promoted their coverage of the events in the Facebook comment stream.
Continue reading

Thinking differently about word-of-mouth

The current approach to WOM is to try to stimulate positive WOM while addressing or countering negative WOM. A sort of “accentuate the positive, eliminate the negative and don’t mess with Mr In-Between” strategy.

But what if we could do it a different way?

This idea stems from a conversation I had back in February with Martin Kelly and Andy Cocker of Infectious Media. Since that time I’ve chatted it through a couple of times with various interesting people. It’s not properly thought through yet, but following a chat a couple of weeks ago with Ketchum London’s new Head of Digital, the excellent Fernando Rizo, I’ve decided to put the idea out into the public domain to gauge what (if any) interest there is and whether I should continue to work on it.

“Word of Mouth” is hard to do well

I’ve read lots of word of mouth marketing case studies (there’s a great list over at WOMMA) and it strikes me that WOM is hard to do well for a few reasons. I don’t want to go into these in too much detail, but here are a couple of the structural issues:

  1. Unless I’m a journalist, an A-list blogger or media personality or have some kind of platform, I probably have a very low reach.

    Despite everything pointing towards personal contact being the best impetus for positive word of mouth, most word of mouth campaigns compensate for my low reach by trying to get me to self-service my relationship with the brand and the campaign.

  2. “Viral” distribution just doesn’t work the way most people seem to think it does; and this is particularly true when it comes to WOM.

    While I’m quite likely to tell stories about my personal experience of a brand and fairly likely to tell stories that involve a mutual friend, I’m much less likely to tell stories about other friends’ experience, and not likely at all to tell stories about friends-of-friends.

    Furthermore because of the ‘clumpiness’ of most people’s social graphs, geometric progression (the “I tell two people and they each tell two people and so on” effect) just doesn’t happen.

Homophily

One of the many reasons that WOM works is a thing called homophily — which roughly translates to “birds of a feather flock together”, or “you can tell a man by the company he keeps.”

I’ve written about examples of this before: for example, my analyses of twittering US Congresspersons and Westminster MPs which showed that one can predict with some reasonable degree of accuracy the political colouration of any given twitter account based on their mutual friends and follows (if you want to know more about the methodology, it’s worth reading Robert Hanneman’s chapter on cliques and subgroups.)

But there’s another side to the homophily coin; the social pressure to conform to the group’s norms.
Continue reading

Today’s “Integration Triangle” presentation

These are the slides from a presentation I did this morning on the topic of the Integration Triangle. I’ve talked about this here before in the article “5 Straightforward Ways To Integrate Your Communication Activities” — this includes some quick case studies.

I created these slides to support the presentation I was giving: they aren’t the presentation itself. This means that while you’ll be able to have a good guess at what I was saying most of the time, there will be moments when my meaning is opaque.

There are 70 slides in the presentation, including the front and back cover. Nevertheless, I gave the presentation in under 25 minutes. To save you doing the maths, that averages out at around 3 slides every minute (actually, there was a 4 minute delay in the middle of the presentation — so it’s more like 3-and-a-half slides per minute.)

In fact, my slides fall into two categories — those on which I spend fewer than 5 seconds, and those on which I spend more than a minute. This is more an artistic decision than anything else — I think that lots of slides going past very quickly give an appearance of pace and energy (which I dearly need first thing in the morning), but can rapidly become exhausting to watch and hard to follow without the occasional pause for breath.

Even with 70 slides, there’s so much more that I can say about the “Integration Triangle” as a planning tool — but I was trying to keep this to a single simple message. I’m hoping that (whatever they thought about my presentation, and no matter whether they liked it or believed what I was saying) the audience will remember what it was that I was saying, and be able to tell a version of the story themselves.

There’s just so much that we can talk about when it comes to the whole Digital PR thing that it all becomes rather overwhelming. I’ve just got off the phone to a colleague in Vienna (where I’m speaking next week) who wants me to talk to his audience about “Facebook and Twitter and Blogs” (oh my!) And I’ve got 45 minutes to do this. Of course I can do it. But what on earth is the “one thing” I want them to remember?

What we can learn from the real evangelists?

This is a description of Billy Graham crusades from an academic study I’ve been reading. I’m interested in how real evangelists work (after all, I use the term often enough when talking to colleagues and clients):

Counselors begin their work after the singing, testimonials, collection and Billy Graham’s sermon, which culminates in the altar call. At the moment of Graham’s invitation to “come forward to Christ.” counselors and choir members begin moving forward to an area usually in front of the speaker’s platform or rostrum. To a naive member of the audience or a television viewer, this movement creates an illusion of a spontaneous and mass response to the invitation. Having been assigned seating in strategic areas of the auditorium or arena and given instructions on the staggered time-sequencing for coming forward, the counselors move forward in such a fashion so as to create the illusion of individuals “flowing” into the center of the arena from all quarters, in a steady outpouring of individual decision. Unless an outsider or observer of these events has been instructed to look for the name tags and ribbons worn by those moving forward it is all too easy to infer from these appearances the “charismatic” impact of Graham and his invitation. These strategies promote the respectability of making a public commitment and represent methods calculated to manipulate the consent of the passive, the uncertain, the wary, and the indecisive.

(from: David L. Altheide and John M. Johnson, Counting Souls: A Study of Counseling at Evangelical Crusades, The Pacific Sociological Review, Vol. 20, No. 3, (Jul., 1977), pp. 323-348)

Momentum

A recent (and criticised) study by Tubemogul on the short shelf life of online video reminded me of some research into views on YouTube videos I did back in 2006. I only looked at about 130 random YouTube videos for the first 20 days of their life cycle, while TubeMogul’s methodology was somewhat more sound (they tracked more than 10K videos for around three months, among other things.)

Here’s the chart from my analysis: Continue reading