Archive for May 2010

The problem with automated sentiment analysis


social-media-monitoring-toolsSentiment analysis is a complex beast. Even for humans. Consider this statement: “The hotel room is on the ground floor right by the reception”. Is that neutral, or is it positive or negative? Well the answer is probably that it is different things to different people. If you want a high room with a view away from the noise or reception the review is negative. If have mobility issues and need a room with easy access it is positive. And for many people it would just be information and so neutral. Sentiment analysis is difficult even in human analysts in ambiguous or more complex situations. For social media monitoring tools it is also complicated and not always as simple or as clear-cut as we might like or expect.

As part of our review of social media monitoring tools we compared their automated sentiment analysis with the findings of a human analyst, looking at seven of the leading social media monitoring tools – Alterian, Brandwatch, Biz360, Neilsen Buzzmetrics, Radian6, Scoutlabs and Sysomos. And the outcome suggests that automated sentiment analysis cannot be trusted to accurately reflect and report on the sentiment of conversations online.

Understanding where automated sentiment analysis fails

On aggregate, automated sentiment analysis looks good with accuracy levels of between 70% and 80% which compares very favourably with the levels of accuracy we would expect from a human analyst. However this masks what is really going on here. In our test case on the Starbucks brand, approximately 80% of all comments we found were neutral in nature. They were mere statements of fact or information, not expressing either positivity or negativity. This volume is common to many brands and terms we have analysed we would typically expect that the majority of discussions online are neutral. These discussions are typically of less interest to a brand that wants to make a decision or perform an action on the basis of what is being said online. For brands the positive and negative conversations are of most importance and it is here that automated sentiment analysis really fails.

No tool consistently distinguishes between positive and negative conversations

When you remove the neutral statements, automated tools typically analyse sentiment incorrectly. In our tests when comparing with a human analyst, the tools were typically about 30% accurate at deciding if a statement was positive or negative. In one case the accuracy was as low as 7% and the best tool was still only 48% accurate when compared to a human. For any brand looking to use social media monitoring to help them interact with and respond to positive or negative comments this is disastrous. More often than not, a positive comment will be classified as negative or vice-versa. In fact no tool managed to get all the positive statements correctly classified. And no tool got all the negative statements right either.

Why this failing matters to brands

This real failing of automated sentiment analysis can cause real problems for brands, especially if they are basing any internal workflow or processes on the basis of your social media monitoring. For example, imagine that you send all your negative conversations to your Customer Care team to respond to where relevant. If two-thirds (or maybe more) of the ‘negative’ conversations sent over are actually positive then this process starts to break down. Perhaps more importantly, a lot of the negative conversations will never make it to the Customer Care team in the first place (having been incorrectly classified as positive). Unhappy customers don’t get routed to the right people and don’t get their problems dealt with. The complete opposite of why many of our clients want to use social media monitoring in the first place.

So what can we do

As with any test, our experiment with the Starbucks brand won’t necessarily reflect findings for every brand and term monitored online. Our test was for a relatively short time period and we only put a randomised, but relatively representative, sample of conversations through human analysis. However, even with these limitations, we were surprised by the very high level of inaccuracy shown by the social media monitoring tools investigated. For businesses looking to make decisions or perform actions on the basis of a conversation being positive or negative this is potentially quite dangerous.

Of course there is much that can be done here and over time the tools can be trained to learn and to improve how they assess conversations about a given brand. But the overall message remains: automated sentiment analysis fails in its role of helping brands to make real decisions and to react to conversations about it online.

Read the other posts from our social media monitoring review 2010.

Facebook, privacy settings and taking control of your personal brand online


Facebook Logo sticker
Image by jaycameron via Flickr

Facebook today announced new features to address the criticism that is has faced recently for its privacy settings and processes. In December 2009 and then again in April this year, the site made a number of changes to its privacy options and settings. In essence they opened up more data to users beyond your friends and immediate networks and changed some of the default settings. This led to the situation where users had 50 different settings and 170 options to control the levels of access to and sharing of the data and information on their profiles.

As somebody with quite strict levels of access and privacy on Facebook, I know the complexity of these controls and the amount of time and effort needed to control access to your profile and your content. Sites like Openbook, which searches publicly viewable status updates, highlighting the vast amount of content that is out there for everybody to see. In many cases this isn’t because people have actively chosen to share this information outside their friendship groups and networks, but a result of not changing or fully managing your privacy settings.

What Facebook’s privacy changes mean

Today’s announcements are designed to make it simpler for users to see what their Facebook privacy settings are, and to manage them. The changes, to be rolled-out over the coming weeks, will mean that users will be able to:

  • have one simple control over who sees their content - everybody, friends-of-friends or just your own friends
  • easily see what their profile looks like to others
  • opt out of sharing their information with third-party applications
  • opt out of sharing your friends and pages

How easy the process will be, and how much you will actually be able to change will be fully understood as the new privacy settings roll-out, and there are already discussions about the ‘Recommended’ settings shown by Facebook. These will suggest that users share with everybody their status, photos and posts, biographies, family and relationship information. This may be more than some are willing to do.

The real test: will people manage their brand online

However, the real test of the new privacy settings will be the extent to which users actually make use of the ability to edit what they share about themselves and the information they add to Facebook. The previous settings did not help people to make these decisions and changes and to take control of their brand on Facebook without a lot of hassle. The power of the new changes will be if they encourage people to take control of their brand online. This may not mean that everybody stops sharing things, but is more likely to see people making sensible decisions about what they share and why. And this can only be a good thing.

We use social media tools, such as Facebook, for different reasons. Maybe we use it to keep up to date with school-friends, or maybe as a personal organiser for our lives right now, or maybe we just document our holidays with photos. Different people use Facebook for different reasons and so a single approach to privacy settings is not appropriate. That is why it is good that Facebook lets users manage their own settings - we each own our own brand online and should make sensible decisions about how we interact with and share from any social media tool we use.

The real test of the new privacy settings on Facebook will not be how many people share more or less of their data. The real test will be how many people take control of their personal brand and make sensible, and often personal, decisions about what to share, with whom and in what circumstances.

The British Airways strike, the union boss and Twitter


British airways B777
Image by griffs0000 via Flickr

British Airways cabin crew are on strike for the second of what could be a number of strikes this year. Last minute talks were taking place over the weekend until they broke up. And BA CEO, Willie Walsh, is blaming the collapse of the talks on Twitter. Or more specifically on the Tweets of Derek Simpson, the boss of Unite the Union, the trade union representing the cabin crew, sent whilst the negotiations were happening. These formed something of a running commentary of proceedings, with messages such as “Talks still ongoing …. Still hard going and progress hard won”, Arguments over the 8 sacked workers” and Fear of more sackings to come”.

Hardly the most revealing and confidential of information, but enough to frustrate Willy Walsh who, in his words:

[...] was shocked and angry when I found out that Derek was doing that. Sending out his version of events to the wider audience, that really did undermine my confidence in his desire to resolve this situation. It is a really serious issue.

The Tweets also seem to have caused disagreement among members of Simpson’s own Union. They served to broadcast not only what was being said (or at least one side of that story) but perhaps more importantly the location of the talks between the two parties. This led to militant protesters storming the talks later in the day, them being abandoned and the strike going ahead.

All because of a Tweet. Or perhaps, more accurately, all a consequence in part of this great example of hypertransparency.

One of the real benefits of social media tools is that they let us connect with people who share a similar interest and tell our story, share our ideas and ask questions. Twitter is a great tool that lets us reflect on and share stories about what is happening to us, what we have seen, what we think or what we find interesting. These stories are shared with anybody interested in reading them and in real time. These stories can also carry information such as the location from which they were sent.

What these social media tools lead to is a change in behaviour and communications. We share more, with more people and do so more quickly than ever before. Previously, a meeting such as this would have ended and each party released a press release giving their side of events. This release would have been written and issued after talks had ended and allow the author to reflect on the full process of talks and on the eventual outcome, and then convey this to their audience. This type of communication is reflective and allows a message to be refined and developed based on a series of discussions and decisions. Twitter is very different. It allows you to send snapshots of opinions at a point in time. Not after talks have concluded or after any decisions have been made.

This is a very different type of communication and a very different tool in the hands of people in the meeting. It means you can share an opinion formed on the basis of just a few moments of discussion. You can rely one side of a story or you can react strongly to something somebody says quickly but is then forgotten. Your messages out to the external world, those not privy to the full flow of discussion, will be difficult to interpret and evaluate as anything but a reflection on the meeting, the discussions and the potential decisions. This is a kind of hypertransparency - a powerful tool for the person sending the messages but also a difficult one. You have to think about your audience, what they will read and if they will understand what you are saying and sharing in your sporadic messages from an event. The chances are it will be difficult to portray the discussions and to share what is going on fairly of consistently. And this poses problems.

Simpson showed a great example of hypertransparency. On one hand what he was doing could be seen as a positive thing. He was sharing events in the negotiation room with members of his Union, keeping them up-to-date. Of course the reality is that his messages could only provoke a response that may not fully reflect the flow of discussion throughout the session.

So was Simpson right to send them? Hypertransparency is a dangerous thing but a growing trend. It requires the person sharing their stories to fully understand what they were doing and this new communications medium. But overall it was probably for other reasons that Simpson’s sharing on Twitter was misplaced. Sharing without the knowledge of the other people in the room could only serve to frustrate and to alienate the people he was quoting. They were playing by one set of rules (the traditional ones) and he was playing with hypertransparency. It’s not surprising they were annoyed when they found out what he was doing.

Stephen Fry to judge the most beautiful Tweet ever written


Tweet Me
Image by TPorter2006 via Flickr

For ten days every summer in the UK, more than 100,000 people gather in Hay, a small Welsh town on the edge of the Brecon Beacons National Park, for a festival of literature. Bill Clinton once described The Hay Festival as the “Woodstock of the Mind”. This year’s festival starts on 27th May, and, along with authors including Nadine Gordimer, Martin Amis and Phillip Pullman, the line-up includes a special place for Twitter. In fact a special prize for Twitter, as Stephen Fry will be awarding a prize for the most beautiful Tweet ever written.

As the festival’s founder and director, Peter Florence remarks:

The definition of most beautiful tweet could fall into a number of different categories: it could prove the most eloquent; the most impassioned; the best demonstration of a clever pun or metaphor; the most evocative description of a place or emotion, or perhaps prove that brevity is conducive to levity, and be the wittiest tweet ever committed to the Twittersphere

I’m not sure many of my Tweets (@mattrhodes) are eloquent or impassioned and I don’t often weave a clever pun or metaphor into them. But maybe now is time to start. I love the concept of the competition, but perhaps more I love that Twitter is being discussed and reviewed alongside more traditional understandings of the word ‘literature’. Too many people, often those who have never used it, think that Twitter is full of people saying such things as “I just ate a sandwich”. The truth is that most people are using Twitter to convey meaning, information, opinions and emotion. It is a new and growing communication medium which has spawned its own style of writing and expression. An award for the most beautiful Tweet ever written is exactly what we need.

How is the most beautiful Tweet being chosen?

Judging begins on Monday 21st May and nominations can be sent to the official festival Twitter account: @hayfestival. A shortlist will then be posted on the festival website and the winner announced on the 6 June.

So choose the Tweet you think most beautiful and send it the the judges now to see if it wins.

Facebook, privacy and social media for financial services


How do I delete my Facebook accountI have just typed “How do I” in Google today, and the fourth suggestion that it proposes is “…delete my Facebook account.”

There’s clearly some discontent out there. What’s the cause of these rumblings? Well, it was reported last week that there has been a significant review of security policy at Facebook HQ. Facebook user profiles are publicly accessible by default, and it seems that a growing number of commentators such as Jason Calacanis, chief executive of the question-and-answer website Malaho, are calling for a boycott of what is now a “not trustworthy” site.

And yet, this is against the backdrop that Facebook will shortly announce over 500 million users, and that’s 40% of everybody on the internet.

So, the dichotomy for businesses that have online security as a top priority, such as in the financial services or pharmaceutical industries for example, is how they should engage in social media when Facebook, the most popular of social media tools, is so open. And this question is always most loudly voiced in the Boardroom of the banks and insurers, where the decision-makers for a social media strategy will be immediate detractors because they consider the simple equation is “Social Media = Facebook”, and they can see no further!

The important resolve at the Boardroom must be that the social media strategies for banks and insurance companies should not focus upon social media tools. Instead, the message for the Boardroom is that the best uses of social media will demonstrate that it can yield amazing results without compromising security or the confidence of your customers. And to achieve this it can be better to think of more creative ways to engage people. We’ll be looking at some of these in the coming weeks.

Read all our posts about social media for financial services