A sentiment experiment: This week’s #BBCQT panellists on Twitter

Tweet

Sentiment is a complex beast, even for humans to decode. How many times are you never really sure if somebody is being positive or negative about something? How many times do you have to use non-verbal cues like their body language or facial expression? Cultural and linguistic factors play a huge role in our ability to understand what is meant. And this is why it is a difficult process to automate.

This is why sentiment reporting for social media discussions is problematic - giving a single ‘positive’ or ‘negative’ rating to a comment risks missing the real nuance. Even at a Tweet-level, assessing 140 characters as positive or negative can be wrong as often as it is right.

There is another way of looking at sentiment. Not to look at the comment, but to identify known elements within the sentence and look at how they are discussed. For example, let’s consider the following Tweet:

As much as I absolutely adore BBC Question Time, Steve Coogan is making me hate this week’s show so much. He’s making me really angry #bbcqt

As we read this, we know that it is neither positive or negative. It expresses both things depending on the object being discussed:

  • ‘BBC Question Time’ is clearly being described positively (‘I absolutely adore [it]‘)
  • ‘Steve Coogan’ is clearly negative (‘making me really angry’)
  • ‘This week’s show’ is also clearly negative (‘[I] hate [it]‘)

So by breaking down the Tweet into elements like this we get a much more nuanced view of sentiment. And probably a much more useful one. If I am analysing what people say about an episode of BBC Question Time, for example, I might be more interested in comparing how people talk on Twitter about the issues raised, or the guests on the panel than I am generically about the tone of discussions during the show. Looking at sentiment at this object-level is more insightful and more actionable.

So for the episode first broadcast on 27 September 2012, we conducted an experiment to explore sentiment. Not of the show or general discussions but specifically to investigate people’s sentiment towards the five guest panellists.

What we analysed

  • Using DataSift, we recorded all Tweets including the hashtag #bbcqt during the time the show was on air. This was a total of 21,651 tweets.
  • A random sample of 20% of these was then taken, giving us a total sample of 4,266 tweets.
  • This sample of Tweets was analysed using Semantria - this identifies the things (they call them ‘entities’) discussed in the Tweet and then gives a positive or negative score based on the context in which that entity is discussed.
  • We isolated entities that were the five guests on the show - using all possible spellings of the following names:
    • Danny Alexander
    • Harriet Harman
    • Jacob Rees-Mogg
    • Kirstie Allsopp
    • Steve Coogan
  • We then took a mean score for how positive or negative the context is in which each of these entities is discussed.

What we found

  • The most discussed guest panellist this week was writer and comedian Steve Coogan who was explicitly mentioned in almost 7% of all Tweets about #bbcqt. But he was also discussed most negatively.
  • The most positively discussed panellist was Labour MP Harriet Harman. She was also the only panellist who had a positive sentiment score overall.
  • Liberal Democrat MP and Minister Danny Alexander was the second most negatively discussed panellist, with Jacob Rees-Mogg and then Kirstie Allsopp above him.

What we can learn about sentiment analysis

What can we learn from this? As with all research it is important to understand the biases of our sample - it could be that the audience who view the programme and discuss it on Twitter are more left-wing and so more sympathetic to Harman’s point of view. It may be that the discussions about Steve Coogan were coordinated by a small group of individuals who had an agenda against him and so biased his score down. And it may be that the relatively small instance of mentions of Kirstie Allsopp makes her score less reliable.

All of these are areas of potential bias that should be explored. But analysing sentiment at the object level like this gives us a much more nuanced understanding of how people were discussing BBC Question Time last night. And it allows us to have much more valuable discussions than just knowing that Tweets during the show were positive or negative.

Sentiment is a complex beast, as are the humans that are expressing it. To inform a real discussion and to have a real understanding of what may be happening in discussions online we need to stop thinking in terms of Tweets and posts and comments, and to start disaggregating the individual objects discussed and explore those instead.

What’s hot in social media – February 2012 round up

Tweet

February was a busy month in social media: Pinterest rocketed in popularity so much that some are (wrongly) calling it “the next Facebook”, while Facebook itself announced the roll-out of Timeline for brand pages. Here’s a few other things that have caught our eye this month, which you may have missed:

Twitter sentiment analysis heats up

  • Datasift historic tweetsTwitter and UK based data technology company Datasift came to an agreement to release tweets going back two years. Until now, marketers have only been allowed to see tweets from up to 30 days ago. Datasift will be taking in about 250 million tweets every 24 hours and analysing them for sentiment, location and influence. The effect of this arrangement to access the Twitter archive has led to concerns about privacy, as well as conjecture that it could be a step towards being able to predict future events.
  • And speaking of predicting the future, HP and Organic took advantage of this month’s Oscars to play with some real time sentiment analysis. Similar to XFactorTracker from Professor Noreena Hertz, The Awards Meter used language analysis to monitor Twitter during the run up to the Oscars and ranked nominees according to popular or negative opinion on Twitter. At FreshNetworks we believe that you can’t necessarily take sentiment analysis at face value - automated tools need deeper analysis and understanding of the tool’s inherent biases to really dig in for insights.  However, simple tools like the Awards Meter do hint at how useful it can be to look at social media for viewing overall trends and, and are a great way to demonstrate the technology.

Social influencers are the new darlings of social media

  • PeerIndex, the social influence company has released a service targeted towards people who are ranked highly in specific subjects to offer them related discounts. Essentially a free sampling service, �?PeerPerks’ aims to differentiate itself by ensuring that free samples only go to people who are really influencers in their product fields – with the aim being that if they then talk about the products in their social circles, the uptake will be much greater. As Ian Carrington, mobile sales director at Google UK said during Social Media Week, consumers are 300% more likely to buy something when it is recommended by a friend, so it will be interesting to see whether PeerPerks takes off.
  • Boo Facebook's most influential dogAnd as we’re involved with Park Bench, a community for dog owners, we like to keep a handle on the non-human influencers in social media too – and with almost 3.5 million fans, Boo is possibly the most famous dog on the planet. Interestingly, it looks like even he is now endorsing products in social media with the recent mention on his Facebook page of a new American Apparel hoodie. Will other brands be jumping on the Boo bandwagon?