A sentiment experiment: This week’s #BBCQT panellists on Twitter

Tweet

Sentiment is a complex beast, even for humans to decode. How many times are you never really sure if somebody is being positive or negative about something? How many times do you have to use non-verbal cues like their body language or facial expression? Cultural and linguistic factors play a huge role in our ability to understand what is meant. And this is why it is a difficult process to automate.

This is why sentiment reporting for social media discussions is problematic - giving a single ‘positive’ or ‘negative’ rating to a comment risks missing the real nuance. Even at a Tweet-level, assessing 140 characters as positive or negative can be wrong as often as it is right.

There is another way of looking at sentiment. Not to look at the comment, but to identify known elements within the sentence and look at how they are discussed. For example, let’s consider the following Tweet:

As much as I absolutely adore BBC Question Time, Steve Coogan is making me hate this week’s show so much. He’s making me really angry #bbcqt

As we read this, we know that it is neither positive or negative. It expresses both things depending on the object being discussed:

  • ‘BBC Question Time’ is clearly being described positively (‘I absolutely adore [it]‘)
  • ‘Steve Coogan’ is clearly negative (‘making me really angry’)
  • ‘This week’s show’ is also clearly negative (‘[I] hate [it]‘)

So by breaking down the Tweet into elements like this we get a much more nuanced view of sentiment. And probably a much more useful one. If I am analysing what people say about an episode of BBC Question Time, for example, I might be more interested in comparing how people talk on Twitter about the issues raised, or the guests on the panel than I am generically about the tone of discussions during the show. Looking at sentiment at this object-level is more insightful and more actionable.

So for the episode first broadcast on 27 September 2012, we conducted an experiment to explore sentiment. Not of the show or general discussions but specifically to investigate people’s sentiment towards the five guest panellists.

What we analysed

  • Using DataSift, we recorded all Tweets including the hashtag #bbcqt during the time the show was on air. This was a total of 21,651 tweets.
  • A random sample of 20% of these was then taken, giving us a total sample of 4,266 tweets.
  • This sample of Tweets was analysed using Semantria - this identifies the things (they call them ‘entities’) discussed in the Tweet and then gives a positive or negative score based on the context in which that entity is discussed.
  • We isolated entities that were the five guests on the show - using all possible spellings of the following names:
    • Danny Alexander
    • Harriet Harman
    • Jacob Rees-Mogg
    • Kirstie Allsopp
    • Steve Coogan
  • We then took a mean score for how positive or negative the context is in which each of these entities is discussed.

What we found

  • The most discussed guest panellist this week was writer and comedian Steve Coogan who was explicitly mentioned in almost 7% of all Tweets about #bbcqt. But he was also discussed most negatively.
  • The most positively discussed panellist was Labour MP Harriet Harman. She was also the only panellist who had a positive sentiment score overall.
  • Liberal Democrat MP and Minister Danny Alexander was the second most negatively discussed panellist, with Jacob Rees-Mogg and then Kirstie Allsopp above him.

What we can learn about sentiment analysis

What can we learn from this? As with all research it is important to understand the biases of our sample - it could be that the audience who view the programme and discuss it on Twitter are more left-wing and so more sympathetic to Harman’s point of view. It may be that the discussions about Steve Coogan were coordinated by a small group of individuals who had an agenda against him and so biased his score down. And it may be that the relatively small instance of mentions of Kirstie Allsopp makes her score less reliable.

All of these are areas of potential bias that should be explored. But analysing sentiment at the object level like this gives us a much more nuanced understanding of how people were discussing BBC Question Time last night. And it allows us to have much more valuable discussions than just knowing that Tweets during the show were positive or negative.

Sentiment is a complex beast, as are the humans that are expressing it. To inform a real discussion and to have a real understanding of what may be happening in discussions online we need to stop thinking in terms of Tweets and posts and comments, and to start disaggregating the individual objects discussed and explore those instead.

Live TV and real-time chat: X Factor and Strictly Come Dancing

Tweet

The End Of Telly-
Image by l-b-p-2010 via Flickr

Watching TV is almost always a social experience.  We talk to the people in the room with us. We talk to our friends on the phone, by instant messenger or on Facebook. We talk to people with similar interests in forums and chat rooms. Some of us even just shout at the TV on our own. However we do it, TV often makes us want to talk, share opinions and express ourselves. And some TV programmes make us want to do this more than others.

This week in the UK we saw one TV programme that drove many of us to chat in online communities and social media during the show. Thursday’s Question Time on the BBC featured the leader of the British National Party, saw a record number of viewers and reportedly 12.5 Tweets every second about what people were watching on their screens. Tonight we have two shows which typically attract and a much greater volume of discussions in chat rooms, forums and social media: the X Factor and Strictly Come Dancing.

The discussions and chat that accompanied these shows have been on other sites and using other tools. Tonight, for the first time, both shows have incorporated chat and social media functions into their own sites. This is a significant step for TV broadcasting in the UK. Consumer patterns have changed. We no longer watch a programme with friends and relatives and then discuss it with others the next day or read reviews in newspapers. We discuss and share our opinions in real-time through social media. The discussions and chats that accompany the show are, for the viewers, an integral part of the experience. By integrating chat and social functions into their sites, the broadcasters are hoping to recapture the viewers’ attention and give them the full experience they want.

Strictly Social

Of the two shows, the BBC’s Strictly Come Dancing offers a richer experience. Their Strictly Social site allows you to watch the show via BBC iPlayer and chat in real time with viewers alongside the screen. If you don’t want to join the discussions (as many people won’t), you can express your opinions by ‘reacting’ - clicking on ‘wow’, ‘boo’ or ‘gasp’ and seeing the word gain more prominence on screen. You can also guess what votes the judges will give the acts and vote in polls.

The Strictly Social site is clearly designed to appeal to a wide range of Strictly Come Dancing viewers. Both to provide a space for those who want to chat during the live show to do this alongside the show itself. And also to provide tools for other viewers to get engaged. It will be interesting to see how popular these other tools are and how many people use the site this week and in coming weeks. There is much talk that this is the future for the BBC’s website - providing a richer experience for the viewer by combining activities and tools they can use online during the show. The Strictly Social site could be the start of a shift in the BBC’s integration of broadcast and social medias.

X Factor Chat

The X Factor has a simpler site with ITV’s X Factor Chat. The chat site does not sit alongside an online broadcast of the show itself and instead relies on people sitting in front of a TV with their computer. Something we know many people do anyway. On the site, viewers can chat about what they are seeing and this chat is punctuated with polls, controlled by the moderator.

The site replicates more closely the experience viewers would have using tools on other sites. A real-time chat function that allows people to discuss and debate what they see on screen. The difference is that being on the ITV site gives this chat more credence. The role of the moderator should be critical here - being the official online host of the X Factor and letting the viewers and chatters feel that they are getting exclusive access and exclusive discussions.

Overall it will be interesting to see which format is most successful for the broadcasters. Which manages to engage people and, perhaps most critically, keep them viewing the show throughout the show this week, next week and for the rest of their respective runs. TV viewers have always been social creatures. For many this has involved the use of social media, online communities and other tools. Tonight the BBC and ITV caught up with them.