Twitter is censoring the discussion of #Wikileaks

05

Dec

Twitter, the very popular 140 character social networking site, has a feature called “Trends” and is supposed to capture what the most popular topics of discussion are, at any given time. When people “Tweet” about a given topic, they can insert what is called a hash tag into their Tweet. For example, if I wanted to Tweet about Richard Feynman, and I wanted other people interested in Richard Feynman to be able to find it, I could put something like “#Feynman” within my post. Twitter would then automatically categorize this post under “Feynman” and voila, people can search for it on Twitter. This is how “Trends” are calculated. If say, within a given time span of perhaps 10 minutes, a million people put the tag #Christmas into their tweets, and this would be a very popular Twitter topic and should make it into the “Trends” list. Simple enough.

Today, as I was scouring the web for everything I could find about the censoring of Wikileaks and Julian Assange’s legal status I had a growing feeling of unease that Twitter was actively censoring the discussion of Wikileaks. New Tweets filed under the tag #Wikileaks were popping up at an astounding pace. If I left the page alone for literally 30 seconds, 40 new Tweets tagged as #Wikileaks would be there when I returned. It just seemed like #Wikileaks would be one of the most popular topics and therefore be reported as a “Trend”, but I really had no way to quantify this.

But then I found an excellent applet hosted by trendistic.com. It allows one to type in whatever hash tag terms they like and it will then compile a graph of how popular that hash tag term was over the course of whatever time period you select. In theory, since #Wikileaks is not a “Trend” as recognized by Twitter, every single term that Twitter lists as a Trend should outperform #Wikileaks. However, this simply is not the case. My web programming skills are not so good as my other talents, so I’ve yet to program this in the dynamically updating form that I envision, but here goes:

As of 10:48 EST, the officially trending topics on Twitter, worldwide, are

From what I can find on how Trends are calculated, rankings are determined simply by the relative amount of Tweets that contain the respective phrase or hashtag. Specifically:

“If you’re looking for a quick snapshot of the most tweeted-about topics on Twitter, refresh the your timeline and direct your eyes to the sidebar at the right of your homepage. You can see an even deeper history of the most popular topics of the minute, day, and week on the logged out twitter.com homepage. From news stories to the latest buzz about movies or music, the trending topics reflect what new or newsworthy topics are occupying the most people’s attention on Twitter at any one time.”

- Twitter Help Center

If that’s so, there would be every expectation that every single phrase listed as a trending topic by Twitter would outperform the hashtag #Wikileaks, since it is omitted from this list. However, trendistic.com allows me to plot this data, and it simply is not the case.

Below, I have plotted the prevalence of the term #Wikileaks along with the prevalence of the Official Top 5 Trends according to Twitter (#TheWalkingDead, #thingsimiss, #noonelikesyoubecause, #rappersthatmightbehomeless and #Vnezuelan♥Biebs).

The term #Wikileaks blows the other terms out of the water over the entire course of the day. It’s not like it’s even close. On average, it is nearly 3 times as popular as any of the other terms.

Why then is #Wikileaks not listed as a “Trend” on Twitter. What explanation could there for this? While Twitter is not explicitly removing posts about Wikileaks, it is actively preventing people from realizing the true scope of Wikileaks discussion, unless they’re already involved with it or seeking it out. This is no small detail. It effectively prevents Twitter from acting as a meaningful social utility.

Isn’t the point of social media to spread information and in a grander sense, democratize societies? While looking through the Twitter Help Center to research this post, I noticed that they list a contrived, hopeful post about Iranian politics as an example of a thing to hash tag on Twitter:

Are they serious? It seems rather clear from this analysis that Twitter is actively and covertly suppressing the spread of WikiLeaks discussion, joining the lowly ranks of Amazon and PayPal. At least Amazon and PayPal have owned up to their decisions. Twitter is hiding their cowardly behavior while feigning Free Speech enthusiasm in public.

Also, while Amazon and PayPal owe much of their success to their faithful customer base, and not necessarily the First Amendment, Twitter’s entire premise is enabling the quick flow of information. Perhaps its most storied role was played in giving the entrenched dissenters in Tehran a far reaching voice during the hostile election season in 2009. Weird, shadow suppression of the simple discussion of Wikileaks seems to be the sort of thing that Twitter would want not to distinguish itself with.

First Update
The website “TweetStats.com” assembles statistics on the top terms used on Twitter each day and also over longer periods of time. They display their data in the popular cloud format, where each term is sized according to its popularity. Obviously, #Wikileaks is nowhere to be seen. It isn’t clear why “trendistic.com” would display Wikileaks’ while “TweetStats.com” wouldn’t, unless “TweetStats” obtains their data from an official Twitter source which would have already censored the term #Wikileaks and “trendistic” obtains their data in some independent fashion. I’ve contacted the owner of “TweetStats” to confirm or refute this.

Second Update
The owner of “TweetStats” has been incredibly helpful in alerting me to another blog, Student Activism, where this question is also being analyzed and also for his personal insights into the process of phrases becoming trended. First of all, the statistics on “TweetStat” are compiled from Twitter’s official Twitter Trends API which updates trend information by the minute. This definitely explain the correlation between the “TweetStats” trend data and Twitter’s trend selections.

At Student Activism, a representative from Twitter has been engaging the author and making informative posts regarding some generalities of the algorithm that Twitter uses for identifying trends. He also denies that any censorship regarding trend identification is taking place.

“Hi – I work at Twitter on trends and other projects. Twitter hasn’t modified trends in any way to help or prevent wikileaks from trending. #cablegate was trending last weekend and various terms around this issue have trended in different regions over the past week. Trends isn’t just about volume of a term but also the diversity of people and tweets about a term and looking for organic volume increases above the norm. I hope this helps.”

He goes on to say more, and it’s worth the reading, but he essentially states that #Wikileaks could be failing to trend for reasons more innocuous and nuanced than outright censorship. Specifically, it might fail to meet certain contextual requirement of the algorithm such as that the posts on the topic exhibit sufficient diversity in their content. Also, any rises should  found to be “organic” which seems to mean that the rise in interest for a phrase stems from truly legitimate public interest and not self promotion or a spam bot or a predictable recurrence or some such thing. Lastly, the “velocity” of the rise in popularity is important in addition to volume. Presumably what they mean is that the volume increase per unit time is large even after being divided by the initial volume.

Certainly, it would be a project to determine the “organic” quality of the Wikileaks response. Calculating the diversity of Twitter content for a given term too would be a difficult problem. One has to wonder how daunting Twitter’s trend finding task can be. But satisfying those requirements seems manageable. In order to get a sense of scale and of the pattern of past trends, I decided to look at the most popular trend of the summer, Inception, and the most popular even of the summer, the Oil Spill. ( Student Activist does a similar comparison with the phrase “Sundays” which one would expect to be popular every Sunday, and therefore not qualify due to predictable periodic behavior).

Using the free registration at “trendistic.com” it’s possible to create graphs of phrase activity that go back 180 days into the past, which puts us at the end of June.

As one might expect, the phrase “Oil Spill” (in yellow) was already in decline at this point. From eyeballing the graph, the behavior of the phrase “Oil Spill” seems to be well approximated as a linear decrease in time with modest fluctuations about that average behavior. During this decline, the rolling average did not change drastically and so, its level of popularity could be honestly described as “sustained”. Therefore, any “residual popularity” should be detected by the algorithm and it should only trend if the phrase “Oil Spill” came back into use in a truly unique way. Also, there are not really any significant fluctuations in this period except for maybe June ~17 and July ~17. To me, I wouldn’t expect it to “Trend” except for around those two dates. Here is the actual trending record for “Oil Spill” during the period shown (from twendit.com)

Timeline

• 06-9-2010 for 14 hours and 51 minutes
• 06-10-2010 for 1 hour and 15 minutes
• 06-11-2010 for 17 hours and 15 minutes
• 06-12-2010 for 1 hour and 15 minutes
• 06-13-2010 for 60 minutes
• 06-14-2010 for 10 hours and 45 minutes
• 06-15-2010 for 5 hours and 30 minutes
• 06-16-2010 for 15 hours and 45 minutes
• 06-17-2010 for 4 hours and 15 minutes
• 06-18-2010 for 11 hours
• 06-19-2010 for 15 minutes
• 06-20-2010 for 6 hours
• 06-21-2010 for 8 hours and 30 minutes
• 06-22-2010 for 15 minutes
• 06-23-2010 for 1 hour and 45 minutes
• 06-24-2010 for 1 hour and 15 minutes
• 06-25-2010 for 9 hours and 15 minutes
• 06-26-2010 for 30 minutes
• 06-28-2010 for 60 minutes
• 06-30-2010 for 2 hours and 45 minutes
• 07-1-2010 for 30 minutes
• 07-5-2010 for 2 hours and 15 minutes
• 07-15-2010 for 15 hours and 59 minutes
• 07-16-2010 for 7 hours
• 07-21-2010 for 6 hours
• 07-27-2010 for 4 hours and 30 minutes
• 07-28-2010 for 4 hours and 15 minutes

“Oil Spill” managed to trend for nearly all of June and much of July.

Next is “Inception”, in purple. This movie came out during the summer and as Student Activist points out, likely gave rise to diverse opinions amongst those who saw it. However, it’s hard to imagine that responses could be so diverse as to give rise to the number one trend on record. After all, the phrase enjoyed a precipitous rise followed by a rather steady fall along with a significant spike near the end of its run on Twitter. Here is the actual trending record for “Inception” (again from twendit.com)

• 07-13-2010 for 7 hours and 44 minutes
• 07-14-2010 for 4 days, 16 hours and 14 minutes
• 07-18-2010 for 4 days, 13 hours and 44 minutes
• 07-23-2010 for 2 days, 9 hours and 58 minutes
• 07-25-2010 for 17 hours and 15 minutes
• 07-26-2010 for 4 days, 19 hours and 18 minutes
• 07-31-2010 for 20 hours and 33 minutes
• 08-1-2010 for 4 days, 10 hours and 32 minutes
• 08-5-2010 for 2 days, 11 hours and 30 minutes
• 08-8-2010 for 18 hours
• 08-9-2010 for 1 day, 1 hour and 45 minutes
• 08-10-2010 for 21 hours and 25 minutes
• 08-11-2010 for 1 day and 45 minutes
• 08-12-2010 for 21 hours and 45 minutes
• 08-13-2010 for 13 hours and 29 minutes
• 08-14-2010 for 10 hours and 15 minutes
• 08-15-2010 for 1 day, 13 hours
• 08-16-2010 for 17 hours
• 08-17-2010 for 1 day, 4 hours and 30 minutes
• 08-18-2010 for 13 hours and 44 minutes
• 08-19-2010 for 1 day, 16 hours and 8 minutes
• 08-21-2010 for 19 hours and 39 minutes
• 08-22-2010 for 21 hours and 29 minutes
• 08-23-2010 for 9 hours and 45 minutes
• 08-24-2010 for 8 hours and 19 minutes
• 08-25-2010 for 14 hours
• 08-26-2010 for 30 minutes

It managed to trend essentially uninterrupted from August 8 to August 26.  During this stretch, the popularity of the phrase generally fell except for a significant spike around August 17th. It seems strange that Twitter’s algorithm would identify something to be trending in the midst of the sustained fall. Trending during the spike would certainly be plausible, but why for the whole life of the fall? None the less, the fluctuations for “Inception” are about the same order as the average value, and so, the jumpy behavior is fairly contained. And it seems, that for both “Inception” and “Oil Spill”, significant jumps were enough to trigger Twitter’s trend algorithm although some sort of sustained performance was enough to keep it as well.

Now, look at the red curve

From August ~28th until Oct ~17, #Wikileaks had essentially no activity, or at least so little that it is impossible to tell from the trend graph. Then, around mid October, the term usage for #Wikileaks goes through a little arc (presumably in the waiting period for the Iraq trove to be released) and then a major explosion, corresponding to the release of the Iraq War Logs. This rise is incredible, and is essentially the same sort of rise that “Inception” saw in mid July. After the release of the Logs, activity died down significantly, returning to baseline activity. However, around Nov 21 some noise emerges, and then, on Nov 28, we see a precipitous and astounding skyrocket in term usage for #Wikileaks, a legitimate 6-800% increase in popularity, eclipsing, by far the rise for “Inception.” It is hard to imagine this rise not being “organic” as it corresponded precisely to the real event of Wikileaks beginning to release the diplomatic cables. Further, the week after saw huge fluctuations in popularity, much like, but surpassing those seen by “Oil Spill” and “Inception”. It is hard to make any truly quantitative statements about these magnitudes, as I cannot yet download the raw data diles, but it is obvious that the behavior of Wikileaks swamps that of the leading trend setters on Twitter, measured in both “trend magnitude” (Inception) and “Real world impact” (Oil Spill).

However, when we look at the trending of #Wikileaks, we find no record of trending for these events

Timeline

• 07-25-2010 for 16 hours
• 07-26-2010 for 20 hours and 30 minutes
• 07-28-2010 for 1 hour and 45 minutes
• 08-21-2010 for 1 hour and 45 minutes

There has been absolutely no trending for #Wikileaks since August 21. How can this be. While Inception surely generated some interesting Tweets, the response to Wikileaks has been massive and international. People have discussed it with regards to business, diplomacy, the wars, free speech, the constitution, Swedish sexual assault law, Swiss bank accounts, and in every language imaginable. How can it possibly be that with fluctuations of this size relative to previous behavior, the massive variety of commentary and the quantitative base of unique Twitter posters making these Tweets, that Wikileaks does not trend in the face of such formidable hash tags as “#thingsimiss” and “#rappersthatmightbehomeless”?

The only plausible scenario I can imagine where #Wikileaks does not trend in the top 10 with that sort of behavior is if the other members of the top 10 exhibit even more astounding rises and falls. However, that doesn’t seem to be what’s happened.

Therefore, I am forced to a similar conclusion as Student Activist. It might well be that #Wikileaks is failing to trend simply because of the algorithm failing to pick it up for whatever reason. However, I must say, that would imply that Twitter has written perhaps one of the most abysmal Trend Identification algorithms it could have possibly written. If the goal of the algorithm was to pick up events of importance, popularity or any other meaningful social metric, Twitter would have failed miserably in this aim, and would truly start looking into developing a new one.

Some have pointed to the fact the hash tag #cablegate trended from Nov 28-30 as evidence that no censorship is taking place. #Cablegate roughly tracks the activity of #Wikileaks on Twitter, save for a much small magnitude compared to that of #Wikileaks. Therefore, it is said that the “novelty” of #cablegate in comparison to #wikileaks qualifies it for inclusion by the algorithm. However, after the initial rise, #cablegate evolves in essentially the same way the Inception did during its run, except for the fact that #cablegate stopped trending on Nov 30. #Cablegate having trended does not seem like a structured argument against the existence of censorship. One could easily argue that #cablegate continues to outperform #rappersthatmightbehomeless and experience large fluctuations and that its cessation is indicative of non-algorithmic intervention. Twitter could clear this up by publishing some kind of algorithm analysis or examples of the algorithms performance on sample data or a plausible explanation for the vast difference in how the algorithm acted in handling “Oil Spill” and “Inception” and how it handled “Wikileaks”.

I have not paid close attention to instances where I may have added to my own thought with material I encountered on Student Activist. I warmly recommend reading his posts on this issue as well as his interactions with the Twitter Representative. He goes into greater detail on some things. In addition, his analysis of how the algorithm handles “Sundays” is illuminating. I welcome all comments and feedback in figuring out what’s going on with #Wikileaks.

Third Update

I don’t have anything in the way of new analysis, but I have a few general comments. Some of the comments posted have been truly useful and seem to have moved the discussion forward in a meaningful way. So far, the most plausible, innocuous suggestion is that perhaps #Wikileaks Tweeters are entrenched in specific sub-networks and that these networks are not identified by Twitter. As this post clearly shows, the Twitter networks surrounding Wikileaks back in June had high connectivities. Wikileaks though, was nowhere near as well known then as it is now.

I think one way forward is to do a similar analysis of the network structure as it exists today. The ultimate dataset would be the connectivity of the Twitter networks over time, which would allow us to visualize the organization of the Wikileaks networks and make meaningful statements about the character of major Twitter events. We could perform the same analysis for topics that do successfully trend. Even if the algorithm is not known, one could use this to transform the problem into an experimental question. This seems like it should support or refute the “clustered network” explanation.

Ideally, Twitter itself could do something like this, although they’re clearly not obligated to. Other people would probably give them another headache in accusing them of not minding privacy considerations. I am not sure how to start on an analysis like this, if anyone has info or wants to do analysis like this, please say so in the comments. It seems like a tool of this nature would be of broad applicability outside Twitter, if it doesn’t exist already.

Also, Twitter has made two statements concerning Wikileaks. In one, they have refused to comment on the viability of Wikileaks Twitter account in the future. It would be great if they could come out as a major company willing to put the First Amendment in their rhetoric, but that is lofty and overlooks the possibility that they may be trying to steer clear of sanctions so as to retain any ability to promulgate Wikileaks and similar organizations.

They have also flat out denied that any human intervention has a role in the non-trending status of #Wikileaks, in this statement obtained by a commenter

“Thanks for your inquiry about Twitter and #wikileaks — I’ve enclosed a statement below:

Twitter is not censoring #wikileaks, #cablegate or other related terms from the Trends list of trending topics.

Our Trends list is designed to help people discover the ‘most breaking’ breaking news from across the world, in real-time. The list is generated by an algorithm that identifies topics that are being talked about more right now than they were previously.

There’s a number of factors that may come into play when seemingly popular terms don’t make the Trends list. Sometimes topics that are popular don’t break into the Trends list because the current velocity of conversation (volume of Tweets at a given moment) isn’t greater than in previous hours and days. Sometimes topics that are genuinely popular simply aren’t widespread enough to make the list of top Trends. And, on occasion, topics just aren’t as popular as people believe.

–
matt graves
communications director | twitter
follow me: http://www.twitter.com/mgrooves“

I can’t say I’m satisfied with this explanation/denial. It doesn’t shed any light on the process. It would be really great if they could offer some specific explanations to explain why phrases like  ”Inception”, or “Oil Spill” could constantly trend with demonstrably small fluctuations, and more prolonged “constant volume” than #Wikileaks which is spectacularly volatile. The clustered network idea seems like a good way forward. Comment if you’d like to look into that!

This site has a statement from Facebook (which seems to say Wikileaks account is safe) and Twitter as well as some information on indexing of wikileaks.ch between Google and bing.

Some have suggested a mass experiment with new hash tags for Wikileaks related topics to see if these will trend. I think such an experiment would require some careful consideration of the “clustered network” idea raised above and in the comments. Maybe if people made new accounts and Tweeted the new hash tags, it would work/lead to more experimentation with the effective structuring of Twitter networks. Either way this is all interesting and a cooperative effort is definitely worth a shot.

Third and a half update

The entire post (up to the second update) has been translated into Spanish by Cadaunante at their blog (thought it appears, w/o the links). My Spanish is fading, but it seems to be a quality translation. Check out their other entries as well.

Fourth Update

Twitter has made a wholly unsatisfactory blog post attempting to explain why #Wikileaks has not been trending. Like the previous response, it fails to make any meaningful explanation or reveal any specific details that would allow one to explain to themselves what has happened. The same buzz words are thrown around while the writer wanders around for a few paragraphs, leading up to a reassuring final statement that #wikileaks is not being removed from the trends list.

The post links to an article (which is described as “a great article & infographic”) filled with plots containing Julian Assange’s beautiful face, hovering in contest with Justin Beiber’s face. In that article, the main claim is that #wikileaks previous level of attention renders it “uninteresting” to the algorithm. However, as one can see by a look at the graphs in the second update, this would not explain at all whatsoever, the reason that Inception trended for a a month or why Oil Spill was able to trend so frequently over the summer. Furthermore, from the graphs in that article itself, I don’t see how the enormous spike on Nov 28 could be described as “uninteresting” in comparison to the relative non-mention in the 4 weeks leading up to it. These explanations do not address any of the specific observations that have been made.

It seems the effort to counter-protest the protest of Wikileaks by major corporations has been building steam, with the websites of Mastercard and Visa shutting down for some time today. According to third summary bullet of this dailymail article plans next to target Twitter itself.

Advertisement

447 days ago Short URL 239 Comments