Archive for the ‘Guardian’ tag

How news organisations should prepare for data dumps

with one comment

This post was first published at PBS Mediashift Ideas Lab on Monday 2nd August, 2010.

Soon every news organization will have its own “bunker” — a darkened room where a hand-picked group of reporters hole up with a disk/memory stick/laptop of freshly opened data, some stale pizza and lots of coffee.

Last year the U.K.’s Daily Telegraph secreted half a dozen reporters in a room for nine days with about 4 million records of politicians’ expenses. They were hidden away even from the paper’s own employees. Now we learn that reporters from the Guardian, the New York Times and Der Spiegel did the same with Julian Assange of WikiLeaks somewhere in the Guardian’s offices in King’s Cross, London.

There is a wonderful irony that open data can generate such secrecy. Of course the purpose of this secrecy is to find — and protect — scoops buried in the data. From the perspective of many news organizations, these scoops are the main benefit of data dumps. Certainly the Daily Telegraph benefitted hugely from the scoops it dug out of the MPs’ expenses data. Weeks of front pages on the print paper, national uproar, multiple resignations, court cases and much soul searching about the state of parliamentary politics.

The Guardian, the New York Times and Der Spiegel have not been able to stretch the WikiLeaks Afghan logs over multiple weeks, but they did dominate the news for awhile, and stories will almost certainly continue to emerge.

These massive data releases are not going to go away. In fact, they’re likely to accelerate. The U.S. and U.K. governments are currently competing to see who can release more data sets. WikiLeaks will no doubt distribute more raw information, and WikiLeaks will spawn similar stateless news organizations. Therefore news organizations need to work out how best to deal with them, both to maximize the benefits to them and their readers, and to ensure they don’t do evil, as Google might say.

5 Questions

Here are just five (of many) questions news orgs should ask themselves when they get their next data dump:

1. How do we harness public intelligence to generate a long tail of stories? Though the Telegraph succeeded in unearthing dozens of stories from the Parliamentary expenses data, the handful of reporters in the bunker could never trawl through each of the millions of receipts contained on the computer disks. It was The Guardian that first worked out how to deal with this; it not only made the receipts available online but provided tools to search through them and tag them (see Investigate your MP’s expenses). This way it could harness the shared intelligence — and curiosity — of hundreds, if not thousands, more volunteer watchdogs, each of whom might be looking for a different story from the expenses data. As a result, the Guardian generated many more stories and helped nurture a community of citizen scrutineers

2. How do we make it personal? Massive quantities of data can be structured to be made directly relevant to whoever is looking at it. With crime data you can, for example, enable people to type in their postcode and see what crimes have happened in their neighborhood (e.g. San Francisco crimespotting). For MPs’ expenses, people could look up their own MP and scour his/her receipts. The Afghan logs were different in this respect, but OWNI, and Le Monde Diplomatique put together an app that allows you to navigate the logs by country, by military activity, and by casualties (see here). The key is to develop a front end that allows people to make the data immediately relevant to them.

3. How can use the data to increase trust? The expenses files, the Afghan logs, the COINs database (a massive database of U.K. government spending released last month) are all original documents that can be tagged, referenced and linked to. They enable journalists not only to refer back to the original source material, but to show an unbroken narrative flow from original source to final article. This cements the credibility of the journalism and gives the reader the opportunity to explore the context within the original source material. Plus, if published in linked data, the published article can be directly linked to the original data reference.

4. How do we best — and quickly — filter the data (and work out what, and what not, to publish)? Those that are best able to filter this data using human and machine methods are those who are most likely to benefit from it. Right now only a very small number of news organizations appear to be developing these skills, notably the Guardian, the New York Times, and the BBC. The skills, and algorithms, they develop will give them a competitive advantage when dealing with future data releases (read, for example, Simon Rogers on how the Guardian handled the 92,201 rows of data and how Alastair Dant dealt with visualizing IED events at FlowingData). These skills will also help them work out what not to publish, such as data that could put people in danger.

5. How can we ensure future whistleblowers bring their data to us? It’s impossible to predict where a whistleblower will take their information. John Wick, who brokered the MPs expenses disk to the Telegraph, went first to the Express, one of the U.K.’s least well resourced and least prepared national papers. But it is likely that the organizations that become known for handling big data sets will have more whistleblowers coming to them. Julian Assange went to the Guardian partly because the journalist Nick Davies sought him out in Brussels (from Clint Hendler in CJR) but Assange must also have been convinced the Guardian would be able to deal with the data.

The influence of the war logs continues to spin across the globe, particularly following the Afghan president’s comments. But it is not the first — and certainly won’t be the last — big data dump. Better that news organizations prepare themselves now.

Written by Martin Moore

August 3rd, 2010 at 7:48 am

The PCC’s governance review just became a lot more important

without comments

It has not been a good week for the Chair of the PCC. This is a shame because Baroness Buscombe seems like a smart woman who arrived at the PCC in April with a real intention of changing the way it worked.

The trouble started on Sunday evening, when Baroness Buscombe made her first major set piece speech at the Society of Editors conference. It was an odd maiden speech, made up of a mixture of party politics and truisms about press freedom in a democracy, with virtually no mention of press standards or future intentions. The type of speech you might expect from a working politician or a Tory peer in the House, but not from the recently appointed head of an ‘independent regulator’ (how the PCC now describes itself). Those papers that chose to report it, however, mostly did so uncritically. The Guardian was the exception. The editor of the Media Guardian called it ‘disappointing’ and suggested the new Chair’s perspective on self-regulation was ‘rose tinted’.

The following morning Buscombe appeared on the Today programme. Bizarrely, she chose to talk more about our ‘dysfunctional democracy’ than about her plans for press self-regulation. When asked directly about relevant issues – on the PCC’s position regarding the reporting of Gordon Brown’s private exchanges with Jacqui Janes – she refused to answer.

Worse was set to come. Before Buscombe’s Radio 4 appearance she released a written statement via the PCC implying that evidence given to the Commons CMS Select Committee by Mark Lewis, a lawyer for Stripes Solicitors, was false. Lewis had told the committee that DS Maberly of the Metropolitan Police had told him that around 6,000 people were involved in the phone hacking episode (though “It was not clear to me whether that was 6,000 phones which had been hacked, or 6,000 people including the people who had left messages” Lewis said to the select committee). Giving misleading evidence to a select committee was, the PCC statement said ‘an extremely serious matter’. The response from Lewis was fast and furious. Not only did he stand by his evidence and say he had two witnesses to corroborate it, but he noted that the PCC had not approached him before making such a serious accusation and had chosen to believe a ‘hearsay letter constructed on behalf of the Metropolitan Police rather than the first hand evidence that was given by me to the Select Committee’. This, Lewis said, ‘betrayed any semblance of impartiality’ the PCC might have had and he called on the Chair to resign.

Hardly had the ink dried on Lewis’ signature than bloggers were reacting to a report on the Independent website that Buscombe had suggested the PCC was considering regulating blogs. Ian Burrell wrote that the Chair had ‘said that after a review of the governance structures of the PCC, she would want the organisation to “consider” whether it should seek to extend its remit to the blogosphere, a process that would involve discussion with the press industry, the public and bloggers (who would presumably have to volunteer to come beneath the PCC’s umbrella).’ Bloggers were outraged. ‘Very disturbing’, Iain Dale wrote: ‘I see absolutely no need for independently operated blogs to be regulated by the PCC or indeed anyone else. If they want to propose a voluntary system of regulation, fine. But the day they try to mandate it is the day I will give up blogging.’ Guido Fawkes subsequently blogged that she’d never said such a thing. But this did not stop a damning critique of the suggestion in a ‘collective letter’ from Unity at Liberal Conspiracy:

‘While we are grateful for your interest in our activities we must regretfully decline your kind offer of future PCC regulation. Frankly, we do not feel that the further development of blogging as an interactive medium that facilitates the free exchange of ideas and opinions will benefit from regulation by a body representing an industry with, in the main, substantially lower ethical standards and practices than those already practiced by the vast majority of established British bloggers.’

Before the Chair had time to calm a furious blogosphere, news came through that the International Federation of Journalists – the world’s largest organization of journalists, with more than 600,000 members in 100 countries – was commissioning an investigation into the PCC’s handling of its inquiry into the News of the World phone-hacking allegations.

Later that afternoon an even more serious problem erupted closer to home. Alan Rusbridger, the editor of the Guardian, had been voicing significant disquiet about press self-regulation since the PCC released its report into phone hacking at the NotW. That report was, Rusbridger said, ‘worse than pointless, it’s actually rather dangerous to the press… I believe in self-regulation because I cannot imagine a country in which the government regulates the press, or there is statutory regulation. But the press is in a very weak position today because its own regulator, its self-regulation, has proved so weak.’ When Mark Lewis’s letter was published online, Rusbridger tweeted it from his personal account.

Then, on Tuesday, Rusbridger announced he was resigning from the PCC’s Editorial Code Committee – the committee composed mostly of working editors that has responsibility for defining and adapting the press’ code of conduct. In his resignation statement Rusbridger made clear that this was not about politics or personality but borne out of substantive concern for the purpose and role of the PCC.

This is not, by anyone’s measure, a good couple of days. No doubt the rest of the press will, as is their wont went it comes to problems with press self-regulation, ignore it and hope it goes unnoticed. But though speeches can be improved and diplomacy finessed, the substantive issues will not disappear, not least because Baroness Buscombe launched a review of PCC governance in August that will run from now until next spring.

Yet, ironically, the last few days will increase rather than diminish the importance of this review. Not only will the review group need to gather evidence from a wide enough range of individuals and organizations to ensure the review’s credibility, it will have to answer some of the critical questions asked by The Guardian. The Chair of the PCC then has the tricky task – if she accepts the recommendations of the review – of instituting the reforms. If she thinks this week has been difficult, it will be a cakewalk compared to that.

Written by Martin Moore

November 18th, 2009 at 4:58 pm

PCC report shows limits of organisation's remit

without comments

When the Guardian’s story about phone hacking at the News of the World broke in July, the Media Standards Trust called on the press to set up its own independent investigation. Only by doing this, we argued, could the press allay people’s fears that such practices were not widespread at the News of the World, or elsewhere in the industry, and sustain people’s faith in self-regulation.

The problem is, as we said at the time, the way the existing system of self-regulation is currently set up does not allow for such an investigation. The current system – as headed by the Press Complaints Commission – has not the resources, the time, or the remit to conduct the type of inquiry needed.

The PCC cannot call people for evidence, it cannot devote significant amounts of time to in-depth interviews or analysis, and it cannot search through internal emails and correspondence. It has to conduct its inquiries in and around the many other responsibilities it has. It is therefore not a surprise that the report the PCC then produces does not uncover any further evidence of wrongdoing.

This is not a criticism of the day-to-day job the PCC does. Quite the contrary. The PCC does a valuable job dealing with complaints from members of the public about misrepresentation, inaccuracy, harassment and privacy intrusion. Rather, it is a criticism of what the PCC does not do – and cannot do as it is currently structured.

Criticising the PCC in an editorial The Guardian writes that:

‘In reaching its conclusions, it appears the PCC did not interview a single witness or inspect a single document beyond those uncovered by police, the information commissioner or MPs. It did not question Andy Coulson, editor at the time (just as it failed to contact him at the time of Goodman). It did not make inquiries of five other NoW journalists or contractees who had direct knowledge of events – Thurlbeck, Greg Miskiw (who signed the contract), the junior reporter, Goodman or Mulcaire – or, indeed, any other NoW journalist employed at the time. It did not interrogate the bonus contract (News Group said it was confidential). It did not interview – though it said it tried – the detective sergeant or reconcile his remark with other police evidence. Indeed, the solitary successful serious inquiry the PCC itself appears to have made was an exchange of letters with the current NoW editor, Colin Myler, who was not at the paper at the time.’

These are not insubstantial criticisms. They point to an investigation that was limited to letter writing and secondary research (although from the report we do not know the full range of the PCC’s inquiries). An investigation that was, in many ways, similar to the 2007 inquiry following Clive Goodman’s conviction. An inquiry that was itself criticized for not pursuing any of the leads uncovered by the Information Commissioner as a result of Operation Motorman, or for questioning many of the key figures at the News of the World and elsewhere at the time.

The editor of the News of the World, Colin Myler, told the PCC that News International had hired a firm of solicitors, Burton Copeland, to investigate the extent of phone tapping at the News of the World. The newspaper said the firm was given ‘every financial document which could possibly be relevant’ to the paper’s dealings with Mulcaire, and they confirmed that ‘they could find no evidence from these documents or their other enquiries which suggested complicity by the News of the World or other members of its staff beyond Clive Goodman in criminal activities’. Yet one has to ask whether the public are best served by Burton Copeland conducting a private inquiry on behalf of News International, rather than the PCC (or an independent investigator) on behalf of the public.

Press self-regulation, as currently constituted, simply does not allow for the types of investigation necessary to reveal the sorts of privacy intrusion the Guardian alleged, or for giving the public renewed trust in the press.

Self-regulation can work more effectively, and needs to for the sake of the press and the public. The PCC has just started a review of its governance which will, we hope, recommend major reforms to the current system. The Media Standards Trust will be making a submission to this review in which it will set out how we think self-regulation can be made more effective. We would encourage all others who want to see self-regulation work – the Guardian included – to do the same.

Written by Martin Moore

November 9th, 2009 at 12:19 pm

Twitter, Trafigura, and the future of press injunctions

without comments

Goodness knows what it was like from the editors chair. But from where I sat watching the live twitter feed of #trafigura on Tuesday was utterly compelling. First there was the detective work – people trying to figure out, based on the sparse information in the Guardian’s initial article about the absurdly wide-ranging ‘super-injunction’ – which Parliamentary question the paper had been prevented from talking about. The key was ‘Carter Ruck’. Searching through Parliamentary written and oral questions a few bright sparks alighted on Paul Farrelly’s question:

‘”To ask the Secretary of State for Justice what assessment he has made of the effectiveness of legislation to protect (a) whistleblowers and (b) press freedom following the injunctions obtained in the High Court by (i) Barclays and Freshfields solicitors on 19 March 2009 on the publication of internal Barclays reports documenting alleged tax avoidance schemes and (ii) Trafigura and Carter-Ruck solicitors on 11 September 2009 on the publication of the Minton report on the alleged dumping of toxic waste in the Ivory Coast, commissioned by Trafigura.”

They then tweeted about this to see if other people agreed it must be the question, and within hours confirmed with one another this must be it.

From there twitterers started looking for the Minton Report. Again, thanks to the net this was available on Wikileaks – a site unthreatened by injunctions or super-injunctions because its ‘information is distributed across many jurisdictions, organizations and individuals’. As it claims on the site, ‘Once a document is leaked it is essentially impossible to censor’.

Then social media and the power of the ‘link economy’ kicked in. Hundreds, then thousands, of people started posting 140 character messages on Twitter expressing outrage at the injunction and pointing people to the Parliamentary question, articles about Trafigura, and the Minton Report.

Within hours #trafigura had become the number one trending topic in Twitter. In other words more people were tweeting about #trafigura than about anything else in the world. At one point four of the top five trending topics on Twitter were about this story (the fifth was ‘Google Wave’).

Then, shortly after 1pm, the editor of the Guardian posted a tweet saying Carter Ruck had backed down. They would not try to prolong the injunction, and the Guardian was free to mention the parliamentary question.

Much whooping and cheering on Twitter. Stephen Fry, comedian and uber twitterer, tweeted, “Carter-Ruck caves in! Hurrah! Trafigura will deny it had anything to do with Twitter, but we know don’t we?”.

But was it Twitter wot won it? And if so, what does this mean for press freedom and the future of injunctions?

Well, if it wasn’t Twitter then one can safely say it was not the other mainstream media outlets. Almost all other newspapers, and the BBC, remained silent during the course of the morning, prevented from publishing by the ‘super injunction’. The Telegraph broke the silence in the late morning but only to report that ‘Trafigura tops list of Twitter trending topics’. There was no mention in the article of the Guardian, Paul Farrelly, or the injunction.

Newspapers are still relatively easy targets for lawyers. They are institutions, they have their own lawyers. They have editors and journalists who can be sent to jail if they break the law.

Twitterers are a far less easy target. They (or rather ‘we’, since I twitter and was twittering on Tuesday morning) are mostly individuals, not institutions or outlets. To stop twitterers Carter Ruck would have to take on thousands of individuals – many of whom are tweeting pseudonymously. To use a military analogy, it’s like an army fighting a guerilla rather than a conventional war.

Yet will these guerilla twitterers have any substantive impact on the law? Well, if they can do another #trafigura with the next injunction then injunctions may be seen to be increasingly ineffective. But this is unlikely given the Guardian took a risk publishing as much information as it did about this super injunction – and probably only got away with it (if indeed it has got away with it) because the injunction appeared to prevent the paper from reporting on Parliament – a privilege held for over 200 years, since John Wilkes’ famous battle in the 18th century (Carter Ruck subsequently denied they tried to gag Parliamentary reporting). Whether or not a Parliamentary gag was intended, one would certainly hope MPs will now put an end to the so-called ‘super-injunction’.

Most injunctions are not about issues that may be debated in Parliament, and therefore do not raise issues of parliamentary privilege. The Guardian perhaps hopes it can use the Trafigura case as a lever to crank open the whole question of injunctions. Maybe. But Private Eye has been making a lot of noise about injunctions for a long while, yet there use appears to be increasing rather than decreasing.

Moreover, there is an understandable public interest in some injunctions. When a media storm grew around the 13 year old ‘baby father’ in February the family court stepped in and issued an injunction to prevent harm to the children involved in the story.

For commercial injunctions, history has shown that people cling most closely to things they are about to lose. So it is likely to be with these injunctions – at least 12 of which have already been served this year against the Guardian (presumably more against other outlets, though we don’t really know, since they are – by law – secret). Over time, and following similar #trafigura incidents, such injunctions should become less and less effective. But the Carter Rucks of this world will keep doggedly serving them up until it costs their clients more than the benefits of buying a temporary silence.

Still, the twitterati should take heart. This was a victory for press freedom, and a victory won by the power of collective voices.

Written by Martin Moore

October 16th, 2009 at 5:02 pm

Posted in Uncategorized

Tagged with , , ,