When Is Transparency Useful?

The following essay appears in the new O’Reilly book Open Government and attempts to combine and clarify some of the points I made in previous essays. It was written in June 2009.

Transparency is a slippery word; the kind of word that, like reform, sounds good and so ends up getting attached to any random political thing that someone wants to promote. But just as it’s silly to talk about whether “reform” is useful (it depends on the reform), talking about transparency in general won’t get us very far. Everything from holding public hearings to requiring police to videotape interrogations can be called “transparency”—there’s not much that’s useful to say about such a large category.

In general, you should be skeptical whenever someone tries to sell you on something like “reform” or “transparency.” In general, you should be skeptical. But in particular, reactionary political movements have long had a history of cloaking themselves in nice words. Take the Good Government (goo-goo) movement early in the twentieth century. Funded by prominent major foundations, it claimed that it was going to clean up the corruption and political machines that were hindering city democracy. Instead, the reforms ended up choking democracy itself, a response to the left-wing candidates who were starting to get elected.

The goo-goo reformers moved elections to off-years. They claimed this was to keep city politics distinct from national politics, but the real effect was just to reduce turnout. They stopped paying politicians a salary. This was supposed to reduce corruption, but it just made sure that only the wealthy could run for office. They made the elections nonpartisan. Supposedly this was because city elections were about local issues, not national politics, but the effect was to increase the power of name recognition and make it harder for voters to tell which candidate was on their side. And they replaced mayors with unelected city managers, so winning elections was no longer enough to effect change.¹

Of course, the modern transparency movement is very different from the Good Government movement of old. But the story illustrates that we should be wary of kind nonprofits promising to help. I want to focus on one particular strain of transparency thinking and show how it can go awry. It starts with something that’s hard to disagree with.

Sharing Documents with the Public

Modern society is made of bureaucracies and modern bureaucracies run on paper: memos, reports, forms, filings. Sharing these internal documents with the public seems obviously good, and indeed, much good has come out of publishing these documents, whether it’s the National Security Archive, whose Freedom of Information Act (FOIA) requests have revealed decades of government wrongdoing around the globe, or the indefatigable Carl Malamud and his scanning, which has put terabytes of useful government documents, from laws to movies, online for everyone to access freely.

I suspect few people would put “publishing government documents on the Web” high on their list of political priorities, but it’s a fairly cheap project (just throw piles of stuff into scanners) and doesn’t seem to have much downside. The biggest concern—privacy—seems mostly taken care of. In the United States, FOIA and the Privacy Act (PA) provide fairly clear guidelines for how to ensure disclosure while protecting people’s privacy.

Perhaps even more useful than putting government documents online would be providing access to corporate and nonprofit records. A lot of political action takes place outside the formal government, and thus outside the scope of the existing FOIA laws. But such things seem totally off the radar of most transparency activists; instead, giant corporations that receive billions of dollars from the government are kept impenetrably secret.

Generating Databases for the Public

Many policy questions are a battle of competing interests—drivers don’t want cars that roll over and kill them when they make a turn, but car companies want to keep selling such cars. If you’re a member of Congress, choosing between them is difficult. On the one hand are your constituents, who vote for you. But on the other hand are big corporations, which fund your reelection campaigns. You really can’t afford to offend either one too badly.

So, there’s a tendency for Congress to try a compromise. That’s what happened with, for example, the Transportation Recall Enhancement, Accountability, and Documentation (TREAD) Act. Instead of requiring safer cars, Congress simply required car companies to report how likely their cars were to roll over. Transparency wins again!

Or, for a more famous example: after Watergate, people were upset about politicians receiving millions of dollars from large corporations. But, on the other hand, corporations seem to like paying off politicians. So instead of banning the practice, Congress simply required that politicians keep track of everyone who gives them money and file a report on it for public inspection.

I find such practices ridiculous. When you create a regulatory agency, you put together a group of people whose job is to solve some problem. They’re given the power to investigate who’s breaking the law and the authority to punish them. Transparency, on the other hand, simply shifts the work from the government to the average citizen, who has neither the time nor the ability to investigate these questions in any detail, let alone do anything about it. It’s a farce: a way for Congress to look like it has done something on some pressing issue without actually endangering its corporate sponsors.

Interpreting Databases for the Public

Here’s where the technologists step in. “Something is too hard for people?” they hear. “We know how to fix that.” So they download a copy of the database and pretty it up for public consumption—generating summary statistics, putting nice pictures around it, and giving it a snazzy search feature and some visualizations. Now inquiring citizen can find out who’s funding their politicians and how dangerous their cars are just by going online.

The wonks love this. Still stinging from recent bouts of deregulation and antigovernment zealotry, many are now skeptical about government. “We can’t trust the regulators,” they say. “We need to be able to investigate the data for ourselves.” Technology seems to provide the perfect solution. Just put it all online—people can go through the data while trusting no one.

There’s just one problem: if you can’t trust the regulators, what makes you think you can trust the data?

The problem with generating databases isn’t that they’re too hard to read; it’s the lack of investigation and enforcement power, and websites do nothing to help with that. Since no one’s in charge of verifying them, most of the things reported in transparency databases are simply lies. Sometimes they’re blatant lies, like how some factories keep two sets of books on workplace injuries: one accurate one, reporting every injury, and one to show the government, reporting just 10% of them.² But they can easily be subtler: forms are misfiled or filled with typos, or the malfeasance is changed in such a way that it no longer appears on the form. Making these databases easier to read results only in easier-to-read lies.

Three examples:

• Congress’s operations are supposedly open to the public, but if you visit the House floor (or if you follow what they’re up to on one of these transparency sites) you find that they appear to spend all their time naming post offices. All the real work is passed using emergency provisions and is tucked into subsections of innocuous bills. (The bank bailouts were put in the Paul Wellstone Mental Health Act.) Matt Taibbi’s The Great Derangement tells the story.

• Many of these sites tell you who your elected official is, but what impact does your elected official really have? For 40 years, people in New York thought they were governed by their elected officials—their city council, their mayor, their governor. But as Robert Caro revealed in The Power Broker, they were all wrong. Power in New York was controlled by one man, a man who had consistently lost every time he’d tried to run for office, a man nobody thought of as being in charge at all: Parks Commissioner Robert Moses.

• Plenty of sites on the Internet will tell you who your representative receives money from, but disclosed contributions are just the tip of the iceberg. As Ken Silverstein points out in his series of pieces for Harper’s (some of which he covers in his book Turkmeniscam), being a member of Congress provides for endless ways to get perks and cash while hiding where it comes from.

Fans of transparency try to skirt around this. “OK,” they say, “but surely some of the data will be accurate. And even if it isn’t, won’t we learn something from how people lie?” Perhaps that’s true, although it’s hard to think of any good examples. (In fact, it’s hard to think of any good examples of transparency work accomplishing anything, except perhaps for more transparency.) But everything has a cost.

Hundreds of millions of dollars have been spent funding transparency projects around the globe. That money doesn’t come from the sky. The question isn’t whether some transparency is better than none; it’s whether transparency is really the best way to spend these resources, whether they would have a bigger impact if spent someplace else.

I tend to think they would. All this money has been spent with the goal of getting a straight answer, not of doing anything about it. Without enforcement power, the most readable database in the world won’t accomplish much—even if it’s perfectly accurate. So people go online and see that all cars are dangerous and that all politicians are corrupt. What are they supposed to do then?

Sure, perhaps they can make small changes—this politician gets slightly less oil money than that one, so I’ll vote for her (on the other hand, maybe she’s just a better liar and gets her oil money funneled through PACs or foundations or lobbyists)—but unlike the government, they can’t solve the bigger issue: a bunch of people reading a website can’t force car companies to make a safe car. You’ve done nothing to solve the real problem; you’ve only made it seem more hopeless: all politicians are corrupt, all cars are dangerous. What can you do?

An Alternative

What’s ironic is that the Internet does provide something you can do. It has made it vastly easier, easier than ever before, to form groups with people and work together on common tasks. And it’s through people coming together—not websites analyzing data—that real political progress can be made.

So far we’ve seen baby steps—people copying what they see elsewhere and trying to apply it to politics. Wikis seem to work well, so you build a political wiki. Everyone loves social networks, so you build a political social network. But these tools worked in their original setting because they were trying to solve particular problems, not because they’re magic. To make progress in politics, we need to think best about how to solve its problems, not simply copy technologies that have worked in other fields. Data analysis can be part of it, but it’s part of a bigger picture. Imagine a team of people coming together to tackle some issue they care about—food safety, say. You can have technologists poring through safety records, investigative reporters making phone calls and sneaking into buildings, lawyers subpoenaing documents and filing lawsuits, political organizers building support for the project and coordinating volunteers, members of Congress pushing for hearings on your issues and passing laws to address the problems you uncover, and, of course, bloggers and writers to tell your stories as they unfold.

Imagine it: an investigative strike team, taking on an issue, uncovering the truth, and pushing for reform. They’d use technology, of course, but also politics and the law. At best, a transparency law gets you one more database you can look at. But a lawsuit (or congressional investigation)? You get to subpoena all the databases, as well as the source records behind them, then interview people under oath about what it all means. You get to ask for what you need, instead of trying to predict what you may someday want.

This is where data analysis can be really useful. Not in providing definitive answers over the Web to random surfers, but in finding anomalies and patterns and questions that can be seized upon and investigated by others. Not in building finished products, but by engaging in a process of discovery. But this can be done only when members of this investigative strike team work in association with others. They would do what it takes to accomplish their goals, not be hamstrung by arbitrary divisions between “technology” and “journalism” and “politics.”

Right now, technologists insist that they’re building neutral platforms for anyone to find data on any issue. Journalists insist that they’re objective observers of the facts. And political types assume they already know the answers and don’t need to investigate further questions. They’re each in their own silo, unable to see the bigger picture.

I certainly was. I care passionately about these issues—I don’t want politicians to be corrupt; I don’t want cars to kill people—and as a technologist I’d love to be able to solve them. That’s why I got swept up in the promise of transparency. It seemed like just by doing the things I knew how to do best—write code, sift through databases—I could change the world.

But it just doesn’t work. Putting databases online isn’t a silver bullet, as nice as the word transparency may sound. But it was easy to delude myself. All I had to do was keep putting things online and someone somewhere would find a use for them. After all, that’s what technologists do, right? The World Wide Web wasn’t designed for publishing the news—it was designed as a neutral platform that could support anything from scientific publications to pornography.

Politics doesn’t work like that. Perhaps at some point putting things on the front page of the New York Times guaranteed that they would be fixed, but that day is long past. The pipeline of leak to investigation to revelation to report to reform has broken down. Technologists can’t depend on journalists to use their stuff; journalists can’t depend on political activists to fix the problems they uncover. Change doesn’t come from thousands of people, all going their separate ways. Change requires bringing people together to work on a common goal. That’s hard for technologists to do by themselves.

But if they do take that as their goal, they can apply all their talent and ingenuity to the problem. They can measure their success by the number of lives that have been improved by the changes they fought for, rather than the number of people who have visited their website. They can learn which technologies actually make a difference and which ones are merely indulgences. And they can iterate, improve, and scale.

Transparency can be a powerful thing, but not in isolation. So, let’s stop passing the buck by saying our job is just to get the data out there and it’s other people’s job to figure out how to use it. Let’s decide that our job is to fight for good in the world. I’d love to see all these amazing resources go to work on that.

Thanks to Andy Oram and Andy Eggers for their insightful comments on earlier versions of this essay.

You should follow me on twitter here.

February 11, 2010