top | item 5854593

Using Metadata to Find Paul Revere

457 points| decklin | 12 years ago |kieranhealy.org | reply

73 comments

order
[+] startupfounder|12 years ago|reply
Paul Revere, "in the Fall of 1774 and winter of 1775, I was one of upwards of thirty, chiefly mechanics, who formed ourselves into a Committee for the purpose of watching British soldiers and gaining every intelligence on the movements of the Tories."[1]

Paul Revere was essentially one of the founding members of this country's counter intelligence program against his oppressive government. He was the first in a long line of Mark Felts, Bradley Mannings and Edward Snowdens of this country.

If we have a system where these whistleblowers are stopped before they can leak information on the system that catches them before they can whistleblow there is no turning back. This would not be the UNITED STATES OF AMERICA without Paul Revere and Edward Snowden.

[1]http://en.wikipedia.org/wiki/Intelligence_in_the_American_Re...

[+] tzs|12 years ago|reply
Did Paul Revere gather intelligence concerning both sides, and release everything he had to both sides, so that he would help his enemies as much as he helped his friends, and hurt his friends as much as he hurt his enemies?

If no, then the long line Revere is the first in does not include Manning.

[+] mpyne|12 years ago|reply
You do realize that Paul Revere would by that logic be considered a founding member of the country's counter intelligence program against enemies of the U.S. as well, right? I.e. the NSA.

As far as Revere was concerned he was way past fighting against his government; he was fighting to support his country against Britain's government.

[+] flyinRyan|12 years ago|reply
Yes, but the US government is now in the position the * British government* was then. So telling them that their actions will prevent another Paul Revere isn't going to win them to your side.
[+] LiamMcCalloway|12 years ago|reply
Academic version: http://db.tt/6UA1bXZq

DIVIDE AND CONQUER: DISTORTED COMMUNICATION IN NETWORKS, POWER, AND WEALTH DISTRIBUTION Wilson Perez-Oviedo

Cornell University and Banco Central del Ecuador

Abstract In a society composed of a dictator and its citizens, what are the determinants of the political equilibrium between these two? What are the conditions for a successful citizens’ revolt? What kind of strategies do governments follow to prevent such revolts? The situation of these types of societies can be understood as a game played between a leader, who has to decide the distribution of the aggregate income, and a group of citizens who have the opportunity to revolt if they are unhappy with the distribution. Coordinated action by citizens is possible because they form nodes in a communication network. However, communication through the network is distorted, which could preclude the emergence of collective action among citizens. The network structure and the distortion level are determinants of the political equilibrium and wealth distribution. The model explains how the dictator could use propaganda, cooptation, and repression to increase his expected utility. Finally, the model is illustrated by applying it to cases in Nigeria and Zaire/Congo.

[+] dmix|12 years ago|reply
This paper is excellent. Thanks for sharing.
[+] csears|12 years ago|reply
So could we spoof bogus metadata to hide Paul Revere?

In other words, if a large enough number of people started start calling and texting random Verizon customers, tweeting with random people from the middle east, inviting random people with Muslim-sounding names to Google+ Hangouts, and commenting on every Facebook like with "This ____ is the bomb!"... could that tip the signal to noise ratio enough to defeat this type of analysis?

[+] maxerickson|12 years ago|reply
I think if you had enough people engaged that you could do anything to affect the analysis that you might as well work to eliminate the injustice (rather than just smearing it out across more innocent people).
[+] crgt|12 years ago|reply
Maybe a simple mail client that adds 'extra' metadata to everything you send? Extra recipients, etc? Would add more hay to the haystack.
[+] antoko|12 years ago|reply
This is very clever and all, but from the British perspective - this is exactly what they would have been looking for and Revere would be considered a revolutionary and a real threat, so this is the system working as it is supposed to.

This is the equivalent of saying that the NSA can use such metadata to find muslim extremists that want to kill your children and implement sharia law. (hyperbole courtesy of American media)

Changing the timeline such that "americans" are the underdogs rather than the establishment isn't really helpful... is it? Or is this attempt to stoke revolutionary sentiment?

[+] ant512|12 years ago|reply
It spins the story around in such a way that Americans can relate to the dangers of a surveillance state.

You're suggesting that the NSA is just looking for bad guys and that they should be able to capture evil muslim extremists.

But now the US government decides to bring back alcohol prohibition. Drinking, they say, is immoral, and so are the people who drink. Do you still have nothing to fear?

Abortion gets outlawed, and its supporters are branded as immoral. Still got nothing to fear?

Creationism gets accepted as a viable, scientific alternative to evolution and gets taught in all schools, and people who believe otherwise are dangerous subversives.

Are you now one of the bad guys? Are the NSA now interested in you?

I find it interesting that Americans can believe that the right to bear arms will protect them from the world's largest military, should it ever be run by tyrants, but at the same time believe that the government will always share the same moral code that they themselves hold.

[+] zero_intp|12 years ago|reply
I think you miss the point entirely. Of course the classic Revolutionary position was treasonous to the British Empire.

The function of dialogue like this is to push the metadata conversation into the mainstream. We want the People to understand that giving away our privacy in this data is /JUST/ like taking their guns, and for the same reasons.

[+] angersock|12 years ago|reply
I consider this article not as an attempt to discredit these techniques by making patriotic references, but as an excellent starting point to help people start building instincts about what their "useless metadata" could actually be used for.

The great challenge we face in educating the populace (and indeed, ourselves) is that people lack any intuition for what sorts of data they should or should not broadcast. People are hesitant (I hope!) to give out home addresses to strangers without good cause--we have no such innate hunches about the data and metadata we leak.

[+] outside2344|12 years ago|reply
let's say in the future, Christianity is made illegal. Is the above scary now?
[+] danso|12 years ago|reply
I don't know if David Simon's (creator of "The Wire" and "Treme") defense of the Verizon phone records request ever made it onto HN's page:

Here it is, if you hadn't seen it: http://davidsimon.com/we-are-shocked-shocked/

Apparently it was so controversial that his site crashed from the traffic, and he had to tell everyone to chill out: http://davidsimon.com/nsa-and-fisa-commentary-calling-it/

Anyway, why I thought of that in relation to the OP was, that I think some defenders of the NSA and general government surveillance policies are just unaware of how technology can fundamentally change things...As Google leaders have been known to say, "Speed is a feature"...and so it's not the finding of information that makes the establishment of Google time in human civilization, but how fast Google allows us to do it.

So that said, Simon is one of the journalists I have the absolute highest regard for...I'll be one of the many who think "The Wire" is the best TV drama ever, both for its artistic take and for its illustration of how institutions -- the police, the schools, the drug trade -- corrupt even the best of individuals. "The Wire" is heavily based off of the year that Simon embedded himself in the Baltimore homicide department...the book (which spawned a network TV show) is the best book about the practice of journalism I've ever read. After a year following the detectives, you'd think Simon would be pretty much in cahoots with the police...but he followed up "Homicide" with "The Corner", in which he spent a year embedded with drug dealers and their customers...apparently most of the friendsships he made in the Baltimore Police department evaporated after he published a book bringing sympathy to Baltimore's downtrodden.

Anyway, I don't think Simon has a love for government or authority. But I do think he's a little naive when it comes to advances in technology and their consequences. When "The Wire" started, the police were focused mostly on tapping pay phones. By the time "The Wire" ended, the police were surprised at the advent of camera phones. So when Simon says he thinks the NSA and other law agencies won't abuse their wiretap authority, I believe him...because in much of his experience, the practical obstacles (such as, having to have an officer watch a payphone all day) made it basically impossible for blanket surveillance.

But technology is different...I think Simon's -- and others who I respect -- mistake is to think that the game is being played the same as it always is. It may be the case that the NSA is staffed with as people as good and conscientious as anywhere else...but it's naive to think (as was the primary lesson in "The Wire") that the power they have will lead them astray...and to those of us affected by it, it makes no difference if the violations were intentional or accidental.

Anyway, back to Paul Revere and the OP...I think it's a great example. But of course, what makes that educational scenario feasible is technology and the ability to record information (metadata or whatever you want to call it) in an organized way.

Frankly, I kind of thought anyone who read 1984 would understand how technology changes everything. But yeah, I do think there are some well-meaning people don't grasp the technology, and if they did, they'd have a different opinion about the dangers of unchecked surveillance.

* edit: misspelled conscientious as 'contentious'

* edit: as an example of how much Simon continues to challenge the police as a citizen, here's an essay he wrote after the success of the Wire, in which he tried, as a citizen, to get the basic details of a cop-involved shooting, something that has always been public record. He eventually succeeded, and the revelations about the officer involved ended up jeopardizing the prosecution: http://davidsimon.com/in-baltimore-no-one-left-to-press-the-...

But as you can see if you read the piece, Simon is not to thrilled with how the Internet has displaced newspaper journalism

[+] gohrt|12 years ago|reply
Simon wrote in the comments section on his post:

> Next, if you’ve read carefully you know that I’ve saved my criticism not for this data dump, which I do indeed support, but for the all-enveloping secrecy that prevents proper civilian oversight of the FISA court.

This is called being disingenuous. The problem has two halves: (A) the data, and (B) controls over the data.

Simon is calling everyone foolish because he thinks the problem is B, not A. He completely fails to acknowledge that the problem is A+B, and if haven't won on B, we have to fight to defend A.

I don't mind an NSA with all data and no abuse/breaches. But that's not going to happen!

I don't care if the NSA is completely independent and covert, if they don't have any data to analyze!

And from what I know of human nature, I know where I want to plant my defenses. Just like the people who wrote the Constitution, I prefer to limit the power of the government officials, not give them unlimited trust.

[+] grey-area|12 years ago|reply
I agree that technology changes the game because it's a multipler.

There's also an insidious quality to current surveillance in that it is not just pervasive over space, but over time. Once they have a database like that envisioned in PRISM and the Verizon records, it could be used to travel back through any person's intimate life looking for weak points, failures, emotional attachments, and transgressions, very useful in discrediting political opponents or inconvenient journalists, and very useful in quelling any revolts by citizens before they even begin. These are currently the methods that intelligence agencies use to turn assets and use them for their own ends on a small scale, but should we allow them the tools to use these techniques against an entire population simultaneously and retrospectively? What would the long-term outcome of that sort of power be?

I also do wonder why the methods they use should be secret in the first place? I completely understand the need for operational security on specific operations or details of methods, but if a huge broad program like this is done in a nation's name, surely they should know what is being done and approve or disapprove of it? I can't imagine there are any terrorists the professed target of these programs, who are not aware of widespread monitoring, so I find it hard to believe the top secret nature of the programs is about security so much as avoiding oversight. The avoidance of providing specific details to congress from former BAH executive Clapper is a good illustration of this - if congress can't get broad figures out of them, who are they accountable to?

So the combination of power multiplied by technology and the lack of accountability is the difference between this and tapping telephone lines on an individual basis.

[+] yesbabyyes|12 years ago|reply
In the words attributed to Joseph Stalin: "Quantity has a quality all its own."
[+] bmelton|12 years ago|reply
I just read his recap, which seems to mirror season 1 of "The Wire" fairly closely. What I remember distinctly from that season (which I've seen in total at least three times) is the great degree of difficulty they had in obtaining the warrants, in meeting the burden of proof, and in convincing a judge that the burden was met to a high enough level to allow the kind of carte blanche access to such a broad tap.

Further, they were constantly in fear of running into the expiration date, and the battle was waged on whether they could get data of enough substance before the tap's expiry that they would be able to extend it.

The problem in reconciling the two situations is that in the Baltimore case, assuming the show faithfully retold that portion, it was hard, but in the case of the NSA, their approvals come from the FISC, which numerically is reduced down to a rubber stamp agency. Since their inception in 1979, up to 2012 or so, they have approved over 30,000 requests, and rejected only 11. Further, it seems that they're somewhat unique in a warrant-issuing agency in that they've provided assistance on those 11 such that requests were amended, reissued, and in many cases, granted. So of those 11 denials, I am led to believe that the majority of those proved to be slightly delayed approvals in reality.

Beyond that, I think that Mr. Simon overlooks the capacity of metadata at that scale, and how effective it could be in painting a picture in a way that smaller scale, Baltimore-wide data could never do. If, for example, the British had been able to capture 'only the metadata' of our founding father's meetings in Philadelphia, it wouldn't have taken much analysis at all to have proven what was happening (famous statesmen gathering in Philadelphia court house for overnight meetings, etc.) Just knowing the attendee list alone would have likely been enough to condemn them all for treason or assassination, and the United States might never have existed.

[+] jabbernotty|12 years ago|reply
> but it's naive to think (as was the primary lesson in "The Wire") that the power they have will lead them astray...

Is this what you meant to say? I must admit, I had expected

"but it's naive to think (as was the primary lesson in "The Wire") that the power they have will not lead them astray..."

[+] colin_jack|12 years ago|reply
Good point, and you don't even need to be very imaginative about the future, imagine Google Glass but with the government quietly able to pull down what you are or were seeing.
[+] dfc|12 years ago|reply
Where does it say the revelations about the officer ended up jeopardizing the prosecution? And do you know what the final result of the prosecution was?
[+] msandford|12 years ago|reply
This is a really great counterpoint to the "if you have nothing to hide..." crowd.
[+] mseebach|12 years ago|reply
Well, Paul Revere did have something to hide.
[+] quchen|12 years ago|reply
While being an interesting read, the "old world" scenario was kind confusing me every couple of lines. (Could be because I'm not a native speaker.)
[+] aidenn0|12 years ago|reply
If you're not a native speaker, you're probably also not as well versed in the folklore of the US revolution. Paul Revere is a popular hero of the US revolution, so using him as the target terrorist invokes "one mans terrorist is another mans patriot" argument at the same time as it demonstrates the usage of metadata.
[+] kevinpet|12 years ago|reply
This is a lot of fancy math and graphics that don't differ all that much from what a very simple analysis can give you. Only Revere belonged to five of the groups of interest. Of the three who belonged to four groups, two of them are in "top scorers" on centrality, and the third, with one of the two in centrality, is in the final table.
[+] rollo_tommasi|12 years ago|reply
It's an exceedingly simple example used to illustrate the technique. The methods included in the python library are robust enough to work just as well on much larger and more complicated data-sets but that would be too difficult to follow for someone who isn't familiar with the subject.
[+] rickyconnolly|12 years ago|reply
You ʃhould have uʃed the long ʃ in this article
[+] dfc|12 years ago|reply
kjh the author of this link has a greatintroduction to emacs:

http://kieranhealy.org/resources/emacs-starter-kit.html

He also has a lot of good reference material on latex/org-mode/pandoc. Definitely a good resource to have handy if you ever have a less-techie friend who wants to get away from MS word and its ilk.

[+] jroseattle|12 years ago|reply
Makes me think of the ramifications of false positives.
[+] lettergram|12 years ago|reply
This would mean the larger your network, the larger your likely hood of being a terrorist
[+] pseut|12 years ago|reply
From a predictive standpoint, that's not necessarily an error. Think of it as "the larger your network, the higher priority to investigate for terrorism."