Spy Agencies Are Like Old-School Porn — But That's Changing
In the fight against the Islamic State (IS), some pretty
surprising tools have come to the fore. Teamed up with US forces,
Kurdish militias in Syria have been turning to Google-based maps and
Android devices to direct
US air support. With publicly available tools like these, Kurdish
fighters can record the exact GPS coordinates of the enemy and forward a
map to their US partners, hundreds of miles away, who can then rain
terror — and bombs — on the enemy, and can do so with some measurable
degree of accuracy.
This technologically enhanced partnership is only possible, really, because of open-source information and software, like Google Earth. And it marks a gigantic shift in the way the world — and particularly those within the government — views both espionage and this publicly accessible information. As we move further into the 21st century, one of the biggest shifts will be seen in the intelligence community, which is slowly moving toward more open-source information-gathering tools.
Last month, Raymond Cook was appointed the next chief information officer for the US intelligence community — in short, the top IT guy in the free world. In his new role, one of his greatest challenges will be helping the intelligence community find and utilize more tools geared toward finding and synthesizing open-source intelligence.
Open-source intelligence (OSINT) is information that the intelligence community gathers through open, as opposed to covert, means. Whereas most people consider intelligence to be information obtained by spies operating in deep cover, and bits of data that are known to the fewest number of people possible, OSINT is the complete opposite. OSINT comprises social media posts, blogs, newspaper articles, podcasts, and publicly released memos and documents from foreign governments. It's any and every bit of data that the average person can get their hands on — if they know where to look. And the intelligence community, after years of focusing the majority of their attention on covert information, is taking another look at the value of open source intelligence.
National Geospatial-Intelligence Agency (NGA) Director Robert Cardillo has called the changes within his agency a "seismic shift," but open-source intelligence (OSINT) isn't new, per se. Traditional sources of OSINT include newspapers and radio programs, which were utilized during World War II to gather information on the US's enemies. OSINT was also a major part of the intelligence collected on the Soviet Union during the Cold War.
The biggest change, rather, is in the loosening of control over the data and information that makes up "intelligence." With so much more available via open means, the authority over the information is in flux.
"The value with open-source information is it's not classified, so it can be released without the lengthy declassification process that's often required by intelligence organizations," Eliot Higgins, a.k.a. Brown Moses, the founder of Bellingcat, an open-source news and resource website for citizen investigators, told VICE News.
"By presenting open-source intelligence alongside any public claims an intelligence organization makes, it allows [that organization] to present information that supports their claims," he said. "The more transparent they are, the more authority they have."
But open-source intelligence also means that the intelligence community is losing some of its authority. Because if everyone can access the data, then it doesn't really belong to anyone. And if it belongs to everyone, what's the intelligence community's special claim to fame?
"Authority is a social characteristic. It changes over time," Chris Rasmussen, the NGA's public open-source software development lead, told VICE News. "This institution, does not hold a monopoly, forever, on any particular data that they've produced."
This shift is already underway, at least among some of the country's intelligence agencies. In March, CIA Director John Brennan announced an overhaul of his agency's directorates, including the creation of the Directorate of Digital Innovation, which will oversee digital tradecraft, including the gathering and analysis of open-source information.
"The value of open source to the CIA's mission has never been greater," CIA spokesperson Lyssa Asbill told VICE News. "The open-source environment includes an array of sources of information — including social media — that is increasingly valuable in understanding US and foreign actions and policies."
Asbill said the volume of and diversity of publicly available data is growing, and that the CIA is developing "norms, standards, and expertise" to ensure they can objectively assess the value of the open source information they collect, in order to "contribute to a well-balanced perspective on world events."
The Defense Intelligence Agency (DIA) is also making an effort to embrace OSINT.
"The volume of available open-source information has exploded, and DIA has adjusted its analytical methods to take advantage of open-source data in making its assessments of foreign military capabilities," DIA public affairs officer James Kudla told VICE News. "OSINT provides a vast array of additional information and data to sort and analyze as DIA performs its mission of providing military and defense intelligence to warfighters and decision-makers."
The DIA, he added, is working to integrate open-source information into all of its defense intelligence gathering, analysis, and assessment.
Meanwhile, the National Geospatial-Intelligence Agency (NGA) has started the GEOINT (geospatial intelligence) Pathfinder team. GEOINT is basically intelligence that comes from maps, overhead imagery like satellites, and measurements of things seen in those images. The NGS's Pathfinder team is tasked with seeing what intelligence questions can be answered through unclassified means. In other words, they want to know how much militarily and politically useful intel can be grabbed from Google Maps and the like.
As part of its OSINT efforts, the NGA has been releasing open-source software to GitHub for almost a year, with its latest program, Hootenanny, a crowd-sourced mapping toolkit, released to the open software development site last month.
"The advantage of open-source is, one, to get changes to improve the software and to dip into the biggest transparent bucket possible, and GitHub happens to be the biggest platform of talent available to improve the software," Rasmussen, the NGA's public open-source software development lead and Pathfinder program manager, told VICE News. "It's also a gesture kind of telegraphing where the agency's going on the open-source data side as well."
"If you look at the amount of volunteer geographic data being generated, the agency has to figure out its value proposition in that space," he continued. "We have to figure out a way to talk back, not just consume, and the release of Hootenanny was to show… that the agency has an interest in growing open data and we have an interest in trying to improve software openly."
GEOINT as a form of OSINT isn't terribly new. However, publically available things like Google Maps have taken the control of GEOINT out of the hands of the experts and put it in the hands of the general population, providing data that only a few decades ago would have been largely unavailable to them. Further, sites like Bellingcat and Wikimapia have taken all that commonly available map data like that resident on Google Earth and turned those images into widely available geolocation information — a major building block of GEOINT.
What's the difference? Data includes objective facts free of context or meaning. In terms of Google Maps, data is an address, a geographic plot point. Without context or synthesis, it has no real meaning. Information is what you get when you put that data in context. If, for example, you have data about a building — the size, the shape, the location — you can infer from that data the building's purpose, which would be information.
The authority over GEOINT — geospatial data and information — has completely changed in recent years. Whereas once both data and information would have been controlled, to a great degree, by the intelligence community, the free availability of the data has caused their hold on information to loosen as well. It's a domino effect; as the authority over the raw data shifts, soon, so does the control of the information gleaned from it.
As the authority over information changes, the debate has become not whether open-source is needed, but what role OSINT will play in the greater intelligence sphere.
Robert David Steele, a former Marine intelligence officer and CIA case officer, told VICE News that open-source information has always been the main source of data and information.
"This has been said in writing and publicly by [former Directors of Central Intelligence] Allen Dulles and William Colby, and myself, among others," Steele said. "The problem is that open-source information is not expensive, not secret, and not in English, which means that it is less profitable to the military-industrial-intelligence complex, and also more of a challenge."
Think about it this way: Porn, as an industry, relies on a kind of scarcity to attract paying customers. To start with, the data is about stuff that isn't generally seen — women's fashion no longer covers ankles, so there's not much prurient interest in checking out ankles these days. On top of that, the photographer or director has to take whatever it is and make the imagery not just accessible but take the clinical and make it exciting enough that people will pay cash money to see it.
But now that business model has changed. The Internet has made it easy to find nearly every type of pornographic media online — for free. Moreover, because anyone with a cell phone and an exhibitionist streak can produce and distribute their own pornographic film or photos in a matter of minutes, the pros are forced to compete with amateurs who have zero production costs.
Compare that to the Golden Age of porn in the 1970s, when films like Deep Throat were shown in theaters across the country and magazines like Penthouse and Hustler were selling millions of copies each month. Sure, you can still buy skin flicks and nudie mags, but you don't have to. And the professional pornographers know it. Are there some things you still have to pay to see? Of course. But would you pay for something that's widely available, for free, online? Doubtful. So, in essence, the internet and the ability to shoot video quickly and cheaply has drastically curtailed the power of the gatekeepers — the pornographers.
The traditional intelligence agencies are a lot like the old-school porn industry: they're still trying to make and sell porn like it's 1979, even though a lot of what they're selling is just hanging out online, free for anyone to download at their leisure. To stay relevant, they need to stop wasting money on producing material they or their customers can get for free and develop new ways to find, collect, and curate that sexy, sexy free intelligence information.
This would allow them to save up some cash to work on more highly specialized products, like that special-edition platinum version of Caligula, the one complete with never-before-seen interviews and bonus footage that hasn't been uploaded yet. In other words, they could save their scarce investment and resources for information that can only be obtained through covert methods, while utilizing new open-source methods to take in data that's already been made available if you know where to find it.
While there are factions within the intelligence community that still view OSINT as only a small piece of the intelligence puzzle, there are many outside the IC that find the collection and analysis of open-source data and information to be essential to the development of that vague thing we call intelligence.
Take, for example, Higgins's confirmation, in August 2013, that the Syrian government had used sarin gas on its citizens, firing rockets containing the chemical agent at the rebel-held suburb of Ghouta, and killing as many as 1,700 people. While initially only rumored, Higgins was able to confirm the use of sarin gas through a combination of open-source tools, including obtaining photos and videos locals had posted online and poring over the minute details to determine the exact kind of rockets used, and using the images' metadata to match the photos and videos to the locations of the attacks.
Watch VICE News' Selfie Soldiers: Russia Checks into Ukraine:
This technologically enhanced partnership is only possible, really, because of open-source information and software, like Google Earth. And it marks a gigantic shift in the way the world — and particularly those within the government — views both espionage and this publicly accessible information. As we move further into the 21st century, one of the biggest shifts will be seen in the intelligence community, which is slowly moving toward more open-source information-gathering tools.
Last month, Raymond Cook was appointed the next chief information officer for the US intelligence community — in short, the top IT guy in the free world. In his new role, one of his greatest challenges will be helping the intelligence community find and utilize more tools geared toward finding and synthesizing open-source intelligence.
Open-source intelligence (OSINT) is information that the intelligence community gathers through open, as opposed to covert, means. Whereas most people consider intelligence to be information obtained by spies operating in deep cover, and bits of data that are known to the fewest number of people possible, OSINT is the complete opposite. OSINT comprises social media posts, blogs, newspaper articles, podcasts, and publicly released memos and documents from foreign governments. It's any and every bit of data that the average person can get their hands on — if they know where to look. And the intelligence community, after years of focusing the majority of their attention on covert information, is taking another look at the value of open source intelligence.
National Geospatial-Intelligence Agency (NGA) Director Robert Cardillo has called the changes within his agency a "seismic shift," but open-source intelligence (OSINT) isn't new, per se. Traditional sources of OSINT include newspapers and radio programs, which were utilized during World War II to gather information on the US's enemies. OSINT was also a major part of the intelligence collected on the Soviet Union during the Cold War.
The biggest change, rather, is in the loosening of control over the data and information that makes up "intelligence." With so much more available via open means, the authority over the information is in flux.
"The value with open-source information is it's not classified, so it can be released without the lengthy declassification process that's often required by intelligence organizations," Eliot Higgins, a.k.a. Brown Moses, the founder of Bellingcat, an open-source news and resource website for citizen investigators, told VICE News.
"By presenting open-source intelligence alongside any public claims an intelligence organization makes, it allows [that organization] to present information that supports their claims," he said. "The more transparent they are, the more authority they have."
But open-source intelligence also means that the intelligence community is losing some of its authority. Because if everyone can access the data, then it doesn't really belong to anyone. And if it belongs to everyone, what's the intelligence community's special claim to fame?
"Authority is a social characteristic. It changes over time," Chris Rasmussen, the NGA's public open-source software development lead, told VICE News. "This institution, does not hold a monopoly, forever, on any particular data that they've produced."
This shift is already underway, at least among some of the country's intelligence agencies. In March, CIA Director John Brennan announced an overhaul of his agency's directorates, including the creation of the Directorate of Digital Innovation, which will oversee digital tradecraft, including the gathering and analysis of open-source information.
"The value of open source to the CIA's mission has never been greater," CIA spokesperson Lyssa Asbill told VICE News. "The open-source environment includes an array of sources of information — including social media — that is increasingly valuable in understanding US and foreign actions and policies."
Asbill said the volume of and diversity of publicly available data is growing, and that the CIA is developing "norms, standards, and expertise" to ensure they can objectively assess the value of the open source information they collect, in order to "contribute to a well-balanced perspective on world events."
The Defense Intelligence Agency (DIA) is also making an effort to embrace OSINT.
"The volume of available open-source information has exploded, and DIA has adjusted its analytical methods to take advantage of open-source data in making its assessments of foreign military capabilities," DIA public affairs officer James Kudla told VICE News. "OSINT provides a vast array of additional information and data to sort and analyze as DIA performs its mission of providing military and defense intelligence to warfighters and decision-makers."
The DIA, he added, is working to integrate open-source information into all of its defense intelligence gathering, analysis, and assessment.
Meanwhile, the National Geospatial-Intelligence Agency (NGA) has started the GEOINT (geospatial intelligence) Pathfinder team. GEOINT is basically intelligence that comes from maps, overhead imagery like satellites, and measurements of things seen in those images. The NGS's Pathfinder team is tasked with seeing what intelligence questions can be answered through unclassified means. In other words, they want to know how much militarily and politically useful intel can be grabbed from Google Maps and the like.
As part of its OSINT efforts, the NGA has been releasing open-source software to GitHub for almost a year, with its latest program, Hootenanny, a crowd-sourced mapping toolkit, released to the open software development site last month.
"The advantage of open-source is, one, to get changes to improve the software and to dip into the biggest transparent bucket possible, and GitHub happens to be the biggest platform of talent available to improve the software," Rasmussen, the NGA's public open-source software development lead and Pathfinder program manager, told VICE News. "It's also a gesture kind of telegraphing where the agency's going on the open-source data side as well."
"If you look at the amount of volunteer geographic data being generated, the agency has to figure out its value proposition in that space," he continued. "We have to figure out a way to talk back, not just consume, and the release of Hootenanny was to show… that the agency has an interest in growing open data and we have an interest in trying to improve software openly."
GEOINT as a form of OSINT isn't terribly new. However, publically available things like Google Maps have taken the control of GEOINT out of the hands of the experts and put it in the hands of the general population, providing data that only a few decades ago would have been largely unavailable to them. Further, sites like Bellingcat and Wikimapia have taken all that commonly available map data like that resident on Google Earth and turned those images into widely available geolocation information — a major building block of GEOINT.
What's the difference? Data includes objective facts free of context or meaning. In terms of Google Maps, data is an address, a geographic plot point. Without context or synthesis, it has no real meaning. Information is what you get when you put that data in context. If, for example, you have data about a building — the size, the shape, the location — you can infer from that data the building's purpose, which would be information.
The authority over GEOINT — geospatial data and information — has completely changed in recent years. Whereas once both data and information would have been controlled, to a great degree, by the intelligence community, the free availability of the data has caused their hold on information to loosen as well. It's a domino effect; as the authority over the raw data shifts, soon, so does the control of the information gleaned from it.
As the authority over information changes, the debate has become not whether open-source is needed, but what role OSINT will play in the greater intelligence sphere.
Robert David Steele, a former Marine intelligence officer and CIA case officer, told VICE News that open-source information has always been the main source of data and information.
"This has been said in writing and publicly by [former Directors of Central Intelligence] Allen Dulles and William Colby, and myself, among others," Steele said. "The problem is that open-source information is not expensive, not secret, and not in English, which means that it is less profitable to the military-industrial-intelligence complex, and also more of a challenge."
Think about it this way: Porn, as an industry, relies on a kind of scarcity to attract paying customers. To start with, the data is about stuff that isn't generally seen — women's fashion no longer covers ankles, so there's not much prurient interest in checking out ankles these days. On top of that, the photographer or director has to take whatever it is and make the imagery not just accessible but take the clinical and make it exciting enough that people will pay cash money to see it.
But now that business model has changed. The Internet has made it easy to find nearly every type of pornographic media online — for free. Moreover, because anyone with a cell phone and an exhibitionist streak can produce and distribute their own pornographic film or photos in a matter of minutes, the pros are forced to compete with amateurs who have zero production costs.
Compare that to the Golden Age of porn in the 1970s, when films like Deep Throat were shown in theaters across the country and magazines like Penthouse and Hustler were selling millions of copies each month. Sure, you can still buy skin flicks and nudie mags, but you don't have to. And the professional pornographers know it. Are there some things you still have to pay to see? Of course. But would you pay for something that's widely available, for free, online? Doubtful. So, in essence, the internet and the ability to shoot video quickly and cheaply has drastically curtailed the power of the gatekeepers — the pornographers.
The traditional intelligence agencies are a lot like the old-school porn industry: they're still trying to make and sell porn like it's 1979, even though a lot of what they're selling is just hanging out online, free for anyone to download at their leisure. To stay relevant, they need to stop wasting money on producing material they or their customers can get for free and develop new ways to find, collect, and curate that sexy, sexy free intelligence information.
This would allow them to save up some cash to work on more highly specialized products, like that special-edition platinum version of Caligula, the one complete with never-before-seen interviews and bonus footage that hasn't been uploaded yet. In other words, they could save their scarce investment and resources for information that can only be obtained through covert methods, while utilizing new open-source methods to take in data that's already been made available if you know where to find it.
While there are factions within the intelligence community that still view OSINT as only a small piece of the intelligence puzzle, there are many outside the IC that find the collection and analysis of open-source data and information to be essential to the development of that vague thing we call intelligence.
Take, for example, Higgins's confirmation, in August 2013, that the Syrian government had used sarin gas on its citizens, firing rockets containing the chemical agent at the rebel-held suburb of Ghouta, and killing as many as 1,700 people. While initially only rumored, Higgins was able to confirm the use of sarin gas through a combination of open-source tools, including obtaining photos and videos locals had posted online and poring over the minute details to determine the exact kind of rockets used, and using the images' metadata to match the photos and videos to the locations of the attacks.
Watch VICE News' Selfie Soldiers: Russia Checks into Ukraine:
"If you consider the situation on the ground, there was very little access to the sites the attacks occurred at for journalists or independent investigators, so all the information was coming through YouTube, Facebook, and Twitter," Higgins told VICE News. "Within days of the attack, I had over 200 videos collected, multiple photographs, and social media posts with details, including images of the munitions used."
Higgins said that previous videos from others areas on earlier dates showed the Syrian government forces had used the same type of munition in earlier alleged chemical attacks, which he said were "broadly ignored in the media."
"This munition had never been seen anywhere else in the world, and appears to be unique to the Syrian military, and we know this because of social media posts and YouTube videos predating the August 21 attacks," he said.
Open-source information was the only reason Higgins — and later, others — was able to prove, beyond a doubt, that sarin gas had been used by the Syrian government. And the availability of that information was only possible because of the fairly recent technological developments — like social media posts and, primary to open-source data collection in the 21st century, the Internet — that are pushing OSINT to the forefront of intelligence.
Higgins says that without these technological advances, little to none of the information about the Syrian government's gas attacks would have been available.
"The information coming from the areas attacked would be tiny, and it's unlikely the link between the munitions used and early chemical attacks would have been established, and the connection between the munition and the Syrian military [would have been equally unlikely]," he said.
Following Higgins' lead, many intelligence agencies are slowly — very, very slowly, some would say — moving toward incorporating more open-source data in their intelligence. But OSINT has been a point of discussion in the community for decades. In 1949, CIA analyst Sherman Kent commented in the book Strategic Intelligence for American World Policy, that new technology was producing greater amounts of information that needed to be sorted through.
More than 40 years later, Steele called for the international community to pay more attention to open sources. But in 2015, the international community has still not fully take advantage of open-source.
"Think of it as the perennial challenge of every analyst in response to changing information technologies," Dr. Stephen Marrin, a former CIA analyst, told VICE News.
"In absolute terms, the amount of data may be increasing, but [so] is the ability of the analyst in sifting, sorting, and searching through the data," he said. "Because of these types of technological assists, I think the actual analytic challenge remains consistent over time."
What has changed, Marrin said, is how technology is changing the value of openly available information at some cost relative to the value of information acquired secretly.
"As long as there is value in knowing others' secrets, then there will be a need for intelligence organizations precisely because people will hide their secrets from information collection platforms that make their data available to others," Marrin said.
In other words, OSINT isn't magic. It can't tell you things that people aren't putting out there. And as the amount of available information grows, those with real secrets will be a bit more gun-shy about letting those secrets slip. No amount of open-source finagling can get you information that hasn't been made public in some form.
Open Source Intelligence, Marrin said, is not the death of every other type of intelligence as it were. Instead, it's an opportunity for the international community to step up their game with regard to the collection of the ever-more-prevalent open-source information so that they can better utilize the covert material that's so integral to their business.
"There's always going to be a need for the really spooky stuff," Rasmussen told VICE News. "The information that's available in the open needs to be the base and the default. Financially it's the right thing to do, and as far as information flow, it's the right thing to do, and [we] can generate better output by doing so."
Higgins finds the one-two punch of OSINT mixed with more traditional covert intelligence equally compelling.
"As with journalism, I think it can complement and support other sources of information, and also allow intelligence agencies to present more information to the public, rather than relying on 'we have a source, trust us,' because the public doesn't trust them," he said.
While opinions differ on how much open-source information is enough, and what is needed for information to be considered proper intelligence, what everyone can agree on is that finding a way to better utilize OSINT can and will lead to better overall intelligence.
But the challenge, going forward, will not only be for the agencies that make up the intelligence community to find the right balance between open-source and covertly acquired data, but for them to come to terms with their slipping authority over the building blocks of intelligence.
Follow Jennifer Peters on Twitter: @EditrixJen
