Netflix, Facebook — and the NSA: They’re all in it together
On June 9, the Wall Street Journal reported that for the last few years the National Security Agency has been relying on a software program with the quirky name Hadoop to help it make sense of its enormous collections of data. Named after a toy elephant that belonged to the child of one of the original developers of the program, Hadoop, reported the Journal, is a crucial part of a computing and software revolution a piece of free software that lets users distribute big-data projects across hundreds or thousands of computers.
Revolution is probably the most overused word in the chronicle of Internet history, but if anything, the Wall Street Journal undersold the real story. Hadoops importance to how we live our lives today is hard to overstate. By making it economically feasible to extract meaning from the massive streams of data that increasingly define our online existence, Hadoop effectively enabled the surveillance state.
And not just in the narrowest, Big Brother, government-is-watching-everyone-all-the-time sense of that term. Hadoop is equally critical to private sector corporate surveillance. Facebook, Twitter, Yahoo, Amazon, Netflix just about every big player that gathers the trillions of data events generated by our everyday online actions employs Hadoop as a part of their arsenal of Big Data-crunching tools. Hadoop is everywhere as one programmer told me, its taken over the world.
The Journals description of Hadoop as a piece of free software barely scratches the surface of the significance of this particular batch of code. In the past half-decade Hadoop has emerged as one of the triumphs of the non-proprietary, open-source software programming methodology that previously gave us the Apache Web server, the Linux operating system and the Firefox browser. Hadoop belongs to nobody. Anyone can copy it, modify, extend it as they please. Funny, that: A software program developed collaboratively by programmers who believe that their code should be shared in as open and transparent a process as possible has resulted in the creation of tools that everyone from the NSA to Facebook uses to annihilate any semblance of individual privacy. But whats even more ironic, and fascinating, is the sight of intelligence agencies like the NSA and CIA joining in and becoming integral players in the world of open source big data software. The NSA doesnt just use Hadoop. NSA programmers have improved and extended Hadoop and donated their changes and additions back to the larger community. The CIA actively invests in start-ups that are commercializing Hadoop and other open source projects.
Theyre all in it together. The spooks and the social media titans and the online commerce goliaths are collaborating to improve data-crunching software tools that enable the tracking of our behavior in fantastically intimate ways that simply werent possible as recently as four or five years ago. Its a new military industrial open source Big Data complex. The gift economy has delivered us the surveillance state.
Hadoops earliest roots go back to 2002, when Doug Cutting, then the search director at the Internet Archive, and Michael Cafarella, a graduate student at the University of Washington, started working on an open-source search engine called Nutch. But the project did not get serious traction until Cutting joined Yahoo and began to merge his work into Yahoos larger strategic goal of improving its search engine technology so as to better compete with Google. Significantly, Yahoo executives decided not to make the project proprietary. In 2006, they blessed the formation of Hadoop, an open-source project managed under the auspices of the Apache Software Foundation. (For a much more detailed look at the history of Hadoop, please read this four-part history of Hadoop at GigaOm.)
Hadoop is basically a nifty hack. The definition, per Wikipedia, is surprisingly simple: It supports the running of applications on large clusters of commodity hardware. Bottom line, Hadoop provides a means for distributing both the storage and processing of an enormous amount of data over lots and lots of relatively inexpensive computers. Hadoop turned out to be cheap, fast and scalable meaning it could expand smoothly in capacity as the flows of data it was crunching burgeoned in size, simply though plugging in extra computers to the network. Hadoop was also fundamentally modular different parts of it could be easily replaced by custom designed chunks of software, making it seamlessly adaptable to the individual circumstances of different corporations or government agencies.
Hadoops debut was timely, addressing not only the problems Yahoo faced in managing the enormous amounts of data produced by its users, but also those that the entire Internet industry was simultaneously struggling to cope with. Basically, the Internet had become a victim of its own success. The enormous flows of data generated by users of the likes of Facebook and Twitter far overwhelmed the ability of those companies to make sense of it. There was too much coming in too fast. Hadoop helped companies cope with the tsunami it was, in the words of Jeff Hammerbacher, an early employee of Facebook, our tool for exploiting the unreasonable effectiveness of data.
Before Hadoop, you were at the mercy of your data. After Hadoop, you were in charge. You could figure out all kinds of interesting things. You could recognize patterns in the data and start to make inferences about what might happen if you made tweaks to your product. What did users do when the interface was adjusted like this? What kinds of ads made them more likely to pull out their credit cards? What did that batch of millions of Verizon calls reveal about the formation of a potential terrorist cell? Facebook wouldnt be able to exploit the insights of its so-called social graph without tools like Hadoop.
Hadoop has become the de facto standard tool for cost-effectively processing Big Data, says Raymie Stata, who served as chief technology officer at Yahoo before eventually starting his own Hadoop-focused start-up, Altiscale. And the significance of being able to cheaply process Big Data, to accurately measure what your users are doing, he added, is a big deal.
Originally posted here:
Netflix, Facebook — and the NSA: They’re all in it together
- Apple announces software update to AirPods Max, and you can get it for free; heres how - Fortune India - March 25th, 2025 [March 25th, 2025]
- Download Free PDF Reader (free) for Windows, macOS and Linux - Gizmodo - March 25th, 2025 [March 25th, 2025]
- 5 best free alternatives to Adobe creative software you should use instead - XDA Developers - March 25th, 2025 [March 25th, 2025]
- Best free Adobe Illustrator alternatives of 2025 - TechRadar - March 25th, 2025 [March 25th, 2025]
- Ocean Swift revives its free Legacy Synthesizer plugins with VST3 support: part 1 bundle - Synth Anatomy - March 25th, 2025 [March 25th, 2025]
- I've found the easiest way to learn Blackmagic's Da Vinci Resolve 19 and it's free - Creative Bloq - March 25th, 2025 [March 25th, 2025]
- Download PDFgear (free) for Windows, macOS, Android, iOS and Web App - Gizmodo - March 25th, 2025 [March 25th, 2025]
- U-he Tyrell N6 3.0, free Synthesizer plugin gets major update with Apple Silicon support, and more - Synth Anatomy - March 25th, 2025 [March 25th, 2025]
- GIMP 3.0 Is Here The Best Free Graphics Editor Just Got Better - 9Meters.com - March 18th, 2025 [March 18th, 2025]
- File Your Tax Return for Free: What to Know About the IRS Free File Program and Its Limitations - CNET - March 18th, 2025 [March 18th, 2025]
- Freeware image editor GIMP 3.0 arrives after seven years of incubation - Tom's Hardware - March 18th, 2025 [March 18th, 2025]
- A Perfect Day - for iOS - Free download and software reviews - Download.com - March 18th, 2025 [March 18th, 2025]
- Oscar winner Gints Zilbalodis: Its really cool that we can make these films with free software - The Irish Times - March 18th, 2025 [March 18th, 2025]
- A government program made tax filing free and more efficient. Musk and DOGE may get rid of it anyway - MyFoxZone.com KIDY - March 18th, 2025 [March 18th, 2025]
- Winner of Best Animated Film at the 2025 Oscars Was Made on Free Software - The Express Tribune - March 18th, 2025 [March 18th, 2025]
- Deep Research could be the next Gemini feature to hit free users, per report - Android Central - March 9th, 2025 [March 9th, 2025]
- As Flow takes home the Oscar using only free software, fans troll"Disneys worst nightmare is indie animators with talent" - Soap Central - March 9th, 2025 [March 9th, 2025]
- The best animation Oscar winner was made in totally free software that anyone can use - Yahoo Entertainment - March 9th, 2025 [March 9th, 2025]
- Best free WinZip alternative of 2025 - TechRadar - March 9th, 2025 [March 9th, 2025]
- YouTube's affordable ad-free Premium Lite plan officially rolls out in the U.S. - Android Central - March 9th, 2025 [March 9th, 2025]
- 303 Day: Get Rolands TB-303 software version absolutely free but youll have to be quick - MusicTech - March 9th, 2025 [March 9th, 2025]
- Everybody needs a 303, and Roland is giving away its software version free for 303 Day but be quick, theres only 3,030 copies up for grabs -... - March 3rd, 2025 [March 3rd, 2025]
- News: Free QNX Everywhere software resources now available - A3 Association for Advancing Automation - March 1st, 2025 [March 1st, 2025]
- How do I file my taxes for free? Federal and Ohio state services to know about this year - The Columbus Dispatch - March 1st, 2025 [March 1st, 2025]
- How to file your taxes for free in 2025 - CNBC - March 1st, 2025 [March 1st, 2025]
- Microsoft quietly tests free, ad-supported version of Office apps for Windows with limited functionality - Windows Central - March 1st, 2025 [March 1st, 2025]
- Empty Out Your Gmail Inbox and Get Back 15GB of Storage - CNET - March 1st, 2025 [March 1st, 2025]
- Google releases free version of AI platform that speeds coding - Business in Vancouver - March 1st, 2025 [March 1st, 2025]
- H&R Block vs. TurboTax vs. Jackson Hewitt: Whats the Difference? - Investopedia - February 25th, 2025 [February 25th, 2025]
- All the Ways You Can File for Free This Year, From TurboTax to FreeTaxUSA - CNET - February 18th, 2025 [February 18th, 2025]
- Best free video editing software of 2025: Top picks for every project and skill-level - TechRadar - February 14th, 2025 [February 14th, 2025]
- 500,000 U.S. Lawyers Now Have Free Access to Trust Software through Bar Partnerships with Smokeball - LawSites - February 14th, 2025 [February 14th, 2025]
- Best Tax Software 2025: TurboTax Leads the Pack, but These Options May Work Better for You - CNET - February 14th, 2025 [February 14th, 2025]
- Photopea Is a Free Photoshop Alternative That Runs in the Browser - WIRED - February 14th, 2025 [February 14th, 2025]
- Freedom Reimagined: Meet the Free Software Foundations 40th Anniversary Logo - It's FOSS News - January 24th, 2025 [January 24th, 2025]
- Free Software Foundation Marking 40 Years Old With A New Logo - Phoronix - January 24th, 2025 [January 24th, 2025]
- Coros smartwatches just got a big free software update here are the best new features - MSN - January 24th, 2025 [January 24th, 2025]
- Best personal finance software of 2025 - TechRadar - January 24th, 2025 [January 24th, 2025]
- Free Mac Email Apps That Stand Out in 2025: A Comprehensive Guide - PUNE.NEWS - January 24th, 2025 [January 24th, 2025]
- Free-software warriors celebrate landmark case that enforced GNU LGPL - The Register - January 13th, 2025 [January 13th, 2025]
- This free software is topping the Steam charts, but its not a game - Notebookcheck.net - January 13th, 2025 [January 13th, 2025]
- IRS offering free tax filing services to millions starting this week - KSWO - January 13th, 2025 [January 13th, 2025]
- The best Android antivirus apps in 2025 - Tom's Guide - January 13th, 2025 [January 13th, 2025]
- GIMP vs Krita: which free software is best for you? - Creative Bloq - January 6th, 2025 [January 6th, 2025]
- Mensla MS-3, free waveshaper Synthesizer plugin for macOS and Windows - Synth Anatomy - January 6th, 2025 [January 6th, 2025]
- Tesla fixes TPMS issue on nearly 700,000 vehicles with free software update - Drive Tesla Canada - December 25th, 2024 [December 25th, 2024]
- STRACKALINE TO OFFER EXCLUSIVE FREE SOFTWARE ACCESS AT THE 2025 PGA SHOW (BOOTH 2808) - The Golf Wire - December 18th, 2024 [December 18th, 2024]
- The Pixel 6 just got a free software upgrade that makes it my favorite budget Android phone - ZDNet - December 12th, 2024 [December 12th, 2024]
- Google just gave older Pixel phones a free software upgrade that you once could only wish for - ZDNet - December 8th, 2024 [December 8th, 2024]
- Free AI-Powered Software for Radiology Impressions Available from Scriptor Software - Imaging Technology News - December 8th, 2024 [December 8th, 2024]
- Maryland State Bar Members Now Get Free Trust Accounting Software in Deal with Smokeball - LawSites - December 5th, 2024 [December 5th, 2024]
- 7 free and open-source tools that rival the best creative software - XDA Developers - December 5th, 2024 [December 5th, 2024]
- Google Drive Full? Gift Yourself More Digital Storage This Holiday Season - CNET - November 30th, 2024 [November 30th, 2024]
- Tired of controller lock-in? Mixxx is a free DJ alternative; 2.4.2 out now - Create Digital Music - November 28th, 2024 [November 28th, 2024]
- 5 of the best free software for data recovery on Windows - XDA Developers - November 23rd, 2024 [November 23rd, 2024]
- AAVAA Hands-Free Accessibility Devices Now Compatible with Apple Software - The Hearing Review - November 17th, 2024 [November 17th, 2024]
- The best graphic design software - Creative Bloq - November 16th, 2024 [November 16th, 2024]
- VMware makes Workstation and Fusion free for everyone - BleepingComputer - November 16th, 2024 [November 16th, 2024]
- Trimble Expands Access to Advanced Construction Project Management Capabilities with Free Version of ProjectSight Software - StreetInsider.com - November 16th, 2024 [November 16th, 2024]
- The best free video editing software: how to cut clips without the cost - Creative Bloq - November 8th, 2024 [November 8th, 2024]
- Best Free Invoice And Billing Software Of 2024 - Forbes - November 8th, 2024 [November 8th, 2024]
- Amazfit just dropped a massive free software update and these new features are coming to your smartwatch - Tom's Guide - November 5th, 2024 [November 5th, 2024]
- The Free Software Foundation Finally Has AI / Machine Learning Apps On Their Radar - Phoronix - October 24th, 2024 [October 24th, 2024]
- Intuit asked us to delete part of this Decoder episode - The Verge - October 24th, 2024 [October 24th, 2024]
- "100% Free" GNU Boot Discovers Again They Have Been Shipping Non-Free Code - Phoronix - October 24th, 2024 [October 24th, 2024]
- The best antivirus software in 2024 for PC - TechRadar - October 24th, 2024 [October 24th, 2024]
- Stunning software giveaway: Save over $500 on tools for video editing, password recovery, and more its all free! - BetaNews - October 18th, 2024 [October 18th, 2024]
- PSA: Windows 10 has entered its final year of free support here's what you need to know - Windows Central - October 18th, 2024 [October 18th, 2024]
- Best video editing software in 2024: free and paid-for tools - Amateur Photographer - October 18th, 2024 [October 18th, 2024]
- Samsung TVs free update to One UI is already happening here are the changes coming to TVs - TechRadar - October 18th, 2024 [October 18th, 2024]
- The best open-source productivity software: Free tools to boost your workflow - XDA Developers - October 9th, 2024 [October 9th, 2024]
- Best tax software of 2024: File fast and accurately, plus get your maximum refund - CNBC - October 7th, 2024 [October 7th, 2024]
- The IRS is expanding its free tax filing service. Do you qualify? - The Washington Post - October 4th, 2024 [October 4th, 2024]
- Explore Top Free Software Alternatives to Popular Paid Programs for Budget-Friendly Solutions - Gizbot - October 4th, 2024 [October 4th, 2024]
- The best free video players in 2024: watch videos in any format - TechRadar - October 4th, 2024 [October 4th, 2024]
- Ford unveils BlueCruise 1.4: hands-free driving time doubled with new software update - CBT Automotive News - October 3rd, 2024 [October 3rd, 2024]
- Free Photo Viewer for Windows - Free download and software reviews - Download.com - October 3rd, 2024 [October 3rd, 2024]
- Top 10 Cool Free Windows Software (You'll Really Want) - MSN - October 3rd, 2024 [October 3rd, 2024]
- Best free YouTube to MP3 converter of 2024 - TechRadar - October 3rd, 2024 [October 3rd, 2024]
- The best free alternatives for pricey software: Adios, Office and Adobe - PCWorld - September 28th, 2024 [September 28th, 2024]