Anti-Terrorist Data Mining Doesn't Work Very Well 163
Presto Vivace and others sent us this CNet report on a just-released NRC report coming to the conclusion, which will surprise no one here, that data mining doesn't work very well. It's all those darn false positives. The submitter adds, "Any chance we could go back to probable cause?" "A report scheduled to be released on Tuesday by the National Research Council, which has been years in the making, concludes that automated identification of terrorists through data mining or any other mechanism 'is neither feasible as an objective nor desirable as a goal of technology development efforts.' Inevitable false positives will result in 'ordinary, law-abiding citizens and businesses' being incorrectly flagged as suspects. The whopping 352-page report, called 'Protecting Individual Privacy in the Struggle Against Terrorists,' amounts to [be] at least a partial repudiation of the Defense Department's controversial data-mining program called Total Information Awareness, which was limited by Congress in 2003."
Bets....? (Score:5, Insightful)
In other news, (Score:4, Insightful)
Seems (Score:4, Insightful)
What we really need are spies. Not so much in the US, here good old fashioned detective work (with Warrants) should work.
But over seas a standing army isn't going to do anything to quell terrorism. Tanks and plans will only inspire more terrorism. What we need are good old fashioned black ops. Undercover agents penetrating the terrorist groups and talking to the bad guys. Much less collateral damage as well.
We'd get a lot further with a couple guys with silenced pistols rather than a whole army.
Re:In other news, (Score:5, Insightful)
Didn't we already know? (Score:5, Insightful)
I thought we already knew this. If the algorithm comes back with even .1% false positives the system is totally worthless. There's 365 million people in the US, .1% means that the FBI/CIA/NSA would have 365,000 people to investigate. Now go and talk to someone in the AI field and see if even .1% false positive is possible.
I'm betting that if a system is going to catch any decent percentage of terrorists (greater than 50%) the false positive rate will be above 1%. Even if you only apply the system to a relatively small number of people (say people entering a leaving the country) you are going to have hundreds of thousands, if not millions, of people to investigate. Combine any kind of realistic false positive rate with the fact that about .00001% of the population deserves to be investigated and the system is worse than worthless; all it will do is distract from the people who should be investigated.
I'd run on that platform. (Score:5, Insightful)
Vote for me.
I'd take their "no fly" list and identify every single person on it who was a legitimate threat and either have them under 24 hour surveillance or arrested.
The mere concept of a list of names of people who are too "dangerous" to let fly ... but not dangerous enough to track ... that just fucking stupid.
Think about how many people could be killed in the airport terminal itself WITHOUT getting on a plane ... say during the Thanksgiving or Christmas rushes there.
What idiot would let the people on that list (if they were really a threat) into a terminal? Wouldn't you expect them to STOP them BEFORE they get into a position to do that kind of damage?
Re:I'd run on that platform. (Score:5, Insightful)
There's a bigger problem with that. (Score:3, Insightful)
As can be seen in the recent "terrorist" arrests in the US. Once you start paying people to turn in "terrorists", you start a market in "terrorists".
So the guy who wants to sell a "terrorist" to the government finds some idiot who meets the basic criteria (non-Christian, non-white) and encourages that idiot to make inflammatory statements while being recorded.
Ka-CHING!
Re:Seems (Score:5, Insightful)
Re:Bets....? (Score:3, Insightful)
The question is, "What will you replace it with?"
No, they will not listen when you say the obvious, which is "Get a real job."
Re:There's a bigger problem with that. (Score:5, Insightful)
Reminds me of a bit from Discworld.
To summarize, Ankh-Morpork was over run by rats. The obvious solution was to put a bounty on rats, payable per tail. Soon, the rat infestation was under control but the number of tails being brought in kept increasing.
The Patrician's solution: tax the rat farms.
Re:Bets....? (Score:3, Insightful)
Your lack of faith is completely unwarranted. After all, when the polygraph was shown to be unreliable and thrown out as evidence of guilt...
Right. Nevermind.
Re:I'd run on that platform. (Score:4, Insightful)
Right, but flying under your middle name does work. As does claiming that you lost your ID (but if you refuse to show it on principle, you can't fly). As does using one boarding pass with matching ID at security, and a different boarding pass with matching ID at the gate.
The realy sad thing is, the people who the government feels are a real threat based on strong intelligence are *not* on the no-fly list! The government doesn't want to reveal to the real suspects that their being watched.
It's not the Data Mining... (Score:2, Insightful)
Having personally used Multiple Data Mining techniques for several years now - It's not that Data Mining doesn't work, rather it's how its used. Data Mining is great at trend forecasting and if you're really good at what you're doing in it you can factor in probabilities of certain future events. The one key factor in data mining is a "Training Set" of Data to teach the machine(s) how to recognize the patterns. Since I suspect Terrorist come from every walk of life, every know nationality, and are using 1 off events this is throwing them a few headaches. The real key is to of course define what is normal, but if the rest of the world is as normal as are we here in the US they don't have a chance to pin point the Target Data (in this case people).
I would also suspect that the Terrorist Motives might be a key factor, but it's like pulling teeth to get any US Administration to admit that their foreign policy is screw up beyond belief, let alone something like a cruddy foreign policy might just result in cruddy foreign relations or popular uprisings around the world. If they did, then we wouldn't need data mining in the first place.
"May You - Live Long and Prosper in Interesting Times" -- by deweycheetham
Re:There's a bigger problem with that. (Score:5, Insightful)
The GP isn't calling for vigilante groups turning in terrorists. He's calling for old-fashioned cloak-and-dagger HUMINT. It works far, far better than the technological circus we are operating now. Humans will always outsmart machines made by humans. The only real accomplishment of mass government data mining is the oppression of the general public who aren't interesting in outwitting the government. They're just trying to live their lives.
In the old days (Revolution, World Wars, Cold War), when we were aware of our enemies, spies, analysts and cryptographers defeated the enemies with courage, brainpower and skill. Now we've replaced them almost entirely with people in offices. This isn't going to change until we have another wakeup call, and the next one will probably come from Russia. The red bear is back, and we aren't prepared to deal with it (or China). Much of Russia's new technology is ahead of the US, particularly in aerospace submarine areas. We do not have a real missile shield, we do not have space-based weapons, we do not have supercavitating torpedoes (or anything to stop them). About the only encouraging developments we do have are in robotics and lasers.
China isn't very technological (except for those nasty anti-sat weapons), but they have an enormous mountain of people they don't mind sacrificing for whatever they dream up. Their standing army is over 2 million. They're also currently building and testing over one ballistic missile a week.
2005 article [bbc.co.uk] 2007 Article [aviationweek.com] Oct 6, 2008 [heritage.org]
Terrorist data mining won't help much of anything when an EMP hits and the computers are fried.
Re:Seems You're Right (Score:2, Insightful)
Yes, because we need an aircraft that can fly faster than mach 3 above 80,000 feet to penetrate terrorist airspace and evade terrorist fighter jets.
They use Predator drones now because, well, they were not manufactured in the 60s, they cost magnitudes less for the same mission, they can perform more than one type of mission, they are unmanned, etc. etc. etc. etc.
The SR-71 was cool and all, but way too antiquated to keep around. Clinton made a good move killing the SR-71.
By the way, how did the SR-71 help anyway? It gave us zero insight into the Soviets...in fact, it did far worse by giving us false insight. It was a waste of money from conception to retirement. But...damn, it was cool...
Re:False negatives are a worse problem (Score:3, Insightful)
No. False positives "only" cost the government time and money. For the individuals falsely suspected, it could cost them their career, their relationships, their home, and their freedom, depending on how much "time and money" the government spends on them before realizing they are innocent. (If they ever do, since -- as shocking as it sounds -- there have been a few cases reported where individuals were detained indefinitely without charges, or even evidence.)
overfitting (Score:3, Insightful)
I was thinking about this very topic the other day (Score:4, Insightful)
I was wondering whether techniques of commercial data mining could be applied to environmental problems like emerging disease surveillance.
Well, of course they can. The question is how far is it from practical? I think, pretty far from being as practical as it is in business.
First of all, businesses have a great deal of object model in common: they have common concepts like customers, products, sales, brands etc., which form a common framework in which they can do all kinds of creative thinking, or if not thinking you can even discover relationships using some kind of machine learning.
Secondly, when you are dealing with business data, the most important events tend to be common events. The most important common event is when a customer buys something. When you talk about something like a new disease emerging, or somebody committing a crime like hijacking or bombing, the most important events are exceeding rare, but catastrophic. Therefore the connection between events we do have in abundance and the events we are interested in is tenuous, poorly statistically attested to, and in many cases pure conjecture.
Finally, a lot of what businesses use data mining for is tweaking marginal costs and revenue by shifting dollars that were already going to be spent from one place to another. Offer product A to this web visitor instead of B. Stock more of item X in the store rather than Y. If you really don't know a priori whether X or Y will sell more profitably, you probably aren't going to go too wrong.
In something like environmental monitoring, you create expenses that weren't already there. No, you can't drain this lake because the model predicts a 5% marginal increase in the probability of human cases of hantavirus in the area. To somebody counting on the economic value of draining that lake, that's a brand new cost that wasn't there before.
Same goes, even more so, to deciding somebody is a danger to society.
Now let me say that I have no doubt that data mining will lead to more terrorist being thwarted or captured, compared to doing nothing else. Of course so would a lottery, but I suspect that data mining is a great deal better at identifying good suspects than a lottery. However, it is for reasons I noted above not going to be particularly accurate, certainly not compared to probable cause. Furthermore, the marginal cost of false positives gained seems likely to exceed the marginal value of false negatives lost, if such things could be quantified.
The purpose isn't to flag terr'ists (Score:5, Insightful)
If you want to catch terrorists.. (Score:2, Insightful)
Just compile a list of all the extremely illegal and unethical things you're doing as a government and find the groups of people most impacted.
Let those people simmer for 5-10 years under your asshattery and let cool. Presto! A tasty terrorist.. Bon Appetite!