Stories
Slash Boxes
Comments

News for nerds, stuff that matters

AT&T Invents Surveillance Programming Language

Posted by Zonk on Tuesday October 30, @09:45AM
from the code-breathing-on-the-other-end-of-your-phone dept.
An anonymous reader writes "AT&T has long been associated with advances in the programming arts as well as communications. They've recently brought those disciplines together to create a powerful datamining language called Hancock. Hancock is a C variant developed to mine gigabytes of the company's telephone and internet records for surveillance purposes. 'The manual for the language includes a Hello World variant that shows you how to write a program that will parse logs of IP addresses and record them into permanent hashes. The program for parsing millions of records as they flow into permanent data farms sounds oddly close to the data mining the NSA performed after 9/11 to find targets for its warrantless spying on American citizens calls and emails."

Related Stories

Display Options Threshold:
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • Hancock.. worst name ever. (Score:5, Funny)

    by EveryNickIsTaken (1054794) on Tuesday October 30, @09:47AM (#21170687)
    What, was Palmdong taken?
  • Ironic Name (Score:1, Insightful)

    by Anonymous Coward on Tuesday October 30, @09:49AM (#21170705)
    One would think that John Hancock would not be pleased to learn that his name is being applied to a project that violates the Constitution he so famously signed.
  • web server (Score:1, Funny)

    by Anonymous Coward on Tuesday October 30, @09:50AM (#21170735)
    Can I replace php with this? PHP is insecure I heard. Hancock must be secure because it is the government.
  • Heard near Massachusetts... (Score:3, Insightful)

    by UncleTogie (1004853) * on Tuesday October 30, @09:55AM (#21170815)
    (http://127.0.0.1/ | Last Journal: Friday November 02, @08:43PM)

    "Hey, what's that whirring sound?"

    "It's the founding father [wikipedia.org] this programming language is named after...spinning in his grave..."

  • And what do facebook use? (Score:2, Interesting)

    by ztransform (929641) on Tuesday October 30, @09:56AM (#21170827)
    Monitoring communities of interest is no doubt something of interest there..
    • 1 reply beneath your current threshold.
  • We are already working on:

    • Hancock++ - Because a single + was not enough
    • H# - .NET version of the language
    • GNU/Hancock - Returns the results as an open source document and publishes it to the freakin' world
    • GoogleHancock - Datamines Chinese citizens and returns the results to party headquarters and the People's 9mm Ammunition Billing System
    • HancockScript - Great for client side mining
    • Re:Variations by Anonymous Coward (Score:2) Tuesday October 30, @10:02AM
      • Re:Variations by cybermage (Score:2) Tuesday October 30, @10:11AM
    • Re:Variations by morgan_greywolf (Score:3) Tuesday October 30, @10:24AM
    • Re:Variations by Just Some Guy (Score:3) Tuesday October 30, @10:39AM
    • Re:Variations by everphilski (Score:2) Tuesday October 30, @10:39AM
    • Re:Variations by WindBourne (Score:2) Tuesday October 30, @10:47AM
  • Ummm.... (Score:5, Informative)

    by Otter (3800) on Tuesday October 30, @09:59AM (#21170893)
    (Last Journal: Thursday November 08, @06:00PM)
    This is at least a decade old, was published in 2000 (I like the breathless "unearthed today", like it was some sort of secret -- the original Hancock paper is listed as having 29 cites) and has rather obvious applications for marketing, billing and security. The "oddly close to the data mining the NSA performed after 9/11" seems a bit excessive.
  • Don't worry! (Score:5, Insightful)

    by Peter Trepan (572016) on Tuesday October 30, @10:00AM (#21170919)

    If you haven't done anything wrong, then you have nowhere to hide!

    Whoops - I mean nothing. Nothing to hide.

  • Hello world (Score:5, Funny)

    by $RANDOMLUSER (804576) on Tuesday October 30, @10:01AM (#21170929)
    update Users set Status = 'suspicious' where Username in (SELECT Username, ipAddress, MissleAddress from IncomingCalls ic, OutgoinCalls oc where Volume = 'whispering' and Username not in (select Username from RepublicanDonors));
  • not a new language (Score:5, Informative)

    by roman_mir (125474) on Tuesday October 30, @10:04AM (#21170975)
    (http://booktextmark.mozdev.org/)
    this is a collection of libraries and some domain specific keywords/structures, but to say that this is a new language is a stretch of imagination.
  • perl (Score:2)

    by _14k4 (5085) <sullivan,t&gmail,com> on Tuesday October 30, @10:06AM (#21171019)
    Well, at least they didn't realize they could do it in perl and then give perl the bad name...
  • *AFTER* 9-11? (Score:3, Informative)

    by erroneus (253617) on Tuesday October 30, @10:15AM (#21171175)
    (http://slashdot.org/)
    I think we've been seeing a trickling of stories and evidence showing that Bush/Cheney/Addington were ALREADY doing many 'questionable' things prior to 9-11. At the speed of government, doesn't it make you ask how they were able to cobble together the DHS?! And if I recall, some of the surveillance activities declined by Qwest were requested prior to 9-11.

    Bottom line? 9-11 is irrelevant to their intent... 9-11 helped provide some justification in the eyes of some, but the evidence shows that this stuff has been planned WELL in advance of 9-11 and this is not a reaction or over-reaction.
  • Prior Art (Score:2)

    The language already exists: it's Soviet East German.
  • Hello World in Surveillance Language (Score:3, Insightful)

    by Lemming Mark (849014) on Tuesday October 30, @10:19AM (#21171245)
    (http://www.cl.cam.ac.uk/~maw48/)

    The manual for the language includes a Hello World variant

    Never has that program name been so fitting.

  • The program for parsing millions of records as they flow into permanent data farms sounds oddly close to the data mining the NSA performed after 9/11 to find targets for its warrantless spying on American citizens calls and emails.
    Ok, this is going WAY too far! Could you have at lease included some LINKS to actual PROOF that ANY government agency conducted warrantless spying on US citizens? I mean other than the FUD that the NYT and Democrats spread which also is not based on any actual proof. IIRC the warrentless wiretaps were on NON-US citizens that were OUTSIDE of US jurisdiction. The whole "it's common knowledge that" defense is wearing quite thin. If you're going to make outrageous claims, you need to back them up with actual proof. No wonder he posted as AC.
  • I sense a new meme! (Score:1, Funny)

    by Anonymous Coward on Tuesday October 30, @10:44AM (#21171597)
    I've been Hancocked!
  • Oh lord... (Score:2)

    by Unlikely_Hero (900172) on Tuesday October 30, @10:46AM (#21171643)
    I try to stay upbeat about all this, I try to think of ways to fight back against this kind of crap, I do everything I can to not have to deal with this kind of obscenely orwellian garbage...but...sometimes it just hits ya right in the gut...and you feel dreadfully sick...
  • Hancock (Score:2)

    by davidsyes (765062) on Tuesday October 30, @11:15AM (#21172099)
    (http://www.otanashide.com/ | Last Journal: Friday November 02, @12:37PM)
    with a Manual Guide... I wonder when this application will "cock up" (in English parlance) and screw over people not part of the serve.. sir vail-ance ... umm surveillance.

    I thought many of these ISP and ISP parent companies had stated officially that they had issues with excessive data retention (storage space, processing of the enormous data sets, legal issues, etc...). Now, this, from one company that is probably going to make some enemies.

    What good (other than government surveillance and corporate marketing) can come from this without harming privacy?

  • Did anyone read up on the language? (Score:4, Informative)

    by Weslee (1118943) on Tuesday October 30, @11:22AM (#21172205)
    Its basically just C with some generic structures thrown on top of it.

    Also, it was created in 2000.
    Its intent, as some have mentioned, was marketing.
    Basically it does what Google Analytics or WebTrends does for the web.

    It actually seems like a nice language, for those who want to quickly run through gigs of data.

    I see nothing evil about the language itself.
    It, like C, perl, PHP, or any other language you chose to use - Can be used for whatever purpose the programmer chooses.
    Its intent was marketing, and almost every company in existence wants to know more about their customers.
  • Hancock Written Before 2001 (Score:4, Insightful)

    by squidguy (846256) on Tuesday October 30, @11:43AM (#21172569)
    Gee, can you conspiracy theorists take a break for a second and consider that, just perhaps, this was written for commercial telecom management, marketing and fraud detection purposes? It was written and in the public domain before 9-11.
    The US Government uses Linux, so are we to presume that Linus Torvalds is an agent of George Bush and the broad conspiracy to spy on you?
  • Hash indeed... (Score:2)

    by DoofusOfDeath (636671) on Tuesday October 30, @11:50AM (#21172677)

    'The manual for the language includes a Hello World variant that shows you how to write a program that will parse logs of IP addresses and record them into permanent hashes

    Sure, I can just imagine what the hash function is, based on AT&T's recent history:

    long long hashForNsaEyesOnly(long long phoneNum) {
    int wink = -1;
    int nod = -1;
    return (2 * phoneNum * wink * nod) / 2;
    }
  • by Algorithmnast (1105517) on Tuesday October 30, @11:54AM (#21172729)

    If you look here [findlaw.com] and research the case a bit, you'll find that a Maryland company may have actually been more responsible for ATT's abilities than ATT would like to admit. That company is now defunct, unfortunately, and so it's now safe for ATT to pretend that they've done work in the area without answering to more law suits.

    It was a very technically challenging job. We helped to index records for these guys [lexisnexis.com] until mid-2005. We did it in effectively O(n) time - the cool factor was higher than the say-nothing factor.

    And yes - I know that academia will claim that it's not possible, that data correlation must be O(n^2). For the decade that we did it, we were sure glad that academia held to that position.

    Enough reminiscing.

  • A "new" concept? (Score:1)

    by Caerdwyn (829058) on Tuesday October 30, @12:27PM (#21173301)
    Wat, someone reinvented regexp?
  • Great Name!! (Score:2)

    by Seraphim_72 (622457) on Tuesday October 30, @12:46PM (#21173637)
    Because I know the Han(d)s are on my shoulders and I then know where the rest of the name is going!!
  • Ellen Hancock? (Score:1, Interesting)

    by Anonymous Coward on Tuesday October 30, @12:54PM (#21173765)
    Any of you young-uns here remember the clipper chip? President Clinton's ingenious plan to bug every phone in America:

    http://en.wikipedia.org/wiki/Clipper_chip [wikipedia.org]

    Wasn't Hancock the name of the academic stooge that was pushed front and center by the feds to shill for this thing, claiming that we could trust Bill & Algore with the key escrow? I seem to remember a fawining article in the ACM Communications pimping for the clipper chip.

    I wanna say Ellen Hancock, former IBM and Apple exec, but her bio makes no mention of time spent in academia or shilling for bad wiretapping schemes:

    http://en.wikipedia.org/wiki/Ellen_Hancock [wikipedia.org]

    Even if it's not her, I wonder if the programming language namesake and the government spying toady are one in the same.
  • by Gizah (887392) on Tuesday October 30, @01:35PM (#21174409)

    Hancock's Half Hour
    The time taken to scan any arbitrary volume of call detail records, being half that of prior technologies.
  • Oh I love it. (Score:1, Interesting)

    by Anonymous Coward on Tuesday October 30, @01:45PM (#21174613)

    The system was built in the late 1990s to develop marketing leads, and as a security tool to see if new customers called the same numbers as previously cut-off fraudsters -- something the paper refers to as "guilt by association."
    When I was hacking and phreaking back in 1980 to 1987 me and some friends were writing War Dialers for PC's, Apple II and C64's. We had something like 2000 people running our software round the clock, all working on finding long distance codes.

    I started to notice patterns on how people would get busted.
    Kid A finds a valid code.
    Kid A calls his buddy Kid B using the code, and gives him the code.
    Kid B then calls Kid A back using the code.
    From there the dispersion of this code goes out exponentially for 2 weeks till the code gets deactivated.

    I start to see a pattern were this kids would then get Busted, or at least a nasty phone call, maybe even some letters.

    Later I would start to see my software pass over valid codes as not valid.

    The phone networks started to sense we were trying numbers sequentially and deactivate them just before we would try it.
    This had to be done in realtime.
    Next was pseudo random numbers we were using, but this eventually failed in the same way.
    Then I used just the random() function in the basic language our code ran in.
    That also eventually started to fail!

    Noticing this intelligence in the system I came up with what I called the reverse pyramid around 1983.

    I Eventually came up with a scheme for pseudo random generators that wouldn't fail. This was done using a central server(over phone lines) and was the only way to solve this since it had become obvious that telco security people were also getting copies of our war dialers. ( The purpose for pseudo random instead of pure random was that we wanted to get full coverage of a specific number space each kid was scanning. )

      Basically the dialers would not tell the kid running the software they had come across a code.
      Instead it would contact a very small server hidden in the phone network and upload the code it had found.
      Then at a regular interval, all thousands of users would get the same code all at once.
      Most kids probably thought that it was there computer that had found the code, but really it was out of a large pool of codes we had acquired. This solved the "guilt by association." problem.

      So after the code was disbursed widely starting with what would usually be the last people to get a code, it would throw off such fancy analysis tools.
      Then after a week or so the "inner circle" of our core group would start using the code, when they were well in the noise of millions of people stealing long distance using the same code.

    Anyhow, the point of all of this is to finally see some confirmation that they had developed such software that they were using against us, and that out countermeasures had worked.

  • by HiThere (15173) <charleshixsn@@@earthlink...net> on Tuesday October 30, @02:27PM (#21175199)
    This might be an interesting language for many purposes... just because those who created it had one purpose in mind, doesn't mean it can't be used elsewhere. But that depends on the license, which the article doesn't seem to mention.

    The license clearly isn't BSD, but it could, conceivably, be GPL...depending on what the article writer translated into "Hancock's source code and binaries (now up to version 2.0) are available free to noncommercial users from an AT&T Research website. " (In the article there's a link to the source code site...but without knowing the license I'm not looking any further.)
  • Obligatory (Score:2)

    by PPH (736903) on Tuesday October 30, @07:59PM (#21178681)
    Nothing to C here. Move along.
  • Good one... (Score:1)

    by Graywolf (61854) on Wednesday October 31, @04:48AM (#21181233)
    SQL jokes FTW!
  • by twkrimm (802775) on Wednesday October 31, @09:57AM (#21183901)
    I wish the satellite providers would offer (one way) internet service. They could use several TV channels, and continously download channels of information. You could have a geek channel with slashdot, cnet, lwn (linux week news). You could have a news channel. Yes could still get on the internet through other means as well. 0) No extra satellite dish required. 1) We have very large, affordable, hard drives to store the information. 2) We have many channels available on satellite already and most of the existing stuff is useless anyway. Unfortunately the satellite provider would determine what to download. There also may be IP concerns but I think these are minor. What do you think?
  • Re:That's the last thing we need! (Score:5, Interesting)

    by ByOhTek (1181381) on Tuesday October 30, @10:15AM (#21171169)
    (Last Journal: Thursday November 01, @10:16AM)
    While normally I'd quite agree with you, straight from the article (and not Zonk), right at the start:

    From the company that brought you the C programming language comes Hancock, a C variant developed by AT&T researchers to mine gigabytes of the company's telephone and internet records for surveillance purposes.


    less inflamitory, later it states:

    The system was built in the late 1990s to develop marketing leads, and as a security tool to see if new customers called the same numbers as previously cut-off fraudsters -- something the paper refers to as "guilt by association."


    It seems to have been created with slightly better intent (fraud detection, as well as, unfortunately, marketing - your phone company is spyware!).

    A tool may not necessarily be bad, but it can have more bad uses than good, and may be been intended for rathern malevolent purposes. The rack comes to mind (although this language certainly isn't in that league).
  • by jamstar7 (694492) on Tuesday October 30, @10:46AM (#21171633)
    The problem has never been the tool. It's the application of the tool, and the people who wield it.

    Consider, for instance, a baseball bat. Under normal useage, it's used to play a game. Sure, accidents do happen, sometimes erious, but these are minority cases. For the most part, it's all in fun.

    Now consider that baseball bat in the hands of a crazed psychopath who thinks YOUR head is a baseball. See the difference?

  • by vertinox (846076) on Tuesday October 30, @11:27AM (#21172301)
    (http://mp3bat.com/)
    Now if if someone is using a tool in a way you don't like, then just say so, don't try to cast aspersions on the tool itself.

    The problem with power is that if it can be abused... It will be.

    So it often behooves citizens to keep these tools out of the hands of the government as long as possible.

    Not that it will make much difference in the end.
  • by Z00L00K (682162) on Tuesday October 30, @12:50PM (#21173697)
    Too many European countries are also planning for large amount of data collection.

    Anyway - just because it's possible to collect and mine data doesn't mean that the same tools can be used by the "bad" guys too to understand how the technology can be circumvented or made to be misleading.

    But a new tool may find new surprising uses too, so there are nothing that can be considered completely bad by this tool.

  • 8 replies beneath your current threshold.