Stories
Slash Boxes
Comments

News for nerds, stuff that matters

PHP Security Expert Resigns

Posted by samzenpus on Thursday December 14, @01:57AM
from the good-day-sir dept.
juct writes "PHP security holes have a name — quite often it was Stefan Esser who found and reported them. Now Esser has quit the PHP security team. He feels that his attempt to make PHP safer "from the inside" is futile. Basic security issues are not addressed sufficiently by the developers. Zeev Suraski, Zend's CTO of course disagrees and points his finger at inexperienced programmers. But given the number of remote code execution holes in PHP apps this year, Esser might have a point. And he plans to continue his quest for security holes in PHP. Only that from now on, he will publish them after reasonable time — regardless if a patch is available or not."
Display Options Threshold:
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
  • YAY

    (Score:3, Funny)
    by phantomcircuit (938963) on Thursday December 14, @02:01AM (#17233364)
    I for one would like to thank him for the nominal increase in success rates of attacks thanks to him!

    GREAT IDEA!!!!
  • Couple thoughts

    (Score:3, Funny)
    by BadAnalogyGuy (945258) <BadAnalogyGuy@gmail.com> on Thursday December 14, @02:02AM (#17233370)
    First, the language is wide open for editing. It might help to be someone who not only finds bugs but fixes them.

    Second, it's PHP. Add another API or something.
  • PHP Security Expert

    (Score:4, Funny)
    by mrshoe (697123) on Thursday December 14, @02:03AM (#17233374)
    (http://www.mrshoe.org/)
    PHP Security Expert...

    Isn't that an oxymoron?

    • Security expert's advice is very easy. by jez9999 (Score:1) Thursday December 14, @05:39AM
    • Re:PHP Security Expert by plierhead (Score:1) Thursday December 14, @06:02AM
      • Uh-huh, riiiiiiiiight... by Gordonjcp (Score:2) Thursday December 14, @06:37AM
        • Re:Uh-huh, riiiiiiiiight... by vadim_t (Score:2) Thursday December 14, @06:47AM
        • Re:Uh-huh, riiiiiiiiight...

          (Score:4, Interesting)
          by solidox (650158) on Thursday December 14, @06:51AM (#17234480)
          (http://solidox.org/)
          There was an exploit for mambo some time ago, sql injection i believe, perhaps several others also, so mambo is a likely culprit.
          One cannot say it was PHP directly that got the machine compromised. It was an exploit in a script written in PHP.
          A box isn't going to get compromised if PHP was installed alone on the box without any scripts (at least it's very very unlikely).
          Is C the direct cause of your box owned when their is an exploit in say, proftpd for example?

          I mean, I could also say...
          "yeah, you'd have to be mad to run sendmail on a box you don't want to get owned"
          "yeah, you'd have to be mad to run proftpd on a box you don't want to get owned"
          "yeah, you'd have to be mad to run bind on a box you don't want to get owned"
          "yeah, you'd have to be mad to run a linux kernel on a box you don't want to get owned"

          These applications have all had their problems in the past, maybe some still have problems, but overall
          they get fixed when new exploits/bugs are discovered.

          I'm not quite sure why, but a lot of people/webmasters/admins do not check for updates to the 3rd party php scripts
          they have installed, they just install them once and leave them running... Then they wonder why their box was compromised
          due to them running out of date software.
          You wouldn't leave your windows machine unpatched and never check for updates, would you?
      • Re:PHP Security Expert by aaronwormus (Score:2) Thursday December 14, @06:55AM
      • Re:PHP Security Expert by Da Fokka (Score:2) Thursday December 14, @07:18AM
  • On second thought...

    (Score:5, Insightful)
    by phantomcircuit (938963) on Thursday December 14, @02:03AM (#17233376)
    On second thought I would have to agree that the majority of PHP flaws are due to unskilled programming.

    just have a look [milw0rm.com]
  • PHP reminds me of IIS4

    (Score:5, Insightful)
    by 93 Escort Wagon (326346) on Thursday December 14, @02:12AM (#17233412)
    We have a large group of students, staff, and faculty that all have varying degrees of write access to a departmental Apache web server. Every few weeks someone asks why we're not giving people PHP access. Users love PHP because it's so easy; it makes them feel like they're clever programmers. But it seems like security knowledge is never imparted alongside the PHP training. People seem to think it's as benign as plain old HTML. When they ask for PHP I tell them we have a policy about not giving scripting-level access to users without good justification, and they have no idea why that applies to them since "we don't want to do any scripting; we just want to make PHP web pages".

    But even leaving all that aside - it seems like every SANS newsletter has multiple announcements either about a bug in some popular bit of PHP-based software, or else in PHP in general. Until that changes, we're sticking to Perl and Python. It's funny, in a way, since the first time I saw PHP I immediately thought of the days when I was writing Active Server Pages on IIS4, because structurally it is so similar - and now we all realize the similarities on the security side (or lack thereof) as well.
  • Open source is the issue

    (Score:3, Funny)
    by Anonymous Coward on Thursday December 14, @02:28AM (#17233484)
    It's widely acknowledged that open source programs are inherently insecure. Whether the cause is the availability of the "internal blueprints", the free-for-all repository commit access, or the rampant theft of patents, one wonders. By contrast, Microsoft's .NET platform, including the widely praised C#, doesn't have this problem. The guarding of the internal source code, the standards-adhering developers, and the rock-solid legality of its software patents gives Microsoft an advantage versus the haphazard "open source" languages like PHP and Java. One wonders if this is a harbinger of future defections in the open source language camp. Speaking as a patent lawyer, I advise all developers to switch to .NET and Microsoft's enterprise-class C#.
  • Actual announcement

    (Score:5, Interesting)
    by kjart (941720) on Thursday December 14, @02:36AM (#17233516)

    Here's the announcement from the source himself, via his blog [php-security.org]. Based on that post I'd say he sounds pretty disgruntled with how his efforts towards security were received i.e. "he PHP Group will jump into your boat as soon you try to blame PHP's security problems on the user but the moment you criticize the security of PHP itself you become persona non grata"

  • XSS by default

    (Score:5, Funny)
    by Anonymous Coward on Thursday December 14, @02:47AM (#17233556)
    When I looked at Zend's introduction to PHP, the first sample PHP program was Hello World, and the second was a cross-site scripting vulnerability. Right, I'm going to trust these people.
  • As a PHP user....

    (Score:5, Interesting)
    As a PHP user, I have attempted to better the thing by reporting what I think are bugs. I can't name a single one that wasn't closed with a WONTFIX and a terse, non-thankful "that is a feature, not a bug." I honestly have zero disbelief that those same programmers would turn against Esser when he blamed the language, not the user, for the security problem.

    In particular, the late static binding issue (if B extends A then A::staticFunc() ran as B::staticFunc() is ran under class A not B). It's like how it took MySQL took a decade to get stored procedures and views despite many people asking for it. Many people complain about the late static binding issue but last I knew it was still "it's a feature, not a bug."

    Regardless, thanks for your work Mr. Esser...
    • Re:As a PHP user....

      (Score:4, Funny)
      by Shados (741919) on Thursday December 14, @03:02AM (#17233594)
      non-thankful "that is a feature, not a bug."
      Oh boy...Microsoft bought out PHP...
    • PHP ought to be forked

      (Score:5, Interesting)
      by Jesus_666 (702802) on Thursday December 14, @04:37AM (#17233978)
      Someone should fork PHP and do a major rewrite. Drop features like HTML embedding, introduce properly defined packages and make all functionality available in both procedural and OO fashions. Clean up the function names so they're predictable. And make some of the more dangerous functions safer.
      PHP could be turned into a decent general purpose scripting language if someone would fork it. Unfortunately that means that we'd need someone who knows the codebase, has time and is fed up with the current PHP development process. Maybe we could talk Esser into it...
    • Re:As a PHP user....

      (Score:4, Informative)
      by Splab (574204) on Thursday December 14, @07:25AM (#17234618)
      Amen to that.

      I had a fun one where one of my scripts would cause a segmentationfault, after hours of debug I found that they don't check the return from malloc when you call a function, so a very deep recursive function will result in a segfault. Now I had the problem with an actual system with 1000s of lines, so I made the simplest possible:

      function foo($a){
          echo $a . "\n";
          foo($a+1);
      }
      foo(1);

      Now this is of course a stupid function since it will never terminate, but it illustrates the point of the segmentation fault, I don't mind that deep recursive functions can exhaust the memory available, but I do mind the way the system handles the problem.

      The bug got rejected, and that was that. I don't do PHP anymore, so I don't really care about that any more.
    • 1 reply beneath your current threshold.
  • he just left a mailing list...

    (Score:3, Informative)
    by aaronwormus (716976) on Thursday December 14, @03:30AM (#17233702)
    The "news" is that Stefan Esser unsubscribed from the security@php.net mailing list.

    Stefan Esser will continue to work on PHP security through maintaining the Hardened PHP project [1] which is a patchset to PHP which enables some low level security features into the language, as well as the suhosin extension [2] for PHP which can be used without patching PHP and "protects servers and users from known and unknown flaws in PHP applications and the PHP core".

    I am personally of the "full disclosure" security mindset, so if there was indeed an issue with the response time of the "PHP Security Response Team" then some outside pressure would be a good thing.

    More about this on Zeev's blog [3].

    [1] http://www.hardened-php.net/ [hardened-php.net]
    [2] http://www.hardened-php.net/suhosin.127.html [hardened-php.net]
    [3] http://www.suraski.net/blog/index.php?/archives/15 -Stefan-Esser-quits-securityphp.net.html [suraski.net]
  • by cheros (223479) on Thursday December 14, @03:40AM (#17233740)
    Wow, stunningly insightful response "that's caused by inexperienced programmers". He's a clue: it doesn't matter what the origin of the problem is (other than to fix it longterm) - IT STILL NEEDS ADDRESSING. I got news for you: the concept of covering large security related cracks in code with prime bullshit is probably already patented by Microsoft.

    Personally I would wonder if Essers' 'abrasive style' is not a result rather than a reason for not being listened to and if this flags up a major problem in the way PHP is coded and maintained I'm all for this move. There is no excuse for sloppiness.

    So, the reaction discloses the attitude - seems Esser made the right move..
  • Not up-to-date on PHP security . . .

    (Score:4, Interesting)
    by pembo13 (770295) on Thursday December 14, @03:51AM (#17233782)
    (http://www.pembo13.com/)
    can someone explain how it is that the apperently consensus is that PHP is insecure by design, asside from just poor programming? Thank you.
    • Re:Not up-to-date on PHP security . . . by aaronwormus (Score:2) Thursday December 14, @04:26AM
    • Re:Not up-to-date on PHP security . . . by nicklott (Score:2) Thursday December 14, @04:32AM
    • by gbjbaanb (229885) on Thursday December 14, @05:44AM (#17234252)
      One of the biggest 'problems' is the way PHP is generally executed as an apache module. You get a lot of shared webhosts that run php as a module, and so the apache user runs the code. Fine, except that if you want to give your PHP script access to your data, you're effectively giving it access to everyone else's data too. So features like open_basedir were added to restrict this.

      Then there is features like safe_mode that turns off many system functions that an attacker could use to get round the other restrictions, and register_globals which is a feature designed to work around an inherently insecure system of passing variables to php pages.

      and so on, and so forth.. possibly the biggest problem is the ease of coding it, the barrier to entry is so low you will attract coders who (to be polite) don't know as much as they could about programming. So you get a lot of PHP code that is poor quality, makes too many assumptions on things that they should have tightened up (eg, not initialising variables to prevent an attacker from passing them in with their desired values), or checking input to functions from the form or url.

      Its the same issue as VB - it was so easy to code VB apps, my boss could do it. So he did. And they looked, performed and crashed as if a manager had coded them :(
      • 1 reply beneath your current threshold.
    • by dysfunct (940221) on Thursday December 14, @06:14AM (#17234328)
      I actually do a bunch of security consulting for PHP based stuff. A great deal of the issues stems from the very beginning of the PHP language itself. Being designed to be as easy as possible without regard to security has kind of made it the Microsoft of scripting langages. They have not built on insecure code, but rather entire concepts that are inherently insecure (fopen() wrappers that open nearly every data connection they're fed, register globals, SQL string concatenation) and have even for a long time endorsed and taught users those concept.

      Instead of changing concepts midway through they have added security layers and APIs that need to be *explicitly* set - meaning that like Windows (was?) they have a policy of being open per default and having to be explicitly made secure, instead of closed by default and enabling only what you need.

      That's what I think Stefan Esser means when he says "safer from the inside". Many things in PHP are inherently flawed and can only be remedied through changes in concept and nothing else.

      Add to that stuff like $GLOBALS overwrite (more details here [hardened-php.net]) that are/were essentially a WONTFIX. No wonder Essner is getting frustrated.

    • Re:Not up-to-date on PHP security . . . by Jamu (Score:3) Thursday December 14, @06:43AM
    • by kestasjk (933987) * on Thursday December 14, @07:26AM (#17234626)
      (http://kestas.kuliukas.com/)
      I've written lots of PHP code [sf.net] in my spare time, and have written an article [kuliukas.com] on creating "rootkits" to covertly inject into PHP scripts (phpBB2 in particular), so I thought I'd chime in. This'll probably be a long post but hopefully it'll give people some things to look out for.

      Here are the most common security problems you run into in PHP:
      • magic_quotes: This adds slashes to all input so that you don't have to sanitize it before it gets inserted into SQL. The problem is that developers write their code with magic_quotes on, but don't realize that it's often turned off elsewhere, which leads to gaping holes.
      • register_globals: Variables can be placed directly into the global namespace. If you don't explicitly set all variables before using them anything can be injected into them, which brings me on to:
      • Only critical errors are reported: If you use a variable which isn't set it'll just return null, with no error (unless you specifically turn up the error_reporting level). This means that someone who isn't familiar with the problem won't know that a variable in their script can be written to by anyone until it's being exploited, functions which you would expect to return an error and halt the script if they fail can carry on without giving any indication of failure.
      • fopen_urls: By default you can include scripts hosted on other websites! This often makes remote PHP execution, which would otherwise require eval(), much easier.
        Who would have thought "<?php include($var.'/include.php'); ?>" will run any PHP on any server, anyhere? (The attack in the article above leveraged entry using this, coupled with register_globals.)
      • Inconsistencies: What one function does can never be applied to what another function does; you can never assume anything with the PHP library and always have to keep a browser window with the PHP manual handy. Using a function without carefully reading up what it does, even when it's very similar to another function you're familiar with, is asking for trouble in PHP.
        The same goes for just about everything; are you checking whether some input equals some harmless number before passing it on to a SQL query or the browser? Don't forget that (5 == "5 UNION SELECT secret FROM ..."), null == 0 == "" == false, "a" == 4 == true; generally you just have to be on your toes.
      • Input checking is difficult: Do you want htmlentities() or htmlspecialchars() ? Have you remembered to strip_slashes() if magic_quotes is on? Remember the user can input arrays too, are you checking that the input isn't an array? Have you remembered to escape queries with mysql_real_escape_string() ? mysql_escape_string() doesn't account for the character set being used, and so isn't good enough, trying to escape input for yourself is also dangerous. What about null bytes? Remember that the user can input binary data; PHP allows null bytes, and will add a slash to them, but when you send a string with null bytes to some functions, but not others, the null bytes will be silently dropped leaving only slashes.
        To check input in PHP you have to be absolutely rigorous and take no half measures, people who aren't aware of the dangers don't stand a chance.

      To be honest I'm a big fan of PHP, it's very flexible and lets you develop very quickly and easily; if you have the knowledge and self discipline it's an excellent language. But allowing fast, easy development at the cost of security is insane for a server-side web scripting language!
      I was hoping that PHP6 was all about doing a 180 degree turn on security, but this article doesn't bode well..
  • PHP security is a disaster by design

    (Score:2, Interesting)
    by Anonymous Coward on Thursday December 14, @04:09AM (#17233840)
    Variables are untyped, so if you do $a + $b, it's not clear what the result might be. Variables do not have to be declared before use, so if I have code like:

    $authorized = callAuthFunction();
    if(! $authoorized) logoutUser(); // note the misspelling
    mysql_query("UPDATE account SET ...."); // you get the idea
    Woops! Languages that have a permissive syntax make it easy for bugs to hide. And security flaws are just a particular subset of bugs. At a higher level, we have problems such as widespread use of direct DB access all over the place, instead of some kind of persistence layer, which results in likely SQL mistakes, and even injection attacks if the code isn't using correct pear DB. There's no true filter mechanism in PHP. There's no way to annotate objects as requiring a certain user-in-role. The whole thing is a big mess of C code and third party libraries, and there are good old fashioned C buffer overflow vulnerabilities in those areas too. Wee!
  • Here's an eye-catcher

    (Score:5, Insightful)
    >bugs were sometimes not correctly fixed or were re-introduced. This was often not noticed because there was no test-rig for exploits and the idea of having one was categorically rejected.

    If that's accurate, and if there wasn't some unimaginable compelling reason, any security person would be unhappy.
  • by pikkumyy (445891) on Thursday December 14, @04:46AM (#17234012)
    Just because the language is easy is no reason to (attempt) to make it idiot proof. Numerous crappy 'security features' have already been added to the annoyance of decent programmers. Making it more secure by design would only encourage sloppy programming, which already is a big problem.
    • In related news

      (Score:5, Insightful)
      Law makers in Texas are debating a bill to enable people to own nuclear weapons and heavy artillery and to remove safety catches from guns.

      "All you should need is a great big red button that says 'Fire'" said Congressman Bobby Ewing "Its ridiculous that people are prevented from using these things and having to put up with safety devices it just encourages sloppying thinking"

      "By letting people launch nuclear weapons with a big red button we are making sure that everyone is aware of how to properly care for their nuclear weapon and that it is their god given right and responsibility to fire it carefully" said some bloke in a hat "I'm fed up with all the ridiculous procedures I have to go through to fire a gun, let alone blow up France just because a few bleeding heart liberals feel they need to protect stupid people in New Hampshire"

      In related new Iowa has banned the use of indicators, roll cages, air bags, crumple zones and seatbelts as it gives people too much sense of security. California has banned the use of door and window locks and the use of burglar alarms as they make houses "secure by design".

      Secure by design is the only type of security that really counts.
  • No bad dogs, only bad owners

    (Score:3, Informative)
    by ajs318 (655362) <sd_resp2 AT earthshod DOT co DOT uk> on Thursday December 14, @05:20AM (#17234166)
    A bad worker blames their tools and a bad boss blames their workers.

    There's no denying that PHP has things wrong with it. It started out as a bastard son of Perl, tried to be a bit more n00b-friendly and tripped over its own cleverness. The beauty of Perl is its very inconsistency. The functions you use most have the shortest names, and there is no need to clutter things up with unnecessary brackets around arguments. Regular expressions, which you are going to use all the time, have a distinct syntax. Number and string data types can be interchanged with such wild abandon, there have to be separate operators for addition and string concatenation (JavaScript, I'm looking at you). There are constructs to populate arrays quickly. All things are subordinate to the goal of letting a programmer get a job done. Easy things are easy, hard things are possible. Perl is so broad-minded, it even has the Principle of Equivalence built in!

    PHP lures you in, with obviously_named_function($par1, $par2) ..... then trips you up with anotherobviouslynamedfunction($par2, $par1). You could say it's not all PHP's fault, as the functions originate from different shared libraries, and PHP is only providing an interface to them by their original name and with something like their original syntax. But it still smacks of laziness on the PHP developers' part. Short aliases for commonly-used functions (a context-sensitive editor can always expand them for the benefit of the anal retentive), and differently-named work-alikes for functions that take their parameters in a different order than you might expect, wouldn't have hurt. Would they?

    Still, you've got two choices, I suppose. Learn to put up with the idiosyncracies or learn another language. And never forget the Principle of Equivalence; "All Means to the same End are equally valid", nor its corollary, "Means which are not equally valid serve different Ends".
  • Would a suitable headline be "Goaded, Esser Back"?

    Apologies to Douglas R. Hofstadter
    • 1 reply beneath your current threshold.
  • If PGP...

    (Score:3, Funny)
    by Alioth (221270) <dyls@alioth.net> on Thursday December 14, @06:06AM (#17234310)
    (http://www.alioth.net/ | Last Journal: Wednesday December 13, @05:34PM)
    If PGP stands for 'Pretty Good Privacy', I wonder if PHP should really stand for 'Pretty Hopeless Privacy'...
  • my butt would be giving buttloads of major security holes ...

    Something which is used extensively gets more flaws discovered than something that is used less. Get this in your heads.
  • by mw22 (908270) on Thursday December 14, @06:50AM (#17234476)
    So I read the piece from his blog and the heise article, I didn't see any remorse against Stefan from the PHP group. I can see Stefan making that accusation, though. It can be very difficult to fix bugs, and sometimes it can take a very long time. So - with the information I got thus far - I think Stefan is trolling and tries to get some publicity. That seems also be the reason why he wants to do a month of PHP bugs.
  • by robsy (1039796) on Thursday December 14, @08:08AM (#17234974)
    Reading all the comments about PHP throughout the years, I think it's about time that Slashdotters unite against PHP(and Microsoft and...add whatever it is that Slashdotters hate).

    I think the functionality of the language is it's biggest enemy when it comes to security. If the language is simple enough like for instance you can only make programs that can print out "Hello world" then it can be considered very secure. It's maybe not very useful, but very secure.
    I know programmers who should never be allowed to program in anything but Java or C# and then only simple code.
       
  • by Anonymous Coward on Thursday December 14, @02:12AM (#17233410)
    ...for script kids to poop on!
  • Re:php is the best language still

    (Score:2, Interesting)
    by Divebus (860563) on Thursday December 14, @02:22AM (#17233454)

    Huge problem is "default" installs - everyone knows where your sample scripts are. Delete those first thing then move/rename the active libraries.

    Now, where's that Ruby book?

  • Any language is only as good as the programmer using it.

    I use a LAMP stack for the most part, many of the security holes in php aren't due to the language itself but the developers of the various webapps.

    That being said, this requires a repost of the ol Adminspotting [adminspotting.org] thang.

    Choose no life. Choose no career. Choose no family.
    Choose a fucking big computer, choose disk arrays the
    size of washing machines, modem racks, CD-ROM writers,
    and electrical coffee makers. Choose no sleep, high
    caffeine and mental insurance. Choose no friends.
    Choose black jeans and matching combat boots. Choose
    chairs for your office in a range of fucking fabrics.
    Choose SMTP and wondering why the fuck you are logged
    on on a sunday morning. Choose sitting in that swivel
    chair looking at mind-numbing, spirit-crushing web sites,
    stuffing fucking junk food into your mouth. Choose
    rotting away at the end of it all, pishing your last in
    some miserable newsgroup, nothing more than an
    embarassment to the selfish, fucked up lusers Gates
    spawned to replace the computer-literate.

    Choose your future.
    Choose to sysadmin.
    • by eln (21727) on Thursday December 14, @03:18AM (#17233662)
      Yes, bad developers produce insecure code, but let me take you on a brief trip down memory lane.

      Way back when, when the Web was new, and CGI was just starting out, there was some debate as to whether C or Perl should be the language of choice for writing CGI scripts. In the end, Perl became much more widely used because it was just too damn easy to open up major security holes writing in C, because it lacked some of the features of Perl (like making it impossible to commit a buffer overrun, for example). Perl won out in early CGI precisely because a lot of the problems of CGI security were already solved because of inherent features of the language.

      Now, PHP came along and billed itself (and in fact was designed) as an easy way to make secure web scripts. So, if the PHP code has bugs that impact its security in web-based applications, these things should be addressed. Otherwise, it's going to end up being supplanted by another language that is more secure and easier to use to build web apps.

      Blaming the developer for security is only going to take you so far when the language the developer is using is supposed to be SPECIFICALLY DESIGNED for web applications.
    • by quantaman (517394) on Thursday December 14, @04:14AM (#17233862)

      Any language is only as good as the programmer using it.
      I actually have a philosophy when writing applications that is almost the complete opposite of that.

      Anytime the tool does something that the user doesn't want it's a bug.

      This applies to applications, programming languages, heck even cars if you want.

      The fact is that if the user gets something they didn't want, no matter how stupidly they tried to use it, the tool still bears some of the blame. I don't care how dumb a thing the user did, there was something there that made them think they could do that and it's a bug.

      With programming languages if the language allows the user to create a security hole it's the fault of the language on some level. Sure you can get stupid programmers but blaming the programmer entirely discourages the search for a better language. Yeah if I overrun my array in C it's my fault. But can it be entirely my fault when in Java that same bug wouldn't be a security exploit? Hey, if I drive my car straight off a cliff, is that my fault? Yeah. But a car with a computer failsafe driver wouldn't of gone off the cliff (hey, if two jetliners are on a collision course the computer takes over).

      You can never make the perfect tool, even a big green button that will do everything you ever wanted will still have a bunch of people who didn't think to push the button. But it forces you to realize, you can never fix users but you can always fix your code.
  • 9 replies beneath your current threshold.